Test Case Title |
TC8.6 - Snap4City Living Lab Life Cycle |
Goal |
To provide support for the Living Lab in terms of life cycle. The living lab is and environment in which stakeholdes can create their own applications, and share them with others. |
Prerequisites |
Knowing the goals of the Select4Cities. The following functionalities are available only for specific Snap4city users with specific privileges. |
Expected successful result |
Understand the mechanisms to set up and start the snap4city platform in third phase. |
Steps |
In order to create an efficient Living Lab, the following key principles have to be taken into account: i) Value (create value for users and customers as a key aspect for business success: involve SMEs, mitigate the competition, open new markets); ii) Influence (users seen as active, competent partners and domain experts, concretize new services coming from the citizens ideas); iii) Sustainability (e.g., choose right materials, implementing user-friendly approaches, considering the social and economic impact of the innovation), iv) Openess (open collaboration between people with different expertise and backgrounds. Different perspectives could lead to a successful innovation process), v) Realism (the innovation actions are carried out in the real-life, realistic and natural settings, this to increase understanding on how innovation can bring advantages valid in the real market) [8]. Another fundamental aspect is also connected to territorial needs, this is the reason why sometimes it is stated about the Urban Living Labs, that “are environments in which innovation is spatialized, i.e., it is generated within a specific spatial environment. These are environments in which the openness of innovation manages to transcend the organizational infrastructures that are traditionally operating in the city and to invent new institutional figures for, or ways of, dialoguing between citizens and institutions” [29], [30], [31].
Snap4City has been designed starting from Km4City models and tools, adding tools and functionalities to satisfy the requirements of ENOLL, Select4Cities consortium and EIP SCC. Snap4City resulted to be the winning awarded solution from the Select4Cities solutions that seen a large number of competitor improving their solution for 3 year, and comparing in a competitive manner the solutions as described in https://www.snap4city.org/558
In the context of Smart City, Living Labs have to provide a set of key features for an infrastructure capable to manage all the complex aspects identified. A Smart City Living Lab infrastructure have to be: i) able to activate practice-based knowledge production in collective and private environments (e.g., taking into account both Private and Open Data and contexts); ii) able to learn internally but also externally, by experimenting with other cities that share the same problems; iii) aware of new civic engagement models being experimented all over the world; iv) aware of the growing demand for new citizenship models; v) capable of directing investments toward opportunity creation (i.e., experiments) rather than pre-developed solutions, [32]; vi) stimulating the participation of all stakeholders in the activities of data collection and process/solution production. Those types of infrastructures manage a massive set of data (Big Data context) that can be shared, processed, used to generate new knowledge; moreover, are capable to ingest both Open and Personal/Private Data (e.g. the user’s profiles and actions done in the city). All the processes realized by this kind of infrastructures imply the necessity to pose attention on a relevant aspect: the treatment of the privacy rights on data (especially the sensitive ones), underlined also by the General Data Protection Regulation (GDPR), [33], [34], [35], [36].
Snap4City solution is a Living Lab Smart City infrastructure, responding to the key features/requirements enabling the Living Lab paradigm especially in the acquisition of City Data and developing smart city solutions.
Architecture
Smart Cities need to set up a flexible Living Lab to cope with the city evolution in terms of services, city users’ needs and capabilities. To this end, the Snap4City solution provides a set of tools and a flexible method and solution to quickly create a large range of smart city applications exploiting heterogeneous data and stakeholder services also enabled by Internet of Thing (IoT)/Internet of Everting (IoE) technologies and Big Data analytic. The Snap4City solution and all the innovation activities carried out for its development, have been realized involving different kind of Organizations (Universities, SME and Large Industries, Public Administrations) and users (City Operators, Resource Operators, Inhouse companies, Tech providers, Category Associations, Corporations, Research groups, Strat-ups, Early Adopters, large industries, advertisers, City users, Community builders, etc.), thus reflecting the features, described in Quadruple Helix (QH), [7], to facilitate the Living Lab approach in a Smart City, Fig. 1. The innovative aspects of the solution proposed are related to semantic computing of entities for discovering and search information, resources management, parallel and distributed computing and cloud management, applications based on microservices and external services, dashboard and development tool kits, etc. Snap4City flexible enough to support extensions at distinct levels of granularity: data, analytics, tools and applications.
Snap4City Life Cycle.
One of the first activities for creating a Living Lab in a city is the process of setting up the technical infrastructure which in turn is grounded on many valuable enabling tools. They must support the city in: the modeling of data; the upload of context data and open data; the connection of IoT/IoE sources and external services; the creation of Extract Transform and Load (ETL) processes and data analytics algorithms, to arrive at producing smart city dashboards and at starting the production of Snap4City Applications based on Microservices. All these phases must be accompanied and supported by the availability of a set of development tools, easy to use, accessible and open. To this aim, the Snap4City solution has been designed to create a collaborative environment in which different kinds of stakeholders can mutually collaborate. At the same time in which the setup is created, the collaboration among stakeholders can start by creating: agreements, collaborations, networking, producing tutorials, workshops, hackathon, etc. Fig. 1, so at to arrive at involving the stakeholder around case studies, and finally to sign contracts of partnership, licensing, etc. Thus, the delivering of specific solutions to city users, operators, etc., is becoming possible. This process must be driven by the municipality and, on the other hand, the municipality needs support for technical aspects if it is not very large and technological oriented. Typically, the single companies even if participated by the city or the city operators, do not have the view and the mission to put in common a so large multi-domain multiservice framework and environment.
The approach is a methodology to arrive at producing the Snap4City innovation matrix. It is implemented by means of a number of workshops (physical or online) to arrive at producing a deep analysis of the current situations and the needed steps for fast evolution toward the smartening of the processes, by taking the more efficient direction and the most acceptable solutions for the stakeholders. See training part 6 https://www.snap4city.org/577
The scenarios identified during the workshops may be focussed on specific domains (mobility, energy, gov services, water, environment, etc.) and with the aim of improving services to specific user categories. To this end, a process of Data Discovery is also performed by talking with the stakeholders and their technicians to verify the possibility of collecting the needed / available data as well as to create new IoT Network of Devices to measure phenomena that is presently under focus. Please take in mind that, typically the same Scenario (target smart solution) may be realized by starting from different kind of data. So that the scenario is the target, the data are a tool, and the experience may help you in identify among the different data sources those that may be simpler to be acquired and exploited, technically and legally speaking.
From the technological point of view, the above process is released by means of a set of tools, to provide a support for collaboration and sharing at distinct levels. We propose the Snap4City architecture (see Fig. 2) as capable to solve all the problems described above and which consists in:
- A layer to ingest all the different kind of data coming from a smart city that can be classified in: Open Data, Personal Data, IoT and IoE, Social Media, static and real time. The set of data regard many distinct categories such as: transport systems, Mobility, Car park; Public Services, Security, Museums; Sensors, Cameras, IoT, IoE; events; Environment, Water, Energy; Shops, Services, operators; Social Media, Wi-Fi, Networks, etc. The flow of data coming from the city data can be: static, slow, real time.
Snap4City Architecture.
- Collecting Data tools. These tools are both for developers (ETL processes) and for users with no technical skills, and were developed to exploit the possibility of bringing together people with different skills, experiences, roles, expertise, motivations, etc. and many kinds of organizations such as universities, public administrations, SMEs, stakeholders, industries, etc.
- IOT Applications: which are progressively substituting the ETL processes in the development of the data ingestion supporting both PUSH and PULL protocols, and a large set of MicroServices. They alwo allo the full integration with Dat Analytics developed in R Studio, Python and Java. So that, Node/RED IOT Applications are used for data ingestion, data trasnformation, workflow management, and smartening solutions, on cloud and on Edge and on premise.
- ETL processes, for developers and based on the Pentaho Kettle Tool, [38]. The Extract Transform and Load (ETL) processes can be personalized to manage many different kinds of data: open and personal, static and real time or periodic, geo-localized. The ETLs consider that data can come from: any kind of sources and providers, IoT/IoE or Sensor Networks, city users’ devices, social media, open street map and in different protocols and formats. Their final aim is transform data so that it conforms to km4city multi-ontology and load them in the Sna4City Knowledge Base (in the form of RDF Store for e the static data, and in an HBase -NoSQL- store for the real time data, [39]).
- ProcessLoader/ResourceManager and DISCES Scheduler have been developed to manage processes to be executed in specific scheduling applications (for example ETL processes analyzing real time or periodic data which need to be launched every day/Hour/minute, as well as Data Analytics processes in R Studio, Java, Python, etc.).
- DataGate: is a web-based open source management system for the storage, distribution, qualification, reconciliation and aggregation of Open Data. It is an extension of the very diffused Open Data platform CKAN (Comprehensive Knowledge Archive Network), [12].
- Services for:
- executing data analytics and computations that can exploit data to provide advanced smart services one demand, early warning, both periodically and in real time modality.
- creating applications that can be: data driven and/or periodic, based on Micro Services. These Services can be applications running on the platform itself (for example by using NodeRED or Pentaho [40], [38]), such as Dashboards and Mobile or Web applications.
- Data storage layer, collecting data in a Knowledge Base (KB) connected to the Km4City Multi-Ontology and making data indexing to prepare services on the data themselves, such as: data retrieval with the capability of inference and reasoning, search and retrieval, etc. [25]. While the real time Data are collected in a NoSQL database (HBase, [39]).
- Advanced Smart City API, capable to provide access to Snap4City data and services. The APIs can be exploited by web and mobile applications, as well as by many tools and cities, [11], [41].
- A set of Tools for all the Living Lab Actors, useful to test the effectiveness utility of the Snap4City solution in different contexts:
- Tools for Living Lab developers, such as the web applications Service Map and Linked Open Graph for navigating on data results considering both the geographical metadata and the semantics aspects [42].
- Tools for Living Lab testers and final users (community builders, city users, advertisers, Category Associations, etc. Fig. 1) showing and via dashboards and in turn make easy the production of specific dashboards for decision makers, city operators, etc. [41], [42]
In the following paragraphs, Datagate and ProcessLoader, as tools relevant to enable the Living Lab activities in a Smart City, is described.
See for training:
- Follow the training 2020: https://www.snap4city.org/577
- Training of 2019: https://www.snap4city.org/501
- Scenarios: https://www.snap4city.org/4
- Test cases: https://www.snap4city.org/108
- HOW TO: create a Dashboard in Snap4City
- HOW TO: add a device to the Snap4City Platform
- HOW TO: add data sources to the Snap4City Platform
- HOW TO: define privacy rules for personal data, produced by the end-users own device
- HOW TO: Develop Smart Applications, Snap4City development Life Cycle
- HOW TO: HLT vs Ingestion, and HLT vs Widgets
- HOW TO: Develop an IOT Application for Data Ingestion
- HOW TO: Upload data into Knowledge Base, ServiceMap (triple upload)
- HOW TO: Create as set of Devices with BulkProcessing
- HOW TO: Create an IOT Device Model
- HOW TO: Create an IOT Device Instance from IOT Directory tool
Datagate (https://datagate.snap4city.org/ , Fig. 3) has been designed with the aim to offer a collaborative environment enabling the data providers to archive, manage, share datasets. It primarily manages static (or periodic) data. Moreover, it allows the data providers to upload their content, with the following features and advantages, which go behind the simple i) data storage on the portal: ii) data enrichment and geo-localization (with the city street graph or directly with Open Street Map); iii) data aggregation, with all the other data contained in the Datagate portal and with those archived in the Snap4City Knowledge Base, thanks to the semantic aggregation made through the KM4City multi-ontology; iv) data sharing (with the IPR license chosen); v) data visualization using different tools (Service Map, Dashboards, web and Mobile Apps, etc., Fig. 2), Fig. 3. These advantages can be attained in an automatic or semi-automatic modality and are described here after.
DataGate: Organizations.
Data storage and sharing. These features are directly derived from the CKAN standard Open Data portal. Each data provider can be registered on the web portal and upload its datasets. A set of instruments to visualize the data are offered by CKAN (views, statistics on downloads, etc.), [12]. It is what can be viewed in Fig. 3: a set of datasets have been uploaded by their providers and then published on the web and visible to all the public Datagate users.
Data enrichment and geo-localization. This, and the following features have been added by the ‘DataEnhancer’ plugin developed in the Snap4City context. If a dataset is uploaded following the specific template, the ‘DataEnhancer’ features is available on it, in addition to all default services offered by CKAN. The template is written in the form of a csv file and provides a set of fields (both mandatory and optional) to be filled to have the most possible advantages: i) mandatory fields: data name, geometry following the Well-known text (WKT) geometric objects: points (POINT), lines (LINESTRING) and areas (POLYGON) or address (city, street, civic number, etc.); ii) optional fields (description, web page, phone number, links, e-mail, etc.). Some fields are automatically enriched by Datagate (e.g. it can automatically calculate: postal code from address, latitude and longitude from address with civic number, moreover it reports incorrectly formed fields such as: e-mail, web portals, links, etc.), Fig. 4.
Datagate: autocompletion and error reporting. From Datagate to ServiceMap.
Data aggregation and visualization. Once the datasets are uploaded in the Datagate Portal, their responsible can connect the data to the KM4City multi-ontology by selecting one of the categories and subcategories present in the ontology and accessible from Datagate thanks to the possibility of selecting ‘KM4City Categories’ and the ‘KM4City Sub-Categories’ from a drop-down menu, Fig. 4. Click the publish on Snap4City button and upload the data also in the Snap4City Knowledge Base. In this way the data is visible
Process Loader/Resource Manager
Algorithm/Process Loader is a web application, developed for allowing the creation and management of processes to be executed in specific scheduling applications with a user interface that receives input data in the form of compressed files that are analyzed, archived and finally transmitted to the desired scheduler. The main application’s activities are focused on uploading many compressed zip archives containing files and directories required to create and execute a process on an external scheduling application through a series of API requests sent to the application. The processes cab be realized by developers coming from different context and smart cities and realized using different kind of technologies (ETL processes, R Studio, Dashboards, nodeRED, R, etc.).
The user interface provides the following services:
- Ingestion of processes (in the form of compressed files) by authorized users. Each process is analyzed, archived and finally transmitted to the desired scheduler and properly launched.
- Process Execution: once the authorized users have uploaded their processes, they can launch and execute them, thanks to the presence of a scheduler, Fig. 5. The Process Loader users have only to insert some mandatory metadata such as the frequency with which a process has to be launched and other parameters that can depend on each process features (e.g. the web server from which the data are taken, the smart city related to the data, etc.). For example, an ETL process that links to an Open Data portal providing busses’ timetables in a certain city, updated once a month. The user can create an ETL capable of collecting data and transform them as he or she sees fit (for example, to insert them in the Knowledge Base of Snap4city in order to take advantage of all the services offered by this solution). Then he/she can upload the ETL process in the process Loader, set the ETL to run periodically (e.g. once a month). In this way he/she can always have (for example on one of the services offered by Snap4City such as ServiceMap) the updated data as results without having to do any further work.
Archiving and indexing of all the processes, that can be shared and easily retrieved, downloaded, re-used (basing on the license associated to each of them), ranked. All the metadata related to the processes are indexed via Apache Solr [44]. Each process publisher can make public its processes so that they can be shared. A web page (Fig. 6) offers the list of the public processes and an easy form to search the ETLs basing on the metadata associated to them.