The main architecture of Snap4City consists of a set of tools to cope with the following concepts:
- collecting data from open data, real-time data, private data, IOT, stream, data driven, etc.; They can come from any kind of source, protocol and format. Provided by city operators, city users’ devices, social media, open street map, international organization, social media, archive, museum, chambre of commerce, institutions, etc.
- Acting: this means that the solution allows to compute actions and to perform them on users, IOT devices, web server and services, dashboards, via a range of notifications, and other effects.
- storing data collected in a knowledge base and indexing, to prepare the data retrieval with the capability of inference and reasoning, search and retrieval, faceted search, drill down, extracting insight from the data and producing smart data, etc.
- executing IOT applications, ETL processes, and Data Analytics processes, in a reliable and scalable smart city cloud infrastructure and computing services that can exploit data to provide advanced smart services, one demand, periodically and in real-time, even driven.
- creating application, data driven and/or periodic, on the basis of MicroServices, which in turn are based on Snap4City Smart City API. These Applications can be visually produced as Dashboard exploiting data, and/or flows processing data and IOT data running on the platform (for example by using NodeRED) also exploiting and presenting some Dashboard.
- creating services/processes by means of a number of development tools, for creating data analytics, early warning, etc. They can be developed in ETL, R, Python, Java, etc.
- showing and navigating on data, original and those resulting from analytics, by using a set of dashboards and in turn make easy the production of specific dashboards for decision makers, city operators, etc.
- providing access to data and services via Snap4City Smart City API which can be exploited by web and mobile applications, as well as by other tools and cities, according to the rights of the users.
- Supporting GDPR and enforcing security for the data and information which are private of the users, and the users may delegate in access or pass in full control to other users. They can be: dashboards, IOT applications, personal data, annotations, etc.
The development tools provided by Snap4City are useful for accelerating the process of adoption, of feeding data into the platform, setting up the real-time streams, creating maps, creating mobile and web Apps, developing ETL data transformation process, developing data analytics processes, and creating Dashboards for city users. All these instruments and tools are specific and provide suitable functionalities for specific kind of user categories: final users, city users, developers, living lab users, etc.
Most of the DISIT tools are components of Km4City family of tools https://www.km4city.org to strength the modularity and general view. All of them are independent tools, which can be independently downloaded, installed and used for setting up different platforms, such as by developing new solutions and tools on top of them.
In order to start up the solution the structural data regarding streets, civic numbers, etc. of the city/region/province by using OSM (Open Street Map) need to be loaded into the system. This activity is performed with specific tools (also open source) and the final destination of the data is the Knowledge Base, KB, grounded on Km4City Ontology. Typically, the list of civic numbers of the city has to be added to have a more precise view of the city area loaded from OSM. Thus, the KB is ready to collect referential data and open data coming from the open data, or provided by stakeholders. The Data Sets (open and/or private) can be uploaded in an easy manner by using the DataGate/CKAN (https://Datagate.snap4city.org ). Alternatively, of the data set is particular it can be ingested by using specific ETL processes. During the upload of Data Sets the connection of them with the structural information on the KB is needed. This process is called reconciliation and has to be accompanied to some quality improving process, autocompleting process, etc. Following this approach, in a short time, the KB is going to be populated and would be ready to host real-time data attached to their referred locations acquired from Data structural information.
Architecture
In the architecture reported below, on top you can see the major roles of the Living Lab actors and what they do: MicroService Developers, IOT Application Developers, Application / Dashboard developers and users, and finally the traditional web and mobile App Developers. In addition, on the rights, we have IOT/IOE Apps Consumers, and other cities and Operators that would be interested to share information and experience with the city and Living Lab, may be other Living Labs.
The description starts from the less technical role and skill to the most.
IOT/IOE Application Consumers may be interested in accessing to some mobile or web application showing data. They can be directly Dashboards. The Dashboards can be personal view of the city, as well as dashboard dedicated to general public and/or for city operators, which are those adopted and controlled by policy level decision makers. The Dashboards may present data and information and have to stay up and running H24/7 on very ultra-wide flat panels into the control room as well as in the mobile of city Major or assessor, or in the hands of some policemen, etc. While others can be specific for police, fire brigade, first aid, assistance associations, voluntaries, civil protection, etc.
· The level 0 of using Snap4City is just the Dashboard usage in view. To this end, the user would not need to be registered to access at the Public Dashboards.
Application/Dashboard Developers may create their personal Dashboards as City Dashboards (with and without the needs of integrating IOT Applications (formalizing how the data has to be processed) with the Dashboard).
· The Level 1: implies to be a registered user to create a Dashboard just exploiting accessible data as sensors, KPI, microapplications, POI, maps, etc. The creation of a Dashboard is strongly simplified by using the Wizard that help the user in selecting data and the graphics view possible for them.
· The Level 2: The Dashboards can be create exploiting data as above, and in addition using also private data from IOT Devices of the user and acting on some IOT device, notifications on other Dashboards, etc. In order to use personal IOT Device, the user has to register them by using the specific service accessible on the main menu on the left.
These users, once they have created some Dashboard, may decide to make them public to all the IOT/IOE Applications Consumers.
In some cases, the process of Dashboard creation may implies the connection of new public external sources but these activities are sporadic and performed by the administrators. On the contrary, the user can add its own IOT device, and in this is completely autonomous.
IOT Application Developers are considered Level 3 users. When explored the first levels, if a user needs to:
· introduce some data transformation in the middle from data to the dashboard
· add some logic to write some rule from data to results on dashboard, for example IF something happen them DO this and that.
· Perform come computation, algorithm. And can use algorithm we made for help the user in producing: predictions, identifications of conditions, anomaly detection, by using Data Analytics, Statistic Analysis and Machine Learning.
· Perform some automated production of action on other users and devices
· Put in execution those action automatically even when the Dashboard is not controlled
· Manage the Smart City back office and services: control, automating, etc.
· Etc. etc.
So that, they may create Applications to be used from Mobile or Web pages, which are in our case Dashboards exploiting data and IOT devices, plus having some IOT Application including some logic on data flow, produced in NodeRED. They do not need to be programmer, while some tutorial would be needed to start creating the flow according to the NodeRED paradigm of IBM. It is a well-known approach, that in Snap4City is simplified since the MicroServices, that are the blocks composting the flow, are simple and well documented.
In this case, the IOT Applications is created by composing MicroServices as Blocks in the Application Builder.
It is a visual editor in which the data flow can be formalized. Each MicroService / Block has clear inputs and outputs and can be directly connected each other. The IOT applications can be immediately tested and put in execution on the cloud, without any intervention of the user, all automatically and transparent for the user. The Snap4City has produced a huge amount of new MicroServices based on: smart city API, knowledge base, advanced dashboards, integration with DataGate, Resource Manager, server kind of IOT data and actuators, suggestion, machine learning, statistics, data analytics, IOT discovery, personal data in the safe, annotations, etc. and much more. Please note that there are TWO collections of Snap4City MicroServices: one for Final User simpler to be used but limited, and one for Developer, more complete and sometimes complex but very powerful.
The same formalism can be used for creating flow (IOT Applications) executed on IOT Edge (e.g., microprocessor based devices such as Raspberry Pi) and on Android devices.
Each user may have one of more IOT applications, working with one or more Dashboards.
Each IOT application may include a number of data flows for real time data driven processing and with dashboards.
All the provided MicroServices of the Service Area are at their disposal for the creation of Applications according to their level of access. The access to data and services is regulated by the tools of Access Services area, which is accessible for the single users for governing their data access. Please note that Final Users can create their own IOT Applications and can be endowed of City Dashboards, accessible for example from their mobile without installing any App. Moreover, the Final User may access to public Dashboard (for example on totem and from their PC or on flat panels in the station), and may access to service of the smart city by using web and mobile Apps developed by some company or groups that exploit the Advanced Smart City API.
MicroService Developers are those that create massive processes for data ingestion and transformation in the Smart City. They are typically operators and researchers that have to bring and transform a large amount of data. They do not need to be programmer while some training is needed as provided on the Snap4City portal and Test Case. They are entitled to creating ETL (with Penthao Kettle), Data Analytics, dashboard and Applications. When a new process is created it can be scheduled in the system and allocated by the Cloud Elastic Management as container. Ready to use containers/images are made available (configuration of execution engine tools: ETL, Java, Python, NodeRED, etc.).
In the same manner, IOT devices can be registered in the IOT Directory (providing their semantic description or integrating them by the user interface, or collecting them from brokers that list them). Thus, IOT devices can be discovered when they are used in creating Snap4City Applications by the Application Builder. The Applications Builder is calling directly the IOT Discovery tool that integrates the Semantic Search on the KB and the Map interface to apply geospatial search, full text searches, etc. IOT Directory and Discovery tools allow connecting and abstracting from the several IOT Brokers, their management, the protocols and the brokers at which the devices are registered. This is possible thanks to the usage of the IOT Application editor and tools based on KB for the IOT Discovery Tool and the related index created at the IOT registration. Depending on the involved sensors, a data flow will be generated and automatically configured for gathering data from the corresponding IOT brokers, data sources and networks. This solution allows the creation of a complex service by composing MicroServices and not creating a monolithic, difficult to maintain, single piece of code. Moreover, the developer can easily change the suggested blocks with other MicroServices.
Data Analytics Developers are typically data analyst, statistics, mathematicians, engineers, researchers, etc., that may be interested and involved to create algorithms for producing smart solutions: prediction, anomaly detection, optimization, etc. They may create ETL and Data Analytics to be uploaded into the platform via the Resource Manager. It provides automated tools for making them accessible as a MicroService into IOT Applications. They can monitor the exploitation and usage of their tools/processes/solutions, the related faults and their correct working (notifications in case of problems are also automatically sent by the Resource Manager). During the development of MicroServices, the developers may need to access data via a set of tools/dashboards (Developer Dashboard, ServiceMap, DataMapper, etc.) which allow to access and navigate among referential data and at the same time discover links and relationships on the city entities via the Knowledge Base (by using: ServiceMap (https://servicemap.snap4city.org ), LOG visual query https://log.disit.org for navigating into the world of Linked Open Data and not only into the KB, FLINT for SPARQL). They also may access at the Developer Dashboards as AMMA, ResDash, DevDash to understand traffic flow, data flow, and resource consumption, filtering on data, on data kind, drill down on timeline, and on geospace, etc.
Traditional web and Mobile App developers interested and engaged to create applications accessing to data and services provided by the Snap4City via the Advanced Smart City API. They can be applications distributed on some mobile market (Google play, Apple Market) or can be HTML5 Web applications in HTML5 for Windows 10, for example. To facilitate their work, Snap4City provide Km4City App Development Kit which is based on Cordova Apache model of programming which is a similar approach of the City of Helsinki App. It can be accessed from the portal of GitHUB.
Finally, the Snap4City solution is also designed to share (back and forwards) information and data with other cities and Living Labs. This activity is facilitated by:
- Smart City API: queries, knowledge, etc.
- Web Portal: Https://www.snap4city.org
- Resource Manager for publication and share of dashboard, data analytics, processes, ETL, applications, algorithms, procedures, IOT applications, microservices, external services, microapplications.
- DataGate for publication and share data, data sets, API, etc., even by using CKAN API
All the above features and tools are also supported by Knowledge Services, and Big Data Storage Services that maintain linked the semantic model and the information coming from IOT, in real-time, referral data, and data in batch collected by ETL from external services. Thus, referential data and entities relationships are maintained and semantically querable with geospatial and temporal reasoners, also allowing to drill down on data tabular with graphs by using Phoenix on Hbase/HDFS distributed and reliable big data storage. In addition, the Indexing and Search Services allow posing complex queries to structured and unstructured and semantic data to get results. For example, searching a civic number GPS with street name and number, searching for services along a path, searching for services into an area, etc.
