Snap4City platform consists of a set of 100% open source, modular and integrated tools accessible according to the user profile. A technical version of the Snap4City architecture is reported in the next figures, where the main elements are reported and described below.
No proprietary components are needed to be installed into the Snap4City platform.
From Left to right of the above figure.
Data Ingestion and aggregation is performed in push/pull by using: IoT Connectors, IoT Agent, IoT Adapter, IoT Broker, IoT Apps in Node-RED, and data gathering tools as DataGate/CKAN for open data (https://ckan.org ), ArcGIS, and Portia for Web Scraping. The Snap4City platform supports a very large number of protocols push/pull, sync/async among them: MQTT, NGSI, COAP, OneM2M, ModBus, OPC, WMS, WFS, and AMQP. See for the list of the protocols supported Https://www.snap4city.org/65 Data Models, IoT Device Models, IoT Devices, and IoT Broker are registered via IoT Directory which is multiprotocol multibroker tool for IoT Network management. Among the brokers, a mayor role is played by IoT Orion Broker of FIWARE by which the platform support NGSI V1, V2, LD with Services/Tenant and Service Paths. And its elements as IoT Agents, etc. A detailed description of IOT capabilities and network are provided in: https://www.snap4city.org/download/video/course2020/sys/ while a large number of detail regarding the data ingestion are reported in https://www.snap4city.org/download/video/course2020/di/
Data Storage is feed by using NIFI Cluster for massive data aggregation/enrichment (collecting data from any IoT Broker at which it is automatically registered), and saving data to Elastic Search cluster for data storage and indexing, and to the Knowledge Base (based on Km4City ontology) implemented as an RDF store (Virtuoso) which is an index for geo-spatial, relational, and temporal aspects. Every time a new data model is registered in the system, the registration is performed into the Knowledge Base. Knowledge base and Smart City API can be federated creating a mutual connection among friend cities. In the Snap4City Platform, (i) Heatmaps, Orthomaps and Maps are managed by the Heatmap Server which is a GIS (for example GeoServer or ArcGIS if you have one installed with WMS facilities) and can distribute the maps according to WMS/WFS protocols towards Web and Mobiles App and Dashboards; (ii) Buildings 3D shapes, and Floors are managed and distributed into the OpenMAINT tool, standard BIM tools are used for editing and interchange in IFC formats by standard tools as AutoDesk Revit, etc. Buildings and Floors are shown in Dashboards for their integration with maps and IoT Devices time trends. Thus, creating a full Digital Twin model of any part of the city. The tools for accessing to the Digital Twin is the so-called Data Inspector. Details can be obtained from: https://www.snap4city.org/download/video/course2020/di/
Data Transformations (reconciliation, regularization, enrichment, etc.) are implemented via IoT Apps (IoT Applications) which are processes for ingesting, integrating and manipulating data with PUSH/PULL protocols from any External Service and also from IoT Broker, and Virtual IoT Devices of any kind. IoT Apps are developed by using a visual programming data flow and/or data driven flows in Node-RED exploiting a large library of more than 150 Snap4City MicroServices.
Please note that, in Snap4City Terminology an IoT App is a process in Node-RED plus Snap4City library. The Library is based on nodes and call MicroServices, for this reason also the nodes are called MicroServices. So that often it is state that the Snap4City library in Node-RED provide a large number of MicroServices, in the sense that provide access to MicroServices via corresponding nodes.
They are also used for implementing user interface logic, Business Logic behind smart applications and Dashboards also in references of Custom Widgets and Synoptics, which can be very useful in Smart City Control Rooms. Please note that Data Transformation / Integrations may exploit internal and external storages from any kind database, any connectors from/to data sources, any External Service, any direct connection with IoT Devices and Networks, etc. A large number of details regarding the data ingestion are reported in https://www.snap4city.org/download/video/course2020/di/
Data Analytics are processes written in Rstudio or Python which can perform data access and apply statistics, Machine Learning, Artificial Intelligence, deep learning tools, algorithms also exploiting Tensor Flow, CUDA, Keras, etc. It is also possible to develop processes in JavaScript directly into IoT App, and in Java scheduling processes with DISCES (not included in the offer). Snap4City Data Analytics in Rstudio, Python and JavaScript are developed with an online development environment which also allows to put in execution the scripts in Containers which are deployed as API and MicroService for IoT App and other tools in the platform. Off-line development environments are viable as well, if preferred. Data Analytics can produce: predictions, early warning, traffic flow reconstruction, alarms detecting anomalies, KPI and indicators, Typical Time Trends, typical trajectories, Routing and paths, travel plans, query results, simulations, analysis, calibrated heatmaps, smart parking suggestions, car sharing analysis, etc. A large number of examples are provided in: https://www.snap4city.org/download/video/course2020/da/
Smart City APIs include a large collection of services to: exploit queries and reasoning on the storage and Knowledge Base, access/control IoT Network, exploit Data Analytic results, exploit IoT Apps, etc. All the data and services are accessible via the Smart City API which are used by Front End Tools such as Dashboards, Web and Mobile Apps, MicroApplications. Details regarding Smart City API are reported in: https://www.snap4city.org/download/video/course2020/app/ Knowledge base and Smart City API can be federated creating a mutual connection among friend cities.
Dashboards can be created by Dashboard Builder and/or Kibana for different kind of users such as: decision makers, city operators, ICT operators, private for the users, etc., suitable for Smart City Control Rooms with video wall, for city Operators on Desktop of multiple monitors, mobile operators, and Situation Rooms with touch panels. Dashboards can exploit all kind of data and Data Analytic, legacy services, and special tools as traffic flow reconstruction, decision support systems, etc. Dashboards area created by using a large range of ready to use Widgets, and also Custom Widgets creating Synoptics as SVG elements, or exploiting other graphic libraries, such as D3, Highcharts, etc. Examples on dashboard and how they can be authored are reported in https://www.snap4city.org/download/video/course2020/das/
Web and Mobile Applications can be created by developers exploiting Smart City APIs and may be controlled by Snap4City tools. For example, to send on the Mobile Apps: engagements, soundages, stimulus, and thus for monitoring user behaviour, creating origin destination matrices, getting reaction from the city users, informing user at the inception of critical conditions, etc. Details regarding Smart City API and their usage for the development of Web and Mobile Apps with related development environment are reported in: https://www.snap4city.org/download/video/course2020/das/
Aspects of Platform Management and Quality Control of back-office include managing users’ activities, platform setup and control, user auditing, assessment, monitoring, security, process management, and elastic scaling. Details and examples are reported in https://www.snap4city.org/download/video/course2020/sys/
Snap4City Platform Architecture
Snap4City platform can be installed in several different configurations exploiting the same modules. The different configurations may be suitable from small cities to large cities with a high number of users and huge volume of data. The different modules can be provided as VM (Virtual Machines) Appliances or Dockers. The different Snap4City configurations and each component mentioned in the following are described in detail with the instructions for their installation at page: https://www.snap4city.org/471 Please note that each VM is an appliance with a number of components which can be updated directly from GITHUB/DISIT. https://github.com/disit
DataCity-Large which is reported in the following figure, and it is mentioned quite similarly into Tutorial Part 6 https://www.snap4city.org/download/video/course2020/sys/
No proprietary components are needed to be installed into the Snap4City platform.
DCL: DataCity-Large configuration includes installations of:
- Snap4CityMAIN VM/Docker compose with: Dashboard Builder, Wizard, library of Widget, WebSocket Servers, Data Inspector, User Statistics, External Service manager, Resource Manager, Micro Applications manager, LDAP/KeyCloak for Authorization/Authentication, Kafka (which has to be scaled up if the traffic on synoptic is high), MyKPI, MyPOI, Synoptic and Custom Widget/PINs, and IoT Directory;
- KBSSM VM: ServiceMap, Smart City API, Knowledge Base Km4City (LOG, Flint, Virtuoso): which can be duplicated in federated configurations to increase robustness/volume of traffic or in FT;
- Heatmap Server VM with an heatmap manager to generate the GeoTIFF which are loaded into a GIS for their distribution via WFM protocol such as ArcGIS or GeoServer; which can be started in FT;
- Living Lab Support VM with Drupal 7 with Snap4City extensions; which can be started in FT;
- MCLSCount VM which is cluster of VM to manage Docker Containers with Marathon, Mesos and DISCES-EM. They are containers for IoT Apps, DataAnalytics, Web Scraping; This cluster has 3 VM for the masters and a number of VM for the Dockers/containers. It may have sense to have a true cluster if the Dockers are more than 200 Containers, otherwise it may be enough to have a static allocation of them in a few VMs in FT. The Snap4City solution has been tested on thousands of Dockers/Containers and the larger cluster of VM consists presently of 16 VMs for Dockers elastically managed. [Scalable2018], [DataFlow2019].
- Typically, we have an average of 70 Containers for each VM depending on the image size of the container, and VM size. The Dockers for Data Analytics cost about 4 times bigger in terms of CPU and Memory with respect to those for IoT Apps, and have a more sporadic usage of the resources. Their timing is more relaxed.
- IOTDSES a cluster of federated NIFI on Dockers and their balancer, they are allocated on VM as well;
- ESSTORE VM a cluster of Elastic Search and Kibana VMs. The number of VM depends on the size of the global storage which can increase elastically with the need along the operation. In alternative the usage of a shared common SAN and docker could be viable if it is fast enough:
- IOTOBSF VM: IoT Orion Brokers (V1/V2) also on Dockers or VM, internal or external, etc.;
- Rstudio Server VM: for development of R Studio processes; Their size depends on the number of developers you have at the same time working on them;
- Python Server VM: for development of Python processes; Their size depends on the number of developers you have at the same time working on them;
- CKAN DataGate VM: for the connection to CKAN open data network;
- OpenMAINT VM: including OpenMAINT and BIM, for the development of Workflow, BPM and BIM;
See more on: TC10.13- Back office Cluster Container Scalability and Monitoring Scalable Infrastructure with Elastic Management of applications and processes. https://www.snap4city.org/232
Snap4City platform can be installed in several different configurations exploiting the same modules.
Where in the table the numbers are only indicative, and refer to a large deploy of the solution for a large city or for a group of small cities:
- CPU are reported in number of Cores at 2.1 GHz min
- MEM is in GByte
- HD are in GByte are high speed HD or SSD for Production while medium speed for the other configurations
- Red: is the level of redundancy, number of VMs. They can be launched in FT or not
- IOTOBSF could be dockers
- MCLSCount is a cluster with dockers. If the solution has no access to orchestrator VMware, then a VM with Nagios/Zabbix has to installed to control the On/Off of VM from DISCES_EM. The alternative could be to manage the small cluster with always one VM.
- IOTDSES a cluster of federated NIFI on Dockers and their balancer
- ESSTORE VM a cluster of Elastic Search and Kibana VMs. The number of VM depends on the size of the global storage which can increase elastically with the need along the operation.
The Snap4City main includes:
|
Dashboard Builder and Engine |
In Snap4city, it is the main tool for creating Dashboards and connect them with IoT Apps, and other dashboards and custom widgets. It is substantially a Decision Support System tool, since it provides evidence of critical conditions, and may offer solutions. On this regard, it may integrate/exploit artificial intelligence algorithms, for example, reporting prediction, identifying anomalies, manifesting early warning, providing relationships among entities exploiting inference geospatial reasoning about what is located in the city: resources, structure, people, areas, critical infrastructures, etc. See details on training course part 2: https://www.snap4city.org/download/video/course2020/das/ |
|
The Data Inspector is the main tool for the browsing of the information about a data stream entering into the platform. It is directly accessible from the main menu and give access to the Digital Twin representation of data, devices, and all High Level Types of the platform. It provides browsing capabilities to understand:
The Snap4City Data Inspector is the primary point to understand the issues and produce also the reports on those facts. For example, at level of: users, devices, process, dashboard and application. See this document on the early Version [DI-DataInspectorBase]. Now we have deployed since June 2020 the new version of Data Inspector and engine. See more details on part 5 of the training course: https://www.snap4city.org/download/video/course2020/di/ |
|
|
widget Library, |
The library of graphic widgets which can be used to compose the dashboard. They can range from map, time trend, barseries, Kiviat, net, tables, multiserie, but also button, knob, custom widgets, lamp, panel on the road, doors, fan, parking, energy plant, metro map, etc |
|
In Snap4City, widgets are the components of the Dashboard to visualize views on data (with animation, graphics, synoptics, etc.) and also to collect interaction from the users (buttons, faces to be clicked, sliders, keypad, text pad, etc.). see HOW to create custom widgets in SVG, and examples: https://www.snap4city.org/651 https://www.snap4city.org/595 https://www.snap4city.org/644 https://www.snap4city.org/663 How to create widgets by programming: https://www.snap4city.org/153 |
|
|
It is part of the Dashboard Builder, and it provides a support for selecting the best graphic widgets representation of the data your have. You can start from data and it suggest graphics, from maps and suggest data and graphics. Or you can start from graphics widgets and it provides the data which can be rendered on it. Wizards and Data Inspector share the same unified model and user interaction for many aspects. |
|
|
Menu Manager |
It allows to formalize the menu on the left, they are different for user role, and for each organization and for each mobile. The same Menu management system allows to formalize the menu for the Dashboards. |
|
WebSocket Server |
A communication protocol for real time connection. In Snap4City is only used in its TLS version Web Socket Secure. The WS allows to communicate from Server to mobile and dashboard in a secure and event driven manner. |
|
Is a Market Place of Snap4City artefacts: IoT App, Flows/subflows, data analytics, ETL, Kibana Dashboards, etc. They can be searched, shared and promoted via a web portal. |
|
|
Snap4City external services can be any external services reachable with some protocol to get data and services. REST Call can be automatically transformed in MicroServices for the IoT App. The other can be called from the IOT Ap directly using a large range of protocols: https://www.snap4city.org/65 |
|
|
A snap4City set of views implemented in HTML5 JavaScript for realizing specific functionalities. They are substantially views of Web and/or Mobile Apps which can be called independently and placed into Dashboard external content Widget as well as into Totems. https://www.snap4city.org/dashboardSmartCity/management/microApplications.php |
|
|
Ownership Manager, User management, User assessment |
to manage users' ownerships and delegations to access at different kinds of data and resources, delegation to access, revocations, etc It provides API to be used by Web applications and tools and MicroServices to be used by IoT App. It provides services to all the Snap4City applications and it is GDPR compliant. The user and manager can have a unique place in which the Authorization and ACL are reported and can be audited. |
|
Snap4City tool for registering IoT Broker and Devices. Snap4City is almost agnostic about the brokers and accept a number of brokers and protocols registered on IoT Directory of Brokers and Devices. The IoT Directory is capable to browse on internal and external brokers to discover IoT Devices and register them on Knowledge Base. https://www.snap4city.org/115 The IoT Directory is also the tool that manage all the network information about the devices deployed and connected. It is capable to exploit NGSI V1 and V2 protocol aspects to inspecting and managing IoT Orion Brokers of FIWARE, so that to manage a large number of them and automatically performing registration of devices in bulk on Knowledge base. See https://www.snap4city.org/76 https://www.snap4city.org/562, https://www.snap4city.org/647 See training part 3 https://www.snap4city.org/download/video/course2020/iot/ |
|
|
User Statistics |
|
|
Snap4City tool for collecting and managing personal KPI, POI. A MyKPI is a variable with may change over time determining a TimeSeries with variable GPS position at each time instant. https://www.snap4city.org/396 https://www.snap4city.org/414 The Data Manager allows to manage the data that may start as KPI and POI into the platform as private and may be used as a personal storage. They can be converted later on private data. Typical KPI can be personal profile, password safe, etc. |
|
|
for programming notifications to be sent when some event occurs on Dashboard metrics. It is a tool at level of RootAdmin. |
|
|
Event Logger |
It is a tool with API and MicroService collecting data on Elastic Search for the analysis of indexed EventLogger data for exploring different aspects about the workload, traffic flow data, the efficiency of the communications, the identification of the most computationally expensive communications, drill down, faceted filtering, thus to make profiling on App and tools. This tool is based on Rsyslog and tools for the indexing of collected log data. In Snap4City (as in many professional solutions and operating systems), LOGS/logs are produced reporting errors with different level of severity (e.g., warning, severe, etc.), they are produced in standard format as SYSlog. Those logs can be browsed, queried with tools, for example: standard LogStash which is based on Elastic Search, and Snap4City AMMA also based on Elastic Search. https://www.snap4city.org/137 |
|
Linked Open Graph tool (noting to do with Logs). LOG.DISIT is a tool for accessing and browsing Linked Open Data in the world and in the local Knowledge base, KB. https://log.disit.org/service/ |
|
|
Flint++ |
It is a tool for developer to perform sematic queries on KB by using SPARQL. https://www.snap4city.org/168 |
|
Report Generator |
It is a tool to design and produce Reports. |
The data ingested as Data Models are registered on Knowledge Base. The registering the data model is the first step to enable the possibility of loading data instances which can be: Static, quasi static and dynamic real time data, as even driven processes and streams. The main component includes:
|
Km4City means Knowledge Model for the City. It is an Ontological model for the smart city and IoT Applications in smart city and many other domains. It is the core model adopted in Snap4City to design and implement the Knowledge Base, ServiceMap and the federation of ServiceMaps and smart city APIs. Km4City Ontology at the basis of the RDF Store. It is an expert system on the city data and entities in which smart city API can perform queries with spatial, temporal and relational reasoners. https://www.snap4city.org/19 https://www.snap4city.org/download/video/DISIT-km4city-City-Ontology-eng-v5-1.pdf Points of Interest (POIs): includes all services, activities, which may be useful to the citizen and who may have the need to “search-for” and to “arrive-at”. The two-level categorization of services and activities with more than 30 classes and more than 520 subclasses. Public Mobility (as Local public transport) the ontology includes concepts for the modelling of public transport infrastructures, time schedules, real-time records. More recently, the General Transit Feed Specification (GTFS) Linked Open Vocabulary has also been imported and it is used with minor integrations, that has allowed to supersede the local dimension of the original modelling. Sensors: concepts for the modelling of specific types of sensor is part of the ontology since the early times, for traffic and car park occupancy. Time (formerly, Temporal): concepts and relations wrapped that enable the representation of time intervals and instants of time to be associated to detections and predictions. Metadata: collection of metadata associated with datasets and dataset status, such as if they have been ingested and integrated. Such as: data of ingestion and update, licenses information, versioning, and so on. GIS data of any kind, ArcGIS, QGIS; coming with WFS, WMS. Traffic restrictions: in this macro-class concepts and relations are wrapped that enable the representation of forbidden and mandatory directions at crossroads (possibly at the lane level), access restrictions (even temporary/conditioned ones), size, weight, speed limitations, at the maximum possible level of detail, with the semantic of values that is borrowed from the definition of corresponding Open Street Map tags, and therefore public, maintained, rigorous. Not only routing, but also traffic prediction and reconstruction are so enabled, and therefore advanced routing, air quality prediction, and more. |
|
|
In Snap4City, a large collection of services to: exploit queries and reasoning on the storage and Knowledge Base, access/control IoT Network, exploit Data Analytic results, exploit IoT Apps, etc. All the data and services are accessible via the Smart City API which are used by Front End Tools such as Dashboards, Web and Mobile Apps, MicroApplications. Details regarding Smart City API are reported in: https://www.snap4city.org/download/video/course2020/app/ Internal APIs: https://www.km4city.org/swagger/internal/index.html
External APIs: https://www.km4city.org/swagger/external/index.html
|
|
|
Super (Super Service Map), |
See Federated Knowledge Base. A tool on top of Smart City API, which is the API interface of ServiceMap, Knowledge Base. It allows to create a network of Smart Cities with the data exchange in a very simple and elastic manner. It has been recently improved with the possibility of federating smaller cities which do not have a knowledge model but only an IoT Orion Broker by using the Snap4City tool called SSM2ORION presently in testing. |
|
Servicemap WFS integration |
Direct URLs, ArcGIS/GIS: WFS, WMS: such as URI/URLS to perform rest call to external services, such as GIS server. They can be directly accessed to show data on Dashboards, exploiting protocols such as WFS and WMS, which can be also produced by Snap4City ServiceMap. |
|
OSM2SM |
It allows to select and ingest a segment of OSM (open street map) into the Service map knowledge base of Snap4City for shortening the understanding and mapping of data and semantic layer of the area. The process is automated can be easily done for large area in short time. https://www.snap4city.org/175 |
|
SSM2ORION a module for connecting an IoT Orion Broker with its Data Shadow implemented by using Quantum Leap with the Federation of Knowledge Base. This solution allows to connect at the Smart City also other already in place FIWARE solutions which may have local storage. The queries performed on Snap4City Smart City API will provide seamlessly the results also providing the geo data which are stored into that Orion Broker without the need to register the IoT Devices of that IoT Orion Broker into the Knowledge Base. Some limitations are present on the security aspects. See also in the following. |
The main component includes:
- HeatMap Server exposes API for (i) colleting data regarding Heatmaps, (ii) providing information about the value of the map in any GPS point included, the so called heatmap picking, (iii) automated generation of Heatmaps in GeoTiFF format according to tiled which are distributed by a GIS via WMS protocol (they can be ArcGIS or GeoServer). https://www.snap4city.org/536 https://www.snap4city.org/507
- Also exposes API for collecting Copernicus satellite data into Heatmap server and directly converting them into heatmaps
- Integration with GIS solution.
The main component includes:
|
Snap4City Living Lab Drupal Modules |
A CMS, Content Management System, for Living Lab. See for its usage Https://www.snap4city.org and part 6 of the training course since it is the main support for the Living Lab: https://www.snap4city.org/download/video/course2020/sys/ Snap4City Living lab Support and Methodology https://www.snap4city.org/206 https://www.snap4city.org/82 On CRM as Drupal 7 we have
|
|
LDAP/KeyCloak Drupal Modules, GDPR |
Lightweight Directory Access Protocol, for user registration role management https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol For identify and access management, authentication and SAML, and SSO. |
The main component includes Snap4City Maintenance Solution:
- OpenMaint open source solution for property & facility management which is a BPM;
- Inventory of industry assets (movable, logistics, equipment, etc.)
- Tickets management for corrective maintenance
- User management with different levels of access
- BIM Server integrated with OpenMaint;
- Snap4City OpenMaint Extension
- Extended API developed by Snap4City
- Create new tickets
- Manage steps, workflow
- Collecting feedbacks and results from teams
- Manage all phases of the workflow on the fields via IOT Apps and logics
- The integration if via API and MicroServices into IOT App.
- MicroServices integrated with Snap4City via IOT Applications;
- Extended API developed by Snap4City
- Business Intelligence which is the Snap4City tool based on Elastic Search: which work on top of the database of tickets collected on OpenMaint;
- BIMServer integration with Snap4City Dashboards.
The main component includes:
|
Secure Filter (Proxy Filter) |
The Orion Broker Filter (also called: "Security Proxy Filter", "Proxy Filter Security" or "NGSI Security Wrapper") is a NGSI compatible Security wrapper to enforce security connection and authentication ( LDAP KeyCloak) on NGSI protocol. It is supposed to work with IOT Orion Broker, but can be easily adapted to other IOT Brokers. The component has been developed by Snap4City DISIT lab and can be obtained from GitHUB: https://github.com/disit/snap4city/tree/master/OrionBrokerFilter https://www.snap4city.org/468 |
|
FIWARE component: https://FIWARE-orion.readthedocs.io/en/master/ The IoT Orion Broker support natively the NGSI V1, V2 and LD. It usually provides only the last values of the IoT Devices and not the historical data (data shadow). In order to have the Data Shadow, and thus to perform queries on time series, the IoT Orion Broker has to be connected to a storage. In Snap4City, the storage provided is Elastic Search which provides high capabilities for insert, search and retrieval and scalability. See training part 3 https://www.snap4city.org/download/video/course2020/iot/ |
Snap4City can integrate any kind of IoT Broker which connect IoT Networks of IoT Devices.
- A mayor role is played by IoT Orion Broker of FIWARE by which the platform support NGSI V1, V2, LD with Services/Tenant and Service Paths. https://FIWARE-orion.readthedocs.io/en/master/
- The IoT Orion Brokers can be controlled by the same administrators of the Snap4City Platform on your premise (so called Internal IoT Broker) or can be External IoT Broker controlled by third parties. Brokers may exploit: IoT Agent, IoT Adapter, wrappers, IoT Edge Devices.
- Snap4City platform supports many different IoT Broker and more than 70 different protocols push/pull, sync/async [Https://www.snap4city.org/65] among them: MQTT, NGSI, COAP, OneM2M, ModBus, OPC, OPC-UA, Copernicus, and AMQP. See for the list of the protocols supported.
- Supported data models are: FIWARE Data Model, and IoT Device Models. The former are automatically converted for the Knowledge base.
- IoT Devices and IoT Broker are registered into IoT Directory which is multiprotocol multibroker tool for IoT Network management, also providing features of IoT Discovery in collaboration with the Knowledge Base. The registration of IoT Devices into Knowledge Base from IoT Orion Brokers is performed automatically.
The main component includes:
|
Snap4City Process NIFI |
allows to perform a copy of any message arriving from/to IOT Devices via the IOT Brokers registered on the IOT Directory. This feature consists of a cluster of VMs, and it is necessary when large volume of IOT Data comes into the system. The solution is based on a cluster NIFI, Elastic Search and Kibana. The solution is scalable and reliable, and fault tolerant. Elastic Search is the storage in which the IOT data are stored. Once the data is collected, the historical values are made available for the Dashboards, and IOT Applications via MicroServices of the Smart City API and thus of the Knowledge Base. To start you can have one single IOTDSES while the solution can scale up cloning and clustering the IOTDSES . Each NIFI server is managing on a modest VM more than 120.000 msg per minute. https://www.snap4city.org/538 |
The main component includes:
|
Snap4City extensions |
a cluster of Elastic Search and Kibana VMs. The number of VM depends on the size of the global storage which can increase elastically with the need along the operation. In alternative the usage of a shared common SAN and docker could be viable if it is fast enough. This component allows to perform the access control to data on top of elastic search. It can be integrated also into the IOTDSES in some configurations. |
|
A Snap4City tool for monitoring the global flow of data entering into the platform and reaching the Elastic Search. https://www.snap4city.org/198 https://www.snap4city.org/152 https://www.snap4city.org/145 |
The main component includes:
|
IoT App, Container management, Marathon update |
IoT App in Node-RED are used to implement processes of integration, data transformation, conversion, adaptation, and business logic. They can be used to collect data from IoT Broker as even drive, from External Services (e.g., IoT Gateways, web servers, ftp server, social media services, linked open data, linked data, ESBs), by using any kind of Connector, calling any kind of protocol and/or getting/transforming any kind of format into a Data Model / IoT Device Model for the Platform. Ingestion can be in PUSH/PULL to set up periodic/sporadic processes, as well as data driven (real time) processes of data ingestion and reaction/action for the platform towards other internal/external elements including other IoT App. See training part 3 https://www.snap4city.org/download/video/course2020/iot/ IoT Edge Device is an IoT Device capable to execute IoT Apps which also have installed Snap4City Libraries of MicroServices. See also
|
|
Node-RED Snap4City adaptation |
A visual Editor for Node.JS processes from JS Foundation. |
|
Snap4City Libraries MicroServices |
Snap4City platform is based on MicroServices. They are realized on the basis of the API (both internal and external). In Snap4City, the term MicroService is a synonym of Node in the Node-RED terminology. Each Snap4City node in the Snap4city Libraries for Node-RED is a MicroService of the platform. Other MicroServices can be easily added for customization and mapping of REST CALL APIs internal or of third party or of additional services, or from Container with Data Analytics. https://www.snap4city.org/22 https://www.snap4city.org/106 https://www.snap4city.org/129 For the recent list of MicroServices you have to see the documentation into the Library: https://flows.nodered.org/node/node-red-contrib-snap4city-user https://flows.nodered.org/node/node-red-contrib-snap4city-developer |
|
Performed with Plumber: Snap4City RStudio server (open source tool) for developing RStudio processes for machine learning, AI and statistical purpose on data. It can access to data via Smart City API and the RStudio processes can be transformed into Containers. They can exploit Tensor Flow and Keras and CUDA. Any data analytic in R Studio developed on the on line development environment of Snap4City can be deployed into a container managed by IoT App and may exploit smart city API for data analytic. |
|
|
Data Analytic in Python |
Performed with Flask and Jupyter Editor. Any data analytic in Python developed on the on line development environment of Snap4City can be deployed into a container managed by IoT App and may exploit smart city API for data analytic. |
|
Web Scraping |
Web Scraping/Portia: activating a process to grab data from your and/or third parties web pages and convert them in real time to a Data Model in JSON, IoT Device Model for the platform. Web Scraping processes are also controlled by IoT App in Snap4City. |
|
DISCES-EM |
Distributed scheduler for elastic management for Smart City by Snap4City. This tool is used for the elastic management of Container on the Marathon/Mesos cluster of containers. Read more on https://www.snap4city.org/232 |
For the data analytic development, it is possible to access the Big Data store respecting the privacy and the data licensing by using authenticated Smart City APIs. The access has to permit to read historical and real time data, and to save the resulting data provided by the algorithms, for example, heatmap-related predictions, the assessment of data quality, and labels of detected anomalies. The work of the data scientist could finish once the algorithm has been developed which he/she should be aware of. On the other hand, the same algorithm (e.g., for computing heatmaps, parking prediction), should allow to be:
- Used on different services of the same kind located in different places and on the basis of a number of parameters (e.g., target precision and list of data sources). This means that data analytic itself has to be designed with the needed flexibility and generality;
- put in execution from IoT Applications by passing a set of parameters and collecting the results on the Data Storage or as a result of the invocation. The executions can be periodic or event driven —e.g., the arrival of a request or by the arrival of the new set of data values;
- controlled for collecting eventual errors and mistakes, in debug and at run time for logging. This may be for informing the developer and/or the administrator of eventual mistakes and problems by sending notifications; and
- dynamically allocated on cloud in one or multiple instances to plan a massive computation of the same data analytic process on several data sets and services at the same time.
Therefore, the solution developed for Snap4City satisfies all the above described requirements. Data analytic processes can be developed using R Studio or Python. In both cases, the code has to include a library for creating a REST Call, namely: Plumber for R Studio and Flask for Python. In this manner, each process presents a specific API, which is accessible from an IoT Application as a MicroService, that is, a node of the above-mentioned Node-RED visual programming tool for data flow.
Data scientists can develop and debug/test the data analytic processes on Snap4City cloud environment since it is the only way to access at the Smart City API with the needed permissions. The source code can be shared among developers with the tool “Resource Manager”, that also allows the developers to perform queries and retrieve source code made available by other developers.
|
Performed with Plumber: Snap4City RStudio server (open source tool) for developing RStudio processes for machine learning, AI and statistical purpose on data. It can access to data via Smart City API and the RStudio processes can be transformed into Containers. They can exploit Tensor Flow and Keras and CUDA. Any data analytic in R Studio developed on the on line development environment of Snap4City can be deployed into a container managed by IoT App and may exploit smart city API for data analytic. |
|
|
Data Analytic in Python |
Performed with Flask and Jupyter Editor. Any data analytic in Python developed on the on line development environment of Snap4City can be deployed into a container managed by IoT App and may exploit smart city API for data analytic. |
The main component includes:
|
DataGate/CKAN for open data (https://ckan.org ): collecting open data from the Open Data network of CKAN harvesting via different protocols. And integrating those data into the platform. DataGate/CKAN can be also controlled by IoT App in Snap4City. CKAN support any kind of files in the domain of Open Data and any CKAN instance can connect with a network of CKAN in the world by harvesting data and sharing data as files and metadata descriptors. |
|
|
CKAN is an open source solution for Open Data management and distribution https://ckan.org/ In Snap4City is integrated with DataGate module and goes with SSO with the rest of tools. CKAN/DataGate can be controlled by IoT Apps. |
The main component includes:
|
UserProfiler |
Apps Users are profiled in the server according to the GDPR, collecting a range of data as described above. This approach allows us to tune the service according to the collective and personal user behaviour, with the aim of improving the user experience providing personalized (i) main menu arrangement of the functionalities according to the user profile (the user also may change it to have precise personalization), (ii) suggestions (the user may decide to ban some of the suggestion or even a category), (iii) menu search among categories (the user may decide to make further selection). Therefore, the menus offered for each profile kind is modified according to the statistics calculated on the collected data, by defining the positions the most researched categories, buttons, etc., and in some cases, putting off/on some functionality. For example, the triage monitoring may be of interest for citizens and operators and less for tourists, the parking status is interesting for users using the private car and less for those moving with the public busses or in bike, etc. The priority on the creation of the menu profiles is given in first place to the updated menu which can be found on the server. If the server is unreachable, the last saved menu for that specific profile is loaded; if the menu has not been saved, the loaded menu is the menu released with the application update, and so on. |
|
User contrib management |
To collect and manage (accept and moderate) contributions from mobile user such as: images, comments, etc. see tutorial https://www.snap4city.org/download/video/course2020/app/ |
The user profiler module can receive data from mobile Apps via some API and produce the collective profiles which can be used for many purposes (not included) such as computing: recommendations, OD matrices, typical trajectories, hot places, etc.
In the following table, the description of the components listed is reported. In the columns of the following table:
- Service Inside is the name of the service/module identified.
- Ownership the modules of DISIT Lab are clearly identified, and when possible the links to access at the open source are provided. As you can verify, all DISIT Lab components needed to create the platform are open source.
- The components market in green are those not owned by DISIT lab Snap4City (please note that are all open source according to the licence reported in the table).
- Licence: the name / identifier of the licence is reported.
Please note that the following table is divided in sections which correspond to the names/identifiers of the main components of the Snap4City platform offered and described in the previous sections, and also reported / referred in the official web page for installations of Snap4City: https://www.snap4city.org/471 .
No proprietary components are needed to be installed into the Snap4City platform.
|
|
Service Inside |
Ownership |
Licence |
|
Snap4City main |
|
||
|
|
Dashboard Builder and Engine, Data Inspector, Menu Manager, Wizard, widget Library, WebSocket Server |
Affero GPL |
|
|
|
Resource Manager (process loader), External Service Manager, Micro Application Manager |
Affero GPL |
|
|
|
Ownership Manager |
DISIT Lab https://github.com/disit/snap4city/tree/master/Ownership-API |
Affero GPL |
|
|
DISIT Lab https://github.com/disit/snap4city/tree/master/PersonalDataManager |
Affero GPL |
|
|
|
Affero GPL |
||
|
|
Affero GPL |
||
|
|
Affero GPL |
||
|
|
User Statistics |
DISIT Lab https://github.com/disit/snap4city/tree/master/UserStats |
Affero GPL |
|
|
Event Logger |
DISIT Lab https://github.com/disit/snap4city/tree/master/EventLogger |
Affero GPL |
|
|
Affero GPL |
||
|
|
Flint++ |
Affero GPL |
|
|
|
Report Generator |
DISIT Lab |
Affero GPL |
|
|
Flint |
MIT Licence |
|
|
|
LDAP, PHPLDAP admin |
Open LDAP https://www.openldap.org/ https://www.openldap.org/software/release/license.html |
open |
|
|
KeyCloak |
KeyCloak https://www.keycloak.org/ |
Apache License 2.0 |
|
|
Kafka |
Apache https://kafka.apache.org/ |
Apache License 2.0 |
|
|
MySQL |
MySQL |
|
|
|
TOMCAT |
Apache http://tomcat.apache.org/ |
Apache License 2.0 |
|
|
Squid |
GPL |
|
|
|
Zabbix Server |
Zabbix https://www.zabbix.com/ |
GPL |
|
KBSSM |
|
||
|
|
Affero GPL |
||
|
|
Super (Super Service Map), Servicemap WFS integration |
Affero GPL |
|
|
|
OSM2SM |
Affero GPL |
|
|
|
DISIT Lab, in beta testing |
Affero GPL |
|
|
|
Virtuoso |
Virtuoso http://vos.openlinksw.com/owiki/wiki/VOS http://vos.openlinksw.com/owiki/wiki/VOS |
open |
|
|
MySQL |
MySQL |
|
|
|
TOMCAT |
Apache http://tomcat.apache.org/ |
Apache License 2.0 |
|
|
PostgreSQL |
PostgresQL https://www.postgresql.org/ |
Open Source license, similar to the BSD or MIT licenses |
|
Heatmap Server |
|
||
|
|
DISIT Lab https://github.com/disit/snap4city/tree/master/Heatmap |
Affero GPL |
|
|
|
MySQL |
MySQL |
|
|
|
GeoServer |
GeoServer http://geoserver.org/ |
GPL |
|
|
PostgreSQL |
PostgresQL https://www.postgresql.org/ |
Open Source license, similar to the BSD or MIT licenses |
|
Living Lab Support |
|
||
|
|
Snap4City Living Lab Drupal Modules |
DISIT Lab https://github.com/disit/snap4city/tree/master/Drupal-Modules |
Affero GPL |
|
|
LDAP/KeyCloak Drupal Modules, GDPR |
DISIT Lab https://github.com/disit/snap4city/tree/master/DrupalGDPR-Module |
Affero GPL |
|
|
Drupal 7 |
Drupal https://www.drupal.org/ |
GPL |
|
|
MySQL |
MySQL |
|
|
OpenMaint Server |
|
||
|
|
OpenMaint |
OpenMaint https://www.openmaint.org/en/home |
Affero GPL |
|
|
BIMServer |
https://github.com/opensourceBIM/BIMserver/tree/master/BimServer |
Affero GPL |
|
|
Shark Workflow |
GPL |
|
|
|
Alfresco |
https://www.alfresco.com/ecm-software/alfresco-community-editions |
LGPLv3 license |
|
|
PostgreSQL |
Postgres https://www.postgresql.org/ |
Open Source license, similar to the BSD or MIT licenses |
|
|
TOMCAT |
Apache http://tomcat.apache.org/ |
Apache License 2.0 |
|
|
GeoServer |
GeoServer http://geoserver.org/ |
GPL |
|
IOTOBSF |
|
||
|
|
Secure Filter (Proxy Filter) |
DISIT Lab https://github.com/disit/snap4city/tree/master/OrionBrokerFilter |
Affero GPL |
|
|
Affero GPL |
||
|
|
Mongo |
Mongo, Apache Licence |
|
|
|
TOMCAT |
Apache http://tomcat.apache.org/ |
Apache License 2.0 |
|
IOTDSES |
|
||
|
|
Snap4City Process NIFI |
DISIT Lab https://github.com/disit/snap4city/tree/master/NIFI-processors |
Affero GPL |
|
|
NIFI |
Apache https://nifi.apache.org/ |
Apache License 2.0 |
|
|
Squid |
GPL |
|
|
ESSTORE |
|
||
|
|
Snap4City extensions |
DISIT Lab https://github.com/disit/security forked by Open Distro ES security (Open Distro is the Elastic Search + Kibana version of AWS) |
Affero GPL |
|
|
DISIT Lab |
Affero GPL |
|
|
|
Open Distro (Elastic Search 7) |
acutally: AMAZON Open Distro https://opendistro.github.io/for-elasticsearch/faq.html |
Apache 2.0-licensed Elasticsearch and Kibana 7.1 |
|
|
Open Distro (Kibana) |
acutally: AMAZON Open Distro https://opendistro.github.io/for-elasticsearch/faq.html |
Apache 2.0-licensed Elasticsearch and Kibana 7.1 |
|
MCLSCount |
|
||
|
|
IoT App, Container management, Marathon update |
DISIT Lab https://github.com/disit/snap4city/commit/4ab44b869b82cd10eda8270ef76d2828d793df51 |
Affero GPL |
|
|
DISCES-EM |
DISIT Lab https://github.com/disit/snap4city/tree/master/Disces-em |
Affero GPL |
|
|
Snap4City Libraries MicroServices |
DISIT Lab https://github.com/disit/snap4city/tree/master/nodered-snap4city-microservices |
Affero GPL |
|
|
Node-RED Snap4City adaptation |
DISIT Lab https://github.com/disit/node-red https://github.com/disit/snap4city/tree/master/node-red |
Apache License 2.0 |
|
|
Plumber |
Plumber https://www.rplumber.io/ |
MIT |
|
|
Flask |
||
|
|
Portia Web Scraper |
Open Source |
|
|
|
Marathon |
Mesosphere https://github.com/mesosphere/marathon-lb |
Apache License 2.0 |
|
|
Mesos |
Apache License 2.0 |
|
|
|
Apache License 2.0 |
||
|
|
Docker |
Docker https://www.docker.com/ |
Apache License 2.0 |
|
|
Zabbix Agent |
Zabbix https://www.zabbix.com/ |
GPL |
|
RStudio Server |
|
||
|
|
RStudio Server |
RSTUDIO https://rstudio.com/ |
Affero GPL |
|
Python Server |
|
||
|
|
Python Editor |
BSD 3-Clause License |
|
|
|
Docker |
Docker https://www.docker.com/ |
Apache License 2.0 |
|
|
|||
|
|
Affero GPL |
||
|
|
Affero GPL |
||
|
|
PostgreSQL |
Postgres https://www.postgresql.org/ |
Open Source license, similar to the BSD or MIT licenses |
|
UserProfile Management Server |
|
||
|
|
UserProfiler |
DISIT Lab |
Affero GPL |
|
|
User contrib management |
DISIT Lab |
Affero GPL |
|
|
MySQL |
MySQL |
|
|
|
TOMCAT |
Apache http://tomcat.apache.org/ |
Apache License 2.0 |
|
|
Squid |
GPL |
|
As described above we would remark the modules that are used in the proposed solution and are not maintained by Snap4City.
|
Service Inside |
Ownership |
Licence |
|
Alfresco |
https://www.alfresco.com/ecm-software/alfresco-community-editions |
LGPLv3 license |
|
BIMServer |
https://github.com/opensourceBIM/BIMserver/tree/master/BimServer |
Affero GPL |
|
Affero GPL |
||
|
Docker |
Docker https://www.docker.com/ |
Apache License 2.0 |
|
Drupal 7 |
Drupal https://www.drupal.org/ |
GPL |
|
Elastic Search 7 |
Apache License 2.0, Apache License 2.0 compatible license, Elastic License |
|
|
Flask |
||
|
Flint |
MIT Licence |
|
|
Inkscape SVG Editor |
Open Source / Freeware for standard SVG editing |
|
|
Affero GPL |
||
|
Kafka |
Apache https://kafka.apache.org/ |
Apache License 2.0 |
|
KeyCloak |
KeyCloak https://www.keycloak.org/ |
Apache License 2.0 |
|
Kibana |
Apache License 2.0, Elastic License |
|
|
LDAP, PHPLDAP admin |
Open LDAP https://www.openldap.org/ |
|
|
Marathon |
Mesosphere https://github.com/mesosphere/marathon-lb |
Apache License 2.0 |
|
Mesos |
Apache License 2.0 |
|
|
Mongo |
Mongo, Apache Licence |
|
|
MySQL |
MySQL |
|
|
NIFI |
Apache https://nifi.apache.org/ |
Apache License 2.0 |
|
Apache License 2.0 |
||
|
OpenMaint |
OpenMaint https://www.openmaint.org/en/home |
Affero GPL |
|
Plumber |
Plumber https://www.rplumber.io/ |
MIT |
|
PostgreSQL |
PostgresQL https://www.postgresql.org/ |
Open Source license, BSD or MIT licenses |
|
Python Editor Jupyter Hub |
BSD 3-Clause License |
|
|
RStudio Server |
RSTUDIO https://rstudio.com/ |
Affero GPL |
|
Shark Workflow |
GPL |
|
|
Squid |
GPL |
|
|
TOMCAT |
Apache http://tomcat.apache.org/ |
Apache License 2.0 |
|
Virtuoso |
||
|
Zabbix Agent |
Zabbix https://www.zabbix.com/ |
GPL |
|
Zabbix Server |
Zabbix https://www.zabbix.com/ |
GPL |
DISIT Snap4City Components that may be optional
|
|
Service Inside |
Ownership |
Licence |
|
Snap4City Mobile App Dev. Kit |
|
||
|
|
Complete mobile app from the “… in a Snap” serie as accessible from Google Play and Apple store |
DISIT Lab provided in Open Source only with limitation to be exploited on your area in which you have coverage with your snap4city installation at platform level. Please request very simple agreement to be signed. |
Affero GPL, with non exploitation agreement for the same area to avoid duplications in the same area. No licence fee. |
|
|
Small App Kit |
Affero GPL |
|
|
|
Swagger staff |
DISIT Lab https://github.com/disit/snap4city/tree/master/swagger |
Affero GPL |
|
Other Snap4City Tools |
|
||
|
|
DISIT Lab https://github.com/disit/engager https://github.com/disit/snap4city/tree/master/Engager |
Affero GPL |
|
|
|
Affero GPL |
||
|
|
What-IF Router, Routing Server |
Fork from Graphhopper https://www.graphhopper.com/ |
GNU |
|
|
SmartDS |
Affero GPL |
|
|
|
Resilience DS, FRAM DS |
Affero GPL |
|
|
|
WLODE |
Affero GPL |
|
|
|
recommender |
Affero GPL |
|
|
|
ETL processes |
DISIT Lab https://github.com/disit/snap4city/tree/master/ETL |
Affero GPL |
|
|
Data Analytics in R |
DISIT Lab https://github.com/disit/snap4city/tree/master/Statistics |
Affero GPL |
|
|
DISIT Lab https://github.com/disit/sce-frontend https://github.com/disit/sce-backend |
Affero GPL |
|
|
|
Data collection from Mobile App, SSM |
DISIT Lab https://github.com/disit/km4c-sensor-server-manager |
Affero GPL |
|
|
Twitter Vigilance RT (real time) |
DISIT Lab |
Proprietary, with source code accessible |
|
|
Twitter Vigilance |
DISIT Lab |
Proprietary |
|
|
Traffic Flow reconstruction |
Proprietary, with source code accessible |
|
|
|
Other Specific Data Analytics not included above |
DISIT Lab: anomaly detection, parking predictions, bike sharing prediction, NOX predictions, OD production, trajectory production, etc. |
Proprietary, with source code accessible |
|
|
DORAM |
tool for the analysis of demand of mobility vs offer of transportation; Tools; beta development |
Proprietary, with source code accessible |
