HOW TO: Deploy & Installation of Snap4City DataCity-Large 2020

Snap4City platform consists of a set of 100% open source, modular and integrated tools accessible according to the user profile. A technical version of the Snap4City architecture is reported in the next figures, where the main elements are reported and described below.

No proprietary components are needed to be installed into the Snap4City platform.

From Left to right of the above figure.

Data Ingestion and aggregation is performed in push/pull by using: IoT Connectors, IoT Agent, IoT Adapter, IoT Broker, IoT Apps in Node-RED, and data gathering tools as DataGate/CKAN for open data (https://ckan.org ), ArcGIS, and Portia for Web Scraping. The Snap4City platform supports a very large number of protocols push/pull, sync/async among them: MQTT, NGSI, COAP, OneM2M, ModBus, OPC, WMS, WFS, and AMQP. See for the list of the protocols supported Https://www.snap4city.org/65     Data Models, IoT Device Models, IoT Devices, and IoT Broker are registered via IoT Directory which is multiprotocol multibroker tool for IoT Network management. Among the brokers, a mayor role is played by IoT Orion Broker of FIWARE by which the platform support NGSI V1, V2, LD with Services/Tenant and Service Paths. And its elements as IoT Agents, etc. A detailed description of IOT capabilities and network are provided in: https://www.snap4city.org/download/video/course2020/sys/ while a large number of detail regarding the data ingestion are reported in https://www.snap4city.org/download/video/course2020/di/

Data Storage is feed by using NIFI Cluster for massive data aggregation/enrichment (collecting data from any IoT Broker at which it is automatically registered), and saving data to Elastic Search cluster for data storage and indexing, and to the Knowledge Base (based on Km4City ontology) implemented as an RDF store (Virtuoso) which is an index for geo-spatial, relational, and temporal aspects. Every time a new data model is registered in the system, the registration is performed into the Knowledge Base. Knowledge base and Smart City API can be federated creating a mutual connection among friend cities. In the Snap4City Platform, (i) Heatmaps, Orthomaps and Maps are managed by the Heatmap Server which is a GIS (for example GeoServer or ArcGIS if you have one installed with WMS facilities) and can distribute the maps according to WMS/WFS protocols towards Web and Mobiles App and Dashboards; (ii) Buildings 3D shapes, and Floors are managed and distributed into the OpenMAINT tool, standard BIM tools are used for editing and interchange in IFC formats by standard tools as AutoDesk Revit, etc. Buildings and Floors are shown in Dashboards for their integration with maps and IoT Devices time trends. Thus, creating a full Digital Twin model of any part of the city.  The tools for accessing to the Digital Twin is the so-called Data Inspector. Details can be obtained from: https://www.snap4city.org/download/video/course2020/di/

Data Transformations (reconciliation, regularization, enrichment, etc.) are implemented via IoT Apps (IoT Applications) which are processes for ingesting, integrating and manipulating data with PUSH/PULL protocols from any External Service and also from IoT Broker, and Virtual IoT Devices of any kind. IoT Apps are developed by using a visual programming data flow and/or data driven flows in Node-RED exploiting a large library of more than 150 Snap4City MicroServices.

Please note that, in Snap4City Terminology an IoT App is a process in Node-RED plus Snap4City library. The Library is based on nodes and call MicroServices, for this reason also the nodes are called MicroServices. So that often it is state that the Snap4City library in Node-RED provide a large number of MicroServices, in the sense that provide access to MicroServices via corresponding nodes.

They are also used for implementing user interface logic, Business Logic behind smart applications and Dashboards also in references of Custom Widgets and Synoptics, which can be very useful in Smart City Control Rooms. Please note that Data Transformation / Integrations may exploit internal and external storages from any kind database, any connectors from/to data sources, any External Service, any direct connection with IoT Devices and Networks, etc. A large number of details regarding the data ingestion are reported in https://www.snap4city.org/download/video/course2020/di/

Data Analytics are processes written in Rstudio or Python which can perform data access and apply statistics, Machine Learning, Artificial Intelligence, deep learning tools, algorithms also exploiting Tensor Flow, CUDA, Keras, etc. It is also possible to develop processes in JavaScript directly into IoT App, and in Java scheduling processes with DISCES (not included in the offer). Snap4City Data Analytics in Rstudio, Python and JavaScript are developed with an online development environment which also allows to put in execution the scripts in Containers which are deployed as API and MicroService for IoT App and other tools in the platform. Off-line development environments are viable as well, if preferred. Data Analytics can produce: predictions, early warning, traffic flow reconstruction, alarms detecting anomalies, KPI and indicators, Typical Time Trends, typical trajectories, Routing and paths, travel plans, query results, simulations, analysis, calibrated heatmaps, smart parking suggestions, car sharing analysis, etc. A large number of examples are provided in: https://www.snap4city.org/download/video/course2020/da/

Smart City APIs include a large collection of services to: exploit queries and reasoning on the storage and Knowledge Base, access/control IoT Network, exploit Data Analytic results, exploit IoT Apps, etc. All the data and services are accessible via the Smart City API which are used by Front End Tools such as Dashboards, Web and Mobile Apps, MicroApplications. Details regarding Smart City API are reported in: https://www.snap4city.org/download/video/course2020/app/ Knowledge base and Smart City API can be federated creating a mutual connection among friend cities.

Dashboards can be created by Dashboard Builder and/or Kibana for different kind of users such as: decision makers, city operators, ICT operators, private for the users, etc., suitable for Smart City Control Rooms with video wall, for city Operators on Desktop of multiple monitors, mobile operators, and Situation Rooms with touch panels. Dashboards can exploit all kind of data and Data Analytic, legacy services, and special tools as traffic flow reconstruction, decision support systems, etc. Dashboards area created by using a large range of ready to use Widgets, and also Custom Widgets creating Synoptics as SVG elements, or exploiting other graphic libraries, such as D3, Highcharts, etc. Examples on dashboard and how they can be authored are reported in https://www.snap4city.org/download/video/course2020/das/

Web and Mobile Applications can be created by developers exploiting Smart City APIs and may be controlled by Snap4City tools. For example, to send on the Mobile Apps: engagements, soundages, stimulus, and thus for monitoring user behaviour, creating origin destination matrices, getting reaction from the city users, informing user at the inception of critical conditions, etc. Details regarding Smart City API and their usage for the development of Web and Mobile Apps with related development environment are reported in: https://www.snap4city.org/download/video/course2020/das/

Aspects of Platform Management and Quality Control of back-office include managing users’ activities, platform setup and control, user auditing, assessment, monitoring, security, process management, and elastic scaling. Details and examples are reported in https://www.snap4city.org/download/video/course2020/sys/

Snap4City Platform Architecture

Snap4City platform can be installed in several different configurations exploiting the same modules. The different configurations may be suitable from small cities to large cities with a high number of users and huge volume of data. The different modules can be provided as VM (Virtual Machines) Appliances or Dockers. The different Snap4City configurations and each component mentioned in the following are described in detail with the instructions for their installation at page: https://www.snap4city.org/471 Please note that each VM is an appliance with a number of components which can be updated directly from GITHUB/DISIT. https://github.com/disit

DataCity-Large which is reported in the following figure, and it is mentioned quite similarly into Tutorial Part 6  https://www.snap4city.org/download/video/course2020/sys/

No proprietary components are needed to be installed into the Snap4City platform.

DCL: DataCity-Large configuration includes installations of:

  • Snap4CityMAIN VM/Docker compose with: Dashboard Builder, Wizard, library of Widget, WebSocket Servers, Data Inspector, User Statistics, External Service manager, Resource Manager, Micro Applications manager, LDAP/KeyCloak for Authorization/Authentication, Kafka (which has to be scaled up if the traffic on synoptic is high), MyKPI, MyPOI, Synoptic and Custom Widget/PINs, and IoT Directory;
  • KBSSM VM: ServiceMap, Smart City API, Knowledge Base Km4City (LOG, Flint, Virtuoso): which can be duplicated in federated configurations to increase robustness/volume of traffic or in FT;
  • Heatmap Server VM with an heatmap manager to generate the GeoTIFF which are loaded into a GIS for their distribution via WFM protocol such as ArcGIS or GeoServer; which can be started in FT;
  • Living Lab Support VM with Drupal 7 with Snap4City extensions; which can be started in FT;
  • MCLSCount VM which is cluster of VM to manage Docker Containers with Marathon, Mesos and DISCES-EM. They are containers for IoT Apps, DataAnalytics, Web Scraping; This cluster has 3 VM for the masters and a number of VM for the Dockers/containers. It may have sense to have a true cluster if the Dockers are more than 200 Containers, otherwise it may be enough to have a static allocation of them in a few VMs in FT. The Snap4City solution has been tested on thousands of Dockers/Containers and the larger cluster of VM consists presently of 16 VMs for Dockers elastically managed. [Scalable2018], [DataFlow2019].
    • Typically, we have an average of 70 Containers for each VM depending on the image size of the container, and VM size. The Dockers for Data Analytics cost about 4 times bigger in terms of CPU and Memory with respect to those for IoT Apps, and have a more sporadic usage of the resources. Their timing is more relaxed.
  • IOTDSES a cluster of federated NIFI on Dockers and their balancer, they are allocated on VM as well;
  • ESSTORE VM a cluster of Elastic Search and Kibana VMs. The number of VM depends on the size of the global storage which can increase elastically with the need along the operation. In alternative the usage of a shared common SAN and docker could be viable if it is fast enough:
  • IOTOBSF VM: IoT Orion Brokers (V1/V2) also on Dockers or VM, internal or external, etc.;
  • Rstudio Server VM: for development of R Studio processes; Their size depends on the number of developers you have at the same time working on them;
  • Python Server VM: for development of Python processes; Their size depends on the number of developers you have at the same time working on them;
  • CKAN DataGate VM: for the connection to CKAN open data network;
  • OpenMAINT VM: including OpenMAINT and BIM, for the development of Workflow, BPM and BIM;

See more on: TC10.13- Back office Cluster Container Scalability and Monitoring Scalable Infrastructure with Elastic Management of applications and processes. https://www.snap4city.org/232

Snap4City platform can be installed in several different configurations exploiting the same modules.

Where in the table the numbers are only indicative, and refer to a large deploy of the solution for a large city or for a group of small cities:

  • CPU are reported in number of Cores at 2.1 GHz min
  • MEM is in GByte
  • HD are in GByte are high speed HD or SSD for Production while medium speed for the other configurations
  • Red: is the level of redundancy, number of VMs. They can be launched in FT or not
  • IOTOBSF could be dockers
  • MCLSCount is a cluster with dockers. If the solution has no access to orchestrator VMware, then a VM with Nagios/Zabbix has to installed to control the On/Off of VM from DISCES_EM. The alternative could be to manage the small cluster with always one VM.
  • IOTDSES a cluster of federated NIFI on Dockers and their balancer
  • ESSTORE VM a cluster of Elastic Search and Kibana VMs. The number of VM depends on the size of the global storage which can increase elastically with the need along the operation.

 

Snap4City main VM

The Snap4City main includes:

Dashboard Builder and Engine

In Snap4city, it is the main tool for creating Dashboards and connect them with IoT Apps, and other dashboards and custom widgets. It is substantially a Decision Support System tool, since it provides evidence of critical conditions, and may offer solutions. On this regard, it may integrate/exploit artificial intelligence algorithms, for example, reporting prediction, identifying anomalies, manifesting early warning, providing relationships among entities exploiting inference geospatial reasoning about what is located in the city: resources, structure, people, areas, critical infrastructures, etc. See details on training course part 2: https://www.snap4city.org/download/video/course2020/das/

Data Inspector

The Data Inspector is the main tool for the browsing of the information about a data stream entering into the platform. It is directly accessible from the main menu and give access to the Digital Twin representation of data, devices, and all High Level Types of the platform. It provides browsing capabilities to understand:

  • Data (and their Digital Twin information): data are heterogeneous, large volume, several protocols, legacy data systems, several semantics, real time, multiple domains, etc., such as: any kind of data for the smart city. They belong to the so called High Level Types:
  • Processes: several of different kinds, aperiodic, periodic, event driven, …. Managed by different tools (IoT Brokers, ServiceMap, HeatMap Server, Web scrapers, IoT Apps, etc.):
    • Data Ingestion: the models and mechanism for data gathering
    • Data Transformation: IOT App, integrations, ..
    • Data Storage, indexing, etc.
    • Data Analytic: predictions, anomaly detection, computing, etc.
    • Business Logic behind Dashboards
    • Data rendering, Dashboarding, interacting
  • Users: owners (who register), responsible, providers, developers, etc.
  • Relationships: among data, processes, users and mixed.

The Snap4City Data Inspector is the primary point to understand the issues and produce also the reports on those facts. For example, at level of: users, devices, process, dashboard and application. See this document on the early Version [DI-DataInspectorBase]. Now we have deployed since June 2020 the new version of Data Inspector and engine. See more details on part 5 of the training course: https://www.snap4city.org/download/video/course2020/di/

widget Library,

The library of graphic widgets which can be used to compose the dashboard. They can range from map, time trend, barseries, Kiviat, net, tables, multiserie, but also button, knob, custom widgets, lamp, panel on the road, doors, fan, parking, energy plant, metro map, etc

Custom widget, Synoptics

In Snap4City, widgets are the components of the Dashboard to visualize views on data (with animation, graphics, synoptics, etc.) and also to collect interaction from the users (buttons, faces to be clicked, sliders, keypad, text pad, etc.). see

HOW to create custom widgets in SVG, and examples: https://www.snap4city.org/651  https://www.snap4city.org/595  https://www.snap4city.org/644 https://www.snap4city.org/663

How to create widgets by programming: https://www.snap4city.org/153

Wizard, Dashboard Wizard

It is part of the Dashboard Builder, and it provides a support for selecting the best graphic widgets representation of the data your have. You can start from data and it suggest graphics, from maps and suggest data and graphics. Or you can start from graphics widgets and it provides the data which can be rendered on it. Wizards and Data Inspector share the same unified model and user interaction for many aspects.

Menu Manager

It allows to formalize the menu on the left, they are different for user role, and for each organization and for each mobile. The same Menu management system allows to formalize the menu for the Dashboards.

WebSocket Server

A communication protocol for real time connection. In Snap4City is only used in its TLS version Web Socket Secure. The WS allows to communicate from Server to mobile and dashboard in a secure and event driven manner.

Resource Manager (process loader)

Is a Market Place of Snap4City artefacts: IoT App, Flows/subflows, data analytics, ETL, Kibana Dashboards, etc. They can be searched, shared and promoted via a web portal.

https://www.snap4city.org/27        https://www.snap4city.org/205

https://www.snap4city.org/188      https://www.snap4city.org/134

External Service Manager

Snap4City external services can be any external services reachable with some protocol to get data and services. REST Call can be automatically transformed in MicroServices for the IoT App. The other can be called from the IOT Ap directly using a large range of protocols: https://www.snap4city.org/65

Micro Application Manager

A snap4City set of views implemented in HTML5 JavaScript for realizing specific functionalities. They are substantially views of Web and/or Mobile Apps which can be called independently and placed into Dashboard external content Widget as well as into Totems.

https://www.snap4city.org/dashboardSmartCity/management/microApplications.php

https://www.snap4city.org/54        https://www.snap4city.org/99

Ownership Manager, User management, User assessment

to manage users' ownerships and delegations to access at different kinds of data and resources, delegation to access, revocations, etc It provides API to be used by Web applications and tools and MicroServices to be used by IoT App. It provides services to all the Snap4City applications and it is GDPR compliant. The user and manager can have a unique place in which the Authorization and ACL are reported and can be audited.

IoT Directory

Snap4City tool for registering IoT Broker and Devices. Snap4City is almost agnostic about the brokers and accept a number of brokers and protocols registered on IoT Directory of Brokers and Devices. The IoT Directory is capable to browse on internal and external brokers to discover IoT Devices and register them on Knowledge Base.

https://www.snap4city.org/115 The IoT Directory is also the tool that manage all the network information about the devices deployed and connected. It is capable to exploit NGSI V1 and V2 protocol aspects to inspecting and managing IoT Orion Brokers of FIWARE, so that to manage a large number of them and automatically performing registration of devices in bulk on Knowledge base. See https://www.snap4city.org/76  https://www.snap4city.org/562, https://www.snap4city.org/647 See training part 3 https://www.snap4city.org/download/video/course2020/iot/

User Statistics

  • User vs registration: LDAP and KeyCloak
  • User Limits management
  • User vs consumption of resources
  • Content vs publication
  • Auditing data access try-out, Elements and Ownerships, personal data, accesses authentications, user activities, queries, articles, web pages, dashboards, IoT Directory, etc-
  • Org vs Groups, user vs orgs
  • Chat management

DataManager, KPI, MyKPI, MyPOI, Personal Data

Snap4City tool for collecting and managing personal KPI, POI. A MyKPI is a variable with may change over time determining a TimeSeries with variable GPS position at each time instant.  https://www.snap4city.org/396     https://www.snap4city.org/414

The Data Manager allows to manage the data that may start as KPI and POI into the platform as private and may be used as a personal storage. They can be converted later on private data. Typical KPI can be personal profile, password safe, etc.

Notificator

for programming notifications to be sent when some event occurs on Dashboard metrics. It is a tool at level of RootAdmin.

https://www.snap4city.org/140

Event Logger

It is a tool with API and MicroService collecting data on Elastic Search for the analysis of indexed EventLogger data for exploring different aspects about the workload, traffic flow data, the efficiency of the communications, the identification of the most computationally expensive communications, drill down, faceted filtering, thus to make profiling on App and tools. This tool is based on Rsyslog and tools for the indexing of collected log data. In Snap4City (as in many professional solutions and operating systems), LOGS/logs are produced reporting errors with different level of severity (e.g., warning, severe, etc.), they are produced in standard format as SYSlog. Those logs can be browsed, queried with tools, for example: standard LogStash which is based on Elastic Search, and Snap4City AMMA also based on Elastic Search. https://www.snap4city.org/137

disit.org">LOG.DISIT.ORG

Linked Open Graph tool (noting to do with Logs). LOG.DISIT is a tool for accessing and browsing Linked Open Data in the world and in the local Knowledge base, KB.    https://log.disit.org/service/

Flint++

It is a tool for developer to perform sematic queries on KB by using SPARQL.  https://www.snap4city.org/168

Report Generator

It is a tool to design and produce Reports.

 

The data ingested as Data Models are registered on Knowledge Base. The registering the data model is the first step to enable the possibility of loading data instances which can be: Static, quasi static and dynamic real time data, as even driven processes and streams. The main component includes:

ServiceMap

Km4City means Knowledge Model for the City. It is an Ontological model for the smart city and IoT Applications in smart city and many other domains. It is the core model adopted in Snap4City to design and implement the Knowledge Base, ServiceMap and the federation of ServiceMaps and smart city APIs. Km4City Ontology at the basis of the RDF Store. It is an expert system on the city data and entities in which smart city API can perform queries with spatial, temporal and relational reasoners. https://www.snap4city.org/19

https://www.snap4city.org/download/video/DISIT-km4city-City-Ontology-eng-v5-1.pdf

Points of Interest (POIs): includes all services, activities, which may be useful to the citizen and who may have the need to “search-for” and to “arrive-at”. The two-level categorization of services and activities with more than 30 classes and more than 520 subclasses.

Public Mobility (as Local public transport) the ontology includes concepts for the modelling of public transport infrastructures, time schedules, real-time records. More recently, the General Transit Feed Specification (GTFS) Linked Open Vocabulary has also been imported and it is used with minor integrations, that has allowed to supersede the local dimension of the original modelling.

Sensors: concepts for the modelling of specific types of sensor is part of the ontology since the early times, for traffic and car park occupancy.

Time (formerly, Temporal): concepts and relations wrapped that enable the representation of time intervals and instants of time to be associated to detections and predictions.

Metadata: collection of metadata associated with datasets and dataset status, such as if they have been ingested and integrated. Such as: data of ingestion and update, licenses information, versioning, and so on.

GIS data of any kind, ArcGIS, QGIS; coming with WFS, WMS.

Traffic restrictions: in this macro-class concepts and relations are wrapped that enable the representation of forbidden and mandatory directions at crossroads (possibly at the lane level), access restrictions (even temporary/conditioned ones), size, weight, speed limitations, at the maximum possible level of detail, with the semantic of values that is borrowed from the definition of corresponding Open Street Map tags, and therefore public, maintained, rigorous. Not only routing, but also traffic prediction and reconstruction are so enabled, and therefore advanced routing, air quality prediction, and more.

Smart City API

In Snap4City, a large collection of services to: exploit queries and reasoning on the storage and Knowledge Base, access/control IoT Network, exploit Data Analytic results, exploit IoT Apps, etc. All the data and services are accessible via the Smart City API which are used by Front End Tools such as Dashboards, Web and Mobile Apps, MicroApplications. Details regarding Smart City API are reported in: https://www.snap4city.org/download/video/course2020/app/

Internal APIs: https://www.km4city.org/swagger/internal/index.html

External APIs: https://www.km4city.org/swagger/external/index.html

Super (Super Service Map),

See Federated Knowledge Base. A tool on top of Smart City API, which is the API interface of ServiceMap, Knowledge Base. It allows to create a network of Smart Cities with the data exchange in a very simple and elastic manner. It has been recently improved with the possibility of federating smaller cities which do not have a knowledge model but only an IoT Orion Broker by using the Snap4City tool called SSM2ORION presently in testing.

Servicemap WFS integration

Direct URLs, ArcGIS/GIS: WFS, WMS: such as URI/URLS to perform rest call to external services, such as GIS server. They can be directly accessed to show data on Dashboards, exploiting protocols such as WFS and WMS, which can be also produced by Snap4City ServiceMap.

https://www.snap4city.org/368

OSM2SM

It allows to select and ingest a segment of OSM (open street map) into the Service map knowledge base of Snap4City for shortening the understanding and mapping of data and semantic layer of the area. The process is automated can be easily done for large area in short time. https://www.snap4city.org/175

SSM2ORION

SSM2ORION a module for connecting an IoT Orion Broker with its Data Shadow implemented by using Quantum Leap with the Federation of Knowledge Base. This solution allows to connect at the Smart City also other already in place FIWARE solutions which may have local storage. The queries performed on Snap4City Smart City API will provide seamlessly the results also providing the geo data which are stored into that Orion Broker without the need to register the IoT Devices of that IoT Orion Broker into the Knowledge Base. Some limitations are present on the security aspects. 

See also in the following.

 

The main component includes:

  • HeatMap Server exposes API for (i) colleting data regarding Heatmaps, (ii) providing information about the value of the map in any GPS point included, the so called heatmap picking, (iii) automated generation of Heatmaps in GeoTiFF format according to tiled which are distributed by a GIS via WMS protocol (they can be ArcGIS or GeoServer). https://www.snap4city.org/536 https://www.snap4city.org/507   
  • Also exposes API for collecting Copernicus satellite data into Heatmap server and directly converting them into heatmaps
  • Integration with GIS solution.

 

The main component includes:

Snap4City Living Lab Drupal Modules

A CMS, Content Management System, for Living Lab. See for its usage Https://www.snap4city.org and part 6 of the training course since it is the main support for the Living Lab:  https://www.snap4city.org/download/video/course2020/sys/

Snap4City Living lab Support and Methodology

https://www.snap4city.org/206 https://www.snap4city.org/82

On CRM as Drupal 7 we have

  • User Management registration and mailing
    • LDAP connection role management
    • KeyCloak Authentication (OpenID Connect)
    • Management of user profile
  • Content management for Organizations and Groups
    • Indexing of all content and search
    • Distribution of content and Video
    • Reports and views
    • Tracking and monitoring
  • Open to full contributions and comments
    • Workflow, expandable as all the CRM

 

LDAP/KeyCloak Drupal Modules, GDPR

Lightweight Directory Access Protocol, for user registration role management

https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol

For identify and access management, authentication and SAML, and SSO.

https://www.keycloak.org/

 

The main component includes Snap4City Maintenance Solution:

  • OpenMaint open source solution for property & facility management which is a BPM;
    • Inventory of industry assets (movable, logistics, equipment, etc.)
    • Tickets management for corrective maintenance
    • User management with different levels of access
    • BIM Server integrated with OpenMaint;
  • Snap4City OpenMaint Extension
    • Extended API developed by Snap4City
      • Create new tickets
      • Manage steps, workflow
      • Collecting feedbacks and results from teams
      • Manage all phases of the workflow on the fields via IOT Apps and logics
      • The integration if via API and MicroServices into IOT App.
    • MicroServices integrated with Snap4City via IOT Applications;
  • Business Intelligence which is the Snap4City tool based on Elastic Search: which work on top of the database of tickets collected on OpenMaint;
  • BIMServer integration with Snap4City Dashboards.

 

The main component includes:

Secure Filter (Proxy Filter)

The Orion Broker Filter (also called: "Security Proxy Filter", "Proxy Filter Security" or "NGSI Security Wrapper") is a NGSI compatible Security wrapper to enforce security connection and authentication ( LDAP KeyCloak) on NGSI protocol. It is supposed to work with IOT Orion Broker, but can be easily adapted to other IOT Brokers.  The component has been developed by Snap4City DISIT lab and can be obtained from GitHUB: https://github.com/disit/snap4city/tree/master/OrionBrokerFilter https://www.snap4city.org/468

IoT Orion Broker Fiware

FIWARE component: https://FIWARE-orion.readthedocs.io/en/master/

The IoT Orion Broker support natively the NGSI V1, V2 and LD. It usually provides only the last values of the IoT Devices and not the historical data (data shadow). In order to have the Data Shadow, and thus to perform queries on time series, the IoT Orion Broker has to be connected to a storage. In Snap4City, the storage provided is Elastic Search which provides high capabilities for insert, search and retrieval and scalability. See training part 3 https://www.snap4city.org/download/video/course2020/iot/

 

Snap4City can integrate any kind of IoT Broker which connect IoT Networks of IoT Devices.

 

The main component includes:

Snap4City Process NIFI

allows to perform a copy of any message arriving from/to IOT Devices via the IOT Brokers registered on the IOT Directory. This feature consists of a cluster of VMs, and it is necessary when large volume of IOT Data comes into the system. The solution is based on a cluster NIFI, Elastic Search and Kibana. The solution is scalable and reliable, and fault tolerant. Elastic Search is the storage in which the IOT data are stored. Once  the data is collected, the historical values are made available for the Dashboards, and IOT Applications via MicroServices of the Smart City API and thus of the Knowledge Base. To start you can have one single IOTDSES while the solution can scale up cloning and clustering the IOTDSES . Each NIFI server is managing on a modest VM more than 120.000 msg per minute. https://www.snap4city.org/538

 

The main component includes:

Snap4City extensions

a cluster of Elastic Search and Kibana VMs. The number of VM depends on the size of the global storage which can increase elastically with the need along the operation. In alternative the usage of a shared common SAN and docker could be viable if it is fast enough. This component allows to perform the access control to data on top of elastic search. It can be integrated also into the IOTDSES in some configurations.

DevDash, My Dashboard Kibana

A Snap4City tool for monitoring the global flow of data entering into the platform and reaching the Elastic Search.

https://www.snap4city.org/198    https://www.snap4city.org/152    https://www.snap4city.org/145

 

The main component includes:

IoT App, Container management, Marathon update

IoT App in Node-RED are used to implement processes of integration, data transformation, conversion, adaptation, and business logic. They can be used to collect data from IoT Broker as even drive, from External Services (e.g., IoT Gateways, web servers, ftp server, social media services, linked open data, linked data, ESBs), by using any kind of Connector, calling any kind of protocol and/or getting/transforming any kind of format into a Data Model / IoT Device Model for the Platform. Ingestion can be in PUSH/PULL to set up periodic/sporadic processes, as well as data driven (real time) processes of data ingestion and reaction/action for the platform towards other internal/external elements including other IoT App. See training part 3 https://www.snap4city.org/download/video/course2020/iot/

IoT Edge Device is an IoT Device capable to execute IoT Apps which also have installed Snap4City Libraries of MicroServices. See also

Node-RED Snap4City adaptation

A visual Editor for Node.JS processes from JS Foundation.

https://nodered.org/ https://flows.nodered.org/

Snap4City Libraries MicroServices

Snap4City platform is based on MicroServices. They are realized on the basis of the API (both internal and external). In Snap4City, the term MicroService is a synonym of Node in the Node-RED terminology. Each Snap4City node in the Snap4city Libraries for Node-RED is a MicroService of the platform. Other MicroServices can be easily added for customization and mapping of REST CALL APIs internal or of third party or of additional services, or from Container with Data Analyticshttps://www.snap4city.org/22      https://www.snap4city.org/106    https://www.snap4city.org/129

For the recent list of MicroServices you have to see the documentation into the Library:

https://flows.nodered.org/node/node-red-contrib-snap4city-user

https://flows.nodered.org/node/node-red-contrib-snap4city-developer

Data Analytics in R Studio

Performed with Plumber: Snap4City RStudio server (open source tool) for developing RStudio processes for machine learning, AI and statistical purpose on data. It can access to data via Smart City API and the RStudio processes can be transformed into Containers. They can exploit Tensor Flow and Keras and CUDA. Any data analytic in R Studio developed on the on line development environment of Snap4City can be deployed into a container managed by IoT App and may exploit smart city API for data analytic.

https://rstudio.com/

Data Analytic in Python

Performed with Flask and Jupyter Editor. Any data analytic in Python developed on the on line development environment of Snap4City can be deployed into a container managed by IoT App and may exploit smart city API for data analytic.

Web Scraping

Web Scraping/Portia: activating a process to grab data from your and/or third parties web pages and convert them in real time to a Data Model in JSON, IoT Device Model for the platform. Web Scraping processes are also controlled by IoT App in Snap4City.

DISCES-EM

Distributed scheduler for elastic management for Smart City by Snap4City. This tool is used  for the elastic management of Container on the Marathon/Mesos cluster of containers. Read more on  https://www.snap4city.org/232

 

For the data analytic development, it is possible to access the Big Data store respecting the privacy and the data licensing by using authenticated Smart City APIs. The access has to permit to read historical and real time data, and to save the resulting data provided by the algorithms, for example, heatmap-related predictions, the assessment of data quality, and labels of detected anomalies. The work of the data scientist could finish once the algorithm has been developed which he/she should be aware of. On the other hand, the same algorithm (e.g., for computing heatmaps, parking prediction), should allow to be:

  • Used on different services of the same kind located in different places and on the basis of a number of parameters (e.g., target precision and list of data sources). This means that data analytic itself has to be designed with the needed flexibility and generality;
  • put in execution from IoT Applications by passing a set of parameters and collecting the results on the Data Storage or as a result of the invocation. The executions can be periodic or event driven —e.g., the arrival of a request or by the arrival of the new set of data values;
  • controlled for collecting eventual errors and mistakes, in debug and at run time for logging. This may be for informing the developer and/or the administrator of eventual mistakes and problems by sending notifications; and
  • dynamically allocated on cloud in one or multiple instances to plan a massive computation of the same data analytic process on several data sets and services at the same time.

Therefore, the solution developed for Snap4City satisfies all the above described requirements. Data analytic processes can be developed using R Studio or Python. In both cases, the code has to include a library for creating a REST Call, namely: Plumber for R Studio and Flask for Python. In this manner, each process presents a specific API, which is accessible from an IoT Application as a MicroService, that is, a node of the above-mentioned Node-RED visual programming tool for data flow.

Data scientists can develop and debug/test the data analytic processes on Snap4City cloud environment since it is the only way to access at the Smart City API with the needed permissions. The source code can be shared among developers with the tool “Resource Manager”, that also allows the developers to perform queries and retrieve source code made available by other developers.

Data Analytics in R Studio

Performed with Plumber: Snap4City RStudio server (open source tool) for developing RStudio processes for machine learning, AI and statistical purpose on data. It can access to data via Smart City API and the RStudio processes can be transformed into Containers. They can exploit Tensor Flow and Keras and CUDA. Any data analytic in R Studio developed on the on line development environment of Snap4City can be deployed into a container managed by IoT App and may exploit smart city API for data analytic.

https://rstudio.com/

Data Analytic in Python

Performed with Flask and Jupyter Editor. Any data analytic in Python developed on the on line development environment of Snap4City can be deployed into a container managed by IoT App and may exploit smart city API for data analytic.

 

The main component includes:

DataGate

DataGate/CKAN for open data (https://ckan.org ): collecting open data from the Open Data network of CKAN harvesting via different protocols. And integrating those data into the platform. DataGate/CKAN can be also controlled by IoT App in Snap4City. CKAN support any kind of files in the domain of Open Data and any CKAN instance can connect with a network of CKAN in the world by harvesting data and sharing data as files and metadata descriptors.

https://www.snap4city.org/26

CKAN

CKAN is an open source solution for Open Data management and distribution https://ckan.org/ In Snap4City is integrated with DataGate module and goes with SSO with the rest of tools. CKAN/DataGate can be controlled by IoT Apps.

 

The main component includes:

UserProfiler

Apps Users are profiled in the server according to the GDPR, collecting a range of data as described above. This approach allows us to tune the service according to the collective and personal user behaviour, with the aim of improving the user experience providing personalized (i) main menu arrangement of the functionalities according to the user profile (the user also may change it to have precise personalization), (ii) suggestions (the user may decide to ban some of the suggestion or even a category), (iii) menu search among categories (the user may decide to make further selection).  Therefore, the menus offered for each profile kind is modified according to the statistics calculated on the collected data, by defining the positions the most researched categories, buttons, etc., and in some cases, putting off/on some functionality. For example, the triage monitoring may be of interest for citizens and operators and less for tourists, the parking status is interesting for users using the private car and less for those moving with the public busses or in bike, etc.  The priority on the creation of the menu profiles is given in first place to the updated menu which can be found on the server. If the server is unreachable, the last saved menu for that specific profile is loaded; if the menu has not been saved, the loaded menu is the menu released with the application update, and so on.

User contrib management

To collect and manage (accept and moderate) contributions from mobile user such as: images, comments, etc. see tutorial https://www.snap4city.org/download/video/course2020/app/

The user profiler module can receive data from mobile Apps via some API and produce the collective profiles which can be used for many purposes (not included) such as computing: recommendations, OD matrices, typical trajectories, hot places, etc.

In the following table, the description of the components listed is reported. In the columns of the following table:

  • Service Inside is the name of the service/module identified.
  • Ownership the modules of DISIT Lab are clearly identified, and when possible the links to access at the open source are provided. As you can verify, all DISIT Lab components needed to create the platform are open source.
    • The components market in green are those not owned by DISIT lab Snap4City (please note that are all open source according to the licence reported in the table).
  • Licence: the name / identifier of the licence is reported.

Please note that the following table is divided in sections which correspond to the names/identifiers of the main components of the Snap4City platform offered and described in the previous sections, and also reported / referred in the official web page for installations of Snap4City: https://www.snap4city.org/471 .

No proprietary components are needed to be installed into the Snap4City platform.

 

Service Inside

Ownership

Licence

Snap4City main

 

 

Dashboard Builder and Engine, Data Inspector, Menu Manager, Wizard, widget Library, WebSocket Server

DISIT Lab https://github.com/disit/dashboard-builder 

Affero GPL

 

Resource Manager (process loader), External Service Manager, Micro Application Manager

DISIT Lab https://github.com/disit/process-loader

Affero GPL

 

Ownership Manager

DISIT Lab https://github.com/disit/snap4city/tree/master/Ownership-API

Affero GPL

 

DataManager, KPI, MyKPI

DISIT Lab https://github.com/disit/snap4city/tree/master/PersonalDataManager

Affero GPL

 

Custom widget, Synoptics

DISIT Lab https://github.com/disit/synoptics

Affero GPL

 

Notificator

DISIT Lab https://github.com/disit/notificator

Affero GPL

 

IoT Directory

DISIT Lab https://github.com/disit/iot-directory

Affero GPL

 

User Statistics

DISIT Lab https://github.com/disit/snap4city/tree/master/UserStats

Affero GPL

 

Event Logger

DISIT Lab https://github.com/disit/snap4city/tree/master/EventLogger

Affero GPL

 

disit.org">LOG.DISIT.ORG

DISIT Lab https://github.com/disit/linked-open-graph

Affero GPL

 

Flint++

DISIT Lab https://github.com/disit/sparql-interface

Affero GPL

 

Report Generator

DISIT Lab

Affero GPL

 

Flint

https://github.com/TSO-Openup/FlintSparqlEditor

MIT Licence

 

LDAP, PHPLDAP admin

Open LDAP  https://www.openldap.org/    https://www.openldap.org/software/release/license.html

open

 

KeyCloak

KeyCloak https://www.keycloak.org/

Apache License 2.0

 

Kafka

Apache https://kafka.apache.org/

Apache License 2.0

 

MySQL

MySQL https://www.mysql.com/products/community/

MySQL

 

TOMCAT

Apache http://tomcat.apache.org/

Apache License 2.0

 

Squid

Squid http://www.squid-cache.org/

GPL

 

Zabbix Server

Zabbix https://www.zabbix.com/

GPL

KBSSM

 

 

ServiceMap, Smart City API

DISIT Lab https://github.com/disit/servicemap

Affero GPL

 

Super (Super Service Map), Servicemap WFS integration

DISIT Lab https://github.com/disit/superservicemap

Affero GPL

 

OSM2SM

DISIT Lab https://github.com/disit/osm2km4c

Affero GPL

 

SSM2ORION

DISIT Lab, in beta testing

Affero GPL

 

Virtuoso

Virtuoso http://vos.openlinksw.com/owiki/wiki/VOS   http://vos.openlinksw.com/owiki/wiki/VOS

open

 

MySQL

MySQL https://www.mysql.com/products/community/

MySQL

 

TOMCAT

Apache http://tomcat.apache.org/

Apache License 2.0

 

PostgreSQL

PostgresQL https://www.postgresql.org/

Open Source license, similar to the BSD or MIT licenses

Heatmap Server

 

 

Heatmap Manager

DISIT Lab https://github.com/disit/snap4city/tree/master/Heatmap

Affero GPL

 

MySQL

MySQL https://www.mysql.com/products/community/

MySQL

 

GeoServer

GeoServer http://geoserver.org/

GPL

 

PostgreSQL

PostgresQL https://www.postgresql.org/

Open Source license, similar to the BSD or MIT licenses

Living Lab Support

 

 

Snap4City Living Lab Drupal Modules

DISIT Lab https://github.com/disit/snap4city/tree/master/Drupal-Modules

Affero GPL

 

LDAP/KeyCloak Drupal Modules, GDPR

DISIT Lab https://github.com/disit/snap4city/tree/master/DrupalGDPR-Module

Affero GPL

 

Drupal 7

Drupal https://www.drupal.org/

GPL

 

MySQL

MySQL https://www.mysql.com/products/community/

MySQL

OpenMaint Server

 

 

OpenMaint

OpenMaint https://www.openmaint.org/en/home

Affero GPL

 

BIMServer

https://github.com/opensourceBIM/BIMserver/tree/master/BimServer

Affero GPL

 

Shark Workflow

https://sourceforge.net/projects/sharkwf/

GPL

 

Alfresco

https://www.alfresco.com/ecm-software/alfresco-community-editions

LGPLv3 license

 

PostgreSQL

Postgres https://www.postgresql.org/

Open Source license, similar to the BSD or MIT licenses

 

TOMCAT

Apache http://tomcat.apache.org/

Apache License 2.0

 

GeoServer

GeoServer http://geoserver.org/

GPL

IOTOBSF

 

 

Secure Filter (Proxy Filter)

DISIT Lab https://github.com/disit/snap4city/tree/master/OrionBrokerFilter

Affero GPL

 

IoT Orion Broker Fiware

FIWARE https://www.fiware.org/

Affero GPL

 

Mongo

Mongo https://www.mongodb.com/community/licensing

Mongo, Apache Licence

 

TOMCAT

Apache http://tomcat.apache.org/

Apache License 2.0

IOTDSES

 

 

Snap4City Process NIFI

DISIT Lab https://github.com/disit/snap4city/tree/master/NIFI-processors

Affero GPL

 

NIFI

Apache https://nifi.apache.org/

Apache License 2.0

 

Squid

Squid http://www.squid-cache.org/

GPL

ESSTORE

 

 

Snap4City extensions

DISIT Lab https://github.com/disit/security forked by Open Distro ES security (Open Distro is the Elastic Search + Kibana version of AWS)

Affero GPL

 

DevDash, My Dashboard Kibana

DISIT Lab

Affero GPL

 

Open Distro (Elastic Search 7)

acutally: AMAZON Open Distro https://opendistro.github.io/for-elasticsearch/faq.html

Elastic https://github.com/elastic/elasticsearch

Apache 2.0-licensed Elasticsearch and Kibana 7.1

 

Open Distro (Kibana)

acutally: AMAZON Open Distro https://opendistro.github.io/for-elasticsearch/faq.html

Elastic https://github.com/elastic/kibana

Apache 2.0-licensed Elasticsearch and Kibana 7.1

MCLSCount

 

 

IoT App, Container management, Marathon update

DISIT Lab https://github.com/disit/snap4city/commit/4ab44b869b82cd10eda8270ef76d2828d793df51

Affero GPL

 

DISCES-EM

DISIT Lab https://github.com/disit/snap4city/tree/master/Disces-em

Affero GPL

 

Snap4City Libraries MicroServices

DISIT Lab https://github.com/disit/snap4city/tree/master/nodered-snap4city-microservices

Affero GPL

 

Node-RED Snap4City adaptation

DISIT Lab https://github.com/disit/node-red https://github.com/disit/snap4city/tree/master/node-red

Apache License 2.0

 

Plumber

Plumber https://www.rplumber.io/

MIT

 

Flask

Flask https://www.flaskapi.org/

BSD 2-Clause license.

 

Portia Web Scraper

https://github.com/scrapinghub/portia

Open Source

 

Marathon

Mesosphere https://github.com/mesosphere/marathon-lb

Apache License 2.0

 

Mesos

Apache https://github.com/apache/mesos

Apache License 2.0

 

Node-RED

Node-red https://nodered.org/

Apache License 2.0

 

Docker

Docker https://www.docker.com/ 

Apache License 2.0

 

Zabbix Agent

Zabbix https://www.zabbix.com/

GPL

RStudio Server

 

 

RStudio Server

RSTUDIO https://rstudio.com/

Affero GPL

Python Server

 

 

Python Editor
Jupyter Hub

https://github.com/jupyter/jupyter/

BSD 3-Clause License

 

Docker

Docker https://www.docker.com/ 

Apache License 2.0

CKAN DataGate

 

 

DataGate

DISIT Lab https://github.com/disit/datagate

Affero GPL

 

CKAN

CKAN https://ckan.org/

Affero GPL

 

PostgreSQL

Postgres https://www.postgresql.org/

Open Source license, similar to the BSD or MIT licenses

UserProfile Management Server

 

 

UserProfiler

DISIT Lab

Affero GPL

 

User contrib management

DISIT Lab

Affero GPL

 

MySQL

MySQL https://www.mysql.com/products/community/

MySQL

 

TOMCAT

Apache http://tomcat.apache.org/

Apache License 2.0

 

Squid

Squid http://www.squid-cache.org/

GPL

As described above we would remark the modules that are used in the proposed solution and are not maintained by Snap4City.

Service Inside

Ownership

Licence

Alfresco

https://www.alfresco.com/ecm-software/alfresco-community-editions

LGPLv3 license

BIMServer

https://github.com/opensourceBIM/BIMserver/tree/master/BimServer

Affero GPL

CKAN

CKAN https://ckan.org/

Affero GPL

Docker

Docker https://www.docker.com/ 

Apache License 2.0

Drupal 7

Drupal https://www.drupal.org/

GPL

Elastic Search 7

Elastic https://github.com/elastic/elasticsearch

Apache License 2.0, Apache License 2.0 compatible license, Elastic License

Flask

Flask https://www.flaskapi.org/

BSD 2-Clause license.

Flint

https://github.com/TSO-Openup/FlintSparqlEditor

MIT Licence

Inkscape SVG Editor

https://inkscape.org/  https://github.com/inkscape/inkscape

https://sourceforge.net/projects/inkscape/

Open Source / Freeware for standard SVG editing

IoT Orion Broker Fiware

FIWARE https://www.fiware.org/

Affero GPL

Kafka

Apache https://kafka.apache.org/

Apache License 2.0

KeyCloak

KeyCloak https://www.keycloak.org/

Apache License 2.0

Kibana

Elastic https://github.com/elastic/kibana

Apache License 2.0, Elastic License

LDAP, PHPLDAP admin

Open LDAP  https://www.openldap.org/

https://www.openldap.org/software/release/license.html

Marathon

Mesosphere https://github.com/mesosphere/marathon-lb

Apache License 2.0

Mesos

Apache https://github.com/apache/mesos

Apache License 2.0

Mongo

Mongo https://www.mongodb.com/community/licensing

Mongo, Apache Licence

MySQL

MySQL https://www.mysql.com/products/community/

MySQL

NIFI

Apache https://nifi.apache.org/

Apache License 2.0

Node-RED

Node-red https://nodered.org/

Apache License 2.0

OpenMaint

OpenMaint https://www.openmaint.org/en/home

Affero GPL

Plumber

Plumber https://www.rplumber.io/

MIT

PostgreSQL

PostgresQL https://www.postgresql.org/

Open Source license, BSD or MIT licenses

Python Editor Jupyter Hub

https://github.com/jupyter/jupyter/

BSD 3-Clause License

RStudio Server

RSTUDIO https://rstudio.com/

Affero GPL

Shark Workflow

https://sourceforge.net/projects/sharkwf/

GPL

Squid

Squid http://www.squid-cache.org/

GPL

TOMCAT

Apache http://tomcat.apache.org/

Apache License 2.0

Virtuoso

Virtuoso http://vos.openlinksw.com/owiki/wiki/VOS

http://vos.openlinksw.com/owiki/wiki/VOS

Zabbix Agent

Zabbix https://www.zabbix.com/

GPL

Zabbix Server

Zabbix https://www.zabbix.com/

GPL

DISIT Snap4City Components that may be optional

 

 

Service Inside

Ownership

Licence

Snap4City Mobile App Dev. Kit

 

 

Complete mobile app from the  “… in a Snap” serie  as accessible from Google Play and Apple store

DISIT Lab provided in Open Source only with limitation to be exploited on your area in which you have coverage with your snap4city installation at platform level. Please request very simple agreement to be signed.

Affero GPL, with non exploitation agreement for the same area to avoid duplications in the same area. No licence fee.

 

Small App Kit

DISIT Lab https://github.com/disit/snap4cityAppKit

Affero GPL

 

Swagger staff

DISIT Lab https://github.com/disit/snap4city/tree/master/swagger

Affero GPL

Other Snap4City Tools

 

 

Engager

DISIT Lab https://github.com/disit/engager  https://github.com/disit/snap4city/tree/master/Engager

Affero GPL

 

ServiceMap3D

DISIT Lab https://github.com/disit/servicemap-3d

Affero GPL

 

What-IF Router, Routing Server

Fork from Graphopper https://www.graphhopper.com/

GNU

 

SmartDS

DISIT Lab https://github.com/disit/smart-ds

Affero GPL

 

Resilience DS, FRAM DS

DISIT Lab https://github.com/disit/resilience-ds

Affero GPL

 

WLODE

DISIT Lab https://github.com/disit/WLODE

Affero GPL

 

recommender

DISIT Lab https://github.com/disit/smartCityRecommender

Affero GPL

 

ETL processes

DISIT Lab https://github.com/disit/snap4city/tree/master/ETL

https://github.com/disit/smart-city-etl

Affero GPL

 

Data Analytics in R

DISIT Lab https://github.com/disit/snap4city/tree/master/Statistics

Affero GPL

 

DISCES

DISIT Lab https://github.com/disit/sce-frontend https://github.com/disit/sce-backend

Affero GPL

 

Data collection from Mobile App, SSM

DISIT Lab https://github.com/disit/km4c-sensor-server-manager

Affero GPL

 

Twitter Vigilance RT (real time)

DISIT Lab

Proprietary, with source code accessible

 

Twitter Vigilance

DISIT Lab

Proprietary

 

Traffic Flow reconstruction

DISIT Lab https://www.snap4city.org/drupal/node/619

Proprietary, with source code accessible

 

Other Specific Data Analytics not included above

DISIT Lab: anomaly detection, parking predictions, bike sharing prediction, NOX predictions, OD production, trajectory production, etc.

Proprietary, with source code accessible

 

DORAM

tool for the analysis of demand of mobility vs offer of transportation; Tools; beta development

Proprietary, with source code accessible