|
Test Case Title |
TC10.20 - The Snap4City solution is Interoperable |
|
Goal |
Communicates with existing and new systems of the cities Provides mechanisms and tools to perform analytics on combined data, including different device and data sources and streams Uses accepted and emerging standards and open |
|
Prerequisites |
Using a PC or Mobile with a web browser. The following functionalities are available only for specific Snap4city users with specific privileges. |
|
Expected successful result |
The verification of other TCs. |
|
Steps |
Snap4City is capable to interoperate since it can:
- Acquire and send data from/to several sources (any protocol and format) as described in Section K3.10 of Iteration 3 final Document and referred links. Snap4City support a very large number of protocols for collecting data and for exposing data towards other services and tools: https://www.snap4city.org/drupal/node/65
- According to the assessment: the process of adding a new protocol has been considered too complex. Now we have performed a Simplification of the process to add a new IOT protocol and thus for becoming interoperable with other platforms. This latter feature is accessible to any level of users and thus for All User Group members of the cities, as described in K3.15.
- Included External Services as Rest Calls: They can be made accessible into Snap4City IOT Applications as MicroServices. The user can register them via a simple interface. A List of the available MicroServices from External Services is also provided in to the Resource Manager https://processloader.snap4city.org/processloader/ssoLogin.php?redirect=page.php%3FshowFrame=false
- Exploit External Services Web Applications (distinguished for organization of the registered user), exploiting them on Dashboard and MicroServices. For External Services we intend Web based Tools that may expose web pages on which one can interact: a weather portal, a specific view on traffic, the status of the port, etc. This means that the Snap4City Platform can integrate in any Dashboard the legacy External Services and Tools to produce integrated views. The Users of the platform can register new External Services on Snap4City and they become immediately accessible in the Dashboard Wizard for their composition into Dashboards. A Dashboard can integrate any number of External Service.
- https://www.snap4city.org/dashboardSmartCity/management/externalServices.php
- Examples of dashboards with External Services are:
- Additional new example for Antwerp with a large number of External Services integrated into the same Dashboard:
- Federated Open Data Portals via DataGate/CKAN that presently presents now more than 13800 data sets linked for the cities of Helsinki and Antwerp.
- https://datagate.snap4city.org/organization
- Federation, Harvesting interface is: https://datagate.snap4city.org/harvest
- Work with top 10 used DBs and more, as: Mongo, Azure, MySQL, Microsoft SQL, MariaDB, As400, Cassandra, Elastic Search, Oracle, IBM DB2, Postgres, AWS, etc. and via standard connectors such as: ODBC, JDBC, etc. via IoT App.
- Join, publish and harvest Open Data Portals via DataGate/CKAN.
- Interoperate with ERP via IoT App: https://flows.nodered.org/search?term=ERP
- Interoperate with BPM and BIM via OpenMaint: https://www.openMAINT.org/en/home and see https://www.snap4city.org/625
- Acquire data from WEB Portals of third parties. This is possible according to their terms of use and national legislations (it may be different in different states). The approach of data scraping/crawling from Web Portals is based on an integrated Web Scraper based on Portal:
- https://www.snap4city.rg/drupal/node/478 ,
- TC9.16 – Web Scraping t get data from web pages
- The WebScraping tool directly produces a process which is allocated on Cloud for scraping the data periodically and bring the data into the IOT Applications. Thus the user may decide to save this data, use in some Data Analytic or bring them on Dashboard directly.
- Federate Multiple Snap4City Knowledge Bases. This allows the creation of mobile applications that may move from multiple cities and area transparently. This solution is presently in place among the Knowledge Bases of: Antwerp/Helsinki, Tuscany/Firenze, Sardegna, etc. The resulting Service is called Super Service Map and it is integrated in the Smart City API.
- Exploit external Maps via WMS protocols (among them also MAP provided for free from ESRI) to show them directly into Dashboards, see for example:
- Exploit WFS sources to show them directly on Dashboards
- Integrate/command ISEMC for Video Wall management, for implementing Map Command and Control Center, Smart City Control Room: (https://www.snap4city.org/621 ) Control Room as: https://www.snap4city.org/531
- Interoperate with ArcGIS ESRI via API and protocols such as WFS, WMS, etc. https://www.snap4city.org/368 It is also integrated with ArcGIS ESRI via IoT App and Node-RED: https://flows.nodered.org/search?term=arcgis
- Exploit external Maps via WMS protocols (among them also MAP provided for free from ESRI) https://www.snap4city.org/368 to show them directly into Dashboards, see for example: https://www.snap4city.org/368 and Exploit WFS sources to show them directly on Dashboards: https://www.snap4city.org/368
- Exploit WFS sources to ingest them into the Snap4City storage via IoT Apps: https://www.snap4city.org/368
Snap4City is capable to interoperate since it can Expose its data and services via:
- Advanced Smart City API which can be confined into a single Smart City installation or Federated as well as for Super Service Map
- Smart City API which can be confined into a single Smart City installation or Federated as well as for Super ServiceMap WFS service of Snap4City on top of Federated Smart City API or simple Smart City API of a single ServiceMap (smart City installation). This solution permits to GIS applications and platforms (such as ArcGIS OnLine ESRI, ArcGIS Enterprise ESRI, ArcGIS Map/pro Desktop, QGIS, GeoServer, etc.) to access at Snap4City data. See interoperability with ArcGIS and GIS in general https://www.snap4city.org/368
- WFS service of Snap4City on top of Federated Smart City API or simple Smart City API of a single ServiceMap (smart City installation). This solution permits to GIS applications and platforms (such as ArcGIS OnLine ESRI, ArcGIS Enterprise ESRI, ArcGIS Map/pro Desktop, QGIS, GeoServer, etc.) to access at Snap4City data. For Example, via:
- WMS service of Snap4City for publishing maps and heatmaps, provided by an installed GeoServer third party open source tool. For example, via:
- Federated Multiple Snap4City Knowledge Bases. This allows the creation of mobile applications that may move from multiple cities and area accessing data and making queries transparently. This solution is presently in place among the Knowledge Bases of: Antwerp/Helsinki, Tuscany/Firenze, Sardegna, etc. The resulting Service is called Super Service Map and it is integrated in the Smart City API. For example, via:
- Federated Open Data Portals via DataGate/CKAN that presently presents now more than 13800 data sets linked for the cities of Helsinki and Antwerp.
- https://datagate.snap4city.org/organization
- Federation, Harvesting interface is: https://datagate.snap4city.org/harvest
- Interoperability with Heatmaps and WMS Snap4City is working with Heatmaps and Maps in WMS standard. Heatmaps are saved in terms of points in the Heatmap manger server and automatically produced on the basis of the set colormap, the maps/heatmaps are provided to dashboards in standard WMS by means of the Snap4City Heatmap Server which includes a reference to a GIS (GeoServer or ArcGSI) and tools for automated generation of GeoTiff images: https://www.snap4city.org/368
Interoperability with Heatmaps and WMS
Snap4City is working with Heatmaps and Maps in WMS standard. Heatmaps are saved in terms of points in the Heatmap manger server and automatically produced on the basis of the set colormap, the maps/heatmaps are provided to dashboards in standard WMS by means of a GeoServer.
You can see the difference comparing the heatmaps in the two examples:
- Correct adopted Dashboard with calibrated heatmaps on A5 case: https://www.snap4city.org/dashboardSmartCity/view/index.php?iddasboard=MTQwNw==
- Former Dashboard with gaussian heatmaps, at the end accessible but in most cases not used since the gaussian distribution change on the basis of the zoom, and thus the colormap is strongly confusing for the users. Deprecated model: https://www.snap4city.org/dashboardSmartCity/view/index.php?iddasboard=MTMxMQ==
Other information (may be older)
See Supported Protocols for the full list of supported protocols.
Interoperability can be demonstrated by showing how the solution proposed can be integrated with already in place solutions. See for example the list of Open Source tools from Apache and well known other distributors on which the solution proposed is based. Being based on Apache Hadoop, Phoenix, Hbase, Penthao Kettle, NodeRED, MySQL, Tomcat, DRUPAL, CKAN, etc. the solution is interoperable in principle with many solutions based on the same fundamental tools. Please see on this regard the section about ACPaaS in which a possible integration has been described. The effective interoperability is also shown by providing the effective integration of the solution proposed in different context. The solution proposed is natively integrated in Km4City and thus is proposed for free to all the partners already using the Km4City solution as a simple add on, bringing the IOT exploitation to a next level with respect to the present solution.
- The prototype is able to communicate with existing and new systems of the cities
The Snap4City solution is easily integrated with already in place solutions for the large overage of protocol, see above the list on Q6. This means that the Snap4City solution is capable to exploit the solutions which are in place of the city operators using a range of tools: IOT Brokers, ETL, NodeRED, CKAN.
In order to test the satisfactory of this feature it has been already shown in other TCs that:
- ETL can access to data from OSM, GTFS, FTP, WS, etc.
- ETL and IOT applications can read files of any format
- ETL and IOT applications can work on new External Services
- ETL and IOT applications can work with external services
- IOT applications Dashboards can be integrated in third party tools and into City Dashboards
- Dashboards created by Dashboard Builder can be included into third party Web Portal
- See the many dashboard hosting and integrating external services:
- The prototype provides mechanisms and tools to perform analytics on combined data, including different device and data sources and streams
The Snap4City solution allows Data Analytics processes:
- defined and executed by using different kind of languages (programming paradigm):
- Executed in Containers or independent processes, scheduled by DISCES;
- call External Services from a large number of languages;
- call MicroServices coming from the several Snap4City APIs from a large number of languages. For example, specific Blocks for NodeRED and/or for ETL are provided (not yet available)
- Executed on the basis of Data Driven, via IOT events, via Publish / Subscribe protocol, Snap4City Applications,
- Executed on demand from other services: Web Portal, ETL, etc. via DISCES
- Executed programmatically and periodically by DISCES according to different policies.
A VM as for Data Analytics and ETL (Data Processing Development Environment) is provided as a specific Snap4City tools. Once an ETL or Data Analytics process is produced it can be uploaded and monitored on the platform by using the ProcessLoader tool on web. Please note that ETL processes can be used to ingest data coming from several protocols and services as described in Q6.
- The prototype uses accepted and emerging standards and open platforms (e.g. The Open Group IoT standards O-DF, O-MI, AIOTI Architectures suggestions, IOT EIP pre-standardisation work, XML, JSON, SOAP, REST, etc.)
The proposed solution is capable to use well established standards as described in Q6, including potential new standards mentioned in TD2, by means of the solution proposed in Q6. UNIFI DISIT lab is part of the ISO TMP Smart City Standardization group. The EIP is an activity followed by UNIFI DISIT.
The prototype is able to consume services that are provided using open standards as demonstrated in other TC.
The prototype is able to cope with both new and legacy protocols and impact of adding new protocols is minimal, for the coverage of NodeRED+ETL+ORION with respect to a wide selection of protocols.
