HOW TO: add data sources to the Platform

One of the first steps to set up a Smart City or add new services can be the data ingestion, and thus connections with data sources/recipients, that in Snap4City can be performed in several manners. All the connections can be bidirectional and based on their own protocols. Snap4City is capable to ingest data via those protocols but also produce data back with the same or other protocols.



Snap4City platform can interact with legacy (of any vender and format) installations on IoT Networks, and thus with IoT Edge Devices, Gateway, etc. They can communicate via their own protocols or using the direction connection with Snap4City exploiting protected communications. IoT Edge devices with Node-RED can directly install Snap4City library to exploit from the platform: data ingestion processes, data model registration, Data Analytic, dashboards, direct interaction with dashboards and users.



Main channels for data ingestion/interaction are:

  • Data Table Loader: this tool allows you to ingest excel files and trasnform them in IOT Device Model, IoT Devices and their instances automatically in few minutes. This approach is very useful for structured data coming from tables of KPI representing the territory, the indicator of city, the origin destination, the people flows of arrivals, the economy of tourism, etc. They have typically some date (year for example) reporting the time in which the data have been collected. Once ingested the data can be directly used into Dashboard so that it is a no-coding approach.
  • POI loader: this tool allows to load / ingest into the platform excel files containing data regarding Point of Interest, POI. They are typically any geolocated element of  the city without a time series of data attached to some variable. They are static or quasi static data. A typical POI can be the description and location of a museum, of a bank, etc.
  • IoT App in Node-RED: collecting data from External Services (e.g., IoT Gateways, web servers, ftp server, social media services, linked open data, linked data, ESBs), by using any kind of Connector, calling any kind of protocol and/or getting/transforming any kind of format into a Data Model / IoT Device Model for the Platform. Ingestion can be in PUSH/PULL to set up periodic/sporadic processes, as well as data driven (real time) processes of data ingestion and reaction/action for the platform towards other internal/external elements including other IoT App. See Section 3.6 and Section 3.7 for interoperability.
  • IoT Broker: which connect IoT Networks of IoT Devices.
  • DataGate/CKAN for open data ( ): collecting open data from the Open Data network of CKAN harvesting via different protocols. And integrating those data into the platform. DataGate/CKAN can be also controlled by IoT App in Snap4City. CKAN support any kind of files in the domain of Open Data and any CKAN instance can connect with a network of CKAN in the world by harvesting data and sharing data as files and metadata descriptors.
  • Web Interface: to add/change data via web page, for example: POI, MyKPI, etc.
  • Web Scraping/Portia: activating a process to grab data from your and/or third parties web pages and convert them in real time to a Data Model in JSON, IoT Device Model for the platform. Web Scraping processes are also controlled by IoT App in Snap4City.
  • Files: of any format they can be loaded via: (i) specific IoT App to set up specific data ingestion processes such as via Web page, FTP, SSH, DropBox, Google Drive, etc., (ii) DataGate/CKAN. Files loaded can be any format and size.
  • Web and Mobile Apps: they typically connect with the platform to get data and suggestions, and also they may send data back as preferences, trajectories, queries on sequences, clicks, etc. (if the user agree, data as sent anonymously or according to a signed consent in agreement to GDPR). This huge amount of data can be used for providing contextual services and tuning the service to the city users. The Mobile App users can be registered on the same pool of platform users or may be decoupled from the platform central user registration. See section 3.5.
  • Dashboards: they typically connect with the platform to get data and suggestions, and also they may send data back. The dashboards may present Actuator (Virtual IoT Device Sensor Actuators) to send control data to the platform such as: dimmer, know, switches, selector, clicks, text, forms, etc.  This information can be consumed on the fly by IoT App which are the Business Logic of the User Interface, UI, and/or they can be saved into the platform for further usage. See Section 3.10
  • Direct URLs: such as URI/URLS to perform rest call to external services, such as GIS server. They can be directly accessed to show data on Dashboards, exploiting protocols such as WFS and WMS, which can be also produced by Snap4City ServiceMap.
  • Data Analytic processes may create new data, which can be saved into the platform and may contribute to the vision of the city via Dashboards and to produce innovative and smart services. For example, they can produce: predictions, early warning, traffic flow reconstruction, alarms detecting anomalies, KPI and indicators, Typical Time Trends, typical trajectories, Routing and paths, travel plans, query results, simulations, analysis, calibrated heatmaps, smart parking suggestions, car sharing analysis, etc. See Section 3.9.
  • Reports: may create consumptive views on the platform status on specific programmable aspects for users and administrators, for example at level of IoT Devices, Dashboards.

---------------- training and HOW to links ---------------------

This page is focussed on guiding the users in adding data source to the platform. Data can arrive to the platform by using different kinds of data sources, different protocols, formats, etc. In most of the IOT smart city platforms, many of the following cases are not addressed, and thus their approach could be at the first glance simpler, while in reality it is only strongly limited.

So that, in Snap4City all cases are addressed and are from the simplest to the more advanced are:

In all these cases, the data ingested are saved into your private storage as MyKPI data on which you can manage access and rights, use them for Dashboards, perform data analytic, etc.

Once the device and the data are added to the platform, they can be used  to create dashboards together with other data sources already available on the platform:

Please start a fully guided training cases: