One of the first steps to set up a Smart City or add new services can be the data ingestion, and thus connections with data sources/recipients, that in Snap4City can be performed in several manners. All the connections can be bidirectional and based on their own protocols. Snap4City is capable to ingest data via those protocols but also produce data back with the same or other protocols.
Snap4City platform can interact with legacy (of any vender and format) installations on IoT Networks, and thus with IoT Edge Devices, Gateway, etc. They can communicate via their own protocols or using the direction connection with Snap4City exploiting protected communications. IoT Edge devices with Node-RED can directly install Snap4City library to exploit from the platform: data ingestion processes, data model registration, Data Analytic, dashboards, direct interaction with dashboards and users.
Main channels for data ingestion/interaction are:
- Data Table Loader: this tool allows you to ingest excel files and trasnform them in IOT Device Model, IoT Devices and their instances automatically in few minutes. This approach is very useful for structured data coming from tables of KPI representing the territory, the indicator of city, the origin destination, the people flows of arrivals, the economy of tourism, etc. They have typically some date (year for example) reporting the time in which the data have been collected. Once ingested the data can be directly used into Dashboard so that it is a no-coding approach. https://www.snap4city.org/drupal/node/729
- POI loader: this tool allows to load / ingest into the platform excel files containing data regarding Point of Interest, POI. They are typically any geolocated element of the city without a time series of data attached to some variable. They are static or quasi static data. A typical POI can be the description and location of a museum, of a bank, etc. https://www.snap4city.org/drupal/node/731
- IoT App in Node-RED: collecting data from External Services (e.g., IoT Gateways, web servers, ftp server, social media services, linked open data, linked data, ESBs), by using any kind of Connector, calling any kind of protocol and/or getting/transforming any kind of format into a Data Model / IoT Device Model for the Platform. Ingestion can be in PUSH/PULL to set up periodic/sporadic processes, as well as data driven (real time) processes of data ingestion and reaction/action for the platform towards other internal/external elements including other IoT App. See Section 3.6 and Section 3.7 for interoperability.
- IoT Broker: which connect IoT Networks of IoT Devices.
- A mayor role is played by IoT Orion Broker of FIWARE by which the platform support NGSI V1, V2, LD with Services/Tenant and Service Paths.
- The IoT Orion Brokers can be controlled by the same administrators of the Snap4City Platform on your premise (so called Internal IoT Broker) or can be External IoT Broker controlled by third parties. Brokers may exploit: IoT Agent, IoT Adapter, wrappers, IoT Edge Devices.
- Snap4City platform supports many different IoT Broker and more than 70 different protocols push/pull, sync/async [Https://www.snap4city.org/65] among them: MQTT, NGSI, COAP, OneM2M, ModBus, OPC, OPC-UA, Copernicus, and AMQP. See for the list of the protocols supported.
- Supported data models are: FIWARE Data Model, and IoT Device Models. The former are automatically converted for the Knowledge base.
- IoT Devices and IoT Broker are registered into IoT Directory which is multiprotocol multibroker tool for IoT Network management, also providing features of IoT Discovery in collaboration with the Knowledge Base. The registration of IoT Devices into Knowledge Base from IoT Orion Brokers is performed automatically.
- DataGate/CKAN for open data (https://ckan.org ): collecting open data from the Open Data network of CKAN harvesting via different protocols. And integrating those data into the platform. DataGate/CKAN can be also controlled by IoT App in Snap4City. CKAN support any kind of files in the domain of Open Data and any CKAN instance can connect with a network of CKAN in the world by harvesting data and sharing data as files and metadata descriptors.
- Web Interface: to add/change data via web page, for example: POI, MyKPI, etc.
- Web Scraping/Portia: activating a process to grab data from your and/or third parties web pages and convert them in real time to a Data Model in JSON, IoT Device Model for the platform. Web Scraping processes are also controlled by IoT App in Snap4City.
- Files: of any format they can be loaded via: (i) specific IoT App to set up specific data ingestion processes such as via Web page, FTP, SSH, DropBox, Google Drive, etc., (ii) DataGate/CKAN. Files loaded can be any format and size.
- Web and Mobile Apps: they typically connect with the platform to get data and suggestions, and also they may send data back as preferences, trajectories, queries on sequences, clicks, etc. (if the user agree, data as sent anonymously or according to a signed consent in agreement to GDPR). This huge amount of data can be used for providing contextual services and tuning the service to the city users. The Mobile App users can be registered on the same pool of platform users or may be decoupled from the platform central user registration. See section 3.5.
- Dashboards: they typically connect with the platform to get data and suggestions, and also they may send data back. The dashboards may present Actuator (Virtual IoT Device Sensor Actuators) to send control data to the platform such as: dimmer, know, switches, selector, clicks, text, forms, etc. This information can be consumed on the fly by IoT App which are the Business Logic of the User Interface, UI, and/or they can be saved into the platform for further usage. See Section 3.10
- Direct URLs: such as URI/URLS to perform rest call to external services, such as GIS server. They can be directly accessed to show data on Dashboards, exploiting protocols such as WFS and WMS, which can be also produced by Snap4City ServiceMap.
- Data Analytic processes may create new data, which can be saved into the platform and may contribute to the vision of the city via Dashboards and to produce innovative and smart services. For example, they can produce: predictions, early warning, traffic flow reconstruction, alarms detecting anomalies, KPI and indicators, Typical Time Trends, typical trajectories, Routing and paths, travel plans, query results, simulations, analysis, calibrated heatmaps, smart parking suggestions, car sharing analysis, etc. See Section 3.9.
- Reports: may create consumptive views on the platform status on specific programmable aspects for users and administrators, for example at level of IoT Devices, Dashboards.
---------------- training and HOW to links ---------------------
This page is focussed on guiding the users in adding data source to the platform. Data can arrive to the platform by using different kinds of data sources, different protocols, formats, etc. In most of the IOT smart city platforms, many of the following cases are not addressed, and thus their approach could be at the first glance simpler, while in reality it is only strongly limited.
So that, in Snap4City all cases are addressed and are from the simplest to the more advanced are:
- IOT Device data via some IOT Broker. (see HOW TO: add a device to the Platform). The IOT Device can be added to an:
- internal Snap4City IOT Broker (a broker managed by Snap4City for security aspects). In this case, the data are immediately accessible, and you can find them into the list of your data in the Data Inspector view, for Dashboards, etc., go in the Data Inspector to search your data by GPS location, name, nature as you like.
- external Broker (a broker managed by a third organization for security aspects). This means that the IOT Device has to be accessing only with specific authentication mechanisms. See HOW TO: add IOT Device data source from external broker to the platform.
- Files on your local computer. If the file is located on your computer (for example a private date set), it can be uploaded and ingested into the platform as well. The simplest solution is to ingest it by using an IOT Applications as follows. Once uploaded you can ingest the data into your MyKPI (for example), and from them on Dashboards.
- WebPages with data source inside. In this case, the data are published on some third party web page. If you would like to access to this data (according to some licence), you can use the Internal Web Scraping Tool which can help you visually create the web scraping process and save the data in some storage, for example into your MyKPI for further reuse:
- TC9.16 Web Scraping to get data from web pages
- Please note that some Web Scraping can be performed also by using ETL processes as well as by using IOT Applications. Those approaches are suitable for programmers, while the Internal Web Scraping Tool can be visually programmed and thus it is easier.
- Open Data file on some Portal. The Open Data can be ingested into the Portal according to different mechanisms, they can be static (do not change) or dynamic (change with some rate, e.g., one every day, once per week). If your Open Data are Accessible on some
- Web page or Web Service or FTP service or API Rest Call, etc. In this case, the best solutions can be to create a periodic or sporadic script to ingest them. You have two possibilities:
- CKAN service of your city, you can set the Snap4City DataGate to access at that CKAN to get them automatically. See:
- Web page or Web Service or FTP service or API Rest Call, etc. In this case, the best solutions can be to create a periodic or sporadic script to ingest them. You have two possibilities:
- ExternalServices data sources: are typically Web Services or FTP services, or REST Call services API. They can be ingested as described above. In this case, the best solutions can be to create a periodic or sporadic script to ingest them. You have two possibilities:
- REST Call services API as MicroServices data sources. They can be treated as ExternalServices (see above) or they can be registered on the platform to be queried directly from IOT Application as MicroService. In this case you need to register them:
In all these cases, the data ingested are saved into your private storage as MyKPI data on which you can manage access and rights, use them for Dashboards, perform data analytic, etc.
Once the device and the data are added to the platform, they can be used to create dashboards together with other data sources already available on the platform:
Please start a fully guided training cases:
