Processing Logic allows to develop processes to perform:
- Data Extraction: Ingestion, gathering, harvesting, grabbing
- Several examples are reported in the training course at the following links: https://www.snap4city.org/download/video/course/di/
- Node-RED + Snap4City environment provide more than 160 protocols IoT and Industry 4.0, web Scraping, external services, any protocol database, etc.
- Ingesting data via the development of ETL or Processing Logic (IoT App). They can collect data from several different format sources using several different protocols. FTP, WS, HTTP/HTTPS, etc.
- Web Scraping tools, to collect data from Web pages and send them to some Processing Logic.
- ingesting data form excel tables and exploiting some visual tool for data ingestion, which lead to pass data into a dedicated Processing Logic.
- getting data from Open Data channel as CKAN with some Processing Logic (IoT App).
- Data Storage/save/load and Retrieval/search: load to storage, retrieve from storage
- See https://www.snap4city.org/download/video/course/iot/
- Data access: save/retrieve data, query search on expert system, geo-reverse solution, search on expert system Km4City ontology, etc.
- Get data and info from Storage:
- Smart City API and Services: transport, parking, POI, MyKPI, personal data, scenarios, etc., via dedicated nodes.
- Brokers providing data in event driven or the last data, via dedicated nodes.
- External Services and resources, via dedicated nodes or directly with REST Calls.
- Several platform managers via API: heatmaps, scenarios, typical time trend, multi series, calendar, maps, OD, TV Cam, trajectories, BIM, 3D shapes, etc.
- Save data on platform Storage of:
- Data Transform: transcoding, decoding, converting, Production, generation, reformatting, etc.
- See https://www.snap4city.org/download/video/course/iot/
- Data Transformation/transcoding: binary, hexadecimal, XML, JSON, String, any format.
- Also the exploitation of Data Analytics in Python and RStudio.
- Data publishing: post in channels of any kind, etc.
- Integration with any service, servers, provider, gateway, brokers, etc.
- Integration: CKAN, Web Scraping, FTP, Copernicus satellite, Twitter Vigilance, Workflow OpenMaint, Digital Twin BIMServer, any external service REST Call, etc.
- Data Analytic management:
- Using Data Analytic nodes which are instantiated as Container from the Processing Logic: managing Python native, calling and scheduling Python/Rstudio containers as snap4city microservices (predictions, anomaly detection, statistics, etc.).
- Calling independent Data Analytics processes which are executed as services, and can be called as REST Calls. The scheduling and the management of these processes has to be performed by some administrator of the platform, or they are located in third party servers/hosts.
- Using Python Node-RED embedded processes, just executing a Python into a node-red and not into a dedicated container. This approach has a limited amount of memory space and use the same container or VM of the Proc.Logic/IoT App executing the node-RED.
- Data Interoperability, establishing connections with other services:
- From the Snap4City platform, with the aim of
- providing data, the best solution is to start communication from the external process, which will have to use the Smart City API, or the API of the Broker (subscription or query), in authenticated manner.
- receiving data in Push the best and safer solution uses the API of the Broker to send that in push on some broker (e.g., MQTT, ORION), in authenticated manner.
- receiving data in Pull the best solution is to start communication protocol from the Processing Logic calling the REST call of the data provider, in agreement with their authentication model and solution.
- The REST Call APIs exposed by Node-RED are strongly discouraged since they are not protected, not scalable, and directly managed into the Processing Logic. In node-red it is possible by using HTTP blocks/nodes to provide a local API interface to receive from other processes a simple REST Calls to get or provide data.
- From the Snap4City platform, with the aim of
- Business Logic (Server-Side) of GUI:
- User interaction on Dashboard: get data and messages from the user interface, providing messages to the user (form, buttons, switches, animations, selector, maps, etc.).
- Custom Widgets: SVG, synoptics, animations, dynamic pins on maps, etc.
- Generate HTML pages and collect data entry from those HTML pages.
- Event Management and production:
- Create a listener to get event driven data from brokers: MQTT or ORION broker, or other kind of brokers and listeners.
- Receive events from: Dashboards, changes on MyKPI, changes on Web Sockets, etc.
- Send/post messages to: Telegram, Twitter, Facebook, SMS, emails, etc.
- Hardware Specific Devices:
- Getting local IoT Data (entity instances) from Raspberry Pi, Android, Philips, video wall management, etc., such as: I/O, temperature, CPU workload, connected custom sensors, etc.
- Sending events from the platform to local HW specific devices such as: audio, vibrations, SMS, etc.
Thus, as a remark, the data ingestion can be performed via:
- Brokers: installing new IoT Devices, registering them on some internal broker, they will send data to Broker which in turn will provide data messages in push directly into the platform storage.
- Snap4City may provide you entry point for Internal ORION Broker, MQTT Brokers, and on demand other brokers kind.
- Snap4City can be connected to any Kind of External Broker.
- Proc.Logic: Connecting the platform to your data sources by using from Processing Logic as described in the sequel. The Proc.Logic can get data from any kind of source with more than 160 protocols.
- You can create your custom data ingestion/loader process.
- Dedicated Python process: Using a Snap4City FastDataLoader in Python processes to get data from MQTT brokers, files and databases to send them to the Orion Broker of the platform in high-speed manner. https://www.snap4city.org/831
- Using advanced tools such as Data Table Loader, POI Loader, etc.
- Etc.
