- Development: Data Processes, aka Processing Logic (IoT Applications)
- Capabilities of Proc.Logic / IoT App in Snap4City
- Examples: Processing Logic / IoT App: typical patterns
- Typical strange patterns that may be not efficient in most cases, or wrong
- Example of Trigger / Sync Pattern
- Test and Verify Data Ingestion
- Test and Deploy of Proc.Logic / IoT App Node-RED Libraries
The development of Data Processes focusses on the functional aspects and includes the implementation of data flow procedures for data: ingestion, transformation, production, publication, load/retrieve, etc. These activities are the classical ETL/ELT of data warehouses and data lakes.
In Snap4City, this activity is strongly simplified since all these aspects are easily developed via Processing Logic (IoT App) that is visual programming Node-RED where the usage of JavaScript is quite limited. For massive data ingestion, for example when large data sources need to be loaded into the Snap4City platform, it is possible to use dedicated Python processes to perform ETL processes. They can be put in execution as stand-alone processes or just activated and controlled via IoT App which can also send them specific parameters.
So that: Processing Logic (IoT App) high flexibility, rapid development any kind of applications, while dedicated Python processes for high performance data ingestion, for example when large amount of data communing from historical repositories has to be loaded in the platform for setup.
Processing Logic (IoT App) Development allows exploiting a large collection of macros/nodes (Node-RED Nodes) and most of them refer to MicroServices which are: the standard nodes provided into the Node-RED editor also called Processing Logic Editor (IoT App) in the context of Snap4City. Those provided by Snap4City (more than 180 MicroServices, which can be installed directly from the Node-RED library, repository of Palette); any other MicroServices/Node from other Libraries of Node-RED nodes and from the web since Node-RED is very diffuse.
Processing Logic (IoT App) of Snap4City can be used for creating data flow integrating in the same flow multiple activities of Data Ingestion, Data Transformation, Data Analytics and Server-Side Business Logic.
- Processing Logic = Node-RED + Snap4City Libraries
- Former: IoT App = Node-RED + Snap4City Libraries
- and can be executed on cloud as well as on IoT Edge: Linux, Windows, Arm, AXIS, Raspberry Pi, etc.
For each Independent Flow of each Processing Logic (IoT App), according to the analysis performed to correctly perform this task the developer should provide answer at questions such as:
- Where these processes would be executed? It has to be executed on Cloud Snap4City platform or on some Edge computing of any kind?
- How I have to schedule this process if it not Event Driven? Periodic event?
- Which kind of Processing Logic (IoT App) I need to develop? Which kind of MicroServices I need to have on the Processing Logic node-red among the several possible?
- I have to process historical data or real time data, offline or in real time?
- We need to take them (PULL) or they will arrive (PUSH)?
- Which is the amount of data to be taken? They need to be decoded and how?
- Which are the gateways to be interfaces if any? Which kind of authentication?
- Which are the brokers to be interfaces if any? Which kind of authentication?
- etc.
