HOW TO: Develop an IOT Application for Data Ingestion

×

Warning message

  • You can't delete this newsletter because it has not been sent to all its subscribers.
  • You can't delete this newsletter because it has not been sent to all its subscribers.

Path in Snap4City: “MENU -> IOT Applications

  • To manage the generation of static data and to keep real-time data updated, it’s necessary to create an IoTApp
    • To create an IoTApp simply click on “Create new” button and give it a name
    • Once created, wait a few seconds: the platform is making it available. When the red light turns green, you can open it
  • An IoTApp is a set of Node-Red flows, each placed in a tab (like internet browsers tabs)
    • Each IoTApp should contain some flows concerning a single topic: only one IoTApp per topic and several flows within it that manage the various components of that topic
    • To create new flows (tabs) click on ‘button > Flows > Add’ (  on the above right). If you want to rename it go to ‘button > Flows> Rename’
  • For the purposes of this user manual, it will be necessary to create only two flows (one to generate static data and one to keep real-time data updated)

Example:

    1.  

IoTApp – Static Data

Path in Snap4City: “MENU -> IOT Applications > Your IoTApp”

Type of user:  Area Manger

Features: 

  • CSV file generation for devices bulk loading (before creating the csv file, please read the section ‘Create Devices in Bulk’)

General guidelines for downloading (or compiling) static data:

  • To automatically generate the csv file, it is necessary to read the data that are made available from the data source (via APIs/FTP/etc.). If you are the owner of the devices, the data must be entered manually in a javascript function that will fill in the csv (see the guidelines below to know how to pass data to the CSV adapter function)
    • The reading and adaptation of static data will depend on the context in which you operate, and for this reason.
  • The target of these general guidelines is to have a javascript array in which each element contains a JSON object with static data

An example, related to the charging stations in Florence, is described as a starting point:

The static and real-time data of this example are accessible from a single API in the form of a KMZ file. It’s therefore necessary to transform the KMZ file into an XML file and select the useful data to compile the CSV and the RDF triples. The example flow also shows the real-time data in order to have a single starting base also for the real-time flow.

Static data flow:

  • timestamp (inject node): this node is necessary to start the flow. To start it, click on the square to the left of the node or set an automatic repeat in the properties (double click on the node)
  • KMZ Download (http node): this node makes an http request to the URL indicated in the properties. In this example it is necessary to download the file http://datigis.comune.fi.it/kml/ColonnineRicarica.kmz containing the static and real-time data of the devices
  • KMZ > KML (zip node): since the downloaded KMZ file is a zip file, it must be unzipped to have the KML file
  • Strings replacement (string node): HTML characters are present in the KML file, they must be replaced with Unicode characters

  • XML Converter (xml-converter node): this node converts the KML file (XML-based) into a JSON object editable by Node-Red
  • Parser (function node): this is the core of the data processing from the APIs. Here you need to use javascript to adapt the data from the APIs into a a javascript array in which each element contains a JSON object with static data. In this example, the device data was written by the XML converter in msg.payload.kml.Document.Folder.Placemark . After some processing, this function inserts all devices in a JSON object, each with its own properties
  • Split node and join node: these nodes are necessary to divide the JSON object into multiple parts and to insert each part into the final array

Guidelines to create the csv file containing static data:

In these guidelines it will be assumed that data has already been read from the APIs and it has been adapted to have a javascript array in which each element contains a JSON object with static data like in this example:

  • Now you need to do two steps:
    • Generate a javascript array in which each element is a JSON object referred to a sensor of a device. JSON objects must have all the csv fields as properties, and the values assigned to them as values. Example:

    • Convert the JSON object to a CSV that can be sent by email (to yourself) so that it can be uploaded on the Snap4City platform

CSV generation flow:

CSV generation flow (following previous example):

  • CSV adapting (function node):
    • This node is the fundamental node for the creation of the CSV
    • For simplicity, the commented code (suitably to be modified) of the node is shown below.

var devices = msg.payload;
var keys = ["c6f03a41-880e-46f0-cccc-993501ca6b5","80466a36-2b5a-tttt-a7c3-db12c480da38"] // keys generated by a UUIDv4 generator
var sensorNumber = 5

// INITIALIZATION
var rows = new Array ((devices.length)*sensorNumber); // *sensorNumber (sensor number of each device)

// EXTERNAL CYCLE WITH PARAMETERS OF THE SINGLE DEVICE
for (var device = 0; device < devices.length; device++){
    var id = devices[device].id;
    var mac = "3D:F2:C9:A6:B3:3F"; // MAC fictitious
    var lat = (parseFloat(devices[device].latitude)).toFixed(5);
    var long = (parseFloat(devices[device].longitude)).toFixed(5);
    var K1 = keys[0];
    var K2 = keys[1];

    // INTERNAL CYCLE WITH PARAMETERS FOR EACH SENSOR
    for (var i = device*sensorNumber; i < (device*sensorNumber)+sensorNumber; i++){ // *sensorNumber (sensor number of each device)
        rows[i]= {
            name : "eCharging_" + id,
            devicetype : "ChargingStation",
            mac : mac,
            frequency : 600,
            kind : "sensor",
            protocol : "ngsi",
            format : "json",
            producer : "Comune di Firenze",
            lat : lat,
            long : long,
            K1 : K1,
            k2 : K2
        };

        switch (i % sensorNumber){
            case 0: // chargingState
                rows[i].valuename = "chargingState";
                rows[i].data_type = "string";
                rows[i].value_type = "charging_state";
                rows[i].editable = "false";
                rows[i].value_unit = "-";
                rows[i].health_criteria = "refresh_rate";
                rows[i].healthiness_value = 900;
                break;
            case 1: // chargingStateValue
                rows[i].valuename = "chargingStateValue";
                rows[i].data_type = "integer";
                rows[i].value_type = "charging_state";
                rows[i].editable = "false";
                rows[i].value_unit = "#";
                rows[i].health_criteria = "refresh_rate";
                rows[i].healthiness_value = 900;
                break;

            case 2: // stationState
                rows[i].valuename = "stationState";
                rows[i].data_type = "string";
                rows[i].value_type = "charging_station_state";
                rows[i].editable = "false";
                rows[i].value_unit = "-";
                rows[i].health_criteria = "refresh_rate";
                rows[i].healthiness_value = 900;
                break;

            case 3: // stationStateValue
                rows[i].valuename = "stationStateValue";
                rows[i].data_type = "integer";
                rows[i].value_type = "charging_station_state";
                rows[i].editable = "false";
                rows[i].value_unit = "#";
                rows[i].health_criteria = "refresh_rate";
                rows[i].healthiness_value = 900;
                break;

            case 4: // dateObserved
                rows[i].valuename = "dateObserved";
                rows[i].data_type = "time";
                rows[i].value_type = "timestamp";
                rows[i].editable = "false";
                rows[i].value_unit = "s";
                rows[i].health_criteria = "refresh_rate";
                rows[i].healthiness_value = 900;
                break;
        }

    }

}
msg.payload = rows;
return msg;

  • CSV node:
    • Properties
      • Insert the CSV fields as Columns: name,devicetype,mac,frequency,kind,protocol,format,producer,lat,long,valuename,data_type,value_type,editable,value_unit,health_criteria,healthiness_value,K1,k2
      • Separator: comma
      • Object to CSV options:
        • Tick on include column name row
        • New line: Linux (\n)
  • Email adapting (function node):

    msg.payload = Buffer.from(msg.payload);

    msg.filename = "charging_stations.csv"; // give it a name

    return msg;

    • After doing this you can connect the latter node with the email node to auto-send the csv file on your mailbox
    •  
Guidelines to create the RDF triples file
  • In these guidelines it will be assumed that data has already been read from the APIs and it has been adapted to have a javascript array in which each element contains a JSON object with static data like in this example:
     
  • Now you need to do two steps:
    • Split the array into JSON objects and for each device compile a JSON containing the RDF triples for that device
    • Join all the resulting JSON objects into a single object and convert it to a N3 file that can be sent by email (to yourself) so that it can be uploaded on the Snap4City platform

Triples generation flow:

Final static flow with triples and CSV generation (following previous example):

var subject = "http://www.disit.org/km4city/resource/iot/orionFirenze-UNIFI/Firenze/eCharging_" + msg.payload.id;
 
var rows = [{
                s : "<" + subject + ">",
                p : "<http://schema.org/streetAddress>",
                o : " " + msg.payload.address,
                stop : "."
            },
            {
                s : "<" + subject + ">",
                p : "<http://www.disit.org/km4city/schema#houseNumber>",
                o : " " + msg.payload.number,
                stop : "."
            },
            {
                s : "<" + subject + ">",
                p : "<http://schema.org/addressRegion>",
                o : " " + "FI",
                stop : "."
            },
            {
                s : "<" + subject + ">",
                p : "<http://schema.org/addressLocality>",
                o : " " + msg.payload.municipality,
                stop : "."
            },
            {
                s : "<" + subject + ">",
                p : "<http://www.disit.org/km4city/schema#isInRoad>",
                o : "<" + msg.payload.roadUri + ">",
                stop : "."
            }
        ];
 
msg.payload = rows;
return msg;

  • Join node and Final join (function node): these nodes merge all triples (stored in JSON objects) into a single array
    • Each triple is stored in an element of the array: the size of the array is the number of devices multiplied by the number of triples per device
    • For sivar rows = [];
       
      for (var i=0; i < msg.payload.length; i++){
          rows = rows.concat(msg.payload[i]);
      }
       
      msg.payload = rows;
      return msg;plicity, the code of the Final join node is shown below
  • .n3 (CSV node): contrary to its function, this node is used to create the N3 file containing the RDF triples (and not the CSV file)
    • Properties
      • Columns: s,p,o,stop
      • Separator: space
      • Object to CSV options:
        • Uncheck on include column name row
        • New line: Linux (\n)
  • “” Fix (string node): you need to apply a fix to the newly created buffer (there is a problem with the quotes)
    • Properties
      • replaceAll {space,quotes,space} with {space,quotes} where {} indicates the characters to be entered
  • Email adapting (function node):

msg.payload = Buffer.from(msg.payload);
msg.filename = "charging_stations.n3"; // give it a name
return msg;

    • After doing this you can connect the latter node with the email node to auto-send the csv file on your mailbox

 

IoTApp – DYNAMIC / Real Time Data

Path in Snap4City: “MENU -> IOT Applications > Your IoTApp”

Type of user:  Area Manager

Guidelines to create a flow that automatically keep real-time data updated:

  • The goal of the flow is to process the data from the sensors of the various devices in a format that can be uploaded to the broker
  • This is a sample flow pattern valid for most applications:
    1. Data Download (directly from the device, from the APIs, …), normally http requests
    2. Parsing: adapting of the received data to the format desired by the broker
    3. Uploading data on broker
  • The format accepted by the broker is a JSON object (one for each device) constructed in the following way:

{
   "id": device name,
   "type": device type,
   "attributes":[
      {
         "name": first value name,
         "value": value from downloaded data,
         "type": data type of this first sensor
      },
      {
         "name": second value name,
         "value": value from downloaded data,
         "type": data type of this second sensor
      },
      … and so on …

      {
         "name": last value name,
         "value": value from downloaded data,
         "type": data type of this last sensor
      }
   ]
}

  • Once the result of the flow is several JSON objects built as above, it will be possible to pass the JSON to the fiware-orion-out node
    • Configuring the fiware-orion-out node:
      • There are essentially two parameters to configure: the broker and the keys
      • K1 and K2 are the two keys set in the static information during the bulk load (or automatically generated for the sensors created by the interface)
      • IoT Broker URI and IoT Broker Port can be found by going to MENU -> IOT Directory and Devices > IoTDevices and selecting the desired device (or one of the set), created previously during the static information upload
    • Now you can connect the node to your flow (where msg.payload = JSON described above) and, by starting it or setting an automatic repeat, automatically the real-time data will be loaded on the broker

Example:

Florence Charging Stations (open data municipality of Florence) – Florence organization

Dynamic data flow:

  • timestamp (inject node): this node is necessary to start the flow. To start it, click on the square to the left of the node or set an automatic repeat in the properties (double click on the node)
  • KMZ Download (http node): this node makes an http request to the URL indicated in the properties. In this example it is necessary to download the file http://datigis.comune.fi.it/kml/ColonnineRicarica.kmz containing the static and real-time data of the devices
  • KMZ > KML (zip node): since the downloaded KMZ file is a zip file, it must be unzipped to have the KML file
  • Strings replacement (string node): HTML characters are present in the KML file, they must be replaced with Unicode characters
  • XML Converter (xml-converter node): this node converts the KML file (XML-based) into a JSON object editable by Node-Red
  • Parser (function node): this is the core of the data processing from the APIs. Here you need to use javascript to adapt the data from the APIs into a a javascript array in which each element contains a JSON object with real-time data. In this example, the device data was written by the XML converter in msg.payload.kml.Document.Folder.Placemark . After some processing, this function inserts all devices in a JSON object, each with its own properties
  • Split node: it’s necessary to divide the resulting JSON because the fiware-orion-out node accepts only one device at a time (and not an object containing all the devices)
  • IoT Device adapting (function node): this is the function that generates the required json so that it can be passed to the fiware-orion-out node

 

 

Comments

luigi scopelliti's picture

is it possible to have a clear picture  and a description of the function implemented in the Parser (function node).

What is li? li[0]? vl? Where are defined'