TC9.9 - Indexing IOT and ETL data changes into a unique location for search via Developer Dashboard (Multiprotocol Subscriber and Indexer)

×

Warning message

You can't delete this newsletter because it has not been sent to all its subscribers.

Test Case Title

TC9.9 - Indexing IOT and ETL data changes into a unique location for search via Developer Dashboard (Multiprotocol Subscriber and Indexer)

Goal

See all IOT device changes from a unique location to search them via Developer Dashboard (already shown in previous TC)

Have in a unique location IOT device changes and ETL changes, index them, browse them and search them via the Developer Dashboard.

Modify the data driven filtering for saving / indexing data

See results on Developer Dashboard.

Prerequisites

Using a PC or Mobile with a web browser. Access to the back office since this process for changing the flow if needed. The implementation is performed into a data driven VM based on NIFI that can be scaled up.

The following functionalities are available only for specific Snap4city users with specific privileges.

Expected successful result

See the resulting referral data a real-time data on the developer dashboard

Steps

 

 

The ETL data is stored in tabular form on Apache Phoenix cluster while IOT streams are retrieved by querying the different IOT broker URLS . In addition, the SOLR index is defined as following json structure

{"measureType":"",

        "src":"<ETL|IOT>",

        "latlon":"",

        "unit":"",

        "deviceName":"",

        "className":"",

        "subMeasureType":"",

        "kind":"",

        "dateTime":"",

        "serviceUri":"",

        "id":"",

        "value":",

        "sensorID":""

        }

Because one IOT device data observation correspond to multi measures (for example “Sensore via Bolognese” measures road temperature, air humidity, etc .. ), a process of denormalization (i.e. having single row for each measure) is needed to permits faceted search on the corresponding SOLR index.

Apache NIFI Framework is used to process the data gathered from Apache Phoenix tables by SQL queries and IOT brokers, denormalize and then update the SOLR index dynamically.

In the figure below is depicted a proof-of-concept of the process of data-retrieve, enrich and indexing. The process is composed of three parts: ETL management, IOT management and store and index management.

 

For each table stored in Apache Phoenix produced by an ETL process is identified a NIFI process Group (e.g.: weatherSensorObsrvation, smartbenchobservation etc … See Figure below).

Each Nifi Process Group is composed by a part of data retrieving and a part of data manipulation permitting to capture each  measure per measure per type e.g. road temperature, pressure.

An example of a filter is drawn the figure below.