|
Test Case Title |
TC9.9 - Indexing IOT and ETL data changes into a unique location for search via Developer Dashboard (Multiprotocol Subscriber and Indexer) |
|
Goal |
See all IOT device changes from a unique location to search them via Developer Dashboard (already shown in previous TC) Have in a unique location IOT device changes and ETL changes, index them, browse them and search them via the Developer Dashboard. Modify the data driven filtering for saving / indexing data See results on Developer Dashboard. |
|
Prerequisites |
Using a PC or Mobile with a web browser. Access to the back office since this process for changing the flow if needed. The implementation is performed into a data driven VM based on NIFI that can be scaled up. The following functionalities are available only for specific Snap4city users with specific privileges. |
|
Expected successful result |
See the resulting referral data a real-time data on the developer dashboard |
|
Steps |
The ETL data is stored in tabular form on Apache Phoenix cluster while IOT streams are retrieved by querying the different IOT broker URLS . In addition, the SOLR index is defined as following json structure
{"measureType":"",
"src":"<ETL|IOT>",
"latlon":"",
"unit":"",
"deviceName":"",
"className":"",
"subMeasureType":"",
"kind":"",
"dateTime":"",
"serviceUri":"",
"id":"",
"value":",
"sensorID":""
}
Because one IOT device data observation correspond to multi measures (for example “Sensore via Bolognese” measures road temperature, air humidity, etc .. ), a process of denormalization (i.e. having single row for each measure) is needed to permits faceted search on the corresponding SOLR index.
Apache NIFI Framework is used to process the data gathered from Apache Phoenix tables by SQL queries and IOT brokers, denormalize and then update the SOLR index dynamically.
In the figure below is depicted a proof-of-concept of the process of data-retrieve, enrich and indexing. The process is composed of three parts: ETL management, IOT management and store and index management.
For each table stored in Apache Phoenix produced by an ETL process is identified a NIFI process Group (e.g.: weatherSensorObsrvation, smartbenchobservation etc … See Figure below).
Each Nifi Process Group is composed by a part of data retrieving and a part of data manipulation permitting to capture each measure per measure per type e.g. road temperature, pressure.
An example of a filter is drawn the figure below.
The interesting data are extracted and a custom json is constructed with valid SOLR data type and then committed to SOLR index and Hbase table.
Click on https://devdash.snap4city.org/
See the resulting Developer dashboard:
