1. Home
  2. TC3.1 - AMMA Dashboard, monitoring data traffic flow in the platform and EventLog

TC3.1 - AMMA Dashboard, monitoring data traffic flow in the platform and EventLog

 

Test Case Title

TC3.1 - AMMA Dashboard, monitoring data traffic flow in the platform and EventLog

Goal
  • Login in the Snap4City portal and access the AMMA tool.
  • monitor and analysing traffic flow volumes and communication events from a unique location, the AMMA (Application & MicroService Monitor and Analyser) tool.
  • customize widgets and views, data drill down and faceted filters capabilities.
  • log any kind of traffic RX/TX, with respect to database, IOT, external services, etc., IOT and/or ETL, geolocations, source/destination, etc.
  • decide the level of logging and the monitoring view, time resolution, etc.
  • save, share and reuse the views of the AMMA (Application & MicroService Monitor and Analyser) tool.

Prerequisites

Using a PC or Mobile with a web browser.

Indexed data are collected by the EventLogger tool, which is called through dedicated API (described in the Appendix) on event-driven basis by IoT, ETL and Data Analytics. ETL and Node-RED blocks for EventLogger have been designed for developers, in order to allow them to embed logging capabilities in their Microservices.

Collected data may present:

  • Values referring to data flow traffic (received or transmitted) by the single agents (ETL, Microservice, Data Analytic application etc.)
  • Values related to system information of the single agent (IP, process name or Container ID, communication type, i.e. if it is transmitting or receiving data)

Values referring to GPS coordinates of the involved service as a static geolocalized element of the city, which can be located on the Map via ServiceMap and Knowledge Base on Km4City model. For example: traffic flow or weather sensors, IoT devices

Expected successful result

See the dashboard and interact with its customizable panels and widgets. The user can access the dashboard and perform a number of actions to drill down on data, on time, facet filtering on different data types etc. The resulting combination of filters and widgets can be saved as a new dashboard/view locally or on the server, and it can be shared with other colleagues.

Steps

 

 

Snap4City is monitoring communication and database activities by using Application & MicroService Monitor and Analyser services, AMMA tool. The AMMA dashboard has been designed and realized mainly for developers and researchers. Initially developed on the basis of SOLR/Banana, now is operative in the versione ElasticSearch/Kibana.

The main service enabling this tool is the EventLogger services/API. Typical events to be log are: access to data storage, communication to devices (IOT Devices, ETL process, REST Call, ODBC, JDBC, WS, FTP, etc.), management kind (see formal definition on the portal articles, and on Swagger), processing kind (see in the following for the definition), they can be sent with some SMTP process/protocol as well. 

The EventLogger is based on NIFI collecting data in stream and RSyslog saving them into Hbase, while a SOLR Cloud sharded Index is created in real-time. Thus, the AMMA tool allows accessing the data and drilling down on them with faceted web interface. Then an elastic search solution is released, for Kibana, so that the following snapshots of Banana have been repplicated on Kibana also.

  • applications instances to be monitored and analysed
  • IOT devices to be monitored and analysed
  • ETL process kind or related instances to be monitored and analysed
  • data stores to be monitored and analysed
  • time period of the analysis
  • events/communications for management kind (subscription, registration, setting, config, process management,..);
  • events/communications for processing kind (rest call, get/set, notification, push,..);
  • Assess the performance and the distribution of the workload
  • Show the resulting diagrams on some graphic user interface as dashboard, control panel
  • Etc.

Presently the EventLogger is used as MicroService that the developer can use explicitly. On the other hand, the same service in terms of API can be directly used by the standard MicroService and tools to all almost all the impose the level of logging directly in the views, in the MicroService (sender) or in the logger filter.

Example 1: Open and use the AMMA Dashboard for communication analysis

This dashboard shows real-time and historical data related to traffic data flow and communication events among Microservices, ETL (and Data Analytics applications). The AMMA dashboard is embedded in the Snap4City portal with through single-sign on access.

To access the AMMA tool, log in the Snap4City portal and click on the “Management” menu item on the left-side bar. A sub-menu opens, in which you can find a list of management tools, including “Traffic Analyzer: AMMA” (Application and MicroService Monitor and Analyser), Click on it and you now have access to the AMMA dashboard:

 

 

The AMMA Tool includes the following widgets:

  • TIMEPICKER (1): date/time range selection. The user can identify FROM TO date/time;

  • QUERY (2): text search. The user can pose a query for retrieving specific indexed data.

  • HITS (3): reporting some metrics on the selection performed by filtering, for instance total counts.

  • FACET (4): a selection of facet fields, in this case: agent (describing the type of agent, e.g.: IoT, ETL, Data Analytics); ip_local (the IP of the agent); ip_ext (the IP of the external agent to / from which the local agent is transmitting / receiving data, if present); motivation (a codified string describing the aim of a single event, for instance IoT_Service, DB_Storage, FileSysyem_Storage, Dashboard, ASCAPI, i.e., Advanced Smart City APIcall, Eternal Services etc.); pid_local (representing the Process ID container for IoT agents or the process name for ETL); pid_local (representing the Process ID); com_mode (describing if the local agent is transmitting – “TX” – or receiving data – “RX” –); service_uri (the Km4City URI of the service involved in the communication/data flow); service_scope (describing if each monitored service is Internal, that is whenever a communication occurs between local and external IPs which are both in the internal back-end architecture, or External, otherwise). The user can make a selection for each Facet Field, thus filtering and drilling down on kind.

  • FILTERING (5): displaying the applied filters, which can be enabled, disabled or removed to allow different configurations and visualizations.

  • LINES-HISTOGRAM (6) with the stacked distribution of traffic data (it can be a measure in Kbytes or counts, according to the filter applied), to drill down on time line.

  • TERMS (7, 8, 9, 10, 11, 12, 14): these are some histogram and pie charts in which several kinds of distribution can be shown on the basis of the Facet fields. For each facet field, both count message and payload traffic in KB are shown. In this case the following Facet Fields are monitored: motivation (10a, 10b), agent (11a, 11b), communication type (12a, 12b) and service scope (14) in six pie charts; eight histogram graphs represent the local IP (8a, 8b) and the external IP (7a, 7b) of the most engaged agents, as well as the most instantiated process kinds (9a, 9b). Please note that widget which are annotated with “a” in the figure represent the message count, while the ones annotated with “b” the traffic payload in KB. The number of displayed records can be adjusted in the widget configuration.

  • HEATMAP (13): This is a heatmap for representing the count message values between ip_local and ip_ext of monitored data records. These values are mapped against a grey or coloured scale array.

  • SUNBURST (15, 16, 17): the Sunburst representation is a multi-level ring/pie chart which allow to easily visualize multi-level faceting diagrams, depending on the faceting input order. In this case, such a widget has been considered very useful to provide a powerful representation of Origin/Destination of local and external IPs, on the basis of different facet fields. However, in order to be suitable for data traffic monitoring, the widget has been customized in order to provide sum values statistics, and not only count values (see Example 3 in test case TC3.11 for further details on Sunburst panel customization and usage). Sunburst charts are also very useful for the purpose of applying more than one facet filter at time, allowing a very deep level of drill-down on faceted data.

  • SMARTCITYMAP (18): this is a newly created panel (see Example 2 in test case TC3.11 for further details on implementation and usage) which has been realized and made available from Banana panel configuration, in order to show service URIs enriched with additional information (service details, description and real-time data visualization, if available) retrieved by calling the Smart City API. Thus, a full integration (both on data and visualization style) has been reached with the map widgets built by using the Dashboard Builder.

  • TABLE (19): a table with data coming from the SOLR index, with a selection of columns and the possibility to order by column values. The “service_uri” field is now treated as a clickable web link, pointing to a page showing the service details as an instance of the Smart City Knowledge Base (showing properties and attributes in the form of Smart City ontological triples). Therefore, it is now possible to pass from real-time data to the Knowledge base description and context.

 

 Example 2: Play with the panels and widgets included in the AMMA Dashboard

 

  • Timeline Window (1)

It is possible to customize the selection of the time range by clicking on the configuration wheel on the left of the panel.

  • Search Query (2)

 

  • Hits (3)

 

The user can choose among different kind of metrics, not only count the data. By clicking on the configuration wheel in the upper-right corner, the following window appears. By selecting the Panel tab, the user can choose to compute metrics such as mean, maximum and minimum, standard deviation etc.:

  • Facet (4): In the facet panel, users can select facet fields to automatically filter all the other widgets and panels instantiated in the same dashboard.

 

Facet filtering is allowed according to the following fields:

        • Agent: it describes the type of agent, e.g.: IoT, ETL, Data Analytics;
        • Lang: language (if present) of analysed data.
        • ip_local: it is the IP of the local agent;
        • ip_ext: it is the IP of the external agent to / from which the local agent is transmitting / receiving data, if present;
        • motivation: a codified string describing the aim of a single event, for instance IoT_Service, DB_Storage, FileSysyem_Storage, , ASCAPI (Call to advanced SmartCity API), Notificator etc.;
        • pid_local: it represents the Process ID container for IoT agents or the process name for ETL;
        • com_mode: it describes if the local agent is transmitting (TX) or receiving data (RX);
        • service_uri (the Km4City URI of the service involved in the communication/data flow).
        • service_scope (describing if each monitored service is Internal, that is whenever a communication occurs between local and external IPs which are both in the internal back-end architecture, or External, otherwise).
  • Histogram: Traffic Data Flow monitoring (6)

The visualization of traffic data flow among IoT/ETL/Data Analytics is stacked and grouped on the basis of a field which can be chosen by the user, by clicking on the configuration wheel and selecting, in the Panel tab of the configuration window, the desired field in the “group” box, as shown in the following picture (in this case, data are grouped by pid_local, i.e. the Process Container ID for IoT or Process name for ETL):

  • Terms (IP_EXT): # Msg Count & Payload Traffic (KB) by Agent External IP – These two histograms show both the message counts (7a) and the payload traffic in KB (7b) grouped by external IP (ip_ext):

 

                               
(7a)                                                                           (7b)

 

  • Terms (IP_LOCAL): #Msg Count & Payload Traffic (KB) by Agent Local IP – These two histograms show both the message counts (8a) and the payload traffic in KB (8b) grouped by local IP (ip_local):

 

                      
(8a)                                                                           (8b)

 

  • Terms (PID_LOCAL): # Msg Count & Payload Traffic (KB) by Process ID – These two histograms show both the message counts (9a) and the payload traffic in KB (9b) grouped by Process ID (pid_local):

 

                              
(9a)                                                                      (9b)

 

  • Terms (MOTIVATION): # Msg Count & Payload Traffic (KB) by Motivation – These two pie charts show both the message counts (11a) and the payload traffic in KB (11b) grouped by Motivation:

 

              
(10a)                                                  (10b)

 

  • Terms (AGENT): # Msg Count & Payload Traffic (KB) by Agent – These two pie charts show both the message counts (11a) and the payload traffic in KB (11b) grouped by Agent:

 

                        
(11a)                                                                     (11b)

 

  • Terms (COM_MODE): # Msg Count & Payload Traffic (KB) by Communication Type – These two pie charts show both the message counts (12a) and the payload traffic in KB (12b) grouped by Communication Type (com_mode):

 

      
(12a)                                                    (12b)

 

  • Heatmap (IP_LOCAL vs IP_EXT): a heatmap for representing the count message values between ip_local and ip_ext of monitored data records. These values are mapped against a grey or coloured scale array (13), and offer a view suitable for origin/destination analysis.

 

 

  • Terms (SERVICE_SCOPE): #Msg Count by Service Scope (14) – This pie chart shows the message counts grouped by Service Scope (that is, if the service is INTERNAL or EXTERNAL):

 

 

  • Sunburst (COM_MODE): IP traffic (KB and counts) by Communication Mode. The following Sunburst pie charts show the IP traffic, in terms of both payload traffic in KB (15a) and counts (15b), grouped by Communication Mode. The represented circular rings, ordered from inner to outer, are: Communication mode (com_mode), ip_local and ip_ext.

 

  
(15a)                                                                                    (15b)

  • Sunburst (PID_LOCAL): IP traffic (KB and counts) by Process ID. The following Sunburst pie charts show the IP traffic, in terms of both payload traffic in KB (16a) and counts (16b), grouped my Process ID. The represented circular rings, ordered from inner to outer, are: Process ID (pid_local), ip_local and ip_ext.

    
(16a)                                                                                    (16b)

  • Sunburst (MOTIVATION): IP traffic (KB and counts) by Motivation. The following Sunburst pie charts show the IP traffic, in terms of both payload traffic in KB (17a) and counts (17b), grouped by Motivation. The represented circular rings, ordered from inner to outer, are: Motivation, ip_local and ip_ext.

 

   
(17a)                                                                                    (17b)

 

In the above illustrated cases, the user can easily have an instant view of which local IPs are transmitting (TX) or receiving (RX) TO / FROM which external IPs, as well as visualizing all the other possible combinations for motivation and process ID facets. For example, in the following three pictures, it is shown how easily the user can see which local and external IPs are committed to read and write data by the process ID “Sensore_Ghiaccio_ViaBolognese_RT” (a process which receives real-time data from a weather sensor), which are the most used, both in terms of message counts and data traffic.

          

  • SmartCity Map (18): this newly created panel (see Example 2 in test case TC3.5 for further details and information) allows to show service URIs with custom style (icons and colours) and enriched information, retrieved by exploiting the Smart City API. The new widget has been mutuated from the Dashboard Builder, in order to easily view different type of services, and which data can be navigated from each of them. Now it is possible to click on each geolocated marker (representing a specific service URI) and retrieve additional information (details, description and visualization of real-time data, if available). In the next figures, as an example, it is shown how to visualize the time trend of data related to air humidity for the past 7 days, coming from a Service with URI: http://www.disit.org/km4city/resource/SensoreViaBolognese (which receives data from a weather sensor):

 

 

Depending on the different types of Service URIs, different kind of information can be visualized. As an example, in the next figure, data related to a Bike-Sharing Rack service (in particular, the weekly trend of available bikes):

In the next figure, the weekly trend of transit counts for a Smart Bench service is depicted:

Finally, the next image shows the monthly filling rate for a Smart Waste container service:

 

  • Table (19), event logged related to the context of analysis, paging and ordering options are active. In addition, the “service_uri” field has now been defined and managed as a Hyperlink Column. In this way, each service URI acts now as a clickable web link, pointing to a page showing the service details as an instance of the Smart City Knowledge Base (showing properties and attributes in the form of ontological triples). Therefore, it is now possible to pass from real-time data to the Knowledge base description and context.

When the user clicks on the ServiceURI hyperlink column, he is redirected to the corresponding Smart City Knowledge base description page:


 

 Example 3: facet filtering and drill down operations on data.

For instance, select a period in the timeline in the Traffic LINE-HISTOGRAM, by dragging a time window of the desired length, while holding the left button of the mouse clicked.

The selected period is highlighted in the line-histogram, as shown in the following figure.

When the left button of the mouse is released, also all the other widgets/panels in the dashboard is uploaded accordingly, and a new search term is added in the FILTERING Panel (5), as shown in the following image:

To remove filter blocks, click on the “X” on the up-right-corner of the filtering you like to remove.

Click on some other widget/panel, for example from TERM (7, 8, 9, 10, 11, 12, 14), to create other filters that are added to FILTERING panel (5), and the other panels change accordingly. For example, if you click on the “agent” field in the FACET Panel (4), you can visualize, for instance, only ETL Microservices, like shown in the following pictures:

 

Furthermore, applying another facet on the communication type (“com_mode”), we can detect only the ETL applications which are transmitting (TX) or receiving (RX) data, or in a similar way, we could be interested in view the data subset related to a specific Process ID (e.g. “Smartwaste” or “Electric_Vehicle_Charging”).

 

Go on the top menu if you like to export and save the Dashboard on the server or locally on your computer. If you save it locally, it can be reloaded from the local disk when you like; while if you save on cloud, your or other colleagues can use the dashboard you developed.
 

Example 4: Geographical filtering (Geo-Facet).

A new kind of facet functionality has been added to the AMMA Tool: a geographical faceting, implemented in the SmartCity Map widget, which allows the user to geographically filter the visualization of services. To this end, the user can simply adjust with the zoom and the visualization bounds of the map, and finally click on the new Geo-Facet button in order to filter the desired services to be viewed on the SmartCity Map. This is not only a visual filtering, but a full facet functionality. Actually, with this operation, all data displayed by the other panels is filtered, displaying data related to the services which are geolocated within the chosen geographical selection. The next figures illustrate this simple process:

As a first step, the user visualizes the services of a certain geographical area. When the user has chosen some services of interest and want to filter all the other panels data to drill down and visualize only the data related to the services of interest, he has simply to adjust the zoom and the position of the map to include only the desired services. Then, by simply clicking on the Geo-Facet button, all the panels is filtered accordingly.

 

 

Actually, enlarging again the visualized area, it can be noticed that only the selected services are shown, and the panels data are filtered accordingly to this geographical faceting, and the geographical filter is shown in the Filter Panel, as illustrated in the next figure.

 


 

Example 5: Customize the AMMA Tool widgets to create real-time dashboards.

Developers and researchers may be interested in a finer tuning in monitoring data traffic flows in REAL-TIME. To this end, you can suitably customize the AMMA Tool widgets, in order to create real-time dashboards.

If you enable the Auto-refresh option in the Time Window (TIME PICKER (1) panel), you can easily add real-time monitor capabilities to your dashboard, by selecting opportune time range and measure resolution to display.

As an example, we created the AMMA Dashboard 1min Snap4City-30minview, available at the following URL:

AMMA Dashboard 1min Snap4City-30minview: https://www.snap4city.org/management/iframeApp.php?linkUrl=https://amma.snap4city.org/&linkId=1mLink&pageTitle=Traffic%20Analyzer:%20AMMA&fromSubmenu=managLink

The dashboard view is shown in the following figure:

In this example, the Time Window range is set to 30 minutes, and the user can visualize trend changes on 1 minute-sampled stacked traffic data live on the screen, every 15 seconds. All the facet filter and drill down capabilities on data present in the AMMA Tool dashboard are maintained. If you want to adjust the Auto-refresh time, you can click on the configuration wheel of the Time Window panel TIMEPICKER (1), then select the “Panel” tab of the configuration window and set the desired Auto-refresh rate, as shown in the following picture:
 

Example 6: How to Explicitly Log from IoT Application in Node-RED

If you are a Node-RED developer, a dedicated NodeJS block called “event-log” is available to perform log operations by calling the EventLogger. It can be connected to the input, as well to the output of the block the developer intends to monitor through the EventLogger (and later see the results in the AMMA Dashboard Details), depending on the communication mode, and related to the specific needs and requirements of the developer or researcher. Details for its usage and configuration are reported in the following table:

Graphic Representation

 

Configuration

 

Required parameters for calling the EventLogger API (see the Appendix for reference) are automatically retrieved by the event-log block, except for the identification Name, the communication mode (TX for transmitting, RX for receiving) and the motivation (see the EventLogger API in the Appendix for reference), which can be chosen by the developer/researcher. Currently, the EventLogger NodeJS block is integrated in all the Snap4City NodeJS blocks available to users to build their own apps. However, it can still be used as a separate block for those users who may create their own new NodeJS blocks and want to log their data.
 

Example 7: How to explicitly Log from ETL

If you are an ETL developer, a dedicated block for calling the EventLogger has been realized and made available, as shown in the following picture:

Also in this case the event-log block can be connected to the input, as well to the output of the block the developer intends to monitor through, depending on the communication mode. The developer/researcher can configure the parameters needed for calling the EventLogger API by clicking on the block and assigning the required fields in the Script Values window.

Presently the EventLogger is used as MicroService that the developer can use explicitly. On the other hand, the same service in terms of API can be directly used by the standard MicroService and tools to all almost all the impose the level of logging directly in the views, in the MicroService (sender) or in the logger filter.

  • Share