1. Home
  2. TC3.1 - AMMA Dashboard, monitoring data traffic flow in the platform and EventLog

TC3.1 - AMMA Dashboard, monitoring data traffic flow in the platform and EventLog

×

Warning message

You can't delete this newsletter because it has not been sent to all its subscribers.

 

Test Case Title

TC3.1 - AMMA Dashboard, monitoring data traffic flow in the platform and EventLog

Goal
  • Login in the Snap4City portal and access the AMMA tool.
  • monitor and analysing traffic flow volumes and communication events from a unique location, the AMMA (Application & MicroService Monitor and Analyser) tool.
  • customize widgets and views, data drill down and faceted filters capabilities.
  • log any kind of traffic RX/TX, with respect to database, IOT, external services, etc., IOT and/or ETL, geolocations, source/destination, etc.
  • decide the level of logging and the monitoring view, time resolution, etc.
  • save, share and reuse the views of the AMMA (Application & MicroService Monitor and Analyser) tool.

Prerequisites

Using a PC or Mobile with a web browser.

Indexed data are collected by the EventLogger tool, which is called through dedicated API (described in the Appendix) on event-driven basis by IoT, ETL and Data Analytics. ETL and Node-RED blocks for EventLogger have been designed for developers, in order to allow them to embed logging capabilities in their Microservices.

Collected data may present:

  • Values referring to data flow traffic (received or transmitted) by the single agents (ETL, Microservice, Data Analytic application etc.)
  • Values related to system information of the single agent (IP, process name or Container ID, communication type, i.e. if it is transmitting or receiving data)

Values referring to GPS coordinates of the involved service as a static geolocalized element of the city, which can be located on the Map via ServiceMap and Knowledge Base on Km4City model. For example: traffic flow or weather sensors, IoT devices

Expected successful result

See the dashboard and interact with its customizable panels and widgets. The user can access the dashboard and perform a number of actions to drill down on data, on time, facet filtering on different data types etc. The resulting combination of filters and widgets can be saved as a new dashboard/view locally or on the server, and it can be shared with other colleagues.

Steps

 

 

Snap4City is monitoring communication and database activities by using Application & MicroService Monitor and Analyser services, AMMA tool. The AMMA dashboard has been designed and realized mainly for developers and researchers. Initially developed on the basis of SOLR/Banana, now is operative in the versione ElasticSearch/Kibana.

The main service enabling this tool is the EventLogger services/API. Typical events to be log are: access to data storage, communication to devices (IOT Devices, ETL process, REST Call, ODBC, JDBC, WS, FTP, etc.), management kind (see formal definition on the portal articles, and on Swagger), processing kind (see in the following for the definition), they can be sent with some SMTP process/protocol as well. 

The EventLogger is based on NIFI collecting data in stream and RSyslog saving them into Hbase, while a SOLR Cloud sharded Index is created in real-time. Thus, the AMMA tool allows accessing the data and drilling down on them with faceted web interface. Then an elastic search solution is released, for Kibana, so that the following snapshots of Banana have been repplicated on Kibana also.

  • applications instances to be monitored and analysed
  • IOT devices to be monitored and analysed
  • ETL process kind or related instances to be monitored and analysed
  • data stores to be monitored and analysed
  • time period of the analysis
  • events/communications for management kind (subscription, registration, setting, config, process management,..);
  • events/communications for processing kind (rest call, get/set, notification, push,..);
  • Assess the performance and the distribution of the workload
  • Show the resulting diagrams on some graphic user interface as dashboard, control panel
  • Etc.

Presently the EventLogger is used as MicroService that the developer can use explicitly. On the other hand, the same service in terms of API can be directly used by the standard MicroService and tools to all almost all the impose the level of logging directly in the views, in the MicroService (sender) or in the logger filter.

Example 1: Open and use the AMMA Dashboard for communication analysis

This dashboard shows real-time and historical data related to traffic data flow and communication events among Microservices, ETL (and Data Analytics applications). The AMMA dashboard is embedded in the Snap4City portal with through single-sign on access.

To access the AMMA tool, log in the Snap4City portal and click on the “Management” menu item on the left-side bar. A sub-menu opens, in which you can find a list of management tools, including “Traffic Analyzer: AMMA” (Application and MicroService Monitor and Analyser), Click on it and you now have access to the AMMA dashboard:

 

 

The AMMA Tool includes the following widgets:

  • TIMEPICKER (1): date/time range selection. The user can identify FROM TO date/time;

  • QUERY (2): text search. The user can pose a query for retrieving specific indexed data.

  • HITS (3): reporting some metrics on the selection performed by filtering, for instance total counts.

  • FACET (4): a selection of facet fields, in this case: agent (describing the type of agent, e.g.: IoT, ETL, Data Analytics); ip_local (the IP of the agent); ip_ext (the IP of the external agent to / from which the local agent is transmitting / receiving data, if present); motivation (a codified string describing the aim of a single event, for instance IoT_Service, DB_Storage, FileSysyem_Storage, Dashboard, ASCAPI, i.e., Advanced Smart City APIcall, Eternal Services etc.); pid_local (representing the Process ID container for IoT agents or the process name for ETL); pid_local (representing the Process ID); com_mode (describing if the local agent is transmitting – “TX” – or receiving data – “RX” –); service_uri (the Km4City URI of the service involved in the communication/data flow); service_scope (describing if each monitored service is Internal, that is whenever a communication occurs between local and external IPs which are both in the internal back-end architecture, or External, otherwise). The user can make a selection for each Facet Field, thus filtering and drilling down on kind.

  • FILTERING (5): displaying the applied filters, which can be enabled, disabled or removed to allow different configurations and visualizations.

  • LINES-HISTOGRAM (6) with the stacked distribution of traffic data (it can be a measure in Kbytes or counts, according to the filter applied), to drill down on time line.

  • TERMS (7, 8, 9, 10, 11, 12, 14): these are some histogram and pie charts in which several kinds of distribution can be shown on the basis of the Facet fields. For each facet field, both count message and payload traffic in KB are shown. In this case the following Facet Fields are monitored: motivation (10a, 10b), agent (11a, 11b), communication type (12a, 12b) and service scope (14) in six pie charts; eight histogram graphs represent the local IP (8a, 8b) and the external IP (7a, 7b) of the most engaged agents, as well as the most instantiated process kinds (9a, 9b). Please note that widget which are annotated with “a” in the figure represent the message count, while the ones annotated with “b” the traffic payload in KB. The number of displayed records can be adjusted in the widget configuration.

  • HEATMAP (13): This is a heatmap for representing the count message values between ip_local and ip_ext of monitored data records. These values are mapped against a grey or coloured scale array.

  • SUNBURST (15, 16, 17): the Sunburst representation is a multi-level ring/pie chart which allow to easily visualize multi-level faceting diagrams, depending on the faceting input order. In this case, such a widget has been considered very useful to provide a powerful representation of Origin/Destination of local and external IPs, on the basis of different facet fields. However, in order to be suitable for data traffic monitoring, the widget has been customized in order to provide sum values statistics, and not only count values (see Example 3 in test case TC3.11 for further details on Sunburst panel customization and usage). Sunburst charts are also very useful for the purpose of applying more than one facet filter at time, allowing a very deep level of drill-down on faceted data.

  • SMARTCITYMAP (18): this is a newly created panel (see Example 2 in test case TC3.11 for further details on implementation and usage) which has been realized and made available from Banana panel configuration, in order to show service URIs enriched with additional information (service details, description and real-time data visualization, if available) retrieved by calling the Smart City API. Thus, a full integration (both on data and visualization style) has been reached with the map widgets built by using the Dashboard Builder.

  • TABLE (19): a table with data coming from the SOLR index, with a selection of columns and the possibility to order by column values. The “service_uri” field is now treated as a clickable web link, pointing to a page showing the service details as an instance of the Smart City Knowledge Base (showing properties and attributes in the form of Smart City ontological triples). Therefore, it is now possible to pass from real-time data to the Knowledge base description and context.

 

 Example 2: Play with the panels and widgets included in the AMMA Dashboard

 

  • Timeline Window (1)

It is possible to customize the selection of the time range by clicking on the configuration wheel on the left of the panel.

  • Search Query (2)

 

  • Hits (3)

 

The user can choose among different kind of metrics, not only count the data. By clicking on the configuration wheel in the upper-right corner, the following window appears. By selecting the Panel tab, the user can choose to compute metrics such as mean, maximum and minimum, standard deviation etc.:

  • Facet (4): In the facet panel, users can select facet fields to automatically filter all the other widgets and panels instantiated in the same dashboard.

 

Facet filtering is allowed according to the following fields:

        • Agent: it describes the type of agent, e.g.: IoT, ETL, Data Analytics;
        • Lang: language (if present) of analysed data.
        • ip_local: it is the IP of the local agent;
        • ip_ext: it is the IP of the external agent to / from which the local agent is transmitting / receiving data, if present;
        • motivation: a codified string describing the aim of a single event, for instance IoT_Service, DB_Storage, FileSysyem_Storage, , ASCAPI (Call to advanced SmartCity API), Notificator etc.;
        • pid_local: it represents the Process ID container for IoT agents or the process name for ETL;
        • com_mode: it describes if the local agent is transmitting (TX) or receiving data (RX);
        • service_uri (the Km4City URI of the service involved in the communication/data flow).
        • service_scope (describing if each monitored service is Internal, that is whenever a communication occurs between local and external IPs which are both in the internal back-end architecture, or External, otherwise).
  • Histogram: Traffic Data Flow monitoring (6)

The visualization of traffic data flow among IoT/ETL/Data Analytics is stacked and grouped on the basis of a field which can be chosen by the user, by clicking on the configuration wheel and selecting, in the Panel tab of the configuration window, the desired field in the “group” box, as shown in the following picture (in this case, data are grouped by pid_local, i.e. the Process Container ID for IoT or Process name for ETL):

  • Terms (IP_EXT): # Msg Count & Payload Traffic (KB) by Agent External IP – These two histograms show both the message counts (7a) and the payload traffic in KB (7b) grouped by external IP (ip_ext):

 

                               
(7a)                                                                           (7b)

 

  • Terms (IP_LOCAL): #Msg Count & Payload Traffic (KB) by Agent Local IP – These two histograms show both the message counts (8a) and the payload traffic in KB (8b) grouped by local IP (ip_local):

 

                      
(8a)                                                                           (8b)

 

  • Terms (PID_LOCAL): # Msg Count & Payload Traffic (KB) by Process ID – These two histograms show both the message counts (9a) and the payload traffic in KB (9b) grouped by Process ID (pid_local):

 

                              
(9a)                                                                      (9b)

 

  • Terms (MOTIVATION): # Msg Count & Payload Traffic (KB) by Motivation – These two pie charts show both the message counts (11a) and the payload traffic in KB (11b) grouped by Motivation:

 

              
(10a)                                                  (10b)

 

  • Terms (AGENT): # Msg Count & Payload Traffic (KB) by Agent – These two pie charts show both the message counts (11a) and the payload traffic in KB (11b) grouped by Agent:

 

                        
(11a)                                                                     (11b)

 

  • Terms (COM_MODE): # Msg Count & Payload Traffic (KB) by Communication Type – These two pie charts show both the message counts (12a) and the payload traffic in KB (12b) grouped by Communication Type (com_mode):

 

      
(12a)                                                    (12b)

 

  • Heatmap (IP_LOCAL vs IP_EXT): a heatmap for representing the count message values between ip_local and ip_ext of monitored data records. These values are mapped against a grey or coloured scale array (13), and offer a view suitable for origin/destination analysis.