TC10.12 - ResDash, Resource Dashboard for monitoring cloud resources

Test Case Title

TC10.12 - ResDash, Resource Dashboard for monitoring cloud resources

Goal

Configuring involved resources

Monitor and control resources allocated on cloud for Hosts, VM and Containers

Open a Dashboard for Drill down on monitored data regarding consumed resources at level of infrastructure per type (servers, Vm per containers, acquisition processes, indexing processes, IOT applications, dashboards, etc.) over time.

Create a new resource management dashboard

Prerequisites

Using a PC or Mobile with a web browser. A number of Hosts containing the VM allocating resources for the Snap4City solutions. A process collecting data from the cloud resources and feeding a database, thus creating a SOLR index.

Dashboards for monitoring resources already created.

The following functionalities are available only for specific Snap4city users with specific privileges.

Expected successful result

Monitor resource consumption, make a drill down on the consumed resources.

See status of the WEB services. See status of the API consumption. See status of the alert regarding the monitoring of the quality of the data ingested.

Steps

 

 

Please note that some of the following links could be accessible only for registered users.

The monitoring of infrastructure resources is the first step for controlling the system. The present solution allows to monitor all the resource, scheduling process and allocate new resources. The DISCES scheduler of DISIT lab is going to be improved to arrive at managing the scaling up and down of resources. This allows to:

  • plan the allocation of Containers and VM over time
  • optimize the allocation of resources
  • periodically re-plan and change the allocation according to single or major events: the disappear of processes,  the new allocation of applications and processes, etc.
    The monitoring of resources passed from the collection of Host, VM involved and inspecting them it is possible automatically extract all data and the information regarding the deployed containers.

The monitoring and management data collected is structured as:

  • by name of the VM
  • by group of the VM (please note that some of them are real processes of the Florence former solution)
    • IOT-Brokers, IOT-NodeRED
    • Hadoop-Phoenix, Hadoop-Phoenix-Master
    • SM-KB: ServiceMaps
    • SOLR-Index, DataIndexer
    • ETL processes VM
    • Mesos, Mesos-Master, Mesos-Test
    • TwitterVigilance
    • RStudio: data analytics remote access
    • FTP: access for services
    • IOT-Generator
    • NTP , Portal
    • SC-Planner-Optimizator
    • Shortest: routing planner
    • WiFi-Monitoring
  • By IP of the VM
  • By Hosts

Each monitoring is controlling several aspects of: CPU, Memory, Storage, Network. The monitoring is performed by a number of processes that collect these data and automatically feeds a data base and perform a SOLR index accessed by the Resource Monitoring Dashboard https://www.snap4city.org, through the menu Management --> Back Office Res. Analyzer: ResDash.
Detailed network consumption per process/task (IOT, ETL, Data Analytic, etc.) is performed by the AMMA.

 


Example 1: Feeding and configuring the Resource Monitoring and Scheduling
This tool is presently under development. The present version of the tool manages the configuration of VM, Host and Containers and perform the collection of measures regarding CPU, Memory, disk and network as depicted into the Resource Management Dashboard reported in the next Section.
 
   
 

 
The Elastic management is connected to the DISCES scheduler. See in Appendix the reference to its manual.
 
 
 


Example 2: Using the controlling the Resource Monitoring Dashboard

  1. Open the Resource Monitoring Dashboard with real-time data of the present deployed snap4city solution.
  • Menu: Management --> Back Office Res. Analyzer: ResDash
     

     
  • (1) TIMEPICKER: from to date/time. The user can identify FROM TO date/time;
  • (2) QUERY: text search. The user can pose a query;
  • (3) FILTERING: reporting the applied filters. The user can see the applied filters and can remove them returning to a different config, can edit them;
  • (4) HIT: reporting some metrics on the selection performed by filtering. The user can ask to have more data;
  • (5) FACET: a selection of facet fields, in this case: Name (VM name), Group (VM Group), IP (VM IP), Host (VM Host). The user can select each facet field, thus filtering and drilling down on type;
  • (6) HISTOGRAM: reporting CPU usage along the filtered timeline;
  • (7) LINE: a line chart reporting memory;
  • (8) HISTOGRAM: reporting disk reads;
  • Additionally (not in the image but visible on the web page) the dashboard reports disk writes, network rx/tx and tabular data related to the filtered data.

 

  1. Open the above link/dashboard on different terminals
  • https://www.snap4city.org
    • Menu: Management --> Back Office Res. Analyzer: ResDash
  • Please note that this dashboard is fully responsive, and presents several tools so that it may be necessary need to scroll. Shorter examples can be realized, while for developers the scroll would not be a problem, if the information provided is that required by the developers.
     
  1. See the Automated update activated for this dashboard.

The following image shown an example of configuration settings of a panel with the auto refresh set every 60 seconds.
 


 
Resource Monitor Dashboard may update values imposing some timing per dashboard widget.

 
List of Widgets for the Resource Manager Dashboard

  • TimeLine 

 

  • CPU usage

 

  • Memory usage

 

  • Disk reads

 

  • Disk writes

 

  • Network RX

 

  • Network TX

 

  • Table

  • Facets

 

  • Search

 

  • Hits

 


Example 3: Drill Down on data and multiple filtering on the Resource Monitoring Dashboard

  1. Go on the dashboard https://www.snap4city.org
    • Menu: Management --> Back Office Res. Analyzer: ResDash
       
  2. drill down on time
    • Select from the TIMEPICKER (1) a time range with the mouse to drill down on time, the new filter on time is added on the FILTERING panel
    • all the other widgets start recalculating to show the resulting data filtered. The drill down on timeline can be performed (by day, by week, by month, by year, from date/time to date/time, from a given time instant to another) by using Resource Monitor Dashboard.  Once the time selection (filtering) is performed, the other views of the Dashboard are aligned: the counting of selected object is updated, the faceted list of population selected is updated, the map is also updated.
  3. In the widget HISTOGRAM (6) with the left button of the mouse clicked, select a period in the timeline. The selected period is highlighted in the histogram as shown in the following figure.
     

     
  4. When the left button of the mouse is released, all the widget in the dashboard are updated accordingly, and a new search term is added in the FILTERING widget, as shown in the following image.
     

     
  5. Click on a field “group” in the FACET widget and click on “IOT-NodeRED”. You can see the other widgets that is updated accordingly, and the FILTERING widget is updated with a new search term, as in the following image.

     

     
  6. Click on some other widget/panel to create other filters that are added to FILTERING panel, and the other panels change accordingly.
     

     
  7. Insert some text on QUERY panel, thus the filtering is performed on the basis of the text. Type a text in the QUERY widget on the top right in the form: “facet field:name”, for example “ip: 192.168.1.20”. 
  8. Go on the FILTERING panel and click on the up-right-corner of the filtering you like to remove. 
  9. Go on the top menu if you like to save the Dashboard on cloud or locally on the computer. If you save it locally, it can be reloaded from the local disk when you like; while if you save on cloud, your or other colleagues can use the dashboard you developed.

 


Example 4: Creating new views on Resource Monitoring Dashboard data

  1. Access to an empty dashboard by clicking the following link:
  • Menu: Management --> Back Office Res. Analyzer: ResDash
  • Click on the sheet of paper icon on the top right to open the "New" dialog (1).