Test Case Title	TC10.12 - ResDash, Resource Dashboard for monitoring cloud resources
Goal	Configuring involved resources Monitor and control resources allocated on cloud for Hosts, VM and Containers Open a Dashboard for Drill down on monitored data regarding consumed resources at level of infrastructure per type (servers, Vm per containers, acquisition processes, indexing processes, IOT applications, dashboards, etc.) over time. Create a new resource management dashboard
Prerequisites	Using a PC or Mobile with a web browser. A number of Hosts containing the VM allocating resources for the Snap4City solutions. A process collecting data from the cloud resources and feeding a database, thus creating a SOLR index. Dashboards for monitoring resources already created. The following functionalities are available only for specific Snap4city users with specific privileges.
Expected successful result	Monitor resource consumption, make a drill down on the consumed resources. See status of the WEB services. See status of the API consumption. See status of the alert regarding the monitoring of the quality of the data ingested.
Steps

 

Please note that some of the following links could be accessible only for registered users.

The monitoring of infrastructure resources is the first step for controlling the system. The present solution allows to monitor all the resource, scheduling process and allocate new resources. The DISCES scheduler of DISIT lab is going to be improved to arrive at managing the scaling up and down of resources. This allows to:

• plan the allocation of Containers and VM over time
• optimize the allocation of resources
• periodically re-plan and change the allocation according to single or major events: the disappear of processes,  the new allocation of applications and processes, etc.
  The monitoring of resources passed from the collection of Host, VM involved and inspecting them it is possible automatically extract all data and the information regarding the deployed containers.

The monitoring and management data collected is structured as:

• by name of the VM
• by group of the VM (please note that some of them are real processes of the Florence former solution)
  • IOT-Brokers, IOT-NodeRED
  • Hadoop-Phoenix, Hadoop-Phoenix-Master
  • SM-KB: ServiceMaps
  • SOLR-Index, DataIndexer
  • ETL processes VM
  • Mesos, Mesos-Master, Mesos-Test
  • TwitterVigilance
  • RStudio: data analytics remote access
  • FTP: access for services
  • IOT-Generator
  • NTP , Portal
  • SC-Planner-Optimizator
  • Shortest: routing planner
  • WiFi-Monitoring
• By IP of the VM
• By Hosts

Each monitoring is controlling several aspects of: CPU, Memory, Storage, Network. The monitoring is performed by a number of processes that collect these data and automatically feeds a data base and perform a SOLR index accessed by the Resource Monitoring Dashboard https://www.snap4city.org, through the menu Management --> Back Office Res. Analyzer: ResDash.
Detailed network consumption per process/task (IOT, ETL, Data Analytic, etc.) is performed by the AMMA.

 

---

Example 1: Feeding and configuring the Resource Monitoring and Scheduling
This tool is presently under development. The present version of the tool manages the configuration of VM, Host and Containers and perform the collection of measures regarding CPU, Memory, disk and network as depicted into the Resource Management Dashboard reported in the next Section.
 
   
 

 
The Elastic management is connected to the DISCES scheduler. See in Appendix the reference to its manual.
 
 
 

---

Example 2: Using the controlling the Resource Monitoring Dashboard

1. Open the Resource Monitoring Dashboard with real-time data of the present deployed snap4city solution.
   • This dashboard shows Virtual Machines real-time data from the datacenter hosting the Snap4City services.
   • https://www.snap4city.org

• Menu: Management --> Back Office Res. Analyzer: ResDash
   
  
   

• (1) TIMEPICKER: from to date/time. The user can identify FROM TO date/time;
• (2) QUERY: text search. The user can pose a query;
• (3) FILTERING: reporting the applied filters. The user can see the applied filters and can remove them returning to a different config, can edit them;
• (4) HIT: reporting some metrics on the selection performed by filtering. The user can ask to have more data;
• (5) FACET: a selection of facet fields, in this case: Name (VM name), Group (VM Group), IP (VM IP), Host (VM Host). The user can select each facet field, thus filtering and drilling down on type;
• (6) HISTOGRAM: reporting CPU usage along the filtered timeline;
• (7) LINE: a line chart reporting memory;
• (8) HISTOGRAM: reporting disk reads;
• Additionally (not in the image but visible on the web page) the dashboard reports disk writes, network rx/tx and tabular data related to the filtered data.

 

2. Open the above link/dashboard on different terminals

• https://www.snap4city.org
  • Menu: Management --> Back Office Res. Analyzer: ResDash
• Please note that this dashboard is fully responsive, and presents several tools so that it may be necessary need to scroll. Shorter examples can be realized, while for developers the scroll would not be a problem, if the information provided is that required by the developers.
   

3. See the Automated update activated for this dashboard.
   • In https://www.snap4city.org
     1. Menu: Management --> Back Office Res. Analyzer: ResDash
   • Each widget has timer to get updated data set at 15 seconds.

The following image shown an example of configuration settings of a panel with the auto refresh set every 60 seconds.
 

 
Resource Monitor Dashboard may update values imposing some timing per dashboard widget.

• To test click on the link to see a live example:
• https://www.snap4city.org
  • Menu: Management --> Back Office Res. Analyzer: ResDash

 
List of Widgets for the Resource Manager Dashboard

• TimeLine 

 

• CPU usage

 

• Memory usage

 

• Disk reads

 

• Disk writes

 

• Network RX

 

• Network TX

 

• Table

• Facets

 

• Search

 

• Hits

 

---

Example 3: Drill Down on data and multiple filtering on the Resource Monitoring Dashboard

1. Go on the dashboard https://www.snap4city.org
   • Menu: Management --> Back Office Res. Analyzer: ResDash
      
2. drill down on time
   • Select from the TIMEPICKER (1) a time range with the mouse to drill down on time, the new filter on time is added on the FILTERING panel
   • all the other widgets start recalculating to show the resulting data filtered. The drill down on timeline can be performed (by day, by week, by month, by year, from date/time to date/time, from a given time instant to another) by using Resource Monitor Dashboard.  Once the time selection (filtering) is performed, the other views of the Dashboard are aligned: the counting of selected object is updated, the faceted list of population selected is updated, the map is also updated.
3. In the widget HISTOGRAM (6) with the left button of the mouse clicked, select a period in the timeline. The selected period is highlighted in the histogram as shown in the following figure.
    
   
    
4. When the left button of the mouse is released, all the widget in the dashboard are updated accordingly, and a new search term is added in the FILTERING widget, as shown in the following image.
    
   
    
5. Click on a field “group” in the FACET widget and click on “IOT-NodeRED”. You can see the other widgets that is updated accordingly, and the FILTERING widget is updated with a new search term, as in the following image.
   
    
   
    
6. Click on some other widget/panel to create other filters that are added to FILTERING panel, and the other panels change accordingly.
    
   
    
7. Insert some text on QUERY panel, thus the filtering is performed on the basis of the text. Type a text in the QUERY widget on the top right in the form: “facet field:name”, for example “ip: 192.168.1.20”. 
8. Go on the FILTERING panel and click on the up-right-corner of the filtering you like to remove. 
9. Go on the top menu if you like to save the Dashboard on cloud or locally on the computer. If you save it locally, it can be reloaded from the local disk when you like; while if you save on cloud, your or other colleagues can use the dashboard you developed.

 

---

Example 4: Creating new views on Resource Monitoring Dashboard data

1. Access to an empty dashboard by clicking the following link:
   • https://www.snap4city.org  

• Menu: Management --> Back Office Res. Analyzer: ResDash

• Click on the sheet of paper icon on the top right to open the "New" dialog (1).

 

• Select “Time-series dashboard” 

 

2. In the Collection Name field write: “vm” and in the Time Field write: “date” and click the Create green button.
   
    
3. The dashboard could be empty since in the TIME WINDOW widget on the top left the value of 15m is set (1). No data could be available in this period of time. Click the 90d (2) and the other widgets is populated with some data referred to the last 90 days.

 

4. You can see the following result.

 

5. Open the editing cog on the top right corner of the HISTOGRAM widget and change the span from 12 to  8. This will reduce the width of the HISTOGRAM widget.

 

6. Click the small PLUS icon on the right of the HISTOGRAM widget (called panel in banana) to fill the available space with a new widget. We will add a FACET panel.

• In the Select Panel Type drop down menu ((1) in the image below) select FACET. New field is shown below.
• In the Add Facet Field write the FACET terms to be added in the list. Press the “+” button to add the term. In the example below the terms added are: “name”, “group”, “ip” and “host”.
• Click the Add Panel green button.

 

7. Close the Row Settings pop up - You have now added a FACET panel.
   
    
    
8. Scroll to the end of the page and click the ADD A ROW button.
   
    

• In the Title field write Map and set the Height to 350px. Click the Create Row green button to confirm.

You can see that a new line with the title Map has been added as last row in the list. Click the Move up arrow to move the Map line above the Table line and click the Close red button.

 

9. Scroll up on the page to the empty new row that is above the TABLE PANEL and click on the Add panel to empty row button.
    
   
    

• In the Add Panel Tab, open the Select Panel Type drop down menu and select Histogram and add the following settings (see image below):
  • Title: Memory
  • Span: 12
  • Mode: Values
  • Value Field: memory_active
  • Group By Field: group
  • Confirm by clicking the Add Panel green button.
• Click the red Close button. A chart has been added.

 

10. You can save the dashboard by clicking the Save button  on the top right menu of the page.

• In the same menu you can save locally and share the dashboard by selecting the Export to file item. 

11. To save the dashboard click the Save button  on the top right menu of the page and select “Export to File”.
    • A new window is shown asking to save the dashboard as text/json file locally in your PC. After saving the file, it can be shared and sent to anyone.
       
      
       
12. To load a dashboard, click the Load button  on the top right menu of the page.
    • You can: Load a local file from your PC (for example the file saved before);
    • Select one of the listed Saved Dashboard.
      
       
13. A Dashboard can be edited in terms off:
    • Add new rows, by clicking the ADD A ROW button at the end of the page;
       
      
    • Add new panels in an available space in a row, by clicking the Add Panel Tab in an empty row (1) or by clicking the small PLUS icon on the top right of an available space in a row (2);
       
      
       
      
    • Drag and drop panels to change the position, by clicking the panel name on the right top corner and moving the panel to change its position;
       
      
    • Edit existing panel to change some parameter and setting, by clicking the cog icon on the top right corner of a panel;
       
      
       
    • Delete a panel by clicking the X button on the top right corner of the panel to be deleted.