1. Home
  2. Smart City Control Room Dashboards: Big Data Infrastructure, from data to decision support

Smart City Control Room Dashboards: Big Data Infrastructure, from data to decision support

×

Warning message

  • You can't delete this newsletter because it has not been sent to all its subscribers.
  • You can't delete this newsletter because it has not been sent to all its subscribers.
  • You can't delete this newsletter because it has not been sent to all its subscribers.

For an update see training https://www.snap4city.org/drupal/node/944

 

Abstract: Smart City Control Rooms are mainly focused on Dashboards which are in turn created by using the so-called Dashboard Builders tools or generated custom. For a city the production of Dashboards is not something that is performed once forever, and it is a continuous working task for improving city monitoring, to follow extraordinary events and/or activities, to monitor critical conditions and cases. Thus, relevant complexities are due to the data aggregation architecture and to the identification of modalities to present data and their identification, prediction, etc., to arrive at producing high level representations that can be used by decision makers. In this paper, the architecture of a Dashboard Builder for creating Smart City Control Rooms is presented. As a validation and test, it has been adopted for generating the dashboards in Florence city and other cities in Tuscany area. The solution proposed has been developed in the context of REPLICATE H2020 European Commission Flagship project on Smart City and Communities.

 

Keywords: smart city dashboard, decision support system, widget, control room.

 

  1. Introduction

    In the development of a Smart City there is a great emphasis to the set-up of the so-called Smart City Control Room, SCCR. A SCCR is an area in which all the data are collected, aggregated and where high-level data/results are summarized and made accessible for the decision makers and shared to the city operators. In large metropolitan cities, the SCCR includes large panels/monitors (even covering large walls) in which the status of the city is reported in real-time presenting the view of the city with some synthesis, predictions, alert of data regarding: mobility, energy, social activities, environment, weather, public transportation, people flow, health, water, security, ICT, governmental, first aid, civil protection, police (118/112/911), fire brigade, hospital triage, and thus almost all the city resources expressed via KPI (Key Performance Indicator). Most of the KPI are representative of the status of resources deployed in the city and may be not geo-localized. Some of the city monitored resources are representative of the critical infrastructures for the city functionality and for the life of city users such as: transportation, energy, security, health, water, civil protection, ICT, etc. In medium sized cities, the daily management of city resources is performed by a set of city operators, which could be legally independent with respect to the central municipality. Thus, they autonomously manage their control rooms, accessing and rendering their own data to take their own decisions which may be limited in scope, and according to some defined protocols and strategies, [1], [2]. For example, when the energy network has a problem in an area of the city the energy is rerouted to reach the all possible subareas via a different path; when the water service network has a problem on major distribution tubes, the water is provided in other means (for example using thanks); in presence of traffic congestion the red-light timing is acted to facilitate the flow and bus paths are changed/rescheduled according to planned schemas.

Once identified and understood the needs of having an integrated SCCR, it is necessary: (i) to choose what must be shown on the panels on walls; (ii) to choose what must be shown on computers of the operators in the rooms and remotely connected; (iii) to understand how the data have to be collected and computed (in the case of prediction and early warning). The ingestion, aggregation and data analytics processes are very complex to be managed since the information is heterogeneous (different format, providers, protocols, etc.) and the total amount of data is a Big Data problem, moreover, the final indicators, provisions and suggestions calculated by these complex processes must be easily understood by the observers of the panels. It is a problem of data representation which also must consider the competence of the final users: citizens /observers/operators/experts/decision makers/etc. In most cases, the final users have to be trained to understand the data and graphics representations. They must become confident on what they see to understand in deep all the single details represented on the screen. They are not going to have time to learn when a critical event happens. Some data or events are easy to be represented and understood in a dashboard, for example: a traffic representation observing the city map with red, orange, yellow, and green segments on the streets; a sensors measuring some values like temperature, humidity percentage, etc.; while it is more difficult to comprehend other, more complex, kinds of data/results/provisions, for example:  tables of pollution and pollination; traffic flow trends by numbers, etc. To make it easy to read even the most complex metrics, it is possible to use: alarm signals in red, blinking signal, etc. [2].

     From the technical point of view, the tools for rendering information on SCCR are typically called Dashboards and are supported by Big Data aggregation tools [4]. The Dashboards should be capable to present real time data in several different manners with real time updates on screen autonomously H24 7/7 days, according to the refresh time of each data source, and have to be interactive to allow the users to make drill down activities on data to better understand the situation and the context. Dashboards for control rooms should not be confused with business intelligence tools that produce graphics from the combination of data extracted from some sources (database, files, API, etc.). In most cases, business intelligent tools may access to data with faceted indexing and search, for example in SOLR or ElasticSearch, [24], [25] . Those kinds of Dashboards are focused on single view of data, filtering and drilling down on data, rather that representing the city KPI and status. Examples of tools for drilling down on time and facet can be obtained by using Apache Banana ([26]), or HUE ([27]).

    Moreover, the concept of Dashboard for SCCR is also often confused with the data aggregator tool that is a fundamental tool for the Control Room and city control in general and can be regarded as the back office of the Control Room. Many solutions for control rooms and their back-offices has been proposed such as those of IBM [5] on services for citizens, business, transport, communication, water and energy; [6] on governmental, educational, e-health, safety, energy, transport and utilities; etc. Most of these solutions present a multi-tier architecture ranging from 3 to 6 layers [7], [8], [9].

    In this document, the Dashboard solution developed in the context of REPLICATE research and development projects of the European Commission is presented. Replicate is an SCC1 project of the European Commission on H2020 ([10]). The solution proposed is been based on Km4City Smart City Ontology ([11]) and data aggregator [4], [10], [22]. Please note that the Dashboard Management System of DISIT Lab is released as Affero GPL Open Source on GitHub, see DISIT lab page. The present solution is managing more than 1.2 million of complex events/data per day.

    The paper is structured as follows. In Section 2, the requirements of smart city control room are discussed. Section 3 presents the adopted smart city architecture. In Section 4, the dashboard system for the smart city control room is presented with its architecture.  Section 5 reports a set of experimental results and lesson learnt. Conclusions are drawn in Section 6.

 

2. Smart City Control Room Requirements

    In this section, the main requirements of Dashboard systems for SCCRs are summarized. They have been collected during the above-mentioned research project by interviewing a number of operators and decisions makers belonging to several cities and nationalities.

    A SCCR dashboard system is substantially a Decision Support System tool, DSS, since it provides evidence of critical conditions, and may offer solutions. On this regard, it may integrate/exploit artificial intelligence algorithms, for example, reporting prediction, identifying anomalies, manifesting early warning, providing relationships among entities exploiting inference geospatial reasoning about what is located in the city: resources, structure, people, areas, critical infrastructures, etc. [10], [17], [19], [20].

According to our analysis, a Dashboard system for smart city must be capable to:

  • Show dashboard on web browser in a H24/7 modality;
  • show data on widgets according to several graphic paradigms (tables, graphs, histograms, maps, Kiviat, lists, tv camera, heatmaps, weather, critical city events, etc.) with a level of interactivity and animation;
  • show data on autonomous and connected/ synchronized widgets;
  • collect, show and keep update on screen data with automated refresh for each view, and real time data according to the even driven paradigm;
  • show data both  real time and historical, allowing the drill down on time, space and relationships among data and city entities;
  • collect and show data coming from different big data and classic data sources (SQL, NoSQL, RDF, P2P, API, SOLR, etc.) also in aggregated manner;
  • compose the Dashboards as a set of graphic and integrated widgets that can be separately set up assigning several parameters: data source, size, colors, shape, etc.;
  • work with large amount of data providing high performances, as short response time;
  • compute alarms, and provide support by a flexible notification system capable to send alerts, activate tickets for maintenance, automate actuators, post on social, etc.;
  • provide actuators widgets together with showing graphs, and capable to act on IOT Devices;
  • provide support for collaborative production of dashboards and for co-working;
  • provide support for embedding dashboards into third party web pages;
  • provide data engine for collecting connection response time on different protocols, and for verifying the consistency of web pages via HTTP;
  • integrating with IOT Applications by managing real time data and connecting its actuators to real time IOT applications;
  • integrate dashboards in more complex dashboard systems;
  • support authentication and authorization with the most general approaches such as LDAP, Kerberos, etc.;
  • collect and get data from batch resources and in real time, using a large range of protocols and formats.

 

This means that each Dashboard should be composed by several configurable Widgets, each of them can collect data coming from several data sources. On each data stream, one or more criteria as firing conditions should be set up for the notification of alerts, intervention, etc.

     In small cities, with about 100.000 inhabitants the number of relevant data sources to be integrated by the data aggregator and represented in Dashboard can be in order of 20-30 while in larger cities they can rapidly grow for the presence of multiple operators for each utility. Thus, the complexity is also greater as the number of stakeholder actors involved in.

    Therefore, before starting with the development of the proposed Dashboard solution, several state-of-the-art solutions and proposals have been analyzed. As nonfunctional requirements, the Dashboard system must be scalable, interoperable with several tools, open source, robust, usable, secure in protecting data views according to the GDPR [17] and flexible.

A set of solutions, both commercial and open-source, have been analyzed to identify a functional platform to be adopted. Most of the solutions which are present at the state of the art derive from business intelligence solutions (e.g., SpagoBI, Tableux, OpenDataSoft, etc. [13], [14], [16]), in which the tools provide some data mart (data virtualization) tool to access data sources and thus have powerful tools in this sense. On the contrary, they provide limited capabilities and tools on rendering and dashboard for control rooms that must stay H24/7, rendering specific kind of structured data. For these reasons, several specific custom solutions have been proposed by many cities such as: London (https://data.london.gov.uk/), Amsterdam (http://citydashboard.waag.org/), Dublin, etc. On the bases of the analysis made, regarding the solutions available on the market, none of them satisfied all the requested functionalities and functional aspects, above described, and this is the reason why we decided to start the development of our solution, ([14]).

Figure 1: Km4City Architecture for Smart City.

3. Smart City Architecture

     This section presents the overview of the Smart City architecture which is presently in place in the Tuscany area. With the aim of producing a smart city infrastructure for stimulating sustainable mobility, smart energy, and smart utility in the city, a data aggregator has been developed (see Figure 1). It presents a front-end layer for the City Dashboard and control room, Smart City API for web and mobile App, decision support tools, personal assistants via mobile App, participative portals, crowd sourcing, etc. The data aggregation infrastructure also supports Data Analytics and Data Intelligence based on integrated data collected from public administrations open data, private data from operators, and personal data coming from social media and city users. In this paper, the architecture enabling the construction of the Control Room in terms of Dashboards and Data Aggregator is described.

    In the Km4City layer, City Operators and Data Brokers provide data which are collected by using streams, data driven and/or ETL processes which are scheduled on the Big Data processing back office based on DISCES (Distributed Smart City Engine Scheduler) tool. Among the data collected those provided in Open Data from the municipalities, Tuscany region (Observatory of mobility), LAMMA weather agency, ARPAT environmental agency, etc., and several private data coming from City/Regionals Operators: mobility, energy, health, cultural heritage, services, tourism, wine and food services, education, wellness, environment, civil protection, weather forecast, etc. Once the data are collected, the back office activates several processes for improving data quality, reconciliating data and converting data into triples for the RDF store of the Knowledge Base (when needed) [10], [4], implemented by using a Virtuoso triple store. DISCES is allocating processes on several virtual machines allocated on the cloud according to their schedule.

     For semantic aggregation of data and service, it has been decided to exploit and improve the Km4City Ontology ([10])  as the main ontological model. Km4City is modeling multiple domain aspects related to mobility, services, Wi-Fi, cultural services, energy, structure (streets, civic numbers, public structures, geographic locations, green areas), sensors, busses, smart sensors, parking, city services, transportation, events, pharmacies, hospital and real time data of first aid, environment (with pollution and pollination, weather forecast, weather conditions), and private mobility with fuel prices.

     In the smart city architecture, in addition to the RDF store for the knowledge base, a number of noSQL Stores (namely: HBase and MongoDB, [28], [29]) are adopted for storing tabular data as those arriving from sensors and user profiles, respectively.

 

  1. Dashboard System Architecture

In Figure 2, the general architecture of the Dashboard solution is presented. The main components of the architecture are described in the following.

Figure 2: Architecture for Smart City Dashboard Management System integrated with IOT.

Data Aggregator is a set of tools for collecting data from the field and from external services and reconciling them to the same city entities. To this end in the proposed architecture the Km4City Aggregation has been adopted to have all data fit into the Km4City Ontological model Thus. are provided from the Aggregator with Smart City API (rest CALL), SPARQL, SOLR and/or SQL queries.

 

Data Collector is a multi-process engine (also called Dashboard Engine) for acquiring data from multiple data sources: SQL, noSQL, RDF/SPARQL, API, SOLR, etc. ([25],[28],[29],[30],[31]), by using multiple protocols: HTTP, HTTPS, ODBC, WebSocket, etc. The Data Collector needs to have a configuration for each acquisition process, which produce a result, also named Metric or Measure. Some of these Metrics may be saved into a local data base for historical reason. To finalize the queries to be performed for collecting Metrics, specific tools may be used for Data Mart such as database browsers and drilling down into Data Sources. The collected data can be (i) saved into a data base of Metric Historical Values or can be (ii) directly accessed from Widget/Dashboard for their visualization. The Data Collector, with its processes to acquire Metrics, is also capable to perform the real time assessment of firing conditions.

  • Firing Assessment allows to compute the firing conditions on all the data streams reaching the dashboard system. In the case in which, a Firing Condition becomes true, a message event is sent to the Notificator service. Firing Conditions can be estimated:
    • on Metric Historical Values taken in Push from the data base of the builder;
    • on the streams directly arriving from the Data Collector;
    • from the data stream arriving directly from outside,
    • From the external tools embedded as IFRAME into widget if there is some integration.
  • Dashboard Builder is the core tool for creating dashboards by using a graphic user interface. In the tool the user can set up Data Sources: IP address, protocol, user name and password for accessing at each specific the data source. Once the data sources are identified several Metrics can be defined. Then new Dashboards can be created taking interested Metrics and associating them with one or more Widget. A Widget may render/exploit the same Metric by means of different graphic models. For example, a temperature read every 5 minutes, can be visualized as current value on a thermometer, as temperature trend in a graphic along the last 24 hours, last week, last month, etc. Therefore, the composition of a Dashboard consists of placing and configuring a set of Widgets into the Dashboard frame. The Dashboards are created by using the visual interface of the Dashboard Builder.
  • Integration with IOT. This feature has been covered by (i) producing special MicroServices as blocks that can be used into IOT applications developed in NodeRED, (ii) connecting NodeRED applications with several IOT brokers, ([32]). Point (i) implied the development of a layer that allowed the traditional Dashboard Widget to be directly connected to (a) IOT Applications by using WebSockets, (b) IOT Brokers (for example, via NGSI, MQTT, etc.). To this end, one of the most suitable IOT Brokers resulted to be the Fi-Ware Orion Broker.

Dashboards are typically adopted for reporting KPI of the city and thus of specific infrastructures and services. This means that specific alerts and notifications have to be activated and managed at level of single Metric. On the other hand, the same Metric can be used on different dashboards and widgets for different purposes. So that, for each Widget of each Dashboard specific alerts and firing conditions can be set up. For example, when Metric M is adopted in Widget W of Dashboard D, the certain criterion C is saved and computed for firing (M, W, D, C). One or more Criteria can be defined (M, W, D, C1 … Cc), each of them may produce multiple Notifications, N: (M, W, D, C1 (N1, …, Nn), …, Cc (…)).

Therefore, the solution has been to design and develop a Notifier to

  • Accept registrations of possible tuples <m,w,d,c>, to enable the reception of Notification Requests, that are used to send Notifications according to different approaches;
  • Accept registration by third-party tools, in addition to those of the Dashboard Builder, to send alarms about the firing of the registered conditions;
  • Produce emails and REST calls, that can be used for calling SMS, as well as for the activation of maintenance ticketing system on OpenMaint tool for example ([27]);
  • Log all the registrations and Notification Requests for further analysis and security evidence;
  • Maintain and use a list of Notification recipients, that are the users which are going to receive the notifications. This list of uses is just listed as: name, surname, email, telephone (if any), role, organization.

To this end we suggest using specifically development tools, Such as the ServiceMap ([34]) which is used for knowledge base browsing over the city data as Km4City knowledge base, which is RDF store as well, exploiting geospatial reasoning and inference [Bellini et al., 2014]. In addition, the technical browsing on the RDF Graph Store may be needed to discover relationships. To this end, the LOG ([35]) tool for browsing into any RDF store by using SPARQL and visual interface has been developed in the past and now used in this context. This tool allows to browse all the RDF stores accessible in the world which provide a public end point for SPARQL queries, from dbPedia, to Europeana, Geonames, Km4City, Camera, Senato, Getty, etc. [23].

As a conclusion, the generated Dashboards are Dashboard Instances which are available for view and activate corresponding widgets according to their Settings. The saving of data into the database of Metric Historical Values, allows keeping track of what has been visualized/monitored and thus enabling the replay of data logged. On the other hand, it is also possible to adopt widgets that: (i) directly show/provide the data from in/out streams, respectively (for example, Civil protection alert status, etc.); (ii) directly render/visualize web page segments into the Dashboard (for example for showing social media analytics, traffic flow reconstruction tools).

5. Experimental Results and Validation

The solution proposed in this paper has been developed in REPLICATE flagship research and development project SCC1 H2020 of European Commission for Florence City. It has been also used in other large projects such as Sii-Mobility Smart City Nazionale on Mobility and Transport of MIUR, RESOLUTE H2020 project for critical infrastructure and resilience of transport systems, and GHOST MIUR for Cagliari smart city experimentation. The proposed Dashboard solution is strictly connected with a number of tools of the Sii-Mobility/Km4City suite of tools which are used to perform smart city analytics, semantic browsing, and decision support, etc., such as: ServiceMap ([34]), smartDS, system thinking decision support based on Bayesian models (http://smartds.disit.org [21]), Wi-Fi monitoring and predictions, smart parking prediction, traffic flow reconstruction and prediction, energy metering, first aid monitoring, environmental monitoring, social media monitoring and alerting, weather forecast, etc.; most of them based on clustering, machine learning, etc. [4].

In general, the decision makers in the city are politicians, assessors, and director of units. Some of the units have already adopted a high level of technology, for example, the mobility and transport, the civil protection, etc. In other units, the level of control is low since the technical activity is mainly delegated to City Operators such as: energy operators, water management, health care hospitals, environmental agency, waste management, police and alert (112, 118, 911), etc. All of them have their own monitoring system, that is tuned to vertical control their own information and status. In some cases, the general information about weather forecasts and status is shared among the different operators. The dashboard can organize data according to different views/paradigms: horizontal view (synthetic view reporting many different aspects of the city status) and vertical (or thematic).