MicroServices Suite for Smart City Applications

 

Keywords: Smart City, IOT Applications, MicroServices, Node-RED, Dashboard.  

 

1.Introduction

 

The concept of Smart City is becoming increasingly important and pervasive, making it a major area for research, business and policy makers [Gupta, 2019]. Smart Cities are currently required to provide flexible architectures, in order to satisfy new functional and non-functional requirements that arise and constantly evolve in many different contexts and environments. There is also the necessity of scaling up to handle the increasingly number of users, devices, and services. However, the current common approach is still to develop ad-hoc applications for specific domains, which leads to the production of non-interoperable services [Esposte, 2017]. Smart Cities are becoming every day more focused on activities related to IOT and IOE (Internet of Things/Internet of Everything). This has provoked a switch of technology and approach in field data collection. In the past, most of the city generated data  have been collected from domain specific vertical applications, by sending data to dedicated concentrators (for example, the collection of data from traffic lights, traffic flow sensors, parking, etc.). Each and every vertical application was using its own solution for data collection either via radio, or any other means. On the other hand, the microservice architecture paradigm has arisen from wide-scale industries requirements of building and maintaining large-scale distributed systems [Krylovskiy 2015], supporting an adequate availability, scalability, modularity, as well as flexibility [Fernandez 2019]. The new push of IOT is stimulating cities to adopt the same gateways to collect data of multiple services. This means that the infrastructure of the sensor network has to be properly designed and planned by the city operator. LoraWan can be an option [Gomez 2012], while in many cities 5G solutions are coming and would replace former technologies in short time, at least within the large cities [Li 2018].

    However, the arrival of a more aggregated IOT streams via 5G is not a problem for smart city data aggregators serving the Smart City Control Room [Bellini 2018]. Presently, they collect data from several heterogeneous sources, and to have a reduction in the number of data concentrators to be aggregated would be a benefit not a problem. On the other hand, the IOT/IOE paradigm is also strongly impacting on the infrastructure management [Bonomi 1012], [Li2 2018], since the systematic adoption of IOT has also led to the adoption of PUSH approach (event driven protocols) and it has brought forth the possibility of acting on actuators and not only sensing the city, not to mention the possibility of creating Event Driven Applications. They are also called IOT Applications which have to be capable to process messages and produce reactions in real time [Su 2011]. For many aspects Smart City application are one of the most complex cyber physical systems, CPS, due to their complexity in terms of data, data analytics, and interfaces with the real world, physical and digital on the user interface.

     In the state of the art, several different definitions and architectures have been proposed for IOT frameworks [Sethi 2017]. Most of the IOT platforms adopt a code-based development for the design of IOT Apps. Some of them provide development tools as: Eclipse, or JavaScript. Others provide simple tools as IFTTT [IFTTT], relying on rule-based scripts [Badii, 2013]. Other platforms adopt solutions similar to ETL (Extract Transform Load) processes [ETL Penthao] capable of working for event driven processes, or just surrogating it with high rate polling. One of the most promising tools for creating IOT App is Node-RED [NodeRED], based on Node.JS, which has been proposed by JS Foundation  as fully open source. The Node-RED approach is a mix of visual composition of nodes/blocks to compose the so-called flows that are concurrently executed by a Node.JS engine. It is quite diffused, being also directly provided into official releases of IOT devices as the Raspberry Pi family. It is provided with a minimum set of functionalities (the building blocks/nodes), while other blocks can be easily added loading them from a large library made available by the JS Foundation. The classical nodes provided in the standard version can be classified as: input, output, function, social, storage, analysis, advanced, and dashboard; such nodes are not sufficient for creating Smart City IOT Applications. Actually, for the use in the context of Smart City it was not powerful enough to cope with the basic requirements of such domain, even though its use in the field is quite widespread. In the next section, many other solutions have been considered and compared, in order to select the starting point of our research and innovation action, i.e. the Snap4City platform funded by European Commission, as described below.

 

1.1 Related Work

Several Smart City use cases have been presented so far in literature, transforming the collection of huge quantities of data related to urban environments into useful content for various stakeholders (citizens, tourists, local government and companies) [Lim 2018]. Among them, some of the most notable use cases concerns the cities of Santander [Díaz-Díaz 2017], Manchester, Barcelona [Agbali 2019], Singapore [Cavada 2019], Seoul, San Francisco [Lee 2014], Rio de Janeiro [Kitchin 2014] and Florence. Most presented solutions however still focus on specific domains, aiming at resolving particular problems, with little software reuse [Santana 2017].

        In order to manage the high variety of devices and applications, including IOT, mobile, web and service-oriented framework, the Microservice architecture is being increasingly adopted in recent IOT based solutions for Smart Cities. Actually, Microservices architecture focus on the development of simple and loosely coupled services [Zimmermann 2017], enhancing scalability and availability, facilitating maintenance and fast isolated testing. These aspects aim at simplifying the complexity of traditional service-oriented architecture (SOA) [Dragoni 2017].

    The programming of IOT Applications is performed in several different manners by different tools [Yasumoto, 2016]. For example, in AWS IoT ecosystem a complex stack of Amazon’s services is proposed, which enables developers to define any business logic by programming the flow of IoT Devices’ data towards their visualization in proprietary dashboard. The IoT Devices can activate AWS functions written in: Java, Python and C#. These functions can implement business logic or trigger actions in the IOT world or Cloud. Data rendering is flexible and can be used for data inspection and reporting. Azure IoT provides a development environment to build business logics. The integration with mobile applications is performed by using the Xamarin platform in C# and .NET. In addition, it is possible to write application also in Java, Node.js and Python. For the creation of dashboards, developers can use MS Business intelligence tool. In Google IoT platform, several programming languages can be exploited to program the data flows from device to dashboards, like Java, Node.js, Python, Go, Ruby, PHP, C#. In such cases, data flows are implemented by using programming languages. Not any of them provide a visual editor for IoT Application flow definition to be used even without any software programming skills. In addition, these solutions often require the mandatory use of cloud services, and they are not suitable for any autonomous deploy on-premise scenario. As a limit case, they may permit to have some modules configured on premise IOT Edge with reduced capabilities connected with the cloud solution.

       In [Ray, 2017], a survey on Visual Programming Languages (VPL) for IOT has been proposed. In the survey 13 different VPLs have been considered. The analysis has been mainly focused on comparing them on the basis of the programming environment, licensing, project repository, and supported platforms. Some of them are Open Source platforms, while others are proprietary. Among the Open Source platforms (Node-RED, NETLab, Ardublock, Scratch, Modkit, miniBloq, NooDL), only some of them can be programmed using a Web interface and can be also executed on some dockers or virtual machines on cloud. In [Contreras Masse, 2019],  the usage of haptic interfaces in the context of IOT Applications for smart manufacturing has been reviewed. The analysis has produced a classification and the identification of the main elements needed to the exploitation of haptic devices without entering the problems of the microservices and VPL for programming IOT Applications.  

In the assessment of Open Source VPL for IOT, several important aspects should be taken into account, such as:

• performance in terms of number of messages processed per second, per minute;
• diffusion of the tool, namely the size of the communities supporting the tool, both as developers and users developing applications and additional features;
• openness in terms of flexibility to create new blocks to extend the basic functionalities, for example, exploiting external services, developing data analytics, etc. For example, in the domain of smart city or industry 4.0, you could add visual modules representing the IOT Device elements, as well as Data Analytics;
• Cloud/docker support for their use on cloud and/or support for any execution on embedded IOT, thus executability on several operating systems. Some of these VPLs produce code only for specific embedded devices. On cloud the automatic resource provisioning and the elastic scaling of resources are needed to manage large amount of IOT applications;
• level of expressivity: an IOT VPL can be oriented on defining functional or data driven/flow aspects. The first are typically less expressive, since such constructs are related to the programming language: section, assignment, condition, iteration, etc. Data Flow models are more expressive and may be easily extended with new blocks/functions and this lack of expressivity may be solved if the right blocks are identified. Therefore, this can involve as a result to produce simple data flows even for complex problems. In fact, the complexity of the graphs produced for solving problems in the applicative domain can be measured in terms of number of blocks/modules and connections used to solve such problems;
• usability, including visual editing of data flow via web browser or through specific client tools, etc. The fundamental features, such as: inputs, outputs, functions definition, social gathering, storage load/save, data analytic and dashboard (user interface), may be not enough for creating complex Smart City IOT Applications with low effort. On the other hand, one may have a well-designed VPL, and the effort may be vanished due to the lack of expressive and domain suitable MicroServices;
• managing resources: the possibility of developing IOT applications (as well as portions of them) as data analytics and share them in a marketplace. In the context of VPL for IOT, the development of Data Analytics is quite complex, since processes may have a lot of dependencies and the programmers’ skill may imply the knowledge of the data storage and models, in addition to the knowledge of algorithms for data analysing. This approach may require the involvement of experts of sensors, algorithms, infrastructure, programming, knowledge modelling, domain etc.

 

Some solutions that can be suitably used to address such aspects and overcome the problems described above are, for example: Node-RED (it can be executed on multiple operating systems, it has a visual programming via Web interface and the source code can be loaded on several different embedded devices, etc., [NodeRED]), NETLab is an open source development environment for embedded system [NETLAB]; Ardublock is an Open source block programming for Arduino, mainly functional approach and limited to specific devices [ARDUBLOCK]; Scratch is a visual tool of MIT, adopted for code generation for IOT, very low level programming model, not event driven, functional; Modkit is strongly oriented to Embedded IOT devices; miniBloq is a block functional programming at  very low level; NooDL allows the fast prototyping of the user interface and it is less suitable for data analytics; VISUINO is an open source tool to  develop code for embedded IOT, but providing limited capabilities to be used on cloud; Kura is an Open Source Java visual tool for IOT; FLOGO is an Open Source tool for data flow in Go, natively works on AWS Lambda, [FLOGO] or Wyliodrin a development environment for IOT edge embedded devices, deploy and update.

    Most of the above solutions are strongly focused on producing code for embedded devices with functional model and not data driven/flow, such as: Scratch, Modkit, miniBloq, etc. Therefore, they are not suitable for creating IOT Applications that can be put in execution on IOT Edge, as well as on cloud docker containers, addressing the complexity of Data Driven applications and flows; exploiting Data Analytic. Solutions such as AWS, IOT Azure, and IOT Google are more oriented on traditional programming. The VPL tools based on data driven, open to extensions, supported by large communities, that can be put in execution on both edges, could be only a few like: Node-RED and maybe the emerging FLOGO. In [Roukounaki, 2015], Node-RED, flow editor has been used for creating smart city IOT applications for projects such as VITAL. The suite of nodes implemented is not focused on functional aspects of IOT Applications, so that the resulting flows are still too much focused on technical aspects and not user centred. In fact, as depicted in [Roukounaki, 2015], even the creation of simple activities as the identification of traffic flow sensors results into the development of complex workflows with a high number of nodes/blocks. Specialized workflow formalization models have been also proposed, such as in [Chattopadhyay, 2015]. In [Dey, 2015], a Model Driven Development (MDD) process has been proposed consisting of (i) a tool for creating semantic algorithms, and (ii) a workflow generator on the basis of the first results. The user interface is a subset of Node-RED environment.

       There is a few IOT development environments where algorithm repositories are provided for their exchange in the community, such as in [Pal, 2014], [Aldasoro, 2011]. This feature should be mandatory in cloud based IOT development environments such as: Axeda [AXEDA], BlueMix [IBM], ThingWorx [THINGWOX]. On this last aspect, there are two levels in IOT valorisation environments, such as:

• IOT platforms marketplaces, such as BeeSmart City, Fi-WARE, EOSC, etc., which are typically constrained to a class of solutions or open to all. Their target is the promotion and marketing of IOT or Cloud solutions.
• IOT Applications and solutions resources sharing and marketplaces, such as: Node-RED library, which are typically focussed on specific IOT kind of development processes, and on promoting the interchange.

 

In this paper, the issues related to Snap4City IOT development environment and framework are addressed. The experience described herein refers to the design and implementation of Snap4City platform, https://www.snap4city.org [Badii 2018], which is based on Km4City [Badii 2017]. Snap4City is the solution produced in response to a research challenge launched by Select4Cities PCP (Pre Commercial Procurement) H2020 research and development project of the European Commission (https://www.select4cities.eu ). Select4Cities has identified a large number of functional (mainly Smart City IoT) and non-functional requirements (open source, scalability, security, GDPR compliance, working on cloud and on premise, etc.) which are fully described in their web site and aimed at creating the best solution for the modern Smart Cities supporting IoT/IoE, in hands of public administrations and Living Labs. Most of the identified requirements have been taken from the large association of Living Lab ENOLL (European Network of Living Lab association, https://enoll.org), and upon consultation of smart cities at level of European Commission as EIP-SCC (European Innovation Partnership on Smart Cities and Communities, https://eu-smartcities.eu).

Snap4City has responded to the research challenge and could prove to satisfy all the Select4Cities requirements. Snap4City allows the creation and management of users communities, which collaboratively: (i) may create IOT Solutions, (ii) exploit open and private data with IOT/IOE respecting GDPR, and (iii) create/use processes and IoT Applications that could run on Edge, Mobile and cloud, with the capability of interacting one another and with users via messages, Dashboards and Applications. A specific attention has been given to the creation of an integrated development environment for IOT App (cloud and edge), based on VPL, with dashboards, and supporting data analytics and resource sharing, thus minimizing any required technical expertise as programmers. To this end, a large number of visual elements have been identified and developed as MicroServices for Smart City IOT. The starting point has been Node-RED, of which we could identify many breaches to cover the Smart City domain. In addition, the developed platform has a full support to give user assistance during the development life cycle, from data gathering to dashboard production, as well as in IOT App development and sharing of results with other developers and in the community as self-assessment, together with any security support as described in [Badii 2019].

It should be remarked that this paper is presenting aspects that have not been presented in our past articles in the past, and in particular: the suite of MicroServices, their motivation and requirements for smart city IOT solutions, a number of  integrated examples, the Life Cycle of development, and the validation of the approach.

1.2 Structure of the paper

   This paper is structured as follows. In Section 2, requirements for a microservice based programming solution for smart city applications are analysed and reported. Section 3 presents the Snap4City Architecture with a stress on data flow, IOT and MicroService aspects. In Section 4, the Snap4City MicroServices Library is presented and motivated. Section 5 introduces some applications exploiting the MicroServices in the context of emergency management, personal mobility, and crowd of people monitoring with mobile PAX Counters. In section 6, the Snap4City development life cycle has been formalized. Section 7 presents a validation of the solution performed with city officials in developing IOT Applications. Conclusions are reported in Section 8. 

 

2. Requirements for MicroService based Smart City Applications

As above described, the aim of the research reported in this paper has been to design and implement a visual programming environment where city operators may develop Smart City Applications with IOT by means of visual constructs. In this section, the identified requirements of the challenge are listed.

      As a first step, by analysing the state of the art we realized that a VPL for developing IOT Applications has to provide generic requirements, so as to support:

• data communications sending/receiving messages/ data, providing both push and pull modalities. This means that the language has to be capable to perform requests using many different protocols among the available ones [Dizdarevic 2019], such as pull protocols to get data (e.g.: Rest Call, Web services, FTP, HTTP/HTTPS, etc.) and push protocols to receive data via data driven subscriptions (WS, MQTT, NGSI, COAP, AMQP, etc.). In the context of IOT solution, this feature is quite common. For example, in Node-RED library, you can find a large collection of nodes covering tens of protocols, while large platforms as AWS and IOT Azure are typically limited and address just few protocols. In some cases, such limitations concern the supported security and authentication schemas, since in most cases, username and password (or authentication keys) have to be included directly into the flow in clear text, thus making not all the approaches on IOT security satisfactory [Badii 2019]. For example, in Node-RED each single IOT Device can be connected to a flow using specific credentials directly included in the code. On the other hand, a generic service for IOT Discovering and Registration is missing.
• data save and retrieval to/from some bigdata storage. For example, the capability to store data and exploit them to implement algorithms for predictions,  decision making processes,  value trends overviews, etc.
• data transformation and processing via the algorithm formalization and the possibility to start, pause, recall their execution periodically or sporadically, according to the arrival of some event or other firing conditions.
• calls of external services in order to enable the instrument of delegation to external solutions, when it comes to the computing and processing, for instance the computation of some complex Data Analytic algorithms, or the exploitation of external services such as the Social Media Analysis via REST Call. For this purpose, the classic Node-RED provides access to Watson IBM for machine learning and artificial intelligence reasoning. Desirable applications could be, for example, computing sentiment analysis, performing a clustering classification and recognition, computing some predictions on the basis of historical data of some City IOT values.
• the presentation of data via some Dashboards that would be the user interface of the application. Data visualization should be expressive enough, including values in real time, time trends, histograms, pie charts, etc.
• the use of the Dashboard as a user actuator interface, enabling the user to act: (i) on some internal variables, (ii) to send messages/ commands to IOT Devices and flows.

 

The VPL IOT platforms should present a number of non-functional requirements such as demonstrating capabilities of: robustness (in terms of availability and fault tolerance), scalability (to be capable to serve from small  to very large businesses with corresponding volumes of processes per second), security (authentication, authorization, secure connection, etc.), full privacy respect (compliance with data privacy according to the GDPR, General Data Protection Regulation  European guidelines), interoperability (e.g., communicating with any kind of protocols, devices, services), and openness in terms of open source and in terms of possibility of adding new modules and functionalities, etc. On the other hand, most IOT development environments are not web based development platforms, and are focussed only on generating code for specific embedded as Arduino, see the above mentioned: VISUINO, Ardublock, ModKIT, miniBloq, etc.

 

     In addition, by analysing the requirements identified by Select4Cities, ENOLL, EIP, etc., we realized that a large number of smart city IOT specific requirements have to be satisfied by a VPL to enable the easy development of IOT Applications in the Smart City context, at the service of City Operators and advanced City Users (among citizens, stakeholders, third party developers of SME, researchers on data science, etc.). For example, the VPL should provide specific features to address functionalities to develop applications in domains such as: mobility, environment, parking, culture, health, tourism, energy, and general city services, with no need of knowing technical details about the provided services or service supplier identity. Actually, in almost all cities, different services on mobility, environment, tourism, energy, education, etc., are provided by different operators. Moreover, some of these services are also related one another; for instance, the number of free parking lots and the events in the area, the number of weather forecasts and the number of people in city gardens, the traffic flow and the environmental data regarding NOX, etc. That is, when it is supposed to rain, the number of traffic car increase, and thus the number of free parking slots along certain hours of traffic decreases.

      On such basis, a number of specific functionalities have been identified to develop smart city applications via VPLs, which should be available in terms of MicroServices and building blocks, in order to:

• Access to Smart City data entities on the basis of their functional semantic descriptors. For example, status or prediction of “parking square Carlo IV” and not status of variable “Status” of device “sensor45”, etc.
• Discover and exploit city entities’ data regardless of their technical details, such as their push/pull protocols and gathering model, messages format, provider, etc., while focusing the search on functional aspects and ignoring all the details about the query syntax used in the specific storage (e.g.: SPARQL, SQL, ElasticSearch). For example, to provide answers to the following questions: “let me access to all temperature values in this area, path or close to a given GPS point”; or: “give me the sensors measuring PM10 (or any other environmental variables) close to my position or along a path”, regardless of the actual connection of sensor devices which would be connected to the platform by using different IOT brokers, different protocols, etc.
• Create Graphic User Interface for web and mobiles, with visual representation of complex data in dashboards, allowing also to input data into IOT Applications and the whole system. Node-RED provides a number of basic nodes to create GUI for IOT App (Dashboard nodes), whereas the offered solution presents strong limitations as to its usage in the context of smart city. They are not: presenting maps (with pins, paths, areas, etc.), heatmaps, trajectories, origin destination matrices, comparing trends, Kiviats, multi-pie charts, etc. Furthermore, they are not protected by authentication and secure connection.  In smart city domains, advanced data types need to be managed and shown, for instance: maps, paths on maps, heatmaps, collective trajectories, area shapes on maps, areas, ortho-maps, origin destination matrices, etc. Most smart city data types are related to sequences of GPS points.
• Access to Data Analytics and their production. The real need in the context of Smart City is to grant data analysts with the possibility of creating some data analytic processes (it may be in R, Tensor Flow, Python, Java, etc.) and using them into the data flows as MicroServices, with no need for a programmer to step in, nor for an administrator to take action every time. This approach may create a strong flexibility into the IOT Applications. For example, classical Data Analysis algorithms for smart city could include solutions to:
  
  • Compute the routing from point A to point B according to certain criteria, for example: using car, public transportation, bike or pedestrian; shortest or fastest; avoiding specific zones due to road works or other kinds of emergencies. This implies the production of a sequence of points connected by street segments.
  • Collect and compute metrics on social media data from Twitter (such as: counting Tweets, retweets, as well as extracting Natural Language Processing features [Nesi 2015], i.e. verbs, adjectives, citations, hashtags, and performing sentiment analysis on them). For example, computing metrics related to Tweets collected on the basis of a set of keys, Twitter usernames, hashtags, etc.
• Register new IOT Devices, regardless of their protocol, which are provided via some external or internal IOT Broker. This feature is strongly relevant to the registration of personal devices. For example, glucose meter, security devices for home monitoring, etc. The data collected from those devices have to be kept strictly private, and according to the GDPR guidelines [Badii 2019], [Valtolina 2019].
• Save and retrieve personal data, for example data coming from private devices, as those mentioned in the previous points, and thus according to GDPR.
• Load and publish Data Sets, aiming at ingesting open data and/or creating new open data sets, share them with other cities and networks. A data set is typically a file in some open format containing data description. For example, the position of benches in the city, the position and description of all the restaurants in the city, etc.
• Provide different kinds of events within the city. They can be (i) notifications of consolidated events (such as road accidents, fire brigade and/or police calls), (ii) entertainment and sport events, or (iii) potential emergency calls, not yet consolidated events which may be produced by police cops or citizens in the street just to communicate facts (for example, potholes in the street, a balcony with some problems), etc. They need to be profiled with a number of metadata, and in case of emergency events they should be compliant with CAP standard (Common Alerting Protocol) [CAP], in order to provide tools and instruments to enhance the city resilience capabilities, and be interoperable among several operators: fire brigade, local and national police, civil protection, etc.

 

3. Snap4City Architecture

In this section, the general architecture of Snap4City is presented to contextualize the successive sections where VPL aspects are discussed. The Snap4City platform has a Lambda architecture, as reported in Figure 1. Data are received in push and/or pull, and almost every data can be considered IOT data sources. They may come from IOT Brokers, social media, web server, web sockets, streams, etc.

 

Figure 1: Snap4City Functional Architecture, cloud view where the IOT Applications are only on cloud.

 

    In Snap4City, most data provided from External Services or operators are collected in pull via REST Calls by using scheduled processes written in ETL, through the Penthao Kettle tool. The ETL processes are executed (periodically or sporadically/on demand) on a cluster of Virtual Machines by a scalable and distributed scheduler called DISCES [Badii 2017]. The collected data are regularised/reconciled according to the Km4City Knowledge Base [Badii 2017], and then pushed into the Big Data Cluster storage. The reconciliation process allows the reconnection of data to city entities already in place; for example, by connecting POI (Points of Interests) to civic numbers, traffic flow to street segments, etc. The aim is creating a fully connected knowledge base for the city, where semantic queries can be performed by exploiting physical and conceptual relationships.

    On the other hand, data arriving in Push (data driven) are typically real time data streams, managed by a number of IOT Brokers which an Apache NIFI distributed cluster is subscribed to. The NIFI process is in charge of performing a regularization/reconciliation task on data, according to the Km4City Knowledge Base and then it pushes the resulting data into the Big Data Cluster storage, while indexing and thus creating the Data Shadow. The above described parallel solution tends to be normalized in a single approach based on NIFI [NIFI] when the platform is small. Otherwise, when the platform has large data volumes, a distributed data warehouse for data ingestion based on ETL is more effective. The Big Data cluster storage includes a HDFS Hbase, a Phoenix storage, an RDF store for semantic reasoning based on Km4City (which is operatively implemented on Virtuoso store), and an ElasticSearch Index. The whole solution presents textual indexes on all fields, semantic indexes, and elastic aggregations to perform advanced “faceted” filtering queries. The indexing and query supports are exploited by Snap4City Smart City APIs [Badii 2017]. When small installations are performed, the HDFS cluster can be avoided.

     The operative IOT processes (IOT Applications) can be executed on cloud or on premise. In Figure 1, the cloud case (private or public) is depicted. When IOT Applications are executed on IOT Edge that are located on the field, they may directly communicate to the IOT Applications or Dashboards on cloud or by means of IOT Brokers (on which all the other ca be subscribed). On such grounds, without the risk of losing generality, in Snap4City, the IOT Applications for Smart Cities are obtained as:

 

IOT App = Node-RED + Snap4City MicroServices

 

    The IOT App exploits basic nodes of Node-RED Node.JS plus Snap4City MicroServices that are suitable for smart city data transformation and processing.

The Node-RED platform is based on two components: (1) a web-based visual editor to design flows, and (2) a runtime environment that may execute flows. In Snap4City, the Node-RED editor has been improved to:

• communicate with the so-called Resource Manager,
• save and load IOT Applications and manage SSO (Single Sign On) with Snap4City,
• manage a large set of MicroServices, namely the Snap4City Libraries of Nodes that are accessible from Node-RED official library.

Moreover, the runtime engine of Node-RED has been also improved to (i) manage the security according to SSO and Snap4City model, (ii) execute it on Docker on the elastic management solution of Snap4City.

On the other hand, the changes performed on the Node-RED have been released open and are functional only for use on cloud for large scale. While, Snap4City library can be used on the installation of IOT Edge with standard Node-RED tools, without any restriction.

 

The Snap4City solution has formalized and implemented a large set of MicroServices satisfying the above discussed requirements. The MicroServices provide an easy and formalized access to all the Smart City services which are available in cloud from the platform (including the ones to control a part of the platform itself). They are made available into the Node-RED Node.JS environment to create IOT Applications as VPL. Among the MicroServices, the IOT applications need also to access such services allowing exploiting Data Analytics, Visual Analytics and Dashboards. The latter two aspects can be employed to create the Graphic User Interface (GUI) of the IOT Applications. These tools, orchestrated by the IOT Application flows, may automatically inform, announce, act and produce alerts and warnings on IOT Devices, networks, user interface, external services, etc., and provide support to close the loop towards the user acting/reacting on the GUI and/or Devices, including notifications.

     In the deployments of Smart City IOT as a Service, SCIaaS, such as Snap4City, and in large smart city applications, a relevant number of IOT Applications may need to be deployed on the basis of the on-line requests made by users/organizations. Snap4City has been recently accepted by the EOSC (European Open Science Cloud) marketplace of the European Commission and therefore the IOT Applications need to be managed and allocated on demand, on the basis of the users requesting them, as well as to be put in execution on cloud. To this end, an elastic (vertical and horizontal) infrastructure has been created to manage in scalable manner the IOT Applications in containers [Bellini 2019], and mechanisms to guarantee end-to-end security [Badii 2019]. In Figure 2, the user interface allows the user to manage its own IOT Applications, irrespective of whether they are IOT App on cloud or on IOT Edges/field, whether they are child processes/containers for Data Analytics or WebScraping, etc. Please note that different kinds of IOT Applications are represented by different icons.

 

 Figure 2: Snap4City: the IOT Applications manager as seen by a user.

 

      The Snap4City MicroServices abstract low-level details to the programmer using a visual environment, hiding the complexity of sophisticate algorithms and tools. This is useful and suitable, for example, to provide routing, spatio-temporal search and discovery, for data analytic, dashboarding, networking among IOT devices, IOT data abstraction, etc. The Snap4City MicroServices are distributed into two official libraries of Node-RED nodes by the JS Foundation portal (https://flows.nodered.org/?term=snap4city). The two libraries are dedicated to final users (basic), and to developers (advanced). The version dedicated to Users provides outputs of Node-RED nodes easy to be exploited for non-skilled user on JSON. In fact, most of the output produces single variables and not complex JSON structured messages. On the other hand, the version for Developers (to be installed on top of the basic version for final users), presents a number of nodes/blocks that can accept complex input JSON messages to create strongly dynamic IOT Applications.

    Both Libraries of Snap4City Nodes can be directly installed in any Node-RED tool of any operating system: Linux, Raspberry pi, Windows, etc. In addition, we have also developed an Android App executing Node.JS/Node-RED and our libraries to provide use of them on IOT Edge, also exploiting mobile device sensors on the above-mentioned operating systems. In [Badii 2019b], we have demonstrated how Snap4City approach may work for mobility and transport applications, where safety critical communications and solutions have to be set up, involving IOT networks with IOT Devices and IOT Edge, IOT Apps and Dashboards.

 

     In this paper, a deep view of the analysis and design of Snap4City MicroServices for smart city applications is presented. The analysis has been also supported by the evidence about how the MicroServices can be successfully used for IOT Applications implementation, so that they can be used by City Operators and Final Users.

 

4. Snap4City Library of MicroServices of Smart City

In order to satisfy the smart city requirements reported and discussed in Section 2, in Snap4City a collection of more than 150 MicroServices has been developed as Nodes for Node-RED programming environment. The Node-RED philosophy of visual programming allows the creation of event driven data flow applications where the exchanged messages are in JSON format. On the other hand, also periodic processes can be developed by scheduling one or more internal timers. This means that users can develop IOT Applications as Node-RED flows, exploiting both Push and Pull data protocols, with the same visual programming environment. In the context of smart cities, both protocols are needed, while IOT Applications have to be capable to create flows exploiting a large number of features which are typically not available in Node-RED open library, neither by installing a number of libraries from different providers. Moreover, the Snap4City MicroServices are at a level which can grant an easy development of IOT Applications for smart city  even for non-expert users, as assessed in Section 6.

 

The most relevant families of nodes/MicroServices for smart city are listed below and they perform different kinds of activities, useful in the IOT App construction.

 

-- Access to Smart City Entities, that have different data models, and thus, different MicroServices may be required. Some Entities may have simple sets of Metadata for examples the ones describing the POI, e.g.: title, description, web, email, GPS location, images, opening time, etc.; others may have complex information, and/or also specific data types and structures, for example:

• Status of the first aid: number of people under observation for each colour in the triage, waiting time, etc.;
• Bus stops: all the bus lines including geometry, their planned time schedule, the real time delays, etc.
• Sensors with their values, measure units, types, healthiness criteria, etc.;
• Weather forecast associated with an area/region, which consists of a large set of values: temperature, humidity, pressure, wind, etc., for many different time slots in advance;
• Shape of cycling paths, gardens, parks, difficulties, restrictions, etc.;
• Parking areas, with the number of free spaces, predictions, typical daily trend of free spaces, costs, etc.;
• Events of: (i) entertainment, with their description, photo, start date, end date, etc.; (ii) from police officers on the street; (iii) Events of emergency, such as Civil protection early warning in CAP standard, etc.

In order to simplify this complexity, MicroServices like “Service Info” and/or “Service Info Dev” are provided for Final Users and Developers, respectively. In the IOT Application, a search/Discovery has to be performed as described in the next paragraph, otherwise the developer needs to know the so called ServiceURI, which is the unique identifier of all the city entities in the Km4City Smart City Knowledge Base. In the Snap4City development environment, the ServiceURIs can be recovered directly from the graphic configuration of the MicroService or searched by using the ServiceMap visual tool which provides a GUI for query. This means that IOT App programming is 100% visual, even if a single service is used for accessing a single element.