Snap4City Dictionary of Tools and Modules

Here in the following the Dictionary of tools and major Snap4City modules.

• ETL and Data Analytics Processes
  
  • Data processing Development Environment (VM): a virtual machine for developing Snap4City Applications, ETL processes, Data Analytics Processes, a personal SandBox that every developer may use on its own computer and desktop.
  • ETL processes: data transformation processes created, see a large set of examples accessible on DISIT GITHUB. They can be uploaded on the system manually or via the ProcessLoader/Resource Manager. They may become MicroServices. https://devkitenv.snap4city.org/
  • Data analytics Processes: a set of processes that can be written in several different languages, R studio, Java, Python, etc. They can be uploaded on the system manually or via the ProcessLoader, can become MicroServices.
• Application Development Tools and Services
  
  • IOT Application Builder: a tool for creating Snap4City Applications, in most cases based on NodeRED and Snap4City Microservices;
  • IOT Applications: Snap4City applications which are executed into the Snap4City environment, as processes and based on NodeRED, NodeJS, reading and acting on IOT devices, exploiting MicroServices, producing resutls on Dashboards. https://www.snap4city.org/dashboardSmartCity/management/iotApplications.php
  • MicroService Directory: a list of MicroServices as described in the previous sections, and as provided by smart city API and by other tools. https://www.snap4city.org/dashboardSmartCity/management/microApplications.php
  • ExternalServices are services provided by third party on third party services (for example, provided with WS or call REST) https://www.snap4city.org/dashboardSmartCity/management/externalServices.php
• Knowledge Services
  
  • Knowledge Base, KB: a knowledge base implemented as triple store, Virtuoso, on which the Km4City Ontology and data with all their relationships are loaded. It provides support for semantic queries in SPARQL. The KB is implemented a Linked Data repository which can be queried and accessed via Smart City API, rest call. https://servicemap.disit.org/
    • Indexing and Search Services: data loaded into the KB are also indexing for their textual attributes, and for making geoqueries, and queries along time lines.
  • ServiceMap: a tool for navigating by a visual interface, on the knowledge base https://servicemap.disit.org/ .  The servicemap is a developing tool for generating Smart City API call and learning how to pose queries on the KB. The maps are based on leaflet. It allows to see both static data and real-time data, and also their relationships, exploring geographical queries and passing to the graphs of the Linked Data and semantic model with disit.org">LOG.DISIT.ORG 
  • disit.org">Log.Disit.org is a tool for browsing on any Linked Data repository on the world, and browsing on them, also among them. The tool allows sharing configuration and view on the graphs. It facilitates to establish links from the KB of the city and the many other linked data in the worlds, for example: dbpedia, geonames, etc. FLINT, SPARQL: is a tool for making semantic queries in SPARQL.
  • ServiceMap3d is a rendering tool showing data and real-time data on a 3D city environment. Based on Bing.
  • Snap4City Web Portal: Https://www.snap4city.org a DRUPAL based access to the Snap4City services and tools, access to the open sources (via github), manuals, examples, links to DataGates and its examples, links to the ProcessLoader and examples, VM with the development environment, etc.
• Dashboard Services
  
  • Dashboard Builder: a tool for creating Dashboards for decision makers H24/7, from mobile to very large and multiple UHD monitors.
    • Dashboards: a set of Dashboard created that can be manipulated, shared, cloned, delegated, etc. They are created by composing a set of Widgets. They can be stand alone Dashboard as just the user interface of IOT applications.
    • Widgets: a set of graphic widgets, also specialized, and belonw to very large collection that can be shared and improved as well. They are implemented in PHP and have to be compliant to specific rules to be managed as Dashboards elements.
    • Dashboard Engine, one or more engines/processes for data flow management:
      • Collecting metrics from Database Storages and API
      • Assessing Metrics quality conditions and level
      • Assessing Web Portal service quality conditions
      • Collecting Actuators from web pages as IOT Actuators and send their values to IOT Broker in multiple protocol if possible.
  • Notificator: a tools for sending notifications to Dashboard users, to the decision makers, also creating for them a common environment to be informed and making decisions.Each Widget works with a number of metrics and for each of them, a set of Firing Conditions can be defined to activate Notifications via Notificator.
• Application Services (web and mobile)
  
  • Snap4City API: a large collection of API for managing Smart City and tools: https://www.km4city.org/swagger/
  • Advanced Smart City API: a set of Smart City API provided by the KB and other tools. In the API also the possibility of collecting mobile sensors, collecting data and comments, providing suggestions, recommendations, routing, accessing to real-time data, navigating on data of public transportation, parking, pharmacies, pollution, pollination, fuel costs, Point of Interest, making queries, etc.
  • Suggester: a tool for producing suggestions to applications connected on the Advanced Smart City API on the basis of few information of the users (position, past requests, etc.), or even of the user profile.
    • Suggestions received and accepted but are provided anonymously. 
  • Personal Assistant: a tool for providing a Personal Assistant to users connected with an application connected on the Advanced Smart City API on the basis of little information of the users, or even of the user profile, their behaviour now and in the past, they position, velocity, accelerations, etc. This tool can be used for distributing bonus, incentives, tickets, etc.
• Access Services
  
  • Authentication and Authorization: support for Web and mobile applications
    • OAuth support: authentication support for mediated registration via social media
    • Authentication and Authorization also including IOT based Authentication and Authorization: connection to reuse the Authentication and Authorization in the context of the IOT communications.
    • LDAP: open, vendor-neutral, standard application protocol for accessing and maintaining distributed directory information services over an Internet Protocol (IP) network. [Wikipedia]
    • SSO: single sign on active for all the tools supporting LDAP.
  • City User Directory, User Profiler: directory of registered users with related data of the user into the UserProfile. The list can be used for management purpose of providing services as well as to put in contact users (e.g., establish chats around a city problem identified in a Dashboard, discussing on some development aspects, etc.). On the other hand, the users have to provide a specific consent to be connected each other, to receive suggestions, to receive support from the personal assistant, etc. The City User Directory is implemented by using DRUPAL solution with LDAP and keyCloack for SSO based on JWT. 
  • User Profile: The user profile may include:
    • data coming from the registration (e.g., email, name, etc.; or Facebook, twitter, G+ account, etc.), in alternative the user may access to services without performing any registration. In this case, the user profile may be limited to some information collected when the first time a used connect to an up providing if prefer: language, and role (citizen, commuter, student, tourist, etc.)
    • saved data in the PersonalStorage from the personal IOT Application, which can be also delegated in access to other users
    • My personal Annotation: annotations on services of t he city, from bus stops to the singe element in the city, which can be also delegated in access to other users.
    • IOT applications: IOT Applications developed by the user, which can be also delegated in access to other users.
    • Requested services: queries, routing, etc. 
    • Etc.
  • Auditing: a tool exploiting the UserProfile to perform an auditing of the actions performed.
• IOT Sensors and Actuator Services: a set of tools for general management of IOT devices.
  
  • IOT Directory: a tool for registering IOT devices, searching them (by ID and kind, type, location, etc.), independently on the IOT Brokers by which they are collected, and protocol they use. It also sends the information of registration to the Knowledge Base. The IOT Directory provides a unique point for searching and subscribing to IOT devices. The IOT Directory is capable to address multiple brokers and protocols: Orion Broker, Mosquitto, AMQP, etc.
  • IOT Discovery: a tool based on the Knowledge Base and Smart City API (that is ServiceMap) to visually search any kind of IOT devices, exploiting text and geo search. IOT devices along a path, in an area, closer to a point, of a given type/subtype, etc. The result is a set of Identification ID corresponding to their ID into the IOT Directory. This tool is totally integrated into the IOT Directory and into the Block of many MicroService Blocks in the NodeRED IOT application editor. 
  • IOT Brokers: a collection of IOT Brokers of different kind, addressing one or multiple protocols, supporting or not the protected protocols. Among the brokers: 
    • Orion Context Broker: an IOT broker
    • RabbitMQ
    • Mosquitto
    • SigFOX
    • LoraWAN
    • Etc.
  • EventLog: a process to log all the activities performed by IOT devices on the different Brokers in a unique storage (also called broker of brokers). The Storage is indexed with SOLR and accessed with AMMA tools based on Banana.
  • IOT Wrappers: are modules transcoding IOT protocols so as to make possible the connection of certain brokers with: (i) upper level broker, (ii) IOT directory. On the other hand, some of the IOT Wrappers can be also located from the Device to the broker. So as to make possible the registration and work with devices having also unaddressed protocols. 
• Computation Management Services
  
  • DISCES (Distributed Smart City/Cloud Engine Scheduler) is a Scheduler with support for Elastic Management. It allows to (i) collect the scheduling plan for the processes, (ii) create the plan, (iii) monitor all the resources of cloud (Hosts, VMs, Containers), (iv) allocate VM and Containers, (v) turn on/off Hosts, (vi) deploy and shutdown VM and Containers, to perform elastic management of resources. It also exports some MicroServices to be managed by special Snap4City Applications.
  • Container Support: a set of open source tools such as
    • Marathon: is a container orchestrator. It allows the deployment of containers into a distributed environment. It is fully based on REST call and can manage thousands of servers/VMs.
    • Mesos: is an Apache project to manage computer clusters. Mesos is a tool for managing servers.
    • Docker: is a technology for managing containers to make possible to automate the management of resource isolation. Docker is a tool that can package an application and its dependencies in a virtual container that can run on any Linux server.
  • ProcessLoader (also called Resource Manager)  is a web tool and portal for uploading Processes (e.g., they can be ETL, and Data Analytics developed in R, Java, Python, C/C++, etc.). The ProcessLoader allows the developers to (i) upload their processes on the smart city back office; (ii) put/ask for their execution in the back office, (iii) monitor their execution, (iv) publish and share them with other developers, via the support of the Snap4City WEB portal ion DRUPAL. The ProcessLoader works in strict connection with the DISCES Scheduler. The same tools allow to move a process from Testing are to the production area. This cativity has to be performed by a RootAdministrator. 
  • Application and MicroService Monitor and Analyser, AMMA: A tool for the analysis of indexed EventLogger data for exploring different aspects about the workload, traffic flow data, the efficiency of the communications, the identification of the most computationally expensive communications, drill down, faceted filtering, thus to make profiling on App and tools.
    • Event Logger: A service for logging communication activities among applications and MicroServices. This tool is based on Rsyslog and tools for the indexing of collected log data.
• Big Data Storage Services, to manage data coming from several sources, indexing them and accessing to the storage via the indexes and directly, via MicroServices.
  
  • DataStorage: A set of noSQL data store for storing: IOT events, EventLogger, IOT data, Open Data, SOLR indexes, etc. It is mainly based on (i) Hbase/Phoenix over HDFS and thus with the possibility of using MapReduce, (ii) Mongo storage as alternative.
  • DataGate/CKAN: a tool for collecting/uploading open data and automatically see them processed for data ingestion into the Knowledge Base. It is based on CKAN, and thus present also API for automated upload of Open Data Sets, and for the cross sharing of Open Data among different CKAN/DataGates.
  • Developer Dashboards: a tool for accessing to SOLR indexes, drill down, and making cross links among them and with ServiceMap, back and forward, also supporting Auditing per user, per IOT applications, etc. The Developer Dashboard can be developed by Banana. The data to be indexed and Dashboarded for filtering, cross filtering, drill down also in timeline are:
    
    1. IOT Event Logger,
    2. Event Logger for communication aspects,
    3. Data Storage of real-time data disregarding if they are collected by IOT or by ETL.
  • DataMapper: a tool for mapping data sources represented in the Knowledge Base and identified by a ServiceURI (as well as: POI, sensors, actuators, IOT devices, etc.) towards different service as ServiceURI and thus inheriting data and moving them on different GPS locations, with different IDs, and descriptors.
  • My personal Data Annotator: is a tool to perform personal annotations. The personal annotations can be at level of:
    • Single ServiceURI in the knowledge base: POI, Sensor, Device, IOT element, etc.
    • Data Set, and thus are performed in the DataGate.
    • Data point in the time line of the above-mentioned A, B and C data sources.