1. Home
  2. Architecture of Snap4City with Internal API, documented in Swagger

Architecture of Snap4City with Internal API, documented in Swagger

The technical architecture can be described for areas:

  • Big Data Store:
    • Knowledge base, created on the basis of the Km4City ontology and GraphDB Virtuoso. It has a 5th  generation of the ontology with fully integrated IOT/IOE aspects: IOT devices, IOT brokers, Discovery, connection with Data Wizard, etc.
    • Indexing services: index of texts; index of data coming from IOT, and from ETL, etc.; the large index are placed on sharded index on SOLR. The indexing processes are realized by using NIFI mainly and thus can be easily configured.
    • Large data store is fully distributed among several VM based on Hbase/Phoenix. ETL processes and KB can work  with direct access to this store and maintain updated the cross linking among tables with real time data and their descriptions and relations on KB.
    • Among the data also: Events logged for  auditing of any kind (traffic, data, etc.) data mapping at level of KB, personal annotation referring to KB elements, etc.
  • Cluster mapReduce and Hbase/Phoenix:
    • It include a number of VM for computational node hosting processes for DataAnalytics and ETL transformations. They are managed by DISCES scheduler. All these processes in batch: (i) get data from Gateways, web servers, social media, FTP servers, IOT brokers, IOT gateways, IOT edge, and also from KB, Big data indexes, etc.; (ii) transform them in knowledge, (iii) write the new data and knowledge into Big Data Store.
  • The integration with legacy is particularly easy since Snap4City is:
    • capable to ingest data from any kind of data sources: External Services, City Operator services, Legacy Services, Social media, etc. A large collection of ETL processes for integration is published on GITHUB/DISIT lab
    • modular and thus not all modules have to be used. You can select from the whole set of tools those that are missing in your architecture and integrate them in your
    • safe since all the  tools are LDAP, keyCloak and HTTPS compliant for API and services
    • flexible since it is compatible with any kind of protocol and format
    • scalable since it has been designed for scalability and it has been already proof in large are with millions of inhabitants and millions of events per day.
    • fully open source, and license free, and patent free.
    • based on open source components mainly coming from Km4City
    • etc.
  • IOT Brokers, Gateways, Devices, Edge, etc. (in green, green/yellow):
    • Collect IOT data from almost any kind of IOT Gateway and Server with a large range of protocols and format. Some of them are secure such as OneM2M, SigFOX, etc. Others (such as IOT Orion broker) we have been secured with a Proxy Filter Security, which is directly connected with the Authentication and Authorization support. Thus an END-TO-END protection is secured.
    • Supported main IOT gateways are: OneM2M, SigFox, Mosquitto, RabbitMQ, LoraWAN Dragino, IOT Orion Broker, with their corresponding protocols, registration, authentication, telemetry, etc., supports.
    • IOT Devices: any kind of IOT device that can be connected with the above mentioned IOT brokers/gateways or similar.... They may be capable to be connected with our suggested Brokers or with commercial brokers such as SigFOX, OneM2M, Lora, etc.
    • IOT Edge: any kind of IOT Edge device that can support the protocols of the above mentioned IOT brokers/gateways. We suggest the usage of Raspberry Pi IOT Edge that can include inside the same IOT engine we use in the core parts that is NodeRED. In that case, an SD image is provide to simply the production of the IOT Edge.
    • IOT mobile Device: Also mobile phone with Android can host IOT Application engine NodeRED. So that their can play the role of IOT Edge and at the same time
  • IOT Application Cluster (in green): https://www.snap4city.org/drupal/node/22
  • Smart City API and Snap4City MicroServices
  • Tools for Administrators:
    • Process Loader for ETL, and DataAnalytics in any language, based on DISCES
    • Cluster Manager to manage the  execution of clusters:
    • Cloud Elastic management, vertical and horizontal scaling, performed by:
      • DRS, HA based on VMware for automated vertical scaling
      • DISCES-EM for horizontal scaling: allocating/deallocating VM on the basis of the workload and of the number of active IOT Applications
    • ResDash: Resource Dashboard collecting data about resource consumption from each VM and Host involved into the system thus monitoring the cloud resources.
      • ResDash Docker: a version working at level of Containers
  • Tools for Developers:
    • Data processing development Environment: It is a personal or shared SandBox for Developers that may use it  https://www.snap4city.org/drupal/node/24 The platform can put in execution, any process developed in Java, C/C++, Python, mapReduce, etc. and can be put in execution from the DISCES.
      • The processes can be developed:
        • from the cloud via WEB VNC access from the snap4city.org menu on left.
        • on their own computer and desktop downloading the VM and executing it with a VMware player. The users may develop Snap4City Applications directly connecting to their own environment. VM for download to be put in execution via VMware player. https://www.snap4city.org/download/Snap4City-ETL-VM.rar,
      • Can be tested with the access to the SandBox for Developers accessible from the menu of the snap4city.org
    • RStudio: https://www.snap4city.org/drupal/node/25  
      • R Studio, accessible from the menu of snap4city.org
      • The R Studio environment allows to access to: (i) referral data via queries, (ii) indexed data collected from IOT/ETL processes via queries, (iii) knowledge base data via SPARQL queries, (iv) knowledge base data via Advanced Smart City API, etc. Among the possible solutions for data analytics at the state of the art, the R seems to be the most powerful since in R one may found a large range of examples. In addition, the solution provided in R Studio is based on Computer endowed of an NVIDIA Titan Xp (gently donated by NVIDIA, http://www.nvidia.com ). This allows you to develop on R Studio processes that exploit the Tensor Flow capabilities. The Titan Xp is a 3840 GPU processor (CUDA Cores). 
    • AMMA, Application and MicroService Monitor and Analyser: https://www.snap4city.org/drupal/node/43 is an Open Source tool for monitoring and analysing communications aspects in real-time, such as data flows and volumes information among IOT, ETL and Data Analytics (in input and output, iot and storage, and API) collected by the EventLogger module in the whole solution (all ETL processes, IOT applications and MicroServices are monitored). The AMMA tool is based on SOLR index, and Banana plus special widget developed for Snap4City for accessing and displaying indexed data in an intuitive and interactive fashion, allowing applying faceted filtering and data drill down in time and space, displaying stacked data and temporal trends. 
    • DevDash, Developer Dashboard: https://www.snap4city.org/drupal/node/42 is an Open  Source tool for accessing to data collected (by means of ETL and IOT processes) in an interactive and faceted manner. It is based on SOLR to index the data, to enable the drill down, and to dynamically shape the data according to multiple filters. This Dashboard approach is only for developers. 
  • GDPR services integrated into DRUPAL content management system, with safe data of the users, capability of browsing, delete, download, delegate access, etc.
    • User Profiling: a management compliant with GDPR for user profile data
    • Auditing: to see who accessed to the personal data, etc.
    • MyPersonalData: to save and retrieve personal data from IOT Applications and Dashboards
    • MyAnnotation: to save and retrieve personal annotations from IOT Application and from the main menu to leave annotation attached to KB services which can be POI, IOT devices, etc.
    • MyIOTKeys: to save and retrieve personal data regarding username/password and K1/K1 for accessing to IOT Devices of any kind.
  • Security: authentication and authorisation services and supervisors
    • Implemented with KeyCloak and LDAP, OAuth2
    • Including certification for HTTPS/TLS
  • Snap4City Users tools:
    • Dashboard Builder: the main entry for developing Snap4City Applications, it allow to: https://www.snap4city.org/drupal/node/17
    • ServiceMap, ServiceMap3D https://www.snap4city.org/drupal/node/19 is a tool for navigating on the Knowledge Base by a visual interface and producing samples of the Calls to invoke the Advanced Smart City API. The ServiceMap is a developing tool for generating Advanced Smart City API calls and a learning too to train you on how to pose queries on the KB. The maps are based on leaflet. The ServiceMap is a learning tool to:
      • train you on how to pose queries on the Knowledge Base and API.
      • visually browse city map services,
      • generating calls of the Advanced Smart City API, which can be received via email,
      • recalling and sharing them with other programmers,
      • embedding interactive maps into Dashboards, etc.
      • simplify the development of Web and Mobile Applications, and the access to the Smart City API for all, also training the developers in their usage.
    • disit.org">LOG.DISIT.ORG, FLINT https://www.snap4city.org/drupal/node/21 is an Open Source tool of DISIT lab for browsing on any Linked Open Data repositories and Linked Data on the world, and browsing also among them independently on their location exploiting also their interconnection via definition in transparent manner. The tool allows sharing configuration and view on the graphs. It facilitates to establish links from the Km4City based KB of the city and the many other linked data in the worlds, for example many Point of interest have links to the definition of VIP people defined in: dbpedia, geonames, etc., thus very important for tourism and cultural applications. FLINT, SPARQL: is a tool for making semantic queries in SPARQL. It is a technical tool that allows at the developers to pose semantic queries on the RDF stores representing the Knowledge Base, one or more. 
    • IOT Directory: https://www.snap4city.org/drupal/node/40 is a web application developed for monitoring and managing the devices (sensors/actuators) that are available in the city by means of their Context Brokers.  The IOT Directory allows you to see IOT devices at level of devices and single sensors/actutora with their ValueType (temperature, pression, velocity, humidity, etc.): and ValueUnit (float, integer, etc.). So that the value type is concept that goes in the semantic of the data, while the value unit is something more technical that describe how the value is coded and represented. Depending on the Context Broker a device (sensor/actuator) is identified by means of specific id field (as in Orion Context Broker) or through the name of the channel/topic according to which their values are published (as in MQTT, NGSI e AMQP, and other protocols as well).
    • MicroApplications: https://www.snap4city.org/drupal/node/54 are microscenarious derived from the Km4City mobile application, and are developed as HTML5 java script solutions accessible in response mode by all device. They can be used into Dashboard as well as stand alone. They can be activated on any Km4City knowledge base. The accessible micro Applications are those that allow you to access to the following geolocated services in your area, and in particular: Advertising and promotion categories, Accommodations, Agriculture, ..... Cycling Paths, POI and paths, Discovery the City, Entertainment Locations, POI and real time data, First Aid and triage real time, Position of  the hospital and emergency status,  Parking and predictions, Path Finder, Routing and Navigation: car, bike pedestrian and multimodal, Pollen Data and real time data, for the sensor position, Pollution data and real time data, for the sensor position, Public Transport and search: Bus Stops, bus paths, list of courses, time schedule, next arrival, Suggestions near to you, closer to you, etc.
    • ResourceManager: https://www.snap4city.org/drupal/node/27  is a tool accessible via web for uploading/submitting and monitoring Processes (they can be ETL, and Data Analytics developed in R, Java, Python, C/C++, etc.). It is capable to manage Processes (ETL, Data analytics), IOT Applications, web applications, html5 micro applications, R processes, subscription to applications, external services, microservices, and MicroApplications, etc.
    • DataGare/CKAN: as DataSet Manager https://www.snap4city.org/drupal/node/26 , is a special module of CKAN open source portal for data set management and publication. CKAN supports the management of multiple collections, multiple organizations, etc. DataGate has been designed to collect/upload open data sets and process them automatically for data ingestion, field cleaning and regularizing, and for reconciliation of entities towards a Knowledge Base grounded on Km4City. The resulting cleaned data set is also accessible for further reuse and can be automatically published on the DataGate. It is a exceptional tool for shortening time to ingest of:
      • referral data, statistical data, etc.;
      • open and not open data sets with geolocations;
      • descriptive data, that just describe the entities of the city and provide for them metadata, GPS and/or shapes

 

The Internal Snap4City APIs include:

The Internal API that are used for communication among the modules of the platform

They include:

 

  • Share