Installing Snap4City: Appliances, Virtual Machines/Containers ready to use on your premise for Smart City and IOT Applications

×

Warning message

  • You can't delete this newsletter because it has not been sent to all its subscribers.
  • You can't delete this newsletter because it has not been sent to all its subscribers.

2023 HOW TO: Deploy/Install your Snap4City Solution on private or public Clouds, VM with Containers

(02-02-2021) Notes on Installing DataCity-Large 2020: HOW TO: Deploy & Installation of Snap4City DataCity-Large 2020

(10 September 2020) Snap4City DataCity-Small (aligned to version 1.5 of VM plus some new features) has been released as DOCKER Compose, see in the following.

(15 August 2020) Snap4CityMAIN version 1.5 has been released with included: synoptics, more widgets, real time buttons, IOT App updated, plus other VMs have been updated for composing a number of configurations: IOTOBSF,  IOTDSES, KBSSM. See in the following.

Snap4City platform can be freely downloaded from this portal, and thus you can produce a number of services with these tools in the view of Smart City IOT as a Service, SCIaaS. Therefore, if you are interested in using Snap4City.org for trial from this portal you do not need to perform any installation on your computers.

On the contrary, this page is dedicated to who would like to perform a local installation on premise or cloud. In fact, in this page you can find the instructions for the installation and a number of Snap4City configurations. Snap4City is 100% open source, license free solution, and the Appliances that you can download from this page are license free. You can download and install Snap4City tools on premise or on cloud as you like, by starting from the following Appliances and/or from the GITHUB/DISIT sources (Source Code and licenses), and docker compose provided. In the following figures, the deploy of Snap4City solution from small business to large scalable cases are presented. The configurations are listed and coded with a letter to simplify the identification of the configuration. While other mixed solutions and intermedium configurations are also possible. In addition, some of the tools we are mentioning/providing in the following configurations can be substituted with your legacy tools. For example your CKAN, your GIS, ArcGIS server, etc., etc. For shortening your deploy and/or installation activities we have created a number of Virtual Machines, VM, Appliances and Docker containers/compose. The VMs can be put in execution on VMplayer or VMware workstation or Vcenter Vsphere VMware infrastructure or you can convert them for any kind of Cloud infrastructures, while Containers on Docker. The configurations are presented below from the smaller and simpler solutions to the most capable and wide deploy for large scale smart city with living lab support. For each of them an incremental description and user manual for installation. The same tools can also be downloaded from GITHUB/DISIT.

How to proceed:
Revise the 7 Configurations models proposed from A:Alone to F:FullPlatform+LivingLab, but they can be customized as you like.
From this page you are guided to identify the most suitable configuration according to your needs;
Once the most suitable Configuration has been identified, you can:
i.Download and deploy the Appliances provided as Virtual Machines or Containers for the desired configuration, or
ii.Download and install modules from GITHUB according to the recipes for  VM/Container and the user manual of the single tools, or
iii.Mixt of the above (i) and (ii) approaches.
IOT Edge can be created by (1) installing Node-RED, (2) adding Snap4City library of Nodes from the Palette Manager of Node-RED, (3) registering device or messages on Snap4City (cloud or your local installation): https://nodered.org/docs/user-guide/editor/palette/manager











Configuration Kind

Min #VM / Docker

IOT Broker int (any# and kind)

IOT Broker Ext (any# and kind)

MAP WMS web

KBSM semantic engine reasoner

IOT APP, MicroServices #Containers

Storage & IOT Data Shadow

SSO, Roles LDAP

IOT Security full stack end2end

Dash + Wizard, widgets, Data Inspector

GIS input WFS WMS

GIS output WFS WMS

Heatmaps WMS

Mob. App & Micro Apps.

ASCAPI Smart City API, Federated

Living Lab
Support

Optional services supported by the configuration

A: Alone/MAIN

1

 

X

X

 

1 mf

Small

X

X

X

X

 

 

 

(x)

 

ETL, DG, CM

B: Basic

2

X

X

X

 

1 mf

Small

X

X

X

X

 

 

 

(x)

 

ETL, DG, CM

C: CityStart

3-4

X

X

X

X

1 mf

Small

X

X

X

X

X

X

X

X

 

ETL, DG, R, CM, Eng

D: DataCity

5-6

X

X

X

X

70

Medium

X

X

X

X

X

X

X

X

 

ETL, DG, R, CM, Eng, LL

E: ExtensiveCity

6-8+

X

X

X

X

70

Scalable

X

X

X

X

X

X

X

X

 

ETL, DG, R, CM, Eng, LL

F: FullPlatform

12-14+

X

X

X

X

Scalable

Scalable

X

X

X

X

X

X

X

X

 

ETL, DG, R, RS, CM, Eng, LL

F: FullPlatform + LivingLab

13-15+

X

X

X

X

Scalable

Scalable

X

X

X

X

X

X

X

X

X

ETL, DG, R, RS, CM, Eng

mf: multiple flows for each IOT App; + means that may depend on the volume you would  like to manage in flow and storage.
In the following configurations you:
have: Multiple IOT Brokers; unlimited number of data sources; interoperability support; KB
may have: HA, DRS, FT, Balancing, cloning and configuring additional VM/Containers;
may pass at the next configuration without reinstalling the VM/Containers;
may have VM or Containers, please ask to Snap4City@disit.org;
--may add one or more Optional Services as VM/Containers: ETL, DataGate, Routing, Engager, Data Analytics, etc. etc.
From Configuration E:ExtensiveCity and upper, the number of VMs/Containers depends on the volume of DataStorage and on the volume of Data Stream in input.
These two aspects are managed by two independent clusters of VMs/containers and thus they scale independently each other. This allows to satisfy any different combination of volumes in streams and data storage.
We suggest using Living Lab support only on FullPlatform, while it can be installed on Smaller Configurations with some limitations in terms of features

Legenda of the above table for Optional Services

Eeach VM is provided configured with large footprint as 16/24 cores 2.2 Ghz, 16-24 GB Ram, 500 GB HD in thin provisioning, with 25-40 GB HD space used at the first start, in most cases Debian, this initial size is only cautelative to get maximum performances, while the minimum footprint is much smaller (please note that configurations A and B can be executed on 4 cores, 4 GB Ram, 44Gbyte HD free space).
to estimate the needed CPU, RAM, Storage for each configuration it is easy from the #of VM in the configuration.
-- to estimate the size of the platform on the basis of the data that you need to ingest is also very simple, ask for a quotation describing your data (send email to snap4city@disit.org)
-- all the VMs in download are in files RAR of 1-12 Gbytes size, so that you can download from browser. If you have problem we can arrange a different modality.
In solutions, with a very large number of users on the Front End: Smart City API (mobile App users, Dashboard users, a frontend balancer and more FrontEnd servers would be needed). Please note that the Smart City API are also used by DataAnalytics processes and by MicroServices in the IOT Applications.
Typically a thousand of simultaneous users on the front-end can be sustained for each VM/Container
More precise estimations can be performed by knowing the actual demanded workload

A: Alone/MAIN) For Small Size Business, for example an industry 4.0 application you can need to install only one VM: Snap4CityMAIN, ContainerCompose "Alone".

it may be used to create dashboards exploting data coming from external third party IOT brokers, from external data stores / databases of several kinds (ODBC, JDBC, Mongo, RDF, etc.), etc. Please note that the solution released as VM allows you to test also Smart IOT Applications, since Snap4CityMAIN includes one IOT App on which you can load/design multiple flows.

Therefore, 1 (one) VM, one Container Compose with multiple Containers and you can start using.

Snap4CityMAIN (also called dashboard) includes the Snap4City Dashboard Builder and a full starting Snap4City environment for small size businesses as well as for large solutions. Snap4CityMAIN includes tools configured for supporting features in a Snap4City local environment on your premise, so that with limitations only on configuration aspects to make the VM consistent and capable to be executed and provide Snap4City services in your network. it can be the seeding of a larger and scalable solution on your premise. Snap4CityMAIN  can be replicated to create a cluster in balance. The database can be shared from the frist Snap4CityMAIN while it cam be also extracted and put in some network storage.

What you can do with ALONE configuration you can:

  • create a small smart city business, (or industry 4.0 solution) acquiring data by using this VM only, for example to get data with Node-RED and presenting them on Dashboards;
  • create and manage a number of users into your on premise Snap4City platform, according to GDPR;
  • directly connect your Dashboard Builder environment to get data from any your database in your network;
  • register External Services for showing them into your dashboards;
  • register MIcro Applications;
  • Communicate with Dashboard by using Web Socket streams.
  • define you myKPI and POI and manager them on dashboards, they can be time series, and time series with GPS locations;
  • connect the SNAP4CITYMAIN with (a) the IOTOBSF: for ingesting data from the IOT Brokers in push, (bETL for ingesting data from batch processes in PULL; This means that with two VM you can create you Industry 4.0 and Smart City solutions. The only limit is the volume of data, the number of IOT Applications, and the lack of massive IOT device management. 
  • add more IOT or other protocols using the IOT Applications as described into https://www.snap4city.org/65  
  • add Data Analytics as described in: https://www.snap4city.org/501
  • please note that:
    • you can start with Snap4City MAIN anche step by step creating more complete and high performance solutions without the need to restart since any solution include also the MAIN
    • MAIN does not include any knowledge base map management. You can use your GIS or direcly the maps from open sources. for the KB you need to instal KBSSM
  • etc

Alone/MAIN Configuration: Snap4CityMAIN Virtual Machine

Snap4CityMAIN VM can be found on:

The suggested VM player is: https://www.snap4city.org/download/video/vm/VMware-player-12.1.0-3272444.exe

Documentation for installation can be found  on HOW TO: configure/install StartSNAP4CITYVM: starting appliance of Snap4City  and the VM appliance Snap4CityMAIN includes tools as:

  • Dashboard Builder for creating dashboards of several kinds, and distributing them on a large number of users, also in connection with IOT Applications, including Dashboard Wizards and Native Widgets. A large number of widgets updated and/or added in version 1.5 and most of them are dynamic: time series, bar series, kiviat, multiseries, single content, buttons, tables, event driven with synoptics, etc., most of them allow you to browse on time, etc.)
  • Data Inspector: for exploring the data accessible on the solution. A large set of new features have been added on Ver 1.5, for managing the collection of data, links to processes, links to IOT Apps, licensing models, owners, etc.)
  • Dashboard engine, for collecting data from several data sources with a Pull approach and save them on local database of Dashboard Builder. The Dashboard engine is compatible with several protocols
  • Authentication tool, Single Sign On, SSO: for registering users and managing keys (keycloak based), LDAP support for managing roles and organizations.
  • Notificator for programming notifications to be sent when some event occurs on Dashboard metrics.
  • IOT Application, IOT App: one Node-RED Docker for implementing IOT Application flows, a number of them, also supporting Snap4City MicroServices, mutual authentications, etc., working with data in Push and Pull and several protocols. The library of Snap4City MicroServices presents some limitations since the Knowledge Base as Servicemap is missing in the configuration.  With Version 1.5, stronlgy update of nodes for node-red for managin dynamic dashboard widgets, synoptics directly, barseries, timeseries, etc., in addition the compatibility with a number of new protocols
  • MySQL database, local instance for configuration and data management. Which can use also Elastic Search even if not configured yet.
  • External Service manager: to collect external services, legacy servers and tools and exploit them on Dashboards. Please remind that External Services as web pages can be also embedded into Dashboard widgets.
  • MenuManager also called Main Snap4City: to manager different tools connected on a Snap4City environment, also external services, legacy servers, etc. On version 1.5, added the possibility of having menu attached to the single organization and single dashboards.
  • OwnershipsManager: to manage users' ownerships and delegations to access at different kinds of data and resources, delegation to access, revokations, etc. Added in Ver. 1.5 the possibility of controlling ownership and delegation via IOT Applications.
  • MyKPI: for collecting and managing MyKPI, personal MyPOI, etc. It includes the Personal Data Manager, based on GDPR. (updated on ver 1.5)
  • WS Server: WebSocket Server for secure communications among IOT Applications and Dashboard, Synoptics, widgets, etc. (strongly updated on ver. 1.5, there is also Kafka for managing event driven data, plus WS server)
  • MicroApplications: a set of MicroApplications which can be tuned on the content accessible on your area. The MicroApplications may have sense when the Main Snap4City is configured together with KBSSM providing knowledge base of city entities on the maps. Please remind that MicroApplications can be also embedded into Dashboard widgets
  • IOT device management with IOT DirectoryIOT Directory: is a tool for managing a large number of IOT Brokers of different kinds, and supporting different IOT protocols. The IOT Directory needs at least a ServiceMap and to perform IOT Discovery. The IOT Directory allows at the users to add IOT Devices, IOT devices in bulk, to manage delegation on devices and IOT Devices values, etc. 
  • Resource Manager: to publish, exchange and access to a large set of resources developed for other smart cities and Snap4City solutions; The Resource Manager includes market place for IOT Applications and flows, ETL processes, Data Analytic processes, etc. It is a very powerful tool for information exchange and for the publication of smart applications.  The Resource Manager also allows the user to create MicroServices for the IOT Applications, when they are based on rest call of external services. Added on ver 1.5 the Data Dictionary, to define nature, subnature, value types, value units.
  • Synoptic support since version 1.5: a set of tools for creating synoptics and custom Widgets for the dashboards using MyKPI, sensors, costants, variables on WS.
  • User Stats since version 1.6 (coming soon): a set of metrics and statistics on resources consumed by each user for self assessment;
  • we postponed other fetures to  the next versions. On the other hand, a number of features that you see on Snap4City.org can be added on the main taking the source code from GITHUB and make from yourself the configuration needed.

Alone/MAIN Configuration: Snap4CityMAIN Docker Containers

The Docker version for this "Alone Configuration" and is available

The configuration in terms of user names and password is the same of the VM described above: HOW TO: configure/install StartSNAP4CITYVM: starting appliance of Snap4City,  On  the other hand, the Container Compose includes the details of the ports used that should not conflict with the host or server in which you deploy it, so t hat you can change in the file in case of conflict. In addition, you have to insert into the hosts file of your operating system: <your-host-ip> dashboard keycloak wsserver iotapp personaldata
This will allow all processes to refer to compoments: dashboard, keycloak, wsserver, iotapp, personaldata, as localhost by using their names and internal mapping. The same if you have other computers in the same network. Please note that, if you are going to make the solution public you have to provide a domain name, etc., and change the configuration according to your setup.

The cerate group of contaiers are put in execution by the Container Compose (version 1.3) and in particular each of them are accessible from Docker HUB and include:

B: Basic) Addressing  a relevant number of IOT devices/ IOT brokers in your smart city or industry 4.0, a few smart Applications with multiple  flows

If you have a relevant number of IOT Devices you need to have several IOT brokers. They can be registered with their devices into the Snap4City IOT Directory in the Snap4CityMAIN. Moreover, you may need to implement and thus provide some of those IOT Brokers. You can do that by using one or more IOTOBSF, which includes an IOT Orion Broker supporting NGSI and other protocols see :https://www.snap4city.org/467 for Mutual Authentication. In the context of IOT for Industry applications, you may need to have some intelligence and IOT Applications located on the field, as well as on premise. For example in factory in the case of Industry 4.0, on the farm in the case of smart farm, etc. This smart tools can be IOT Applications working on IOT Edge. This configuration alloow have them connected to the Smart4CityMAINVM, thus sending data to the solution at MyKPI, via IOT brokers, etc. see below the list of  the IOT Edge solutions on Snap4City with Security support. You may have IOT brokers and/or IOT Edge, both solutions or  only one, depending on your data and preferences. Please note that the solution allows you to test also Smart IOT Application, since Snap4CityMAIN includes one IOT App on which you can load/design multiple flows.

Therefore, 1 (one) VM + N IOT broker VM and you can start using.

C: CityStart) Addressing  a relevant number of GeoLocated Points of Interest, PIN on maps, and eventually heatmaps, a Small Smart City with a few smart applications on cloud and many on edge

If you need to manage a relevant number of city entities located on maps, we suggest you the following minimal configuration which includes KBSSM modeling a knowledge base for street graphs which can be feed with OSM graph or with your GIS as well (also included into it). You can exploit maps from GIS solutions (see https://www.snap4city.org/368) or you can exploit the Knowledge Base approach which is more powerful in terms of semantic relationships among city entities. To this end you need to have KBSSM appliance into your platform. If the workload on Smart City API become very high the  KBSSM  can be cloned to create a cluster and a balancer. The first KBSSM may keep the storage and share with partners. Please note that the solution allows you to test also Smart IOT Application, since Snap4CityMAIN includes one IOT App on which you can load/design multiple flows.

Therefore, at the minimum, 2 (two) VM and you can start using. If you have also Geoserver 3 VM. if you have a huge amount of mobile apps exploiting the Smart City data or you have to provide dashboard towards a large number of users, you may need to create a balanced workload, etc.

Ds: DataCity-Small) A small size Smart City with a fixed number of smart applications on cloud and 2 IOT brokers, relevant volume of data entering into the cloud

To satisfy small size Smart City, probably you need to have a limited number of smart applications on cloud, for example 4-5 IOT App/containers, and 1-2 IOT brokers, that collected IOT Data from whcih you prefer to the Data Shadow in place. You can start with the following configuration with restricted clustering capabilities that may growth with your city needs as well, reaching the Configuration E as described in the following and without radical changes but progressively. You may continue to use IOT Edge even if they have not been represented in the picture. The 2020 version includes full support for synoptics, data dictionary, resource management, new library of microservices aligned to July 2020 (the update can be easily performed from the IOT App in node-RED), IOT edge any kind, etc..

For this configuration on VM you have to install VM MAIN and the other components.

KBSSM: see above for this component

IOTDSES: including a non clustered solution for data shadown based on NIFI + Elastic Search.  IOT Data Shadow: the IOT data shadow allows to perform a copy of any message arriving from/to IOT Devices via the IOT Brokers registered on the IOT Directory. This feature consists of a cluster of VMs, and it is necessary when large volume of IOT Data comes into the system. The solution is based on a cluster NIFI, Elastic Search and Kibana. The solution is scalable and reliable, and fault tolerant. Elastic Search is the storage in which the IOT data are stored. Once  the data is collected, the historical values are made available for the Dashboards, and IOT Applications via MicroServices of the Smart City API and thus of the Knowledge Base. To start you can have one single IOTDSES while the solution can scale up cloning and clustering the IOTDSES . Each NIFI server is managing on a modest VM more than 120.000 msg per minute.

The Docker compose version of September 2020 (aligned to ver. 1.5 of VM, a little  more advanced) of the whole configuration is: https://github.com/disit/snap4city-docker/tree/master/DataCity-Small 

The configuration contains: MAIN, IOT Directory, two IOT Applications (July version), Notificator, one IOT Broker (IOTOBSF), ServiceMap (KBSSM), Advanced Smart City API with SuperServiceMap, Elastic Search, Kibana, Dashboard Builder, Synoptics, Data Inspector new version, Resource Manager new wversion, MyKPI, MyPOI, Knowledge Base, (for OSM to KB conversion use VM), auditing tools, management tools, external services, micrapplications, etc. Starting resources suggested: a minimum of 16-32 Gbytes RAM, 8 CPU cores at 2 Ghz, and 500 Gbytes of storage (it may work/start even with 6 Gbytes but some of the process could be very limited)
What you can do:

  • manage the city geo data by ingesting data from OSM to ServiceMap (via separate VM), with our tools, exploiting your GIS tools
  • ingesting data driven data from IOT Devices via IOT Orion Broker and related adaptors, it can manage a number of brokers, internal and external.
  • ingesting pull data by developing flows into IOT Applications (each IOT App may have multiple flows), exploiting Snap4City MicroServices for the ingestion. The IOT App may go to get the data and pass them into the IOT broker, thus resulting the data automatically indexed, stored and accessible for all the features
    • storing data into the data shadow based on elastic search and indexing them into the knowledge base automatically
    • the ingested data are automatically made available for the Data Inspector and for the Dashboard wizard and builder
  • create your dashboards exploiting maps, time trends, single content, buttons, bar series, multiseries, etc. IOT devices as sensors and actuators, etc. etc. they are data drive, may change dynamically data  from IOT App directoy on Dashboards, and a number of new widgets have been added
  • Synoptic: production of synoptic templates in SVG (open source editor separated), istantiation of a synoptic connecting them to MyKPI, sensors, contants, etc.
  • adding any other external services, IOT Devices, IOT brokers, etc., extended IOT device model with constant attributes, and ingestion in bulk
  • developing smart IOT Applications for data transformation in the back of your data flow or in the back of the dashboards for business intelligence
  • adding more IOT Applications by changing the Docker Compose or allocating them in other VM or hosts
  • registering local users and managing their profiles, with the user manger allowing local registration
  • data dictionary has been added to define: nature, subnature, value types and value units, and their relationships
  • etc.

D: DataCity) A medium/large size Smart City with a number of smart applications on cloud and a number of IOT brokers, relevant volume of data entering into the cloud

To satisfy a medium size Smart City, probably you need to have a number of smart applications on cloud, for example even 70 IOT App/containers, and a number of IOT brokers, that collected IOT Data from whcih you prefer to the Data Shadow in place. You can start with the following configuration with restricted clustering capabilities that may growth with your city needs as well, reaching the Configuration E as described in the following and without radical changes but progressively. Please note that even a large Smart City usually did not need more than 70 processes. For example, In Helsinki and Antwerp we used no more than 15 processes each. In Florence and whole Tuscany we have about 200 processes, covering the whole region with several relevant cities. So that you can pass at Large Scale Smart Cities maintaining the single MCLSCount and a cluster of IOTDSES for the data shadow of big data: billions and billions of data according to the number of VM you are allocating, not problem on scaling. So that Configuration D with elastic search cluster is called E: ExtensiveCity.
You may continue to use IOT Edge even if they have not been represented in the picture.

At the minimum without any optional, you need to install 4 VM + N IOT broker VM and you can start using.

  • IOTDSES: see above for this component

When you have VMs as: Snap4CityMAIN, KBSSM, IOTBSF, GEOServer and a number of IOTOBSF you can create a smart city as DataCity version minus (that means without a cluster of IOT Applications), that can be statically instantiated on some VM with Docker any way. Thus you have a non scalable solution for smart city that you can size according to your needs.

  • MCLSCont (coming soon): including Container Cluster Manager first VM: Mesos, Marathon, APIs, DISCES-EM. it consits of a set of tools for managing in elastic manner a large number of containers dockers. which are going to include IOT Applications, ETL processes, Web Scraping Processes, Data Analytics in R Studio, etc. This feature can be added including a number of VM for creating a Marathon/Mesos cluster. This VM includes the DISCES-EM scheduler for elastic management of containers on VSphere infrastructure, plus other minor tools. If you do not need to have scalability on a large number of Containers (IOT Applications, Data Analytic, Web Scraping, etc.) for them you can avoid to install a cluster and may use only one of them: TC10.14- Container Cluster Monitoring including: Marathon and Mesos each VM may manage up to 70 Containers for IOT Applications, but depend on the memory size, similarly 30 Data Analytics, 50 Web Scraping, etc.  You can create three of MCLSCont for creating a Master Cluster based on Zookeeper, the additional two MCLSCont have to provide a different configuration with respect to the main MCLSCont. The Data Analytics may include, predictions, heatmap productions, OD matrices estimation, traffic flow reconstruction, GRAL predictions, traffic total flow  computing, anomaly detections, etc. 

Please note that a number of other VMs/Container have been listed in the figure as optionals, such as: CKAN Data Gate, Routing, etc:

  • ETLSDK Serverhttps://www.snap4city.org/download/video/Snap4City-ETL-VM.rar, this ETLSDK Server is a Virtual Machine dedicated to developers of ETL processes. This appliance or eventually a cluster of this VM may be optional and may be needed only if the Smart City has already adopted in the past ETL based data ingestion process. It is strongly similar to the ETL Development environment that Snap4City provide online to its developers. It includes: ETL development environment based on a standard installation of Penthao Kettle, plus Snap4City DISCES for periodic scheduling of data ingestion processes (or at least to learn how to schedule processes in a single or multiple nodes). We suggest you to read documentation about this on: https://www.snap4city.org/24  The Solution is scalable in the sense that this VM can be cloned to create a cluster with the distributed scheduler DISCES to create a Data Cluster. see slides: https://www.snap4city.org/download/video/barc19/ingest/
  • DataSetManager DataGate is additional (please request): the Data Set Manager is an instance of CKAN customized to work with SSO, LDAP released as open source also. It also includes DataGate module for accelerating the data ingestion of data sets. If you have already a CKAN on your premise you can simply install the DataGate module.
  • Routing Server (please request): is an additional VM/Containerwith the OSM gathering and routing algorithms. In Snap4City there are two Routers. The one used for the Mobile App and one used for the What-IF Analysis. Both of them are Open Source.
  • ChatManager (please request): is an additional VM/Container with the Chat Manager based on Rocket Chat Room system with some configuration only!! all the adptations are already included into the Smat4CityMain. It is directly connected with the Snap4CityMAIN for dashboard system.
  • RStudioServer (please request): is an additional VM/Containerwith R Studio, it is a standard deploy of the free version of R Studio Server. This VM/Container or physical server may have separate user management, and assignment of VM for R-studio Server in the main menu of the Snap4City. R Studio Server can be located on Host with NVIDIA Tensor Flow, as well as on other parallel machines. There is a commercial version of R Studio on which it is possible to have SSO with the rest of the platform., that is not commercialized by Snap4City but from an R  Studio third party provider.
  • Engager Server VM (please request): is an additional VM/Container with Engager tools which are based on DROOLS, for assessing user behavior condition from the data coming from mobile apps (typically on MyKPI), and reacting to the changes of status by providing suggestions, recommendation, engagements, questionnaires, soudages, etc. They can be planned and tuned on the basis of the user profile and behavior.
  • advanced Data analytics (please request) that may have a cost such as: traffic reconstruction, environmental data predictions, parking predictions, wifi people flow predictions, offer vs demand of mobility, etc.
  • Python server (please request): a VM/Container for developers for creating Python based processes which are deployed as Containers into the platform and made accessible as MicroServices for IOT Application, or direcly scheduled by DISCES.
  • Java server (please request): a Virtual machine for developers for creating Java based processes which are deployed as Containers into the platform and made accessible as MicroServices for IOT Application, or direcly scheduled by DISCES.
  • your legacy tools can be connected as well.

See DCL: DataCity-Large Structure and Configuration

 

E: ExtensiveCity) A Large size Smart City with smart applications on cloud and a number of IOT brokers, large volume of data streams entering in the cloud

So that you can pass from configuration D to E managing a Large Scale Smart Cities maintaining the single MCLSCount and a cluster of IOTDSES for the data shadow of big data: billions and billions of data according to the number of VM/Container you are allocating, not problem on scaling. So that Configuration D with elastic search cluster is called E: ExtensiveCity. You may continue to use IOT Edge even if they have not been represented in the picture.

F: FullPlatform) For Huge scale Smart City with a Huge number of smart applications and processes on cloud (thousands), any number of IOT brokers

A large scale deploy for several millions of messages per day entering into the cloud can be obtained by using the same components as described in the following figure. This solution add to the E configuration described above a cluster for managing containers including: IOT Applications, Data Analytic, ETL processes, Web Scraping Processes, R Studio processes, Java, etc. created with MCLSCount and NCLSCount. The resulting configuration is for very huge scale smart City and also capable to manage a large Living Lab Community of developers with their personal IOT App, processes of several different kinds. You may continue to use IOT Edge even if they have not been represented in the picture.

F-LivingLab: FullPlatform + LivingLab) For Huge scale Smart City as Full Platform plus Living lab Support

In this case, the schema is practically identical of the F:FullPlatform but with the addition of:

  • LivingLabPortal (please request): a VM/Container including Drupal with all modules intalled for managing the CRM of  the Living Lab, registration and user management, multiple organizations, multiple groups, articles and blogs, comments on alll products, automatic alignement of LDAP users with those of Drupal, Single Sign On, SSO, among  the users of Snap4City and those of Drupal, monitoring of user activity, GDPR support, SOLR index of queries, reports, indexing, etc.

For the DRUPAL modules to be installed to set up the Living Lab support we suggest:

  • install drupal 7
  • take and install our DRUPAL 7 Snap4City Custom modules from: https://github.com/disit/snap4city/tree/master/Drupal-Modules :
    • Simple LDAP (7.x-1.4) (enable Role and User submodules)
    • OpenID Connect (7.x-1.0-beta8+2-dev)
    • OpenIDConnectPatch
    • Organic Groups (7.x-2.10)
    • OrganicGroupSimpleLdap
    • ProfileSnap4City
    • many other DRUPAL standard modules can be installed for: editing content, indexing and search, etc. etc.
  • Setup drupal:
    • add Content type "Organization" with machine name "disitorg" and set it as Group in the Organic groups section
    • add Content type "Organization group" with machne name "disitgroup" and set it as Group and Group content" ...

 

Strongly suggested to install also optional or similar legacy services such as:

What you can  exploit from Https://www.snap4city.org community a set of On Line Services:

  • Help Desk and contacts;
  • updated documentation, test cases, videos, slides, tutorials, Swagger of API, and a community for discussing on them;
  • Virtual Assistant to support the developers in getting the needed documentation;(please request);
  • Development tools as: LOG (linked open graph) also called Knowledge Base Graphs, and Knowledge Base Queries in SPARQL;
  • Several Knowledge Bases of cities and regions: Toscana and all its cities and provinces, Helsinki, Antwerp, Sardegna and many cities, Garda, etc.; 
  • an environment in which the new tools (even your if you propose) are tested and incubated for all;
  • access to DISIT.org Https://www.disit.org 
  • access to Km4City.org https://www.km4city.org 
  • ....

If you have special requests please ask to Snap4City via Contact form https://www.snap4city.org/3

Map of features per VM/Container  see also (Source Code and licenses)

Features included into the VM Appliance: (you can update from Github/disit)

SNAP4CITYMAIN

VMVMVMVMETL

IOT Edge

IOTOBSF KBSSM MCLSCont
NCLSCont
IOTDSES GISGeoserver

 

VM

VMVM

VM, img, Src

VM

VM VM VM VM
                 

Dashboard Builder and its interfaces, wizard

X

 

 

 

       

Data Inspector

X

 

 

 

       

KPI Support, POI support

X

 

 

 

       

SSO: Authentication, Authorization, Roles

X

 

 

 

       

Notificator

X

 

 

 

       

IOT Application = Node-RED + Snap4City MicroServices

X

 

X (Windows, Linux, Raspberry Pi, Android)

 

       

External Services manager

X

 

 

 

       

MicroApplications Manager

X

 

 

 

       

Menu Profiler

X

 

 

 

       

GDPR Data Type Ownership manager

X

 

 

 

       

ETL editor, EDL Development Environment

 

X

 

 

       

DISCES: scheduler

 

X

 

 

       

IOT Broker: Orion Broker + Security Filter

 

 

 

(X)

       
IOT Device Management: IOT Directory (includes IOT brokers management) X              
Resource Manager: auditing, market place X              
Synoptics and custom widgets X (since version 1.5)              
User Statistics and self assessment X (since version 1.6)              
Micro Applications X              
ServiceMap, ServiceMap3D, SuperServiceMap, Knowledge Base, Virtuoso, SmartCity API, from OSM to SM Knowledge base, semantic index: servicemap (includes Smart City API, Swagger), WFS API for GIS integration         X      
IOT Application Manager: Mesos, Marathon, APIs, DISCES-EM, 
IOT Application = Node-RED + Snap4City MicroServices; Web Scraping, DataAnalytics, ETL processes, Java processes, etc. 
(X: only 1 IOT App)         X    
IOT Data Shadow: NIFI + Elastic Search (it can be used as a starting point for clustered NIFI and ES creating large storage)             X  
GIS, GeoServer, heatmap server, heatmap manager, WFS and WMS support         (X: WFS)     X

Main links for the main components. All source code is on GITHUB/DISIT https://github.com/disit see licenses for all: https://www.snap4city.org/7

If you need specific VM/Appliance for the following components please ask to snap4city@disit.org providing a description of your purpose.

Architecture

https://www.snap4city.org/222

https://www.snap4city.org/165

https://www.snap4city.org/501

https://www.snap4city.org/223 

Synoptics + Graphic Custom widgets: SVG editor and tools

https://www.snap4city.org/504

Knowledge base, semantic index: servicemap (includes Smart City API, Swagger)

https://www.snap4city.org/19

https://www.snap4city.org/183

https://www.snap4city.org/181  

IOT Device Management: IOT Directory (includes IOT brokers management)

https://www.snap4city.org/474

https://www.snap4city.org/228

IOT Data Shadow: NIFI Cluster + Elastic Search (large storage)

https://www.snap4city.org/229

https://www.snap4city.org/512

Resource Manager: auditing, market place

https://www.snap4city.org/205

IOT Application Manager: Mesos, Marathon, DISCES-EM, APIs

https://www.snap4city.org/233

https://www.snap4city.org/232

DataSetManager: DataGate + CKAN

https://www.snap4city.org/26

https://www.snap4city.org/185

Data Analytics: RStudio + Plumber

https://www.snap4city.org/162

https://www.snap4city.org/196

https://www.snap4city.org/25

WEB scraping: Portia in connection with IOT Application Manager

https://www.snap4city.org/478

https://www.snap4city.org/431

Monitoring tools: depending on the modules adopted

It is a distributed feature

Chat room on Dashboard: Rocket

https://www.snap4city.org/515 

Web App: the Series “in a Snap” App, MicroApplications

https://www.snap4city.org/493

https://www.snap4city.org/492

ETL Data ingestion and storage: HBASE, Phoenix, DISCES (huge storage)

https://www.snap4city.org/187

https://www.snap4city.org/192

https://www.snap4city.org/24

Engager, user engagement

https://www.snap4city.org/486

https://www.snap4city.org/472

 

Comments

roottooladmin1's picture

Please provide us  your comments and suggestion to go further in providing you this kind of solutions ready to use!!! Snap4City Team.

Hello,
I'm currently having an issue while trying to create an entity through CLI. I was expecting broker to pass the message (entity) to nifi for ingestion but it didn't.

Installation:
I have deployed IOTOBSF and IOTDSES (nifi, elasticsearch and kibana) on the same VM.
I have subscribed nifi to broker, and i created an entity. However i am getting this warning in the docker logs for orion:

orion_1              | time=2021-08-24T07:05:00.836Z | lvl=WARN | corr=9918d6ae-04a9-11ec-a28b-0242ac120003; cbnotif=1 | trans=1629725321-127-00000000009 | from=172.18.0.1 | srv=<none> | subsrv=/ | comp=Orion | op=AlarmManager.cpp[361]:notificationError | msg=Raising alarm NotificationError nifi:1030/ingestngsi: notification failure for sender-thread: Couldn't resolve host name

I don't understand why is that a problem since my hostfile contains this line: <my_host_ip>    iotobsf iotdses-address nifi kibana elastcisearch

After, i replaced hostname with my_host_ip for the subscription and i got the following warning:

orion_1              | time=2021-08-24T08:50:31.304Z | lvl=WARN | corr=567a8324-04b8-11ec-8961-0242ac120003; cbnotif=1 | trans=1629725321-127-00000000052 | from=172.18.0.1 | srv=<none> | subsrv=/ | comp=Orion | op=AlarmManager.cpp[361]:notificationError | msg=Raising alarm NotificationError <my_host_ip>:1030/ingestngsi: notification failure for sender-thread: Couldn't connect to server

I also connected nifi container to the pre-existed iotobsf_default network so the containers could communicate with each other. Or was this unnecessary, since ports are exposed?

Any help will be highly appreciated.
Thanks in advance.

roottooladmin1's picture

we are answering directly on email to you on this topic,

paolo 

roottooladmin1's picture

"I have deployed IOTOBSF and IOTDSES (nifi, elasticsearch and kibana) on the same VM."

They are single  VM so that it is not clear how you have composed into the  same VM?