HOW TO: Deploy/Install your Snap4City Solution on private or public Clouds, VM with Containers

Version 1.4 of 06-09-2021

The Docker Config Generator x Snap4 Tools is in beta testing and it is presently accessible from the main menu under "Deploy and Installation" only for AreaManagers or higher roles in Organizations such as: DISIT and few other organizations.

This  web page aims to prepare you entering into the Docker Config Generator Tool, and to provide you the minimal suggested info of the VMs involved in the installation. 
This page is describing a tool for generating installation files for a number of different configuration models each of which with a set of parameters.
The main idea of the Snap4 Configuration Tool is to:

  • allow you to select a configuration on the basis of the purpose
  • provide you a wizard that is going to ask you information such as: IP, names, IDs, number of features interested 
  • generate for you a set of installation files to perform an almost automated configuration based on Containers on your VMs on any cloud/servers
  • save the installation files to be reused by you in other installations, also modifying some parameters.

The installation files are generated for a number of proposed configurations with a number of scalable parameters.
Depending on the configuration a different number of VMs will be suggested and the configurations will be provided for each VM.

  • We suggest you to use Debian distribution for the VM. You can get the ISO from https://www.debian.org We also suggest to execute your VM on cloud environment as Vmware or similar.
  • in each VM, the docker and docker-compose have to be installed, please verify their correct installation.
  • each VM should have at least 10 Gbyte of RAM, more than 50 Gbyte of HD, but this is going to depend  on the data you would like to have, and 8 cores or virtual cores. The precise size of the VM (in terms of Memory, CPU, Storage) can be computed only at the end of the Docker Config Generator process when all needed information for their computation will be provided by you to the tool, and when the number of VM are also known.
  • The VM have to provide a network connection with the IP  that you have to provide in the file generation process. If you execute the VM into VMWare player, the VM network has to be bridget and not NAT.

Once you decided for a Configuration Model, before entering on the Snap4 Docker Config Generator you need to have the IP addresses of those VMs.
if you are not capable to complete the Docker Config Generator process for the lack of data, the tool is providing you a CODE/Token to enter egain and restart from the configuraiton you performed.
Please note that, the Configurations are anonymous, so that you can use the Configurator without exposing your email or nickname.
Each configuration is saved as a TOKEN and you can access again to it any time, to download the produced configuration/deploy files, and to create other configurations starting from those you have already generated, by changing the detailed parameters but not the size of the configurations in terms of X, Y, Z, K. 

The Configuration Models described in the following are the evolution of the DataCitySmall/Large configurations which is accessible as installation and guidelines in other web pages https://www.snap4city.org/471

For each Configuration Model, we are providing a description of the typical usage and the meaning of the parameters.
The meaning of X, Y, Z of the Configuration Models are identiical in all models:

  • X: is the number of IoT Applications, which are Node-RED containers, X>=1
  • Y: is the number of IoT Orion Broker V2 FIWARE, Y>=1
  • Z: is the number of Knowledge Bases, KB, Service Maps created into the solution. Z>=1; they are typically 1:1 with the Organizations. You can have multiple KBs for the same org. as well. Please note that the KBs are the providers of Smart City API, and Super Smart City API.

The NEW Configuration Models are:

  • Micro X: configurations suitable for solutions for small verticals and industries, single VM, see in the following for the details.
    • it is more complete than the Alone configuration of https://www.snap4city.org/471
    • VM main all in:
      • ​10-16 Gbyte of RAM, more than 50-200 Gbyte of HD (this depends on data size you would like to have, we suggest 500 GByte in thin provisioning), and 8 cores or virtual cores, faster than 2GHz each.
      • larger sizes may depend on the number of users/data you have
  • Normal X,Y: configurations with multple  IoT App and Multiple IoT Brokers. They are on 2 VM, the first VM contains almost  all, while the second only the Brokers and IoT Apps 
    • it is more complete than the Basic configuration of https://www.snap4city.org/471
    • VM1 MAIN: contains the main as Dashboards services, Knowlege Base, Resource Manager, Authentication and authorization, Synoptics, WS, etc.
      • ​10-16 Gbyte of RAM, more than 50-200 Gbyte of HD (this depends on data size you would like to have, we suggest 500 GByte in thin provisioning), and 8 cores or virtual cores, faster than 2GHz each.
      • larger sizes may depend on the number of users/data you have
    • VM2 IOT Apps and Brokers: X IoT Apps, Node-RED, MicroServices; and Y IoT Brokers. 
      • ​16-24 Gbyte of RAM, more than 200 Gbyte of HD (this depends on data size you would like to have, we suggest 500 GByte in thin provisioning), and 8 cores or virtual cores, faster than 2GHz each.
      • please note that each IoT App take about 100 Mbyte of RAM and 10% of a core, in a VM of 24Gbyte, 6 Cores you can have 70 IoT Apps, 100 Gbytes Storage in thin provisioning
      • an IoT Orion Broker can work with 6 Gbyte Ram, 2 Cores, 50 Gbytes.
  • Small X,Y: configurations with multple IoT App and multiple IoT Brokers. It is suitable for solutions in which the storage is growing and thus can be managed into a separate VM, and may be clustered later on.
    They are on 4 VM
    • VM1 MAIN: contains the main as Dashboards services, Knowlege Base, Resource Manager, Synoptics, WS, etc.
      • ​10-16 Gbyte of RAM, more than 50-200 Gbyte of HD (this depends on data size you would like to have, we suggest 500 GByte in thin provisioning), and 8 cores or virtual cores, faster than 2GHz each.
      • larger sizes may depend on the number of users you have
    • VM2: authentication and authorisation: LDAP, KeyCloak, ....
      • ​10-16 Gbyte of RAM, more than 50-100 Gbyte of HD and 8 cores or virtual cores, faster than 2000GHz.
    • VM3 STORAGE: NIFI, Open Distro for Elastic Sarch/Kibana,
      • ​24-32 Gbyte of RAM, more than 200-500 Gbyte of HD (this depends on data size you would like to have, we suggest 500-1000 GByte in thin provisioning), and 8 cores or virtual cores, faster than 2GHz each.
      • larger sizes may depend on the number of data you have, please note that you can replicate and cluster them later
    • VM4 IOT APPs and Brokers: X IoT Apps, Node-RED, MicroServices; and Y IoT Brokers. 
      • ​16-24 Gbyte of RAM, more than 200 Gbyte of HD (this depends on data size you would like to have, we suggest 500 GByte in thin provisioning), and 8 cores or virtual cores, faster than 2GHz each.
      • please note that each IoT App take about 100 Mbyte of RAM and 10% of a core, in a VM of 24Gbyte, 6 Cores you can have 70 IoT Apps, 100 Gbytes Storage in thin provisioning
      • an IoT Orion Broker can work with 6 Gbyte Ram, 2 Cores, 50 Gbytes.
  • DataCitySmall X,Y,Z: configurations with multple IoT App, multiple IoT Brokers, and multiple KB/ServiceMap. It is suitable for more scalable solutions in which the storage is growing and thus can be managed into a separate VM, also IoT App can be managed separately, such as also the IoT Brokers. It is the perfect starting point for replicating VM for storage, Brokers and IoT according to the needs, and thus for supporting MultiTenant solutions.
    They are on 6 VM
    • VM1 MAIN: contains the main as Dashboards services, Resource Manager, Synoptics, WS, etc.
      • ​10-16 Gbyte of RAM, more than 50-200 Gbyte of HD (this depends on data size you would like to have, we suggest 500 GByte in thin provisioning), and 8 cores or virtual cores, faster than 2GHz each
      • larger sizes may depend on the number of users you have
    • VM2: authentication and authorisation: LDAP, KeyCloak, ....
      • ​10-16 Gbyte of RAM, more than 50-100 Gbyte of HD and 8 cores or virtual cores, faster than 2000GHz.
    • VM3 STORAGE: NIFI, Open Distro for Elastic Sarch/Kibana,
      • ​24-32 Gbyte of RAM, more than 200-500 Gbyte of HD (this depends on data size you would like to have, we suggest 500-1000 GByte in thin provisioning), and 8 cores or virtual cores, faster than 2GHz each.
      • larger sizes may depend on the number of data you have, please note that you can replicate and cluster them later
    • VM4 IOT APPs: X IoT Apps, Node-RED, MicroServices.
      • ​16-24 Gbyte of RAM, more than 200 Gbyte of HD (this depends on data size you would like to have, we suggest 500 GByte in thin provisioning), and 8 cores or virtual cores, faster than 2GHz each.
      • please note that each IoT App take about 100 Mbyte of RAM and 10% of a core, in a VM of 24Gbyte, 6 Cores you can have 70 IoT Apps, 100 Gbytes Storage in thin provisioning
    • VM5 IOT BROKERS: IoT Brokers, secure filter, etc.
      • ​8-12 Gbyte of RAM, more than 100 Gbyte of HD (this depends on data size you would like to have, we suggest 100-200 GByte in thin provisioning), and 8 cores or virtual cores, faster than 2GHz each.
      • please note that each IoT Orion Broker can work with 6 Gbyte Ram, 2-4 Cores, 50 Gbytes of storage.
    • VM6 Knowledge Bases: Z KB, ServiceMap, one for each organization, they can be federated each other. 
      • ​24-32 Gbyte of RAM, more than 200-500 Gbyte of HD (this depends on data size you would like to have, we suggest 200-400 GByte in thin provisioning), and 8 cores or virtual cores, faster than 2GHz each.
      • larger sizes may depend on the number of data you have, please note that you can replicate and federate later
  • DataCityMid X, Y, Z: is a mid range solution for medium size cities, with 
    • VM1 MAIN: contains the main as Dashboards services, Resource Manager, Synoptics, WS, etc.
      • ​10-16 Gbyte of RAM, more than 50-200 Gbyte of HD (this depends on data size you would like to have, we suggest 500 GByte in thin provisioning), and 8 cores or virtual cores, faster than 2GHz each
      • larger sizes may depend on the number of users you have
    • VM2: authentication and authorisation: LDAP, KeyCloak, ....
      • ​10-16 Gbyte of RAM, more than 50-100 Gbyte of HD and 8 cores or virtual cores, faster than 2000GHz.
    • a number of VM in cluster for managing STORAGE: NIFI, Open Distro for Elastic Sarch/Kibana, each of them
      • ​24-32 Gbyte of RAM, more than 200-500 Gbyte of HD (this depends on data size you would like to have, we suggest 500-1000 GByte in thin provisioning), and 8 cores or virtual cores, faster than 2GHz each.
      • larger sizes may depend on the number of data you have, please note that you can replicate and cluster them later
    • a number of VM in cluster for managing a large number of IoT Apps, Node-RED, MicroServices, each of them, in the ange of 70-100 for each VM
      • ​16-24 Gbyte of RAM, more than 200 Gbyte of HD (this depends on data size you would like to have, we suggest 500 GByte in thin provisioning), and 8 cores or virtual cores, faster than 2GHz each.
      • please note that each IoT App take about 100 Mbyte of RAM and 10% of a core, in a VM of 24Gbyte, 6 Cores you can have 70 IoT Apps, 100 Gbytes Storage in thin provisioning
    • a number of VM for managing IoT Brokers, secure filter, etc., each of them
      • ​8-12 Gbyte of RAM, more than 100 Gbyte of HD (this depends on data size you would like to have, we suggest 100-200 GByte in thin provisioning), and 8 cores or virtual cores, faster than 2GHz each.
      • please note that each IoT Orion Broker can work with 6 Gbyte Ram, 2-4 Cores, 50 Gbytes of storage.
    • a number of VM for managing Federated Organizations each of which with KB, ServiceMap, each of them
      • 24-32 Gbyte of RAM, more than 200-500 Gbyte of HD (this depends on data size you would like to have, we suggest 200-400 GByte in thin provisioning), and 8 cores or virtual cores, faster than 2GHz each.
      • larger sizes may depend on the number of data you have, please note that you can replicate and federate later
    • a VM for the Heatmap Manager, Traffic Flow Manager, GeoServer, etc. 
  • DataCityLarge

If you have problems with some configuration produced you have to contact us by providing the configuration TKEN you could get at the end of the generation of the configuration files. 

Micro X configurations

Micro X: for example you can realize Micro 3, Micro 1 configurations as explained in the following

  • The Micro X configurations are suitble for solutions for small verticals and industries. 

    • X represents the number of IoT Applications that will be available after the intallation
    • the X # of IoT-Apps is the first information requested
  • The Micro X includes all tools into a single VM so that before starting you have to provide the IP of the VM in which Dockers will have to be installed
    • the IP is the second information requested by the tool to you
  • Click on "Fill Placeholders" will allow you to provide the information needed to finalize the generation of the Docker compose files
    • the placeholders are divided into two categories:
      • mandatory to be provided: that are those that you have to provide to complete the installation
      • Default Fields: which can be changed if needed by selecting "Show default Fields"
    • The Mandatory fields are, for example:
      • Email address of the administrator/owner for events and info
      • sender name of the administrator/owner
      • SMTP server/host 
      • SMTP server port
      • Geo Location, GPS on map or values to be provided, of the unique IoT Orion Broker Fiware created in the solution
    • After setting the Mandatory fields, You can SAVE Configuration and a TOKEN is provided to reload the configuration and create a new one, or get again the file for deploy
    • At the same time a button for Download Configuration is activated. The ZIP file provided includes a subdirectory for each VM with the IP value, 1 VM in this case and thus one IP.
    • For example a VM Micro-2 contains:
  • DEPLOY: To execute the installation/configuration you need to have a VM (for example wth Debian) with Docker and Docker-Compose installed.
    • verify that your Docker and Docker-Compose are operative and well installed
    • Unapack the ZIP file in a directory. Move into the directory of the VM IP with command CD <IP>. There there a number of files that are going to be used in the successive steps, such as
      • docker-compose.yml 
      • apache-proxy.conf
      • post-setup.sh
      • setup.sh
    • Thus you can execute the installation by executing the command:
      • setup directories write permissions and sets vm.max_map_count=262144 for elasticsearch consider adding this option into /etc/sysctl.conf otherwise have to be set after each reboot
        • sudo ./setup.sh
      • deploy the containers
        • sudo docker-compose up
          • to perform a detached process: "docker-compose up -d"
      • setup virtuoso and elasticsearch (to be done only the first time but no problem if you repeat)
        • sudo ./post-setup.sh
  • VERIFY: The process downloads from internet all the docker images needed and deploy/configuration the whole entities and finally putting in execution all containers, for example resulting in 10 minutes the following log on your terminal:
    • Creating 192168125_elasticsearch_1  ... done
      Creating 192168125_virtuoso-kb_1    ... done
      Creating 192168125_synoptics_1      ... done
      Creating 192168125_iotapp-001_1     ... done
      Creating 192168125_nifi_1           ... done
      Creating 192168125_kibana_1         ... done
      Creating 192168125_dashboarddb_1    ... done
      Creating 192168125_dashboard-cron_1 ... done
      Creating 192168125_zookeeper_1      ... done
      Creating 192168125_ldap-server_1    ... done
      Creating 192168125_mongo-001_1      ... done
      Creating 192168125_iotapp-002_1     ... done
      Creating 192168125_keycloak_1          ... done
      Creating 192168125_myldap_1            ... done
      Creating 192168125_orion-001_1         ... done
      Creating 192168125_kafka_1                 ... done
      Creating 192168125_servicemap_1            ... done
      Creating 192168125_dashboard-backend_1 ... done
      Creating 192168125_personaldata_1          ... done
      Creating 192168125_wsserver_1              ... done
      Creating 192168125_dashboard_1             ... done
      Creating 192168125_orionbrokerfilter-001_1 ... done
    • the first time can be useful to put in exceution in that manner to see all the progress, while the successive execution could be performed in background by using the "-d" option for detached.
  • with command "sudo docker container ls" is possible to get the list of containers in execution on your VM. When they are you can something similarly to
    • CONTAINER ID   IMAGE                                   COMMAND                   CREATED          STATUS          PORTS                                                                                  NAMES
      393a3b7c5251   disitlab/orionbrokerfilter:v0           "catalina.sh run"         18 minutes ago   Up 3 minutes    8080/tcp, 0.0.0.0:8443->8443/tcp, :::8443->8443/tcp                                    192168125_orionbrokerfilter-001_1
      93b5ae48c721   disitlab/dashboard-builder:v4           "docker-php-entrypoi…"    18 minutes ago   Up 3 minutes    0.0.0.0:80->80/tcp, :::80->80/tcp                                                      192168125_dashboard_1
      c4bc8bcf46a2   disitlab/websocketserver:v2             "/bin/sh -c 'sh -c \"…"   18 minutes ago   Up 3 minutes    0.0.0.0:9000->9000/tcp, :::9000->9000/tcp                                              192168125_wsserver_1
      5a84e49ef4d2   disitlab/personaldata:v2                "catalina.sh run"         18 minutes ago   Up 3 minutes    0.0.0.0:8080->8080/tcp, :::8080->8080/tcp                                              192168125_personaldata_1
      62aaf5a4c307   disitlab/dashboard-backend:v0           "/bin/sh -c ./run.sh"     18 minutes ago   Up 3 minutes                                                                                           192168125_dashboard-backend_1
      4e16997f7ada   disitlab/servicemap:v1                  "catalina.sh run"         18 minutes ago   Up 3 minutes    0.0.0.0:8090->8080/tcp, :::8090->8080/tcp                                              192168125_servicemap_1
      4550755bcb2c   bitnami/kafka:latest                    "/opt/bitnami/script…"    18 minutes ago   Up 3 minutes    0.0.0.0:9092-9093->9092-9093/tcp, :::9092-9093->9092-9093/tcp                          192168125_kafka_1
      4af522fece3f   fiware/orion                            "/usr/bin/contextBro…"    18 minutes ago   Up 26 seconds   0.0.0.0:1026->1026/tcp, :::1026->1026/tcp                                              192168125_orion-001_1
      642887c8866a   disitlab/preconf-keycloak:v4            "/opt/jboss/tools/do…"    18 minutes ago   Up 3 minutes    8080/tcp, 0.0.0.0:8088->8088/tcp, :::8088->8088/tcp                                    192168125_keycloak_1
      519a878ebef9   osixia/phpldapadmin                     "/container/tool/run"     18 minutes ago   Up 3 minutes    80/tcp, 0.0.0.0:6443->443/tcp, :::6443->443/tcp                                        192168125_myldap_1
      7a534a7f8c79   disitlab/preconf-openldap:v2            "/container/tool/run…"    18 minutes ago   Up 3 minutes    0.0.0.0:389->389/tcp, :::389->389/tcp, 0.0.0.0:636->636/tcp, :::636->636/tcp           192168125_ldap-server_1
      ab7094bc0241   disitlab/snap4city-nodered-adv:v76      "npm start -- --user…"    18 minutes ago   Up 3 minutes    0.0.0.0:1881->1880/tcp, :::1881->1880/tcp                                              192168125_iotapp-002_1
      a13c022eac7a   mongo:3.6                               "docker-entrypoint.s…"    18 minutes ago   Up 3 minutes    27017/tcp                                                                              192168125_mongo-001_1
      e723387d73e3   zookeeper                               "/docker-entrypoint.…"    18 minutes ago   Up 3 minutes    2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 8080/tcp                192168125_zookeeper_1
      f68a87b7d63c   disitlab/dashboard-builder:v4           "docker-php-entrypoi…"    18 minutes ago   Up 3 minutes    80/tcp                                                                                 192168125_dashboard-cron_1
      3fc34e510bbf   mariadb                                 "docker-entrypoint.s…"    18 minutes ago   Up 3 minutes    0.0.0.0:3306->3306/tcp, :::3306->3306/tcp                                              192168125_dashboarddb_1
      3c040d2dfcbd   kibana:6.8.5                            "/usr/local/bin/kiba…"    18 minutes ago   Up 3 minutes    0.0.0.0:5601->5601/tcp, :::5601->5601/tcp                                              192168125_kibana_1
      c3fdb3aaf5e5   disitlab/snap4city-nodered-adv:v76      "npm start -- --user…"    18 minutes ago   Up 3 minutes    0.0.0.0:1880->1880/tcp, :::1880->1880/tcp                                              192168125_iotapp-001_1
      9dce05e7b47f   disitlab/synoptics:v0                   "docker-entrypoint.s…"    18 minutes ago   Up 3 minutes    0.0.0.0:3001-3002->3001-3002/tcp, :::3001-3002->3001-3002/tcp                          192168125_synoptics_1
      b1e4f9a746e8   tenforce/virtuoso:1.3.1-virtuoso7.2.1   "/bin/bash /virtuoso…"    18 minutes ago   Up 3 minutes    0.0.0.0:1111->1111/tcp, :::1111->1111/tcp, 0.0.0.0:8890->8890/tcp, :::8890->8890/tcp   192168125_virtuoso-kb_1
    • the resulting services will be at the following port on the main <IP>
      • 389, 636 for ldap server
      • 1880 for nodered application 1
      • 1881 for nodered application 2
      • 9000 for dashboard builder websocket server
      • 8080 for personaldata service
      • 8088 for keycloak access
      • 3306 to access to mysql (for debugging)
      • 6443 for phpldapadmin
      • 1026 for orion
      • 8443 for orionbrokerfilter
      • 9200 for elasticsearch
      • 5601 for kibana
      • 9090 for nifi
  • OPEN the Interface You can test the correct installation from another computer in the same network. This computer has to present in its etc/hosts file the statement
  • See https://github.com/disit/snap4city-docker/blob/master/DataCity-Small/README.md for

Normal X,Y configurations:

 

Small-X-Y configurations:

 

DataCitySmall-X-Y-Z configurations:

 

DataCityMid-X-Y-Z configurations:

 

DataCityLarge configurations: