1. Home
  2. TC10.3 - Architecture of Snap4City, deploy of major component and cloud details, Distributed, Decoupled, Robust Snap4City Solution

TC10.3 - Architecture of Snap4City, deploy of major component and cloud details, Distributed, Decoupled, Robust Snap4City Solution

×

Warning message

You can't delete this newsletter because it has not been sent to all its subscribers.

Test Case Title

TC10.3 - Architecture of Snap4City, deploy of major component and cloud details, Distributed, Decoupled, Robust Snap4City Solution

Goal

The architecture has been designed fully distributed, avoiding single point of failure, and capable to react at the problems, informing and recovering from failure.

All the resources are monitored and managed in a distributed manner.

All the solutions in place have been designed to have low costs of maintenance, since all HD and structures are redounded in terms of Power, memory, storage, network, cpu, etc.

Prerequisites

Using a PC or Mobile with a web browser.

Expected successful result

Verify the structure of the architecture. It is very difficult to give you an operative TC in this case. On the other hand, all the mechanisms in place to monitor the solution and react are a good point.

Steps

  

see the new version of the installation and deploy procedures   

The architecture has been described in TC10.1 and TC10.2.

Snap4City platform is developed as a distributed and parallel architecture, where all components are decoupled, hot replaceable, and balanced in workload. This approach permits the addition and removal of components at real-time without downtime since all elements is placed on Containers. Each single Service area is allocated on separate VM and/or container minimising dependencies. The solution is based on a set of distributed service subsystems with clear roles. All the components are part of the distributed architecture on cloud and have been designed to preserve scalability, robustness, interoperability and performance on cloud.

In this TC, the deployment solution is described. Snap4City solution is robust with respect to failures in the backend and frontend. The prototype ensures 100% (or near) availability for its services, e.g. zero-downtime

The Snap4City is located in the same data center of Km4City which is continuously assessed by E015 in terms of Availability of  the service for Km4City Smart City API. The rate is almost over 99,88% of uptime.

To increase the uptime, a second datacenter is taken during the third phase.

 

Cloud Management of Resources

The whole solution is deployed on VM which are on cloud using VMware HA (high availability)/ DRS (Dynamic Resource Scheduling)/ FT (fault tolerant) approach on VMware, on hosts with double power and multiple network cards. 

  • HA: High Availability means that the VM are deployed on cluster of Host, and they are located on a NAS while their execution is on one of the Host of the Cluster. The advantage of the HA is the fact that if one of the Host would fault, the VM are automatically put in execution on one of the others.
  • DRS: Dynamic Resource Scheduling, means that the resources allocated on VM are managed dynamically. Each VM has a reservation and a limit, and the solution is automatically managing the vertical scale on the basis of the requests.
  • FT, Fault Tolerant: means that the VM are executed in a replicated cluster in which each VM has a second VM already allocated and ready to be activated running on a different Host of the cluster. FT has a sense for front end VM which are not in balance since VM in balance already support the FT at architectural level, so that is a VM get off the other is exploited by the Balances. The FT has also sense for the balancers.
     
    Therefore, in Snap4City solution
  • Resources of VM executing Containers is monitored by means of VCenter by calling VSphere and by the Resource Monitor, calling directly Mesos and VM, see Https://resdash.snap4city.org already demonstrated in a previous TC.
  • Resource allocator decides where to allocate the Containers or ETL processes (if they are not “containered”) Scheduling is based on planned execution (periodic and aperiodic) and periodic schedule optimization, by DISCES.
  • Vertical scaling at level of VM: VMs have limited resources reserved at the boot while the rest, up to the limit, is allocated when requested by the processes into the VM. This implies the possibility of making overallocation on resources, and vertical scaling.
    • scaling at level of VM is also performed moving the VM on different hosts when the resources available on the current Host are not sufficient for executing all the VM. This approach is provided by DRS solution at level of Vcenter.
  • Horizontal Elastic management at level of VM is performed by
    • increasing the number of VMs in the clusters according to the requested resources for example IOT Application, ETL in containers. These decisions are taken by the DISCES-EM on the basis of the requests of resources and knowing the amount of resource available.
    • allocating new VM to joined them at the cluster managing the Containers to execute ETL processes, DataAnalytics and IOT Applications
    • allocating VMs and/or Containers for the Front End Processes, such as the Web page of the portal, the web pages for the dashboards, etc.

 
The hardware support of the solution is based on High availability, HA, DRS, FT is

  • based on Hosts which have 40-42 cores each (total of 336 cores), 256 Gbyte RAM diskless (4 Tbyte RAM). double Power each, multiple NIC at 10Gbps each;
  • based on Storage NAS with RAID60 and HOTspare, on HD 15K and on SSD. The NAS server has double power and double RAID controller, each NAS has two banks in RAID60 with 12 HD/SSD each, size of them are 400-500 Gbyte for the top front end (15K/SSD), and 1.8 TByte HD at 10K for the mid. Total of 40Tbyte hot plug.
  • supported by multiple network switches with double power supply of 10Gbps and 1Gbps for kernel management;
  • protected by a Frontier FireWall with high performance switch, and including a FW also from the back office and the Lab’
  • deployed on a cluster of DELL hosts in HA (and FT when needed), automated Vmotion and failover, and workload balancing (DRS);
  • managed at level of cloud with VMware 6.5 with ESX from 5.5 to 6.5 on SSD in RAID1;
  • allocated into a controlled datacenter room with multiple sensor system: temperature, humidity, gas, flame, smoke, delta temperature, etc. with automated notification activated;
  • allocated on hosts supported by multiple UPS APC 2 for each RACK all of them network managed, with automated notifications activated, and auto-shutdown in the case of disaster;
  • Allocated on shared cluster of 7 Hosts and 3x2 NAS (as described above) plus a number of Host for sandbox and allocating multiple experiments.

 

Resilience and Disaster recovery

The Resilience aspect for disaster recovery is supported by:

  • automated replica of VM with 7 days versioning on a second Storage daily in the same building
    • every X hours in the running phase 3, in which a second data center would be allocated in a different building
  • automated notifications of replica problems.

 

Deploy architecture

As depicted in the following figure all the components of Snap4City architecture are deployed in VM (represented as a box, according to the VSphere model and convention), and Docker Containers. In particular, the allocation of VM has been performed as follows:

  • Back office VMs for computationally executing Applications, ETL and Data Analytics processes are managed by Marathon/Mesos plus a balancer and a cloud management system for elastic management of VM and Containers, both levels.
    • The set of Mesos/Docker master is managed by three Marathon/Mesos/Zookeeper in cluster. The tools for balancing, scheduling and elastic computing are allocated on the same VMs.
    • The set of Mesos/Docker clients/slaves has to be larger than 2 to have a tolerance to start. When the Containers are allocated the number of VM is proportionally managed, producing them by a template, and shutting down them when they are not needed. Are all identical since are produce by a common template, each of them has 6 Cores of 2.2Ghz, and 17000 Gbyte RAM, no limitation on HD. The Memory, and CPU are dynamically assigned by the VM manager in DRS
    • Three container images kinds are put at disposal of the launcher with 3 main configurations
  • IOT Services are allocated on multiple VMs and they may be located on Containers as well. This means to have multiple IOT brokers, you can see them listed in the IOT Directory.
    • One VM for each IOT Orion Broker instance including its own Proxy Filter Security to wrap the access and guarantee the secure and authenticated access to NGSI protocol and compatible with KeyCloak/LDAP.
    • One VM for each of the other IOT Brokers: Mosquitto, RabbitMQ, etc.
  • Knowledge Base and related storage services are located on fault tolerant NAS with double Power and RAID, multiple connections, etc. (e.g., PowerVault of Dell).
    • The NAS hosts VMs in HA, DRS, FT, balance for the front end.
  • The front end VMs contain: Snap4City.org portal, dashboard, Notificator, Resource Manager, Data Gate, etc.
    • The Dashboards and related storage services are located on a cluster of traditional SQL database: MySQL.
    • Also in this case, the NAS hosts VMs in HA, FT, balance for the front end.
  • The Access Services are on NAS hosts VMs in HA, FT, in balance. Belong to the Access Services: KeyCloak, LDAP.
  • Development Sandbox: they are typically VM which can be accessed from outside for:
    • Data Processing Development Environment: for developing processes and data analytics/data transformations using: ETL, Java, Python, etc. It is presently a single VM that can be replicated to provide this feature to specific users, or groups of them. According to our experience we can typically provide access to the same VM at multiple users. The usage is typically sporadic. The developers develop their Data Processes on their VM on their premise, when they need to finalize for the real data access on the Smart City Data they use the Data Processing Development Environment.
    • R studio: for developing R Studio programme, using or not the R-paralle or Tensor Flow. In the case you can test, you are accessing to a strongly powerful Host with 16 CPU 128Gbyte Ram, TITAN Xp 3800 GPU, and thus accessible and usable by multiple users at the same time. The same Host provides testing processes as MicroServices.
    • These processes are finalized to be back office periodic processes put in execution by DISCES
       

       
  • The indexing processes and index data for AMMA, ResDash and DevDash are allocated on VM for the data collection and indexing process and on 4 VMs for hosting the sharded index fully replicated on all 4 VM. Each of them has 24Gbyte of RAM and 8cores (this part is not reported in the figure)
     

The present deployed architecture consists of:

The following values refer to an infrastructure that is presently managing more than 2 millions of new complex data structure per day coming in push and pull via different protocols and in different formats from about 400 sources. 

  • IOT Applications management and scalability
    • 16xVM with Marathon, Mesos, Docker containers with NodeRED, for million of messages per minute
    • 3xVM Masters with Zookeeper
  • IOT devices and their management
    • 2xVM: Orion Broker A, Orion broker B
    • VM: Mosquitto IOT Broker (in Florence and Milan)
    • VM: RabbitMQ IOT Broker (in Florence and Milan)
    • A number of Raspberry Pi with NodeRED
    • A number of sensors of different kinds
  • Dashboards
  • Data Storage, Knowledge Base, KB, with:
  • Developer Dashboard, AMMA, Resource Manager Dashboard
    • VM: IOT/ETL real-time data collected on Hbase via NIFI, and indexed on SOLR (only for administrators when needed)
    • 4xVM Sharded Elastic Search, with Kibana
    • Host for R Studio: data analytics as a service, Tensor Flow, TITAN Xp (only for data analysis development, in containers or scheduled in processes)
  • General Services:
    • VM: Snap4City.org portal and download web site, virtual hosts
    • VM: DataGate with added Helsinki and Antwerp groups
    • VM: Resource Manager with stand-alone testing DISCES
    • VM: Dedicated SDK with VM for data processing + NodeRED + R Studio + DISCES
    • KeyCloak, LDAP common with other services
    • HTTPS: certifier common with other services
    • NTP: in common with other services for clock sync
  • Cloud
    • VSphere Vcenter 6.5, with multiple ESX for VM
    • VEEAM for Autoreplica of VM, for disaster recovery
    • VM on a cluster with HA DRS
    • 3x (NAS with 12 SSD 600 Gbyte, and 12 HD 15Krps)
    • Intranet Network 10Gps copper
    • Firewall separating the front office and the back office networks

The principles are reported in the following points:

  • All the major components are installed in separated Virtual Machines, VM, or containers in execution;
  • All VM's are set to auto-restart, that means, at VM boot, it automatically starts all assigned services.
  • All the VMs are configured in VMware environment and are ready as Templates. So that new instances can easily be added by recalling from the VCenter bootstrap.
  • All the VM are in HA DRS, a part of them are in HA DRS FT
  • All the Hosts have double power supply, 6 network cards
  • All the NAS have double RAID, RAID 60 on HD hot plug
  • All the power including network are under some UPS, APC 5000KWA
  • The cluster is located into the DISIT DataCenter with: 4 tvcameras, 3 sensors for Gas, 4 sensors for temperature, 4 sensors for humidity, 2 sensors for IR, 3 sensors for smoke, 2 sensors for delta temperature, redundant air conditioning 3 x 22.000 btu.
  • the solution is monitored in terms of Smart City API availability from the Lombaria Regione E015, getting an availability higher than 99% 

Network configuration
Network topology, including the different network protocols. The figure mainly describes the structure of the network into the cloud. While the connection with the IOT is performed via IOT gateways independently on the protocols. Please note the presence of redundant network, of firewalls, and of the internal network for the NAS the scalable back office (VM box), while the frontend may have balancer in the case of heavy workload otherwise a simple redundant solution is enough.
 

  • Share