Test Case Title
|TC6.10 - ETL Applications using multiple protocols, and formats for files and to calling services using REST and WS
Snap4City, provide support for real time analytics which is performed by using a set of services. One of these is the possibility to:
Using a PC or Mobile with a web browser
See the previous TC on the same topics of ETL editing before than this one
The following functionalities are available only for specific Snap4city users with specific privileges.
Expected successful result
Create an ETL process for data transformation directly in the snap4city context
- Login on the snap4city platform
- Open the VM from the snap4city home page
- Create the ETL processes DIRECTLY from the Virtual Machine in the snap4city portal
Users roles for which this test case is available: All
User used to describe this Examples: ‘snap4city’ (role: AreaManager)
- Go to the snap4city home page
- Make the login
- I can see the snap4city home page
- Now you can select one of the functionalities on the left panel
- Note that the list of the functionalities depends on the user role (and on the related permissions).
- See the previous TC on the same topics of ETL editing before than this one
SAMPLE: Florence_School_Canteen_FTP (FTP)
- Follow the same steps viewed in the previous case of ‘Helsinki_youth_subsidies_XLS’
- Open the main job:
An FTP step is present, in this case the data come from the Disit FTP and are downloaded in the Virtual machine
Fig.: Details on the FTP step.
Example 6: To Put the ETL in execution, on demand or periodically, on the back office via DISCES, can be used the Process Loader. It is possible to upload the ETL processes on the Process Loader and scheduling them, and to monitor.
- Go to the Process Loader home page (https://processloader.snap4city.org) and make your login (with the credentials in section 3.3), the you can see a set of ‘your’ ETL already in execution in the scheduler
Fig. z – The snap4city ETLs scheduled on DISCES.
- To Monitor, click on the process name to view the scheduler page containing other information on all the processes running on it.
Fig. x – The snap4city ETLs scheduled on DISCES.
Example 7: To Getting results from the process executed
- See previous points (1, 4). Another modality of verification can be a Database check. For example:
- The ‘Snap4city/Electric_vehicle_charging_kmz’ ETL upload data from an external web server, elaborate data, send data on HBase (thanks to Phoenix). It is possible to verify this, opening the transformation, click the ‘run’ button and look at the logs
Fig. y – New ETL: blocks.
- This ETL writes on our local HBase, for us it is possible to see the results making queries on DB:
Fig. Z – Data on HBase (Phoenix - Electric_vehicle_charging dataset).