1. Home
  2. TC6.10 - ETL Applications using multiple protocols, and formats for files and to calling services using REST and WS

TC6.10 - ETL Applications using multiple protocols, and formats for files and to calling services using REST and WS

Test Case Title

 TC6.10 - ETL Applications using multiple protocols, and formats for files and to calling services using REST and WS

Goal

Snap4City, provide support for real time analytics which is performed by using a set of services. One of these is the possibility to:

create ETL processes capable to manage data coming from web services using multiple protocols, and formats for files and to calling services using REST and WS.

Prerequisites

Using a PC or Mobile with a web browser

See the previous TC on the same topics of ETL editing before than this one

The following functionalities are available only for specific Snap4city users with specific privileges.

Expected successful result

Create an ETL process for data transformation directly in the snap4city context

Steps

  

 

Users roles for which this test case is available: Area Manager, ToolAdmin, RootAdmin
Users roles for which the test case is not available: Manager
User used to describe the Example: ‘snap4city’

 

Examples:

  • Login on the snap4city platform
  • Open the VM from the snap4city home page
  • Create the ETL processes DIRECTLY from the Virtual Machine in the snap4city portal

Users roles for which this test case is available: All

User used to describe this Examples: ‘snap4city’ (role: AreaManager)

  • Go to the snap4city home page 
  • Make the login
  • I can see the snap4city home page
  • Now you can select one of the functionalities on the left panel
    • Note that the list of the functionalities depends on the user role (and on the related permissions).
  • See the previous TC on the same topics of ETL editing before than this one

 

Example 1

SAMPLE: Florence_School_Canteen_FTP (FTP)

  • Follow the same steps viewed in the previous case of ‘Helsinki_youth_subsidies_XLS’
  • Open the main job:

An FTP step is present, in this case the data come from the Disit FTP and are downloaded in the Virtual machine


Fig.: Details on the FTP step.

 

Example 6: To Put the ETL in execution, on demand or periodically, on the back office via DISCES, can be used the Process Loader. It is possible to upload the ETL processes on the Process Loader and scheduling them, and to monitor.


Fig. z – The snap4city ETLs scheduled on DISCES.

  • To Monitor, click on the process name to view the scheduler page containing other information on all the processes running on it.
     

    Fig. x – The snap4city ETLs scheduled on DISCES.

Example 7: To Getting results from the process executed

  • See previous points (1, 4). Another modality of verification can be a Database check. For example:
    • The ‘Snap4city/Electric_vehicle_charging_kmz’ ETL upload data from an external web server, elaborate data, send data on HBase (thanks to Phoenix). It is possible to verify this, opening the transformation, click the ‘run’ button and look at the logs


Fig. y – New ETL: blocks.
 

  • This ETL writes on our local HBase, for us it is possible to see the results making queries on DB:



Fig. Z – Data on HBase (Phoenix - Electric_vehicle_charging dataset).

  • Share
TC6.10 - ETL Applications using multiple protocols, and formats for files and to calling services using REST and WS | Snap4City

Error

×

Error message

  • Notice: Trying to get property of non-object in user_is_logged_in() (line 1576 of /var/www/html/drupal/modules/user/user.module).
  • Notice: Trying to get property of non-object in user_is_logged_in() (line 1576 of /var/www/html/drupal/modules/user/user.module).
  • Notice: Trying to get property of non-object in og_field_access() (line 578 of /var/www/html/drupal/sites/default/modules/og/og.module).
  • Notice: Trying to get property of non-object in user_access() (line 806 of /var/www/html/drupal/modules/user/user.module).
  • Notice: Trying to get property of non-object in user_access() (line 818 of /var/www/html/drupal/modules/user/user.module).
  • Notice: Trying to get property of non-object in user_access() (line 819 of /var/www/html/drupal/modules/user/user.module).
  • Notice: Trying to get property of non-object in user_access() (line 825 of /var/www/html/drupal/modules/user/user.module).
  • Notice: Trying to get property of non-object in user_access() (line 828 of /var/www/html/drupal/modules/user/user.module).
  • Warning: Cannot modify header information - headers already sent by (output started at /var/www/html/drupal/includes/common.inc:2791) in drupal_send_headers() (line 1499 of /var/www/html/drupal/includes/bootstrap.inc).
  • Error: __clone method called on non-object in og_user_access_entity() (line 2249 of /var/www/html/drupal/sites/default/modules/og/og.module).
The website encountered an unexpected error. Please try again later.