ETL Development

ETL  tool. Creating ETL Processes.  (9'.21'' - training video)

The Data Processing Development Environment (VM) can be used for developing Snap4City Applications, ETL processes, Data Analytics Processes. It is a personal or shared SandBox for Developers that may use it:

When data transformations have to be developed for creating new services, we suggest to use ETL and not IOT Applications. The ETL can be developed with the approach described in this document. This approach of direct access to the VM on cloud results to be the most powerful solution and environment for developers. Thus, the developers acceding to the VM for developers can find an integrated environment in which they may develop:

  • ETL processes as above described for data transformation, via ETL developer Tools as Penthao Kettle, for data transformation which is capable to cope with any kind of protocols and formats; In addition, may smart city examples are also provided on GITHUB/DISIT and from Resource Managerhttps://processloader.snap4city.org/ 
  • Karma for developing XML automated mapping, for example for passing from data in MySQL table to RDF triples;
  • R stat, Python, java, for data analytics development, via installed in local
  • NodeRED, for Snap4City Application development, missing IOT directory, Snap4City MicroServices, also available in the VM while we suggest you use directly as IOT application of Snap4City
  • DISCES tool for local scheduling and test. While the real DISCES of the Snap4City back office is accessible only for ToolAdmin. Your process loaded into the local DISCES can be moved by the adimistrator when needed, under your request, for example.  

Once a process is developed it can be tested as scheduled process by using a local DISCES (a local stand alone instance of the Smart Cloud Scheduler). Any process, one tested in local can be loaded on the ProcessLoader to be submitted. So that it can be approved and put in execution in the  back office on the real DISCES automatically.

Please note that the following links could be accessible only for registered users.

On your premise:

User Manual to download:

Source code is  included into the VM, while the ETL source code is accessible on:

----------------------

DOCUMENTAZIONE precedente ed aggiuntiva

MACCHINA VIRTUALE, VMSDETL, GIA' PRONTA

VMSDETL, con Linux Ubuntu 14.04 (root: ubuntu, password: ubuntu)

La VM contiene un sistema di sviluppo preparato con i segunti tool che sono elencati per vostra conoscenza, ma che non devono essere installati, ma in certi casi avviati: