click the image to open the tool (for registered users), see the video
The DataGate/CKAN is a special module of CKAN open source portal for data set management and publication. CKAN supports the management of multiple collections, multiple organizations, etc. DataGate has been designed to collect/upload open data sets and process them automatically for data ingestion, field cleaning and regularizing, and for reconciliation of entities towards a Knowledge Base grounded on Km4City. The resulting cleaned data set is also accessible for further reuse and can be automatically published on the DataGate. It is a exceptional tool for shortening time to ingest and publish of datasets including for example:
- referral data, statistical data, etc.;
- open and not open data sets with geolocations;
- descriptive data, that just describe the entities of the city and provide for them metadata, GPS and/or shapes
DataGate is also federated with other CKAN open data repositories in the world via the mechanism of Harvest: https://datagate.snap4city.org/harvest . So that thousands of data sets are accessible via DataGate and can be published in international networks of open data. The federation of new repositories / harvest can be requested to snap4city@disit.org by https://www.snap4city.org/drupal/contact
Since it is based on CKAN, the solution also presents API for automated upload of Open Data Sets, and for the cross sharing of Open Data among different CKAN/DataGates. In Snap4City, the CKAN APIs are used for automatically managing and publish Data Sets in bulk by:
- extracting data collected from ETL and/or IOT, For example, the proposed solution allows to defined process that publish periodically all data of a given source: the collected sensors data of an area every day, etc. Go on Github of Disit Lab to download a large collection of ETL processes including those to work in automatic manner with DataGate publication and ingestion.
- providing data sets produced as result of Data Analytics algorithms and solutions.
Please, note that some of the following links are accessible only for registered users.
- TC6.1. Managing DataSets via DataGate: ingest, search, download, upload, annotate, share
- TC6.2. Search on DataGate for Data Sets
- TC6.3. Creating ETL processes for automated data ingestion and data transformation
- TC6.4. Managing ETL processes via Resource Manager, upload, execute, monitor
- TC6.5. Managing Heterogeneous File Ingestion via ETL processes
- TC6.6. Producing data-sets in Bundle via ETL
- TC6.7. Producing data-set in Bundle/Bulk via IOT Application
- TC6.8. ETL processes for data transformation, and exploiting MicroServices/API/RestCall
- TC6.9. ETL processes for multiprotocol and format data ingestion, see on GITHUB for library
- TC6.10. ETL Applications using multiple protocols, and formats for files and to calling services using REST and WS
- TC6.11. Add a new ETL coping with a new Protocol
Entry Login URL: https://datagate.snap4city.org/ or https://datagate.snap4city.org/ssologin_handler
User Manual to download:
- https://www.snap4city.org/download/video/Datagate_tool_User_Manual.pdf
- https://datagate.snap4city.org/about (Quick introduction in English)
Collections of Datasets ingested by using the DataGate tool:
- https://datagate.snap4city.org/organization
- Antwerp: https://datagate.snap4city.org/organization/antwerp-s4c
- Helsinki:https://datagate.snap4city.org/organization/helsinki
- Tuscany: https://datagate.snap4city.org/organization/disit-unifi
- Sardinia: https://datagate.snap4city.org/organization/ghost
- Emilia Romagna: https://datagate.snap4city.org/organization/emilia-romagna
- Veneto: https://datagate.snap4city.org/organization/veneto
Most of the datasets have been automatically ingested by DataGate and its ETL processing data mining and are visible by accessing the following link:
- Antwerp: https://antwerp.snap4city.org/
- Helsinki: https://helsinki.snap4city.org/
- Tuscany: https://servicemap.snap4city.org/
- Sardinia, Emilia Romagna, Veneto: https://www.disit.org/smosm/ with many cities as: Cagliari, Sassari, Bologna, etc.