TC6.1 - Managing DataSets via DataGate: ingest, search, download, upload, annotate, share

×

Warning message

  • You can't delete this newsletter because it has not been sent to all its subscribers.
  • You can't delete this newsletter because it has not been sent to all its subscribers.

Test Case Title

TC6.1 - Managing DataSets via DataGate: ingest, search, download, upload, annotate, share

Goal

Search and download referral data and results from the DataGate/CKAN

Upload data set for knowledge base integration. Data Gate automatically assist the user to collects, enriches, integrates, and automatically reconciles referral data towards the Knowledge Base, smart city API, MicroServices

Enrich data set with metadata/annotations

Get authorisation from the moderator to see data uploaded available on the Knowledge Base, on API, on IOT applications, etc.

Ingest and share data from other CKAN portals

Publish data towards other CKAN portals

Accessing data via tabular inspecting tool

Connection with dashboards via HTTPS/HTTP, LDAP

Prerequisites

Using a PC or Mobile with a web browser. Access to DataGate, availability of some examples of datasets. They have been provided into the web site and FTP according to the following story. Access to another CKAN to exchange data sets among them: DataGate/CKAN and the other CKAN. For AUTOMATED ingestion see the ETL processes.

The following functionalities are available only for specific Snap4city users with specific privileges.

Expected successful result

Data ingested on DataGate, once loaded and processed, if accepted they become visible on the knowledge base, and thus on the corresponding ServiceMap, Dashboard, mobile Apps, IOT applications, etc.

Steps

 

 

 

Please note that some of the following links could be accessible only for registered users.

DataGate can be used to:

  • Collect data sets coming from operators with their manual upload;
  • Collect data sets coming from other CKAN;
  • Publish data sets to public and/or registered users.

The possible actions that a DataGate authenticated user can perform on own data are the following:

  • view the list of present Datasets
  • Visualize the list of Resources available in every dataset
  • Download the files of a dataset
  • Visualize resource data
  • Add a new Dataset
  • Modify the dataset metadata
  • Add resources (files) in a Dataset
  • Modify the metadata of a resource
  • Modify and update data of a resource using the Enhance Data view, available only for tabular resources (csv, xlsx, xls), compliant with the DataGate template

You can start testing this requirement by following the actions described in Section 3.3 of the DataGate User Manual: http://www.disit.org/7110.

The user manual is also accessible as file on Google Drive: Datagate_tool_User_Manual.pdf

 


Example 1: To search and download data from the DataGate/CKAN:

  • Go to the Snap4city portal 
  • Make Login:
    • Fig. a) Insert your credentials and click on the ‘Login’ button
    • Fig. b) Search the Data Gate menu and click on it (‘Data Set Manager: Data Gate’)
    • Fig. c) you can see your Data Gate home page: list of datasets

Fig.   – Data Gate user home page (user: snap4city).

  • From your home page you can see your activities on Datagate (click o the ‘Activity Stream’ tab). For example, the snap4city user has scheduled a real time ETL (from the Resource Manager tool) sending data on the portal and also an has also realise and IoTApplication from nodeRED writing in datagate.

 

Fig.   – snap4city activities.

      • To have a simple description related to Datagate, click on the menu ‘About’

 

      • To search for all the Organizations (or group of users, stakeholders, etc.), click on the menu ‘Organizations’

 

      • To search for a dataset, click on the menu (in the bottom of the page) ‘Datasets’

 

Fig.  – Datasets’ list.

  • Search a Dataset:
    1. Case A) you can search the datasets using: i) the search form (e.g. write ‘wifi’)

Fig.   – case A) Dataset list, based on the search done, from the ‘Datasets’ page.

 

      1.  
    1. Case B) you can use the tags on the left block (e.g. click on: ‘Antwerp > Show more tags > wifi), then it is visible a dataset list, based on the search done

 

Fig.   – case B) Dataset list: search with tags.

  1. Case C) This is an alternative modality of Case B to make search and obtaining the same result. Starting from the home page or from your user profile page, click on the ‘Organizations’ tab and select a city (e.g. click on the ‘Antwerp’), search a dataset (use the search form, e.g. writing ‘wifi’). 
  2. Case D) You can Obtain the same results clicking on the ‘Organization’ Main menu, clicking on the ‘Antwerp image, and using the textual filter.

 

  • Select a Dataset:
    1. Click on Explore>Download to download the file

 

        1. Click on Explore>Preview, to visualize the data

Fig.  – Click on File name.

Fig.  – Data visualization.

 

      • Starting from the Data visualization page, you can modify your data, and the  use the lateral scroll bar and go at the bottom of the page, then click on the ‘Download csv’ button (as alterative modality to download the file).

Fig.  – Download button.

 


Example 2: Upload data set for Knowledge base integration, follow these steps:

  • Go to the snap4city home page, make the Login (if you are not already logged in), click on the ‘Data Set Manager: Data gate’ menu (on the left column) and then on the ‘About’ main menu of the Data Gate page. Then click on the Template LINK to download the csv template.

Fig.  m – About page, and the Link to download the csv DataGate Template.

 

    • Create your csv file (Fig. o) based on the DataGate template.

 

Fig.  n – List of the csv DataGate Template rules.

 

Fig.  o – the ‘’ csv file created starting from the DataGate Template (only the mandatory fields are visible). 

  • Go to the home page and select the Tab ‘Datasets’ from the menu, click on the button, if you have the permissions, you wiil see the ‘Add Dataset’ button, put the dataset name (e.g. ‘belangrijk museum’), insert the metadata, tags and upload the csv file (e.g. ‘belangrijk museum.csv’), then click on the button ‘Next: Add data’. As metadata you can put:
    • Dataset name
    • Upload the csv file
    • Description (for example can be useful put the link of the Publisher, (e.g. something like this: “Data taken from: 'http://datasets.antwerpen.be”)
    • A set of tags, that can be useful to retrieve the dataset
    • A Licence
    • The Organization, is the city to which the data are referring (e.g. ‘Antwerp’)
    • Visibility.
      • Privacy, means that the dataset is visible only for the creator
      • Public, means that the dataset is visible for the DataGate users
    • Source: 'http://datasets.antwerpen.be/v4/gis/museumoverzicht.csv'
    • Others non-mandatory: Author, author e-mail, version, etc.

 

Fig.  – Add a New Dataset: Menu/Tab Datasets > Add Dataset button.

 

Fig.  – Metadata to describe a New Dataset.

  •  
  • Upload the csv file from your file system, select the csv format and click on the ‘Finish’ Button.

 

Fig.   – Upload the file for a New Dataset.

  • Click on the file name and go to the data visualization page. Here you can see all the data and modify them also adding rows. To save the changes click on the ‘save’
  • Now you have created a New dataset, you can click on the file name to visualize the data

 

Fig.   – New Dataset Created.

  • Select the Category and the sub category of the KM4City Ontology thanks to the drop down menu (e.g. category ‘Cultural activity’, sub-category ‘Musem’).

 

Fig.   – Select KM4City Ontology categories

  • Modify your data as you prefer, go the bottom of the page and click on the Public (to snap4City) button

 

Fig.   – Publish a dataset on Snap4City Knowledge Base.

 


Example 3: To Enrich a data set with metadata/annotations, follow the steps:

  • Search your dataset, following the steps above in [1] (e.g. search the dataset ‘public-wifi_wifiopenbaar’) and go to its visualization page select the field you want to add or modify and a pop-up is available to made it

Fig.  – Publish a dataset on Snap4City Knowledge Base.

 

  1. To get authorisation from the moderator to see data uploaded available on the Knowledge Base, on API, on IOT applications, etc., you can write an e-mail to the DataGate administer to receive information on which snap4City tools your data is available.
    • Once your data is published to the Knowledge Base (steps [2]), you can see its usages, for example you can see the data in the service Map tool  

Fig.  z – Datasets coming from DataGate and visible on the Service map tool (only Wi-Fi and museums of Antwerp).

 

Fig.  x – Datasets coming from DataGate and visible on the Service map tool (all the datasets ingested from Antwerp).

 


Example 4: ingesting data from other CKAN portals,

  • you can (see the previous point [2]): i) download a dataset from another CKAN portal; ii) convert the dataset following the DataGate template; iii) create a new dataset on DataGate; iv) Publish it to snap4City Knowledge Base; v) see the dataset use on the snap4City tools/services (see point [4])

 


Example 5: sharing data from two CKAN having the same DataGate module.