TC6.1 - Managing DataSets via DataGate: ingest, search, download, upload, annotate, share

Test Case Title	TC6.1 - Managing DataSets via DataGate: ingest, search, download, upload, annotate, share
Goal	Search and download referral data and results from the DataGate/CKAN Upload data set for knowledge base integration. Data Gate automatically assist the user to collects, enriches, integrates, and automatically reconciles referral data towards the Knowledge Base, smart city API, MicroServices Enrich data set with metadata/annotations Get authorisation from the moderator to see data uploaded available on the Knowledge Base, on API, on IOT applications, etc. Ingest and share data from other CKAN portals Publish data towards other CKAN portals Accessing data via tabular inspecting tool Connection with dashboards via HTTPS/HTTP, LDAP
Prerequisites	Using a PC or Mobile with a web browser. Access to DataGate, availability of some examples of datasets. They have been provided into the web site and FTP according to the following story. Access to another CKAN to exchange data sets among them: DataGate/CKAN and the other CKAN. For AUTOMATED ingestion see the ETL processes. The following functionalities are available only for specific Snap4city users with specific privileges.
Expected successful result	Data ingested on DataGate, once loaded and processed, if accepted they become visible on the knowledge base, and thus on the corresponding ServiceMap, Dashboard, mobile Apps, IOT applications, etc.
Steps

 

 

Please note that some of the following links could be accessible only for registered users.

DataGate can be used to:

• Collect data sets coming from operators with their manual upload;
• Collect data sets coming from other CKAN;
• Publish data sets to public and/or registered users.

The possible actions that a DataGate authenticated user can perform on own data are the following:

• view the list of present Datasets
• Visualize the list of Resources available in every dataset
• Download the files of a dataset
• Visualize resource data
• Add a new Dataset
• Modify the dataset metadata
• Add resources (files) in a Dataset
• Modify the metadata of a resource
• Modify and update data of a resource using the Enhance Data view, available only for tabular resources (csv, xlsx, xls), compliant with the DataGate template

You can start testing this requirement by following the actions described in Section 3.3 of the DataGate User Manual: http://www.disit.org/7110.

The user manual is also accessible as file on Google Drive: Datagate_tool_User_Manual.pdf

 

---

• Go to the Snap4city portal 
• Make Login:
  • Fig. a) Insert your credentials and click on the ‘Login’ button
  • Fig. b) Search the Data Gate menu and click on it (‘Data Set Manager: Data Gate’)
  • Fig. c) you can see your Data Gate home page: list of datasets

Fig.   – Data Gate user home page (user: snap4city).

• From your home page you can see your activities on Datagate (click o the ‘Activity Stream’ tab). For example, the snap4city user has scheduled a real time ETL (from the Resource Manager tool) sending data on the portal and also an has also realise and IoTApplication from nodeRED writing in datagate.

 

Fig.   – snap4city activities.

• • • To have a simple description related to Datagate, click on the menu ‘About’

 

• • • To search for all the Organizations (or group of users, stakeholders, etc.), click on the menu ‘Organizations’

 

• • • To search for a dataset, click on the menu (in the bottom of the page) ‘Datasets’

 

Fig.  – Datasets’ list.

• Search a Dataset:
  1. Case A) you can search the datasets using: i) the search form (e.g. write ‘wifi’)

Fig.   – case A) Dataset list, based on the search done, from the ‘Datasets’ page.

 

1. 1. 1.  
   2. Case B) you can use the tags on the left block (e.g. click on: ‘Antwerp > Show more tags > wifi), then it is visible a dataset list, based on the search done

 

Fig.   – case B) Dataset list: search with tags.

3. Case C) This is an alternative modality of Case B to make search and obtaining the same result. Starting from the home page or from your user profile page, click on the ‘Organizations’ tab and select a city (e.g. click on the ‘Antwerp’), search a dataset (use the search form, e.g. writing ‘wifi’). 
4. Case D) You can Obtain the same results clicking on the ‘Organization’ Main menu, clicking on the ‘Antwerp image, and using the textual filter.

 

• Select a Dataset:
  1. Click on Explore>Download to download the file

 

1. 1. 1. 2. Click on Explore>Preview, to visualize the data

Fig.  – Click on File name.

Fig.  – Data visualization.

 

• • • Starting from the Data visualization page, you can modify your data, and the  use the lateral scroll bar and go at the bottom of the page, then click on the ‘Download csv’ button (as alterative modality to download the file).

Fig.  – Download button.

 

---

• Go to the snap4city home page, make the Login (if you are not already logged in), click on the ‘Data Set Manager: Data gate’ menu (on the left column) and then on the ‘About’ main menu of the Data Gate page. Then click on the Template LINK to download the csv template.

Fig.  m – About page, and the Link to download the csv DataGate Template.

 

• • Create your csv file (Fig. o) based on the DataGate template.

 

Fig.  n – List of the csv DataGate Template rules.

 

Fig.  o – the ‘’ csv file created starting from the DataGate Template (only the mandatory fields are visible). 

• Go to the home page and select the Tab ‘Datasets’ from the menu, click on the button, if you have the permissions, you wiil see the ‘Add Dataset’ button, put the dataset name (e.g. ‘belangrijk museum’), insert the metadata, tags and upload the csv file (e.g. ‘belangrijk museum.csv’), then click on the button ‘Next: Add data’. As metadata you can put:
  • Dataset name
  • Upload the csv file
  • Description (for example can be useful put the link of the Publisher, (e.g. something like this: “Data taken from: 'http://datasets.antwerpen.be”)
  • A set of tags, that can be useful to retrieve the dataset
  • A Licence
  • The Organization, is the city to which the data are referring (e.g. ‘Antwerp’)
  • Visibility.
    • Privacy, means that the dataset is visible only for the creator
    • Public, means that the dataset is visible for the DataGate users
  • Source: 'http://datasets.antwerpen.be/v4/gis/museumoverzicht.csv'
  • Others non-mandatory: Author, author e-mail, version, etc.

 

Fig.  – Add a New Dataset: Menu/Tab Datasets > Add Dataset button.