Skip to main content

Data Services Quick Start

This guide provides an introduction to the data services available to users through AmSC.

Obtaining your AmSC Bearer Token​

Navigate to https://my.amsc.energy.gov/ and login with your AmSC/organizational credentials.

Once logged in, you will see your AmSC Profile page. At the bottom of the page you will find your AmSC Bearer Token in the box labeled 'ID Token'. It is good practice to avoid including your bearer token directly in your code. Instead, create an environment variable, and access it in your code like this:

import os

# Paste your AmSC token from https://my.american-science-cloud.org/profile here, or set AMSC_TOKEN 
# environment variable before starting Jupyter
AMSC_TOKEN = os.environ.get("AMSC_TOKEN", "")

Installing the AmSC Client​

Details about installing the AmSC Client can be found at this link

Searching for Data Using the AmSC Client​

Create a Client​

First create a Client object, providing a string containing your AmSC token. The client will be used to make calls to the AmSC API, providing functionality such as searching.

catalog_client = Client(
    base_url="https://api.american-science-cloud.org/api/current",
    token=AMSC_TOKEN,
)

Next, use the client to search for datasets related to Illinois. We can explore information about the search results by iterating over the returned result object:

results = catalog_client.catalog.search('Illinois', limit=200)
for item in results:
    print(' --Result--')
    print(f'Name: {item.name}')
    print(f'type: {item.type}')
    print(f'FQN: {item.fqn}')
    print(f'location: {item.location}')

Running the search produces two related results:

 --Result--
Name: Illinois High Resolution Wind Resource
type: scientificWork
FQN: oedi-amsc-storage.oedi-amsc-data-catalog.Illinois High Resolution Wind Resource
location: https://data.openei.org/submissions/278
 --Result--
Name: 1. Illinois High Resolution Wind Resource shape file.zip
type: artifact
FQN: oedi-amsc-storage.oedi-amsc-data-catalog.Illinois High Resolution Wind Resource."1. Illinois High Resolution Wind Resource shape file.zip"
location: https://data.openei.org/files/278/illinoiswindhighresolution.zip

Looking at the result details , we can see that the result entities are of two different types. The first is a scientificWork entity. A scientificWork is a container type that holds one or more related entities. The second result is an Artifact, which represents a unit of data (often a file).

In this case, the URL of a zip file is provided in the location field of the Artifact, and this zip file may be downloaded directly.

Let's look at another search:

results = catalog_client.catalog.search('Black body grids at HFIR CG-1D and NCNR BT2', limit=200)
for item in results:
    print(' --Result--')
    print(f'Name: {item.name}')
    print(f'type: {item.type}')
    print(f'FQN: {item.fqn}')
    print(f'location: {item.location}')

The again provides two results, a scientificWork and an Artifact:

 --Result--
Name: https_doi_org_10_13139_ORNLNCCS_1870689
type: scientificWork
FQN: olcf-constellation-amsc-storage.olcf-const-data-catalog.https_doi_org_10_13139_ORNLNCCS_1870689
location: https://doi.org/10.13139/ORNLNCCS/1870689
 --Result--
Name: Black_body_grids_at_HFIR_CG_1D_and_NCNR_BT2
type: artifact
FQN: olcf-constellation-amsc-storage.olcf-const-data-catalog.https_doi_org_10_13139_ORNLNCCS_1870689.Black_body_grids_at_HFIR_CG_1D_and_NCNR_BT2
location: globus://57618e0a-2c99-45ff-9694-24141b92fa17/gen101/world-shared/doi-data/ORNLNCCS/202206/10.13139_ORNLNCCS_1870689/

Here, we can see that the location provided for the Artifact entity is a Globus URL. To obtain this data, we will use the Data Movement API.

Data Movement using Globus​

To move the data, we'll use the name and location from the search to define name and source Globus URL for the transfer. You will also need to obtain a destination URI, which will define the location where you would like Globas to deposit the data. These details are then used to create the inputs for the Globus transfer:

from amsc_client.catalog.transfer import GlobusTransferInputs

inputs = GlobusTransferInputs(
    source_url='globus://57618e0a-2c99-45ff-9694-24141b92fa17/gen101/world-shared/doi-data/ORNLNCCS/202206/10.13139_ORNLNCCS_1870689/',
    destination_url='globus://9d6d994a-6d04-11e5-ba46-22000b92c6ec/pscratch/sd/<user directory>', # Replace with your destination URL
    label=f"AmSC data services: Black_body_grids_at_HFIR_CG_1D_and_NCNR_BT2",
)

Once we have defined the parameters of the data movement, we can initiate the transfer with:

transfer = catalog_client.catalog.globus_transfer.start(inputs)
transfer_uuid = str(transfer.transfer_uuid)

print(f"Transfer submitted")
print(f"  UUID:        {transfer_uuid}")
print(f"  Source:      {SOURCE_URL}")
print(f"  Destination: {transfer.destination_url}")