Data Services Quick Start

This guide aims to provide a gentle introduction to the data services available to users through AmSC. Code examples use python, and were tested using python 3.14.3.

Obtaining your AmSC Bearer Token

Navigate to https://my.american-science-cloud.org/profile and login with your organizational credentials. Click "Sign in with Globus", and you should see a dialog similar to this:

Search for your organization in the list and follow the instructions given to login. The login details may be slightly different depending on your institution. More details about logging into and using the AmSC portal may be found here.

Once logged in, you will see your AmSC Profile page. At the bottom of the page you will find your AmSC Bearer Token in the box labeled 'ID Token'. It is good practice to avoid including your bearer token directly in your code. Instead, create an environment variable, and access it in your code like this:

import os

# Paste your AmSC token from https://my.american-science-cloud.org/profile here, or set AMSC_TOKEN 
# environment variable before starting Jupyter
AMSC_TOKEN = os.environ.get("AMSC_TOKEN", "")

Installing the AmSC Client

Details about installing the AmSC Client can be found at this link

Searching for Data Using the AmSC Client

Create a Client

First create a Client object, providing a string containing your AmSC token. The client will be used to make calls to the AmSC API, providing functionality such as searching.

catalog_client = Client(
    base_url="https://api.american-science-cloud.org/api/current",
    token=AMSC_TOKEN,
)

Next, use the client to search for datasets related to Illinois. We can explore information about the search results by iterating over the returned result object:

results = catalog_client.catalog.search('Illinois', limit=200)
for item in results:
    print(' --Result--')
    print(f'Name: {item.name}')
    print(f'type: {item.type}')
    print(f'FQN: {item.fqn}')
    print(f'location: {item.location}')

Running the search produces two related results:

 --Result--
Name: Illinois High Resolution Wind Resource
type: scientificWork
FQN: oedi-amsc-storage.oedi-amsc-data-catalog.Illinois High Resolution Wind Resource
location: https://data.openei.org/submissions/278
 --Result--
Name: 1. Illinois High Resolution Wind Resource shape file.zip
type: artifact
FQN: oedi-amsc-storage.oedi-amsc-data-catalog.Illinois High Resolution Wind Resource."1. Illinois High Resolution Wind Resource shape file.zip"
location: https://data.openei.org/files/278/illinoiswindhighresolution.zip

Looking at the result details , we can see that the result entities are of two different types. The first is a scientificWork entity. A scientificWork is a container type that holds one or more related entities. The second result is an Artifact, which represents a unit of data (often a file).

In this case, the URL of a zip file is provided in the location field of the Artifact, and this zip file may be downloaded directly.

Let's look at another search:

results = catalog_client.catalog.search('Black body grids at HFIR CG-1D and NCNR BT2', limit=200)
for item in results:
    print(' --Result--')
    print(f'Name: {item.name}')
    print(f'type: {item.type}')
    print(f'FQN: {item.fqn}')
    print(f'location: {item.location}')

The again provides two results, a scientificWork and an Artifact:

 --Result--
Name: https_doi_org_10_13139_ORNLNCCS_1870689
type: scientificWork
FQN: olcf-constellation-amsc-storage.olcf-const-data-catalog.https_doi_org_10_13139_ORNLNCCS_1870689
location: https://doi.org/10.13139/ORNLNCCS/1870689
 --Result--
Name: Black_body_grids_at_HFIR_CG_1D_and_NCNR_BT2
type: artifact
FQN: olcf-constellation-amsc-storage.olcf-const-data-catalog.https_doi_org_10_13139_ORNLNCCS_1870689.Black_body_grids_at_HFIR_CG_1D_and_NCNR_BT2
location: globus://57618e0a-2c99-45ff-9694-24141b92fa17/gen101/world-shared/doi-data/ORNLNCCS/202206/10.13139_ORNLNCCS_1870689/

Here, we can see that the location provided for the Artifact entity is a Globus URL. To obtain this data, we will use the Data Movement API.

Data Movement using Globus

To move the data, we'll use the name and location from the search to define name and source Globus URL for the transfer. You will also need to obtain a destination URI, which will define the location where you would like Globas to deposit the data. These details are then used to create the inputs for the Globus transfer:

from amsc_client.catalog.transfer import GlobusTransferInputs

inputs = GlobusTransferInputs(
    source_url='globus://57618e0a-2c99-45ff-9694-24141b92fa17/gen101/world-shared/doi-data/ORNLNCCS/202206/10.13139_ORNLNCCS_1870689/',
    destination_url='globus://9d6d994a-6d04-11e5-ba46-22000b92c6ec/pscratch/sd/<user directory>', # Replace with your destination URL
    label=f"AmSC data services: Black_body_grids_at_HFIR_CG_1D_and_NCNR_BT2",
)

Once we have defined the parameters of the data movement, we can initiate the transfer with:

transfer = catalog_client.catalog.globus_transfer.start(inputs)
transfer_uuid = str(transfer.transfer_uuid)

print(f"Transfer submitted")
print(f"  UUID:        {transfer_uuid}")
print(f"  Source:      {SOURCE_URL}")
print(f"  Destination: {transfer.destination_url}")

Data Streaming using EJFAT

EJFAT (ESnet-JLab FPGA-Accelerated Transport) lets you stream large volumes of scientific data in real time — for example, routing a live detector feed to multiple analysis workers as data is collected. It is designed for high-throughput streaming at N x 100 Gbps, making it a good fit for workflows where data needs to be processed as it arrives rather than transferred as files after the fact.

EJFAT is co-designed as both hardware (FPGA) and software (data plane and control plane) pieces. We have deployed a hardware LB and its associated control plane, accessible via the AmSC data service APIs. This guide covers using the AmSC API to reserve and release an LB instance; for an end-to-end example EJFAT streaming incorporating both data plane and control plane, please refer to the example notebook EJFAT + E2SAR on Perlmutter.

At the center of every EJFAT streaming session is an FPGA-accelerated Load Balancer (LB) instance. The LB receives data from one or more senders and distributes it across your receiving workers automatically. You reserve an LB before your workflow starts, share its connection details with your senders and receivers, and release it when you are done.

Your AmSC Bearer Token (the same one used elsewhere in this guide) is used to create and manage LB instances. Once an LB is created, it returns an Access Token embedded in an environment variable string called the ejfat_uri. Senders use this URI to connect to the LB.

Example: Streaming Detector Data to Analysis Workers

The following example walks through creating and releasing an EJFAT instance:

Reserve an LB instance, registering the IP addresses of your sender nodes
Get the LB status with lb_id
Release the LB when the session is complete

Step 1 — Reserve a Load Balancer

Provide a name for your session and the IP address(es) of the machine(s) that will send data. The API returns an ejfat_uri that encodes everything senders and receivers need to connect.

import os

# Using the same TOKEN and AmSC Client as above
AMSC_TOKEN = os.environ.get("AMSC_TOKEN", "")
# Use the NERSC endpoint for EJFAT — the AmSC data services base_url does not support this service yet
ejfat_client = Client(
    base_url="https://amsc-data-api.nersc.gov/movement/streaming/ejfat",
    token=AMSC_TOKEN,
)

version = ejfat_client.ejfat_version()
print(f"EJFAT version: {version}")

lb = ejfat_client.ejfat_create_loadbalancer(
    name="ejfat-amsc-test",
    sender_addresses=["10.100.100.1", "10.100.100.2"],  
    ip_family="ipv4",
)
lb_id = lb["lb_id"]

print(f"Load balancer reserved")
print(f"  lb_id:            {lb_id}")            # use this lb_id for following queries
print(f"  ejfat_uri:        {lb.get('ejfat_uri')}")

Step 2 — Get the LB status

Retrieve the current state of the reserved load balancer by its lb_id. It will return the same response containing the ejfat_uri when it was created.

lb_status = ejfat_client.ejfat_get_loadbalancer(lb_id)
print(f"Load balancer status for {lb_id}:")
print(lb_status)

Step 3 — Release the Load Balancer

When your workflow finishes, release the LB so the resources become available for other users.

result = ejfat_client.ejfat_delete_loadbalancer(lb_id)
print(f"Load balancer {lb_id} released: {result}")

AmSC EJFAT control plane API Reference

Method	Path	Description
`GET`	`/movement/streaming/ejfat/health`	Check that the EJFAT service is reachable. Returns `"true"`.
`GET`	`/movement/streaming/ejfat/version`	Return the current service version.
`POST`	`/movement/streaming/ejfat/control-plane-ip/loadbalancers`	Reserve a new load balancer.
`GET`	`/movement/streaming/ejfat/control-plane-ip/loadbalancers/{lb_id}`	Get the status and configuration of an existing load balancer.
`DELETE`	`/movement/streaming/ejfat/control-plane-ip/loadbalancers/{lb_id}`	Release a load balancer reservation.

All endpoints require your AmSC Bearer Token as Authorization: Bearer <token>. The ejfat_uri returned when creating an LB takes the form:

ejfat://<ejfat_access_token>@ejfat/lb/<lb_id>?sync=<sync_ip>:<port>&data=<data_ip>:<min_port>-<max_port>

Accessing your AmSC Account and Related Credentials​

Obtaining your AmSC Bearer Token​

Installing the AmSC Client​

Searching for Data Using the AmSC Client​

Create a Client​

Data Movement using Globus​

Data Streaming using EJFAT​

Example: Streaming Detector Data to Analysis Workers​

AmSC EJFAT control plane API Reference​

Accessing your AmSC Account and Related Credentials