Python bindings
GA4GH-SDK Python Bindings¶
This directory provides Python bindings for the GA4GH SDK using
PyO3. Releases use the PyPI distribution
name ga4gh-sdk. The other instructions build the bindings from this
repository. Start at the repository root; after cd python, run later commands
from python/ with the virtual environment active.
Install from PyPI¶
Install a release into a CPython 3.11+ environment:
pip skips pre-releases such as 0.2.0rc1 unless you add --pre or name the
version (ga4gh-sdk==0.2.0rc1). The release workflow builds wheels for Linux
(x86-64 and aarch64, glibc 2.28 or newer), macOS (x86-64 and arm64) and
Windows (x86-64); installing them needs no Rust toolchain. Elsewhere, for
example on musl-based Linux such as Alpine or with an older glibc, pip builds
the bindings from the sdist, which needs the prerequisites below. See the
release guide for the publication process.
Prerequisites¶
To build the bindings from source:
- CPython 3.11 or newer (PyPy and free-threaded builds are not supported)
- The Rust toolchain
The bindings use rustls, so unlike the library's default build, building them
with maturin needs no OpenSSL development files or pkg-config on Linux.
Installation¶
From the repository root, create an environment, build the bindings, and check the import:
cd python
python3 -m venv .venv
. .venv/bin/activate
python -m pip install 'maturin>=1.7,<2'
maturin develop --locked
python -c 'from ga4gh_sdk import TES; print(TES.__name__)'
Use a Python 3.11+ executable if python3 points to an older version. The
distribution is named ga4gh-sdk; the import name is ga4gh_sdk. Earlier
source builds used from GA4GH import ...; update those imports before using
the released package. An uppercase import alias is not shipped because it can
collide with the older ga4gh package on case-insensitive file systems.
Build a wheel¶
Build the Python library wheel:
The bindings use Python's stable ABI (abi3), so a cp311-abi3 wheel can be
installed on CPython 3.11 and newer on the same platform.
Running Tests¶
After maturin develop, install pytest and run the Python binding tests with:
The tests use mock services. Smoke tests of the TRS bindings against WorkflowHub and Dockstore need network access and run only on request:
Supported surface¶
| Python API | Current behavior |
|---|---|
Configuration |
Set one service API root, authentication source and time limits, or read a built-in or custom section of ~/.ga4gh-cli/config.json; get_extra() returns the section's other keys. set_token_provider() supplies dynamic bearer tokens. See Reading the configuration file, Authentication, Timeouts and Token providers. |
ServiceInfo |
Fetch service info as a formatted string; it is not yet a typed Python model. An invalid response raises RuntimeError; see the Service Info contract. |
TES, Task, TesTask, ListTasksParams |
Create, inspect, list, poll, cancel and wait for TES tasks; read full task JSON including unknown fields. See Waiting for tasks. |
AsyncTES, AsyncTask |
The same TES calls for asyncio, as awaitables; see asyncio. |
Transport |
Send relative HTTP requests with JSON, query parameters, or multipart bodies; responses are text. |
DRS, DrsResolver |
Read DRS objects and access URLs, one at a time or in bulk, wait while the server prepares them, and plan access for drs:// URIs; read full JSON including unknown fields. See Using DRS. |
WES, Run, RunRequest |
Start runs (with workflow attachments), list, inspect, poll, and cancel WES runs, and read their tasks; read full JSON including unknown fields. See Using WES. |
TRS, Tool, ToolVersion, FileWrapper, ToolFile, ToolPage |
List tools page by page, read tools, versions, descriptors, test files, file lists, ZIP archives and container files of TRS 2.0.1 and 2.0.2 registries; read full JSON including unknown fields. See Using TRS. |
| Service Registry | Available in the Rust SDK, not yet as a Python client class. |
The Python API is maintained by hand; a new Rust SDK method or model does not appear in Python automatically. For each SDK change, decide whether to expose it here, then update the wrapper, this table, and the mock-backed tests and examples. CI checks the documented Python behavior, but it cannot infer that a new Rust API needs a Python counterpart.
Usage¶
Configuration takes the API root, including the service's path prefix (see
the usage guide).
Run this example from python/ after starting Funnel and Docker as described
in the quick start. The checked-in task is
examples/tes-hello.json from the repository
root, so its path from python/ is ../examples/tes-hello.json. It runs
echo in an Alpine container without external storage.
from ga4gh_sdk import Configuration, ServiceInfo, ServiceType, TES
config = Configuration("http://localhost:8000/ga4gh/tes/v1")
# Or read the TES section of ~/.ga4gh-cli/config.json:
# config.from_file(ServiceType.TES)
print(ServiceInfo(config).get_service_info())
tes = TES(config)
task = tes.create("../examples/tes-hello.json")
print(task.id, task.status())
for t in tes.list(None).tasks:
print(t.id, t.status)
print(tes.get("FULL", task.id).status)
TES.create() accepts a dict, a TesTask, or a path to a JSON task file.
Use TesTask.from_json(text) when the task starts as JSON text.
For a task that is still running, call task.cancel().
from_file replaces the API root. Task.status() and Task.cancel() are methods;
id and status on tasks returned by list() and get() are attributes.
Reading the configuration file¶
config.from_file() reads a section of ~/.ga4gh-cli/config.json (format in
the usage guide). In a file with
profiles, it reads the default profile. Name a
built-in section with ServiceType (TES, DRS, TRS, AAI, WES,
REGISTRY), and any other section with a str. config.get_extra() returns
the keys of the
section that the SDK doesn't use itself, as a dict with the values as they are
in the file, so don't log it if it can hold credentials:
from ga4gh_sdk import Configuration, ServiceType
config = Configuration("https://example.org")
config.from_file(ServiceType.TES)
config.from_file("MY_SERVICE")
queue = config.get_extra().get("queue")
A section can name an environment variable for its token with
oauth_access_token_env or bearer_access_token_env. A missing section or an
unset variable raises ValueError, a missing file FileNotFoundError.
Authentication¶
Set one source before constructing a client: set_basic_auth(username,
password=None), set_oauth_access_token(token),
set_bearer_access_token(token), or a token provider. Use
clear_auth() before choosing a different source. from_file() can read
static credentials from the selected service section.
The client uses that credential for discovery and later operations; invalid or
conflicting sources raise an exception before sending the request. The SDK does
not follow redirects automatically. See the
authentication guide for file keys, injected
transport boundaries and errors. The token-provider section
describes dynamic credentials.
Timeouts¶
config.set_timeouts() sets time limits of requests, in seconds, as keyword
arguments; config.get_timeouts() returns all four as a dict. A section of
the configuration file sets them with its timeouts object:
from ga4gh_sdk import Configuration, TES
config = Configuration("https://example.org/ga4gh/tes/v1")
config.set_timeouts(discovery=30, submission=600)
config.get_timeouts()
# {'connect': 10.0, 'discovery': 30.0, 'request': 60.0, 'submission': 600.0}
tes = TES(config)
connect(default 10): opening each connection.request(default 60): each request that the limits below don't apply to.discovery(default 10): the service check ofTES(config),AsyncTES,WES(config),DRS(config)andTRS(config), andTRS.service_info(), in place ofrequest.submission(default 300):TES.create(),AsyncTES.create()andWES.run(), including uploads, in place ofrequest. A submission that runs out of time after its request started raisesRuntimeErrorwith "submission may have been accepted"; look for the task or run before you submit it again.
Clients and Transport read the limits when they are built, so set them first.
Calls with a timeout argument, such as DRS.resolve_object(), stop at that
timeout as well. The usage guide has the details.
Token providers¶
config.set_token_provider(provider) makes the clients and Transport built from
config ask provider for a bearer token before each request, including the
service check of TES(config) and the other clients. provider(audience) gets the
API root (or the audience given to set_token_provider()) and returns the token
as a str, or as a (str, float) tuple with the seconds for which the SDK may
reuse it:
from ga4gh_sdk import Configuration, TES
def token_provider(audience: str) -> tuple[str, float]:
token, expires_in = get_token_for(audience) # your code
return token, expires_in - 30 # reuse it until 30 seconds before it expires
config = Configuration("https://example.org/ga4gh/tes/v1")
config.set_token_provider(token_provider)
# Or with the audience of a token exchange:
# config.set_token_provider(token_provider, audience="https://example.org/ga4gh/tes")
tes = TES(config)
- Set the provider after
from_file(), which replaces all settings. Clients andTransporttake it when they are built;set_token_provider(None)removes it for those built later. - After
401 Unauthorized, the SDK drops a reused token and asks the provider again; aGETrequest is sent once more right away with a current token, while submissions such asTES.create()andWES.run()are never sent twice. The provider isn't told about the rejection, so don't let it cache tokens itself: return a(str, float)tuple and let the SDK reuse the token instead. - If the provider raises, returns something else or a token that can't be an HTTP
header value, the call raises
RuntimeErrorand the pending request is not sent. A read-only request may already have received401before a failed refresh. The default message omits the provider's exception text, which could contain a secret. HTTP errors such as401also raiseRuntimeError; the message distinguishes them from provider failures. - A provider can't be combined with other credentials from the configuration file
(
basic_auth,oauth_access_token, or a user or password inbase_path): calls then raiseRuntimeErrorbefore anything is sent. - The provider runs on a thread of the SDK's runtime, with the GIL, also for the asyncio classes. It may block, for example on a token endpoint, but each call waits for it.
The usage guide describes the contract.
Network behavior¶
The Python build uses rustls for HTTPS and loads trusted certificates from the host system. On Linux, install the system CA certificate bundle for HTTPS requests; building the bindings does not need OpenSSL development headers.
Synchronous network methods release the Python GIL while Rust waits for I/O.
They share a process-lifetime multi-thread Tokio runtime, so a client's HTTP
connections remain reusable between calls. Inputs are decoded before releasing
the GIL and Python results/exceptions are constructed after reacquiring it.
No Python callback runs on the runtime, except a token provider,
on a blocking thread of its own. The asyncio classes run
their requests on the same runtime. Use multiprocessing's spawn mode rather
than forking a process after starting threaded I/O.
tests/test_io_runtime.py exercises all synchronous network wrappers against a
Python-thread HTTP server, including a heartbeat during delayed I/O, keep-alive
connection reuse, and error recovery. Each scenario runs in a subprocess with a
15-second timeout so a GIL regression cannot hang the test worker.
Configuration.from_file() raises FileNotFoundError when the config file is
missing and ValueError when its contents are malformed. Invalid URLs and JSON
arguments also raise ValueError.
Using TES¶
TES is a client for a Task Execution Service. It takes tasks as dicts, so Python apps
need neither task files on disk nor JSON parsing:
from ga4gh_sdk import Configuration, ListTasksParams, TES, TesState, TesTask
tes = TES(Configuration("http://localhost:8000/ga4gh/tes/v1"))
task = tes.create({
"name": "hello",
"executors": [{"image": "alpine", "command": ["echo", "hello"]}],
})
print(task.id)
if task.status() is TesState.QUEUED:
task.cancel()
details = tes.get("FULL", task.id)
print(details.state, details.executors, details.to_dict())
page = tes.list(ListTasksParams(state=TesState.RUNNING, page_size=50))
for listed in page.tasks:
print(listed.id, listed.state)
TES.create(task)takes adict(or other mapping), aTesTask, or the path of a JSON file (stroros.PathLike). Astris always a path, unlike forTesTask(task); pass JSON text asTesTask.from_json(text). Astrthat starts with{raisesValueError; pass such a path as apathlib.Path. It sends the task's JSON as given, including fields that the TES models don't know, and returns aTask.Taskaddresses a task on the server by itsid.Task.status()returns the current state as aTesState;Task.cancel()cancels the task and returnsNone.Task.wait()waits until the task ends; see Waiting for tasks.TES.get(view, id)returns aTesTask;viewis"MINIMAL","BASIC"or"FULL"(case-insensitive). Invalid views raiseValueErrorbefore a request.TES.list(params=None)returns aListTasksResponsewithtasks(a list ofTesTask) andnext_page_token; see Listing tasks.- Errors: invalid tasks and arguments raise
ValueError, a missing task fileFileNotFoundError(anOSError). Failed requests and responses that don't fit the TES models raiseRuntimeError. If task creation fails after submission, the error may mean the server created the task but its acknowledgement was lost or malformed; do not retry automatically. HTTP failures include the status in the exception message but omit the server's untrusted response body.
Task objects¶
TesTask holds a task: one to create, or one as returned by the server. Build one with
TesTask(task) from a dict or JSON text, or with TesTask.from_dict(d),
TesTask.from_json(text) or TesTask.from_file(path); all of them check the task
against the TES models and raise ValueError if it doesn't fit. The classmethods call
cls(task), so further constructor arguments of a subclass need defaults. Tasks can be
pickled and copied, including the attributes of subclass instances.
Its fields are properties: id, state, name, description, creation_time,
inputs, outputs, resources, executors, volumes, tags and logs. Missing
fields are None, except state, which is TesState.UNKNOWN then. A state that TES
1.1 doesn't define is also TesState.UNKNOWN; to_dict() keeps the server's value.
Nested values, such as executors, come as dicts and lists. The status property is the same as state and
is kept for compatibility. Data objects such as TesTask have properties; Task, which
asks the server, has methods (Task.status()).
task.to_dict() returns the whole task as a new dict and task.json() as JSON text,
both including fields that the TES models don't know. ListTasksResponse has the same
two methods for the whole page.
Task states¶
TesState is a Python enum.Enum of the TES states (UNKNOWN, QUEUED,
INITIALIZING, RUNNING, PAUSED, COMPLETE, EXECUTOR_ERROR, SYSTEM_ERROR,
CANCELED, PREEMPTED, CANCELING). Its members are also strings:
TesState.RUNNING == "RUNNING", and str(), f-strings and json.dumps() give the
state's name. TesState("RUNNING") looks a state up by name. Members can be pickled.
Waiting for tasks¶
Task.wait() polls a task until it ends, so apps need no polling loop of their own:
from ga4gh_sdk import Configuration, TES, TesState, WaitTimeoutError
tes = TES(Configuration("http://localhost:8000/ga4gh/tes/v1"))
task = tes.create({"executors": [{"image": "alpine", "command": ["echo", "hello"]}]})
try:
state = task.wait(timeout=3600)
except WaitTimeoutError as error:
print(f"{error.id} still {error.last_state} after an hour")
else:
if state is not TesState.COMPLETE:
print(task.id, "ended in", state)
# Or get the ended task in a view, for example with its logs:
details = task.wait(timeout=3600, view="FULL")
print(details.state, details.logs)
Task.wait(timeout=None, *, poll_interval=1.0, max_interval=None, view=None)returns once the task is in a terminal state:COMPLETE,EXECUTOR_ERROR,SYSTEM_ERROR,CANCELEDorPREEMPTED. It returns that state as aTesState; with aview("MINIMAL","BASIC"or"FULL", in any case), it returns the task in that view as aTesTask. The minimal view is what it polls, so that view needs no further request.- It uses the Rust SDK's waiter. Polls ask for the task in the minimal view, starting
poll_intervalseconds apart; the interval doubles after each poll, up tomax_interval(by default 10 seconds, orpoll_intervalif that's longer). Other states, including states that TES 1.1 doesn't define, don't end the wait. timeoutis the time in seconds for the whole wait, including the polls and the request for the view;Nonewaits as long as it takes. When the time runs out, it raisesWaitTimeoutError, aTimeoutError, whoseidis the task's ID andlast_statethe last polled state (Noneif no poll succeeded). If the time runs out while it gets an ended task in a view,last_stateis that task's final state. Each request also keeps the configuration's timeouts.- A failed poll, for example an HTTP error or a response without a state, raises
RuntimeError; polls are not retried. Invalid arguments, such as a negative timeout, apoll_intervalof 0 or amax_intervalbelowpoll_interval, raiseValueError. - The wait releases the GIL, so other threads run meanwhile. In the main thread, it
runs signal handlers within about 0.1 seconds: Ctrl-C stops it with
KeyboardInterrupt, as does any exception that a handler raises. Handlers may call the SDK, for exampletask.cancel(). Python runs signal handlers in the main thread only, so a wait in another thread goes on until it ends or times out. - A wait never cancels the task: not when the time runs out, not on Ctrl-C, not when a
poll fails. Call
Task.cancel()for that. To keep waiting, callwait()again, also with a newTask(id, transport)in another process.
asyncio¶
AsyncTES and AsyncTask are the TES client and task for asyncio apps. They have the
same methods as TES and Task, take the same arguments and give the same results
(TesTask, ListTasksResponse, TesState), but each method returns an awaitable:
import asyncio
from ga4gh_sdk import AsyncTES, Configuration, TesState
async def main():
tes = await AsyncTES.connect(Configuration("http://localhost:8000/ga4gh/tes/v1"))
task = await tes.create({"executors": [{"image": "alpine", "command": ["echo", "hello"]}]})
if await task.status() is TesState.QUEUED:
await task.cancel()
details, page = await asyncio.gather(tes.get("FULL", task.id), tes.list())
print(details.to_dict(), [listed.id for listed in page.tasks])
asyncio.run(main())
- The built-in
AsyncTESmethods check the service info before their first TES request.AsyncTES(config)doesn't contact the server: its first method call checks the service, once for concurrent calls. If the check fails, the calls that waited for it raise itsRuntimeError, and a later call tries again.await AsyncTES.connect(config)checks it right away and gives the client. On a subclass,connect()buildscls(config, *args, **kwargs)at once, in the caller's thread and context, and gives it once the check is done. A subclass's__init__can send its own requests before that check completes. AsyncTES.create()gives anAsyncTask;AsyncTask(id, transport)addresses an existing task, likeTask.config,transportandidare plain properties.await task.wait(...)waits for a task likeTask.wait(), with the same arguments, results and errors. Cancelling the awaiting coroutine, for example withasyncio.wait_for(), stops polling; the task on the server goes on. Between polls, a pending wait sleeps on the runtime without a call thread (see below); it takes one only for each request. So an app can wait for many more tasks than there are call threads, and other async calls, includingcancel(), can run while waits sleep. They may still queue if polls or other requests occupy all 256 call threads. Thetimeoutalso holds while a request waits for a call thread: if the time runs out before the first poll,WaitTimeoutErrorhaslast_stateNone.- The requests run on the runtime of the sync API without the GIL, so they don't block
the event loop, and several of them run at the same time. Because the SDK's futures
aren't
Sendyet, each active request takes a thread of a pool of its own: up to 256 requests run at once. A wait releases its slot between requests. Further calls wait for a slot without occupying a pool thread; cancelling a waiting call prevents it from starting. This limits concurrent calls, not the number of futures an app can schedule. - The methods return
asyncio.Futures of the running event loop, so call them from a coroutine or a callback of the loop; outside of one, they raiseRuntimeError. - Cancelling an awaitable, for example with
asyncio.wait_for()orasyncio.Task.cancel(), returns at once and drops its request: the connection is closed. Whether the server already acted on a request that was sent is unknown then. - Invalid arguments raise
ValueError(orOSErrorfor task files) when the method is called; failed requests and invalid responses raiseRuntimeErrorwhen awaited. transportgives aTransportfor further endpoints. Its methods are synchronous: they release the GIL but block the event loop while a request runs, so run them withawait asyncio.to_thread(...)in a coroutine. There is no asyncTransportyet.
Type hints¶
The package ships type stubs (ga4gh_sdk.pyi, with a py.typed marker), so editors and
type checkers such as mypy see every class and method. Their docstrings form the
Python API reference. Where Python exposes a runtime docstring for a class,
method or property, help() shows the same text. Descriptions of attributes and enum
members are available in the API reference. tests/test_stubs.py checks stub coverage
and the docstrings shared with the runtime module.
Using WES¶
WES is a client for a Workflow Execution Service (WES 1.1). Like TES, it takes run
requests as dicts, and workflow files as bytes or paths:
from pathlib import Path
from ga4gh_sdk import Configuration, WES, WesState
wes = WES(Configuration("https://your-wes.example/ga4gh/wes/v1"))
run = wes.run(
{
"workflow_type": "CWL",
"workflow_type_version": "v1.2",
"workflow_url": "main.cwl",
"workflow_params": {"message": "hello"},
"tags": {"project": "demo"},
},
attachments={"main.cwl": Path("main.cwl"), "tools/echo.cwl": b"cwlVersion: v1.2\n..."},
)
print(run.id)
if run.status() is WesState.RUNNING:
run.cancel()
details = wes.get(run.id) # or run.log()
print(details.state, details.outputs, details.to_dict())
page = wes.list(page_size=50)
for summary in page.runs:
print(summary.run_id, summary.state)
more = wes.list(page_token=page.next_page_token)
tasks = run.tasks(page_size=10)
for task in tasks.task_logs:
print(task.id, task.name, task.exit_code)
print(run.task(tasks.task_logs[0].id).stderr)
WES(config)reads the service info and raisesRuntimeErrorif the service isn't reachable or isn't a WES.WES.run(request, attachments=None)starts a run and returns aRun.requestis adict(or other mapping), aRunRequest, or the path of a JSON file (stroros.PathLike); as forTES.create(), astris always a path, so pass JSON text asRunRequest.from_json(text).workflow_type,workflow_type_version,workflow_urlandworkflow_paramsare required;workflow_paramsmust be a JSON object. The request goes to the server as the fields of the WESRunRequestform. Fields that the WES models don't know, such as a server's extension fields, go as further form parts: strings as they are, other values as JSON. Aworkflow_attachmentfield raisesValueErrorbefore a request; pass files withattachmentsinstead.attachmentsmaps filenames to contents, asbytes(orbytearray) or as the path of a file (os.PathLike, such aspathlib.Path), which is streamed rather than read into memory. A list of(filename, content)pairs works too. Filenames are relative paths with/separators, such astools/echo.cwl; absolute paths,.., backslashes, colons and repeated filenames raiseValueErrorbefore a request. Refer to an attachment inworkflow_urlby its filename. The upload and the server's answer must finish within thesubmissionlimit of the configuration, 300 seconds by default (see Timeouts).Runaddresses a run on the server by itsid.Run.status()returns aWesState,Run.cancel()cancels the run and returnsNone, andRun.log()returns aRunLog.Run.tasks(page_size=None, page_token=None)returns aTaskListResponsewithtask_logs(a list ofTaskLog) andnext_page_token;Run.task(task_id)returns oneTaskLog. The task endpoints are new in WES 1.1.WES.get(id)returns aRunLog, likeRun(id, wes.transport).log().WES.list(page_size=None, page_token=None)returns aRunListResponsewithruns(a list ofRunSummary) andnext_page_token. A bare array of runs, as WES 0.2.0 servers (such as the GA4GH Starter Kit) send it, becomes a page without a token.- Errors: invalid requests, attachments and arguments (such as an empty ID,
.or..) raiseValueError; request or attachment files that are missing or unreadable raise anOSErrorwith the path, such asFileNotFoundError. Failed requests and responses that don't fit the WES models raiseRuntimeError. If starting a run fails after submission, the error may mean the server started the run but its acknowledgement was lost or malformed; do not retry automatically.
RunRequest holds a run request, like TesTask holds a task: build one with
RunRequest(request) from a dict or JSON text, or with RunRequest.from_dict(d),
RunRequest.from_json(text) or RunRequest.from_file(path). Unlike WES.run(), which
reads a str as a path, RunRequest(text) parses a str as JSON. Its fields are properties:
workflow_type, workflow_type_version, workflow_url, workflow_params, tags,
workflow_engine, workflow_engine_version and workflow_engine_parameters. Requests
can be pickled and copied.
The results are typed objects with properties, and keep the JSON that the server sent,
including fields that the WES models don't know: to_dict() returns it as a new dict
and json() as JSON text.
RunLog:run_id,request(aRunRequest, orNone),state,run_log,task_logs_url,task_logsandoutputs.RunSummary:run_id,state,start_time,end_timeandtags.TaskLog:id,name,cmd,start_time,end_time,stdout,stderr,exit_code,system_logsandtes_uri.
Missing fields are None, except state, which is WesState.UNKNOWN then. A state
that WES 1.1 does not define also reads as WesState.UNKNOWN; to_dict() keeps
the server's original value. Nested values, such as run_log, come as dicts
and lists. WesState is a str enum of the
WES states (UNKNOWN, QUEUED, INITIALIZING, RUNNING, PAUSED, COMPLETE,
EXECUTOR_ERROR, SYSTEM_ERROR, CANCELED, CANCELING, PREEMPTED) that works like
TesState, but is a separate class.
To read a WES section of ~/.ga4gh-cli/config.json, use
config.from_file(ServiceType.WES).
The checked-in WES example uploads a workflow file, polls the
run and prints its details. Run it from python/ against a WES service:
The Python test suite runs this same file against a mock service.
Runnable example¶
After installing the bindings with maturin develop, run the checked-in
Python script from python/ with Funnel and Docker running:
The output depends on server timing, for example:
The ID, state, and number of listed tasks depend on the server and timing.
The script creates a task from a Python dict and also works when invoked
from the repository root. Set GA4GH_TES_URL to use another TES API root.
CI runs this exact script against a mock service and checks the submitted task body.
Using DRS¶
DRS is a client for a Data Repository Service (DRS 1.4). It reads object metadata and
access URLs, one at a time or in bulk. It never downloads object bytes:
from ga4gh_sdk import Configuration, DRS, PendingResolution
drs = DRS(Configuration("https://your-drs.example/ga4gh/drs/v1"))
obj = drs.get_object("my-object-id")
print(obj.id, obj.size, obj.checksums)
for method in obj.access_methods or []:
print(method["type"], method.get("access_id"))
access = drs.get_access_url("my-object-id", "my-access-id")
# Fetch the bytes from access.url with access.headers; don't log signed URLs.
# The server may still be preparing an answer (202 Accepted):
answer = drs.get_object_once("my-object-id")
if isinstance(answer, PendingResolution):
print("ask again in", answer.retry_after, "seconds")
obj = drs.resolve_object("my-object-id", timeout=120, max_attempts=10)
report = drs.get_bulk_objects(["id-1", "id-2"])
for outcome in report.outcomes:
if outcome.is_success:
print(outcome.object_id, outcome.resolved[0].size)
else:
print(outcome.object_id, outcome.failure_codes, outcome.pending, outcome.errors)
urls = drs.get_bulk_access_urls({"id-1": "my-access-id"})
The checked-in DRS example reads one object without
printing any access URL or header. Run it from python/ after installing the
bindings:
The Python test suite runs this file against a mock DRS service.
DRS(config)reads the service info and raisesRuntimeErrorif the service isn't reachable or isn't a DRS.DRS.get_object(object_id)returns aDrsObject.DRS.get_access_url(object_id, access_id)returns anAccessURL;access_idis theaccess_idof one of the object'saccess_methods. IDs are sent as one URL path segment each. Theget_object_raw()andget_access_url_raw()variants return the complete JSON object as adictwithout applying the generated models.- A DRS server can answer
202 Acceptedwith aRetry-Afterheader while it prepares an object or URL.get_object()andget_access_url()then raiseRuntimeError.get_object_once()andget_access_url_once()send one request and return aPendingResolutioninstead, withretry_after(seconds),endpointandapi_root(without credentials).resolve_object()andresolve_access_url()wait and ask again as the server says, up tomax_attemptsrequests (default 5) andtimeoutseconds in all (default 60), then raiseRuntimeError. DRS.get_bulk_objects(ids, passports=None, policy=None, timeout=60.0)asks for several objects withPOST objects;DRS.get_bulk_access_urls(pairs, ...)asks for access URLs withPOST objects/access.pairsmaps object IDs to access IDs, or is a list of(object_id, access_id)pairs, with one access ID per object per call. Repeated IDs are asked for once.passportsis a list of GA4GH Passport tokens to send in the request body. Both calls return aBulkReportwith oneBulkOutcomeper distinct ID (or pair), sorted by ID, anddiagnosticsfor problems with the server's answer that belong to no ID.dialectsnames the reply shapes seen across the call:standardand, for object lookups,gen3.- The typed bulk calls validate resolved entries against the generated models. The
get_bulk_objects_raw()andget_bulk_access_urls_raw()variants return complete JSONdicts inoutcome.resolvedwhile still correlating IDs and reporting failures. Useget_bulk_objects_raw()for Gen3 objects withcreated_time: null; the typed call records those entries as errors. - A
BulkOutcomehasobject_id,access_id,resolved(theDrsObjects orBulkAccessURLs that the server returned for the ID),failure_codes(such as[404]),pending(aPendingResolutionif the server is still preparing the answer),errorsandis_success(exactly one result and nothing else). Failures of single IDs and of requests are reported in the outcomes, not raised. For a pending outcome, call the bulk method again for its IDs; itsendpointis the bulk endpoint. BulkPolicy(chunk_size=100, advertised_limit=None, concurrency=4, max_requests=1000, max_split_depth=8, max_attempts=1, initial_delay=0.2, max_delay=10.0)limits a bulk call: IDs per request, parallel requests, requests in all, halvings of a request that the server rejects as too large (413), and retries of network errors, 429 and 503.policy.with_service_info(info)also keeps to themaxBulkRequestLengthof the service info, for examplejson.loads(drs.transport.get("service-info")).- Errors: invalid IDs (empty,
.or..), timeouts, attempt counts, bulk inputs and policies raiseValueErrorbefore a request. Failed requests and responses that don't fit the DRS models raiseRuntimeError; model error messages name missing fields without quoting response values, as those may be credentials. - The waiting calls block for up to
timeoutseconds without the GIL; Ctrl+C takes effect after they return.
DrsResolver turns hostname-based DRS URIs (drs://<host>/<object-id>) into an
AccessPlan: the access method to use, its url and headers (a list of (name,
value) pairs, unlike the "Name: value" strings of AccessURL), and its region:
from ga4gh_sdk import Configuration, DrsResolver, DrsUri, Transport
resolver = DrsResolver().with_service(
"drs.example.org", Transport(Configuration("https://drs.example.org/ga4gh/drs/v1"))
)
plan = resolver.access_plan("drs://drs.example.org/my-object-id", protocols=["https", "s3"])
print(plan.method, plan.region) # use plan.url and plan.headers to fetch the bytes
uri = DrsUri("drs://drs.example.org/my-object-id")
print(uri.authority, uri.object_id, uri.default_api_root())
with_service(authority, transport)asks the DRS service oftransportfor URIs with that host, using the transport's API root and settings. Other hosts are asked anonymously athttps://<host>/ga4gh/drs/v1; settings are never sent to them. Add services before sharing a resolver between threads: while another thread plans access with it,with_service()raisesRuntimeError.access_plan(uri, protocols=None, regions=None, method_index=None, timeout=60.0, max_attempts=5)reads the object, picks an available access method (byprotocolsandregions, in order of preference, or the one atmethod_index), and reads its access URL if the method has none. It waits while the server prepares an answer, asresolve_object()does.access_plan_once()sends each request once and returns aPendingResolutioninstead.- Compact identifiers (
drs://prefix:accession) are not supported; they and malformed URIs raiseValueError. No usable access method raisesRuntimeError.
The results are typed objects with properties, and DrsObject, AccessURL and
BulkAccessURL keep the JSON that the server sent, including fields that the DRS models
don't know: to_dict() returns it as a new dict and json() as JSON text.
DrsObject:id,name,self_uri,size,created_time,updated_time,version,mime_type,checksums,access_methods,contents,descriptionandaliases. Nested values, such asaccess_methods, come asdicts andlists.AccessURL:urlandheaders(a list of"Name: value"strings).BulkAccessURL:drs_object_id,drs_access_id,urlandheaders.
Access URLs and headers may hold credentials, such as signatures or bearer tokens. The
repr() of AccessURL, BulkAccessURL, BulkOutcome and AccessPlan doesn't show
them.
To read a DRS section of ~/.ga4gh-cli/config.json, use
config.from_file(ServiceType.DRS). The Rust methods behind these classes are
listed in the usage guide; DRS::resolve_reference, which
resumes from a stored reference, is not bound yet.
Using TRS¶
TRS is a client for a Tool Registry Service (TRS 2.0.1 and 2.0.2), such as WorkflowHub
or Dockstore. It reads tools, their versions, descriptors and files through the Rust TRS
client, so it checks IDs, versions and filters as the Rust SDK and the CLI do:
from ga4gh_sdk import TRS, Configuration
trs = TRS(Configuration("https://workflowhub.eu/ga4gh/trs/v2"))
print(trs.api_version, trs.version_source) # 2.0.1 declared
page = trs.list_tools(tool_class="Workflow", limit=10)
for tool in page.tools:
print(tool.id, tool.name, [version.id for version in tool.versions])
if page.next_page:
page = trs.next_tools_page(page.next_page)
version = trs.get_version("26", "1")
print(version.name, version.descriptor_type)
descriptor = trs.get_descriptor("26", "1", "CWL") # JSON: content, url, checksum
plain = trs.get_descriptor("26", "1", "PLAIN_CWL") # the bare file, as content
other = trs.get_descriptor("26", "1", "CWL", relative_path="ro-crate-metadata.json")
for file in trs.list_files("26", "1", "CWL"):
print(file.file_type, file.path)
archive = trs.get_files_zip("26", "1", "CWL") # bytes, at most 64 MiB by default
The checked-in TRS example reads a version, its descriptor and its
file list. Run it from python/:
The Python test suite runs this file against a mock TRS.
TRS(config, *, api_version="auto", tag_filter="auto", discovery=True)reads the service info and raisesRuntimeErrorif the service isn't reachable or isn't a TRS. It reads responses by the TRS version that the service declares:api_version="auto"uses 2.0.1 or 2.0.2 as declared. A service that declares another version, or none, is read as 2.0.1, andversion_sourceis"baseline": the SDK hasn't checked that the service works with that.api_version="strict"raisesRuntimeErrorfor such a service instead.api_version="2.0.1"or"2.0.2"reads responses by that version, whatever the service declares.discovery=Falsesends no service info request; it needsapi_version="2.0.1"or"2.0.2", and doesn't check that the service is a TRS.tag_filter="enabled"or"disabled"says whether the server supports thetagsfilter oflist_tools(), whatever its TRS version says. Use it only if you know the server. Invalid options raiseValueErrorbefore a request.api_versionis the TRS version that responses are read with,declared_versionthe version that the service info declares (Nonewithout discovery), andversion_sourcewhere the choice comes from:"declared","caller"or"baseline".tag_filteris"specified"(TRS 2.0.2) or"caller_enabled"iflist_tools(tags=...)may be sent; otherwise it is"not_specified","caller_disabled"or"unknown".discovered_service_infois the service info as adict;service_info()reads it again.TRS.list_tools(*, id=None, alias=None, tool_class=None, descriptor_type=None, registry=None, organization=None, name=None, toolname=None, description=None, author=None, checker=None, tags=None, offset=None, limit=None)returns aToolPage. The filters are keyword-only, and unset filters are not sent.limitmust be between 1 and 2147483647, else it raisesValueError.tagsis a list of tags (TRS 2.0.2); they are sent as one comma-separated value, so a tag that is empty or contains a comma raisesValueError, as do tags for a server whosetag_filterdoesn't allow them. Requested filters are never dropped silently.- A
ToolPagehastools(a list ofTool) and the TRS paging headers:next_page,last_page,self_link,current_offsetandcurrent_limit(strs, orNoneif the server didn't send them; a repeated header gives its first value);headers()returns all response headers as adict, by lowercase name, with repeated headers joined by", "and headers that aren't text left out. They can hold cookies, so don't log them.TRS.next_tools_page(page.next_page)gets the next page. It follows only URLs of the tools endpoint of the client's API root, and raisesRuntimeErrorfor others before a request, so credentials never go to another server. TRS.get_tool(id)returns aTool,TRS.list_versions(id)a list ofToolVersions andTRS.get_version(id, version_id)oneToolVersion. Some registries, such as Dockstore, take the version'snameasversion_id.TRS.get_descriptor(id, version_id, descriptor_type, relative_path=None)returns the primary descriptor, or the file atrelative_pathbelow it, as aFileWrapper.descriptor_typeis a TRS descriptor type such as"CWL","WDL","NFL","GALAXY"or"SMK", or the same withPLAIN_, for which the server sends the bare file; it comes as aFileWrapperwith onlycontentset. Types are sent as given, also those that TRS doesn't define, so use the spec's uppercase names.TRS.list_tests(id, version_id, descriptor_type)returns the test parameter files asFileWrappers; forPLAIN_types, a bare test file comes as oneFileWrapper.TRS.get_containerfiles(id, version_id)returns the container specifications, such as Dockerfiles, asFileWrappers.TRS.list_files(id, version_id, descriptor_type)returns the version's files asToolFiles withpathandfile_type.TRS.get_files_zip(id, version_id, descriptor_type, limit=None)downloads them as one ZIP archive and returns it asbytes; it reads at mostlimitbytes into memory (None: 64 MiB), and alimitbelow 1 raisesValueError. Both take descriptor types withoutPLAIN_, such as"CWL". A larger archive, or a response that isn't a ZIP archive (such as the JSON file list of a registry that ignoresformat=zip), raisesRuntimeError. The archive isn't extracted or checked further.TRS.list_tool_classes()returns the registry's tool classes asdicts.- Credentials set on the configuration (see Authentication) go with
the service info request and every TRS request, including
next_tools_page(), which only follows URLs at the client's API root. - IDs, version IDs and relative paths are sent unchanged, as one percent-encoded URL path
segment each, so IDs such as Dockstore's
#workflow/github.com/org/repowork. Empty values,.and..raiseValueErrorbefore a request. - Errors: failed requests, such as a missing tool (
404), and responses that don't fit the TRS models raiseRuntimeError; the message has the HTTP status, not the server's response body. A version'sdescription(TRS 2.0.2) that isn't a string is kept in the JSON, and thedescriptionproperty isNonethen.
The results are typed objects with properties. They hold the JSON of the Rust SDK's TRS
models, including fields that those models don't know: to_dict() returns it as a new
dict and json() as JSON text. Fields of the TRS models that the server sent as null
are left out.
Tool:id,url,aliases,organization,name,toolclass(adict),description,meta_version,has_checker,checker_urlandversions(a list ofToolVersion).ToolVersion:id,url,name,author,description,is_production,images,descriptor_type(a list ofstr),descriptor_type_version,containerfile,meta_version,verified,verified_source,signedandincluded_apps.FileWrapper:content,url,checksumandimage_type. Itsrepr()shows the length ofcontent, not the content.ToolFile:path,file_type(as the server sent it, such as"PRIMARY_DESCRIPTOR") andchecksum.
Nested values, such as images and checksum, come as dicts and lists. To read a TRS
section of ~/.ga4gh-cli/config.json, use config.from_file(ServiceType.TRS). The
streaming ZIP download (responses::files_zip) of the Rust SDK is not bound yet; use
TRS.get_files_zip() with a limit.
Extending the bindings¶
The bindings are meant as a base for clients of services that add endpoints or
fields to the GA4GH APIs. Configuration, ServiceInfo, Transport, TES,
Task, TesTask, ListTasksParams, and ListTasksResponse can be subclassed,
as can WES, Run, RunRequest, RunLog, RunSummary, RunListResponse,
TaskLog, and TaskListResponse.
ServiceType, TesState, and WesState are enums and are not subclassable.
The base classes initialize in __new__. A subclass that needs extra
constructor arguments can define __init__ and call super().__init__()
without arguments. For TES and WES, service-info discovery happens before the
subclass's __init__, so those extra arguments cannot change its first
request. A subclass without its own __init__ accepts only the base arguments.
DRS, DrsResolver, BulkPolicy and DrsUri can be subclassed too. Like TES,
DRS reads the service info before a subclass's __init__ runs.
DrsResolver.with_service() returns the resolver itself, so it keeps its subclass. The
result classes (DrsObject, AccessURL, BulkAccessURL, PendingResolution,
BulkOutcome, BulkReport and AccessPlan) allow subclasses as well, for Rust crates
that extend them, but the clients return instances of the base classes.
TRS can be subclassed in the same way; it reads the service info, unless
discovery=False, before a subclass's __init__ runs. Its keyword arguments
(api_version, tag_filter and discovery) are taken by the base class; a subclass's
__init__ receives them too. The result classes (Tool, ToolVersion, FileWrapper,
ToolFile and ToolPage) allow subclasses for Rust crates, like those of DRS.
Further endpoints¶
Transport sends requests below the configured API root and returns the
response body as text. TES and Task expose their .transport, so a
subclass reuses the client's configuration. The checked-in
extension example adds a service-specific
provenance endpoint, encodes the task ID as one URL path segment, and parses
the response as JSON. Run it from python/ against a TES service that provides
that extra endpoint:
The Python test suite runs this same file against a mock service.
The methods are get(endpoint, params=None), post(endpoint, data=None, params=None),
put(endpoint, data, params=None), delete(endpoint, params=None) and
post_multipart(endpoint, fields=None, files=None, params=None):
endpointis relative to the API root, for exampletasks/123. Encode IDs that you put into it yourself, for example withurllib.parse.quote(task_id, safe="").datais any object thatjson.dumpsaccepts. Astris taken as JSON text, so passjson.dumps(text)to send a JSON string.paramsholds query parameters, as adictor a JSON object string. Values are strings, numbers, booleans or lists of these; a list repeats the parameter ({"tag_key": ["a", "b"]}givestag_key=a&tag_key=b), andNoneor an empty list leaves it out. Keys are sorted. Keys and values are form encoded (a space becomes+), as in the Rust SDK, and are appended after a query already inendpoint.- Other values, such as a
dict, a nested list orNoneinside a list, raiseValueErrorbefore a request is sent. - Behavior change: earlier versions left out
Nonelist items. Remove them yourself, keepingtag_keyandtag_valueitems aligned, since TES pairs them by position. fieldsis a list of(name, value)text parts;filesis a list of(name, filename, content)or(name, filename, content, content_type)tuples, withcontentasbytes. Names may repeat, as for WESworkflow_attachment. This illustrates the multipart API; to start a WES run, useWES.run()instead:
from pathlib import Path
from ga4gh_sdk import Configuration, Transport
transport = Transport(Configuration("https://your-wes.example/ga4gh/wes/v1"))
transport.post_multipart(
"runs",
fields=[("workflow_type", "CWL"), ("workflow_type_version", "v1.2")],
files=[("workflow_attachment", "main.cwl", Path("main.cwl").read_bytes())],
)
A failed request raises RuntimeError; invalid values raise ValueError, and arguments
of the wrong type (for example a dict for fields) raise TypeError.
Fields the models don't know¶
TES.get() and the tasks of TES.list() are TesTask objects. Besides their typed
fields, they keep the task's JSON as the server sent it, including fields that the TES
models don't know: task.json() returns it as JSON text (object keys may come in another
order), task.to_dict() as a dict. TES.create() sends such fields as given, too.
TesTask was called PyTesTask before; the old name still works.
Listing tasks¶
ListTasksParams takes the filters of the TES ListTasks request as keyword arguments:
from ga4gh_sdk import ListTasksParams
params = ListTasksParams(state="running", tag_key=["project"], tag_value=["demo"], page_size=50)
page = tes.list(params)
for task in page.tasks:
print(task.id, task.state)
next_page = tes.list(ListTasksParams(page_token=page.next_page_token))
state accepts a TesState or any TES state name in any case, and raises ValueError
for others; the state property returns a TesState.
view accepts "MINIMAL", "BASIC" or "FULL" in any case; the property and
request use the uppercase TES spelling. Invalid views raise ValueError before a request.
Rust crates¶
The ga4gh crate in this directory builds as a Python module and as a Rust library. A
PyO3 crate (same PyO3 version) can depend on it and extend the classes in Rust, for
example with #[pyclass(extends = PyTES)]. Existing extension crates need to update
their direct pyo3 dependency to 0.30.1 (and pyo3-async-runtimes to 0.30.0 if
used) when updating ga4gh. Migrate removed PyO3 APIs such as PyObject,
Python::with_gil, and Bound::downcast to Py<PyAny>, Python::attach, and
Bound::cast, respectively; see the PyO3 migration guide.
The inner field of each client and handle class holds the
library type (for PyAsyncTES, a LazyTES that connects on first use); result classes
have accessors instead (see Result classes).
Read the state of a library client through its accessors, such as
self.inner.transport() and self.inner.configuration() of PyTES: its
fields, and those of Discovered and Transport, are private (see the
replacements). The Python getters,
such as TES.config and TES.transport, don't change:
// Before: read the former public field
let transport = self.inner.transport.clone();
// After
let transport = self.inner.transport().clone();
let root = &self.inner.configuration().base_path;
The TES calls that the sync and the async classes share are public functions in clients::tes, such as
get_task() and task_status(); an async extension method can run them with
clients::tes_async::spawn_io inside clients::tes_async::future_into_py, as
clients::tes_async does.
The extension module gets its own copies of the base classes,
which don't mix with those of the ga4gh_sdk module: a ga4gh_sdk.Configuration can't be passed
to it. So register every base class that its users need
(m.add_class::<PyConfiguration>() and so on, and m.add("TesState",
tes_state_type(py)?) for the state enum) in the extension module, and import everything
from there. The default feature extension-module builds the Python module; a dependent
crate can turn it off with default-features = false and enable it in its own features
instead.
clients::tes::register(py, m) registers all TES classes at once: the sync and the async
ones, the PyTesTask alias, the TesState enum and WaitTimeoutError. Use it instead of
adding these one by one; m.add_class and tes_state_type remain for an extension module
that needs only some of them.
The DRS classes (PyDRS, PyDrsResolver and so on, in clients::drs) follow the same
pattern; clients::drs::register(py, m) registers all of them.
The WES classes (PyWES, PyRun and so on, in clients::wes) follow the same pattern;
clients::wes::register(py, m) registers all of them, with the WesState enum.
The TRS classes (PyTRS, PyTool and so on, in clients::trs) follow it too;
clients::trs::register(py, m) registers all of them. For Rust code, PyTRS::new(config)
builds a client with the default options and PyTRS::with_options(config, options,
discovery) one with TrsOptions; its inner field holds the library's TRS.
PyToolPage keeps the response's HeaderMap in headers.
PyAsyncTES keeps its state in its transport field:
- The first call connects the
LazyTESwith that transport, so the service check and all built-in calls use it. Theconfiggetter givestransport.configuration(). PyAsyncTES::with_transport(transport)builds a client with a given transport, for example one fromTransport::with_client. Don't replacetransporton a client that may be connected: a connected client keeps the transport it connected with.PyAsyncTES::connection()gives theLazyTESand the transport, for a call to move tospawn_io;clients::tes_async::connected(tes, transport)gives the connected client.- The public
configfield is a copy of the configuration that the client was built with. The SDK doesn't read it, so changing it has no effect.
Migration: earlier versions connected a new TES from the config field, and
connected() took a Configuration. Pass the transport to connected(), and change the
client's configuration or HTTP client through with_transport instead of config.
Result classes¶
The result classes that keep JSON describe one JSON value: PyTesTask,
PyTesListTasksResponse, PyRunRequest, PyRunLog, PyTaskLog, PyRunSummary,
PyRunListResponse, PyTaskListResponse, PyDrsObject, PyAccessURL,
PyBulkAccessURL, PyTool, PyToolVersion, PyFileWrapper and PyToolFile. Read the
value with raw() and its model with inner(); the list responses also return their
items with tasks(), runs() and task_logs(). The typed getters, json() and
to_dict() describe that value, including fields that the models don't know.
For PyTesTask and PyRunRequest, pickling and the request that TES.create()
or WES.run() sends also use that value.
The classes have no setters. To change a value, change a copy of raw() and build a new
instance: with from_value for PyTesTask and PyRunRequest, and with from_json for
the other classes that keep a server's JSON. The TRS classes keep the JSON of their model,
so build them from a changed model with new.
let mut raw = task.raw().clone();
raw["name"] = json!("changed");
let task = PyTesTask::from_value(raw)?;
The inner and raw fields of these classes, and the tasks, runs and task_logs
fields of the list responses, are private. A change to one of them would not change the
others, and the getters, json() and the request sent could then disagree. Python code is
not affected. In an extension crate, replace task.inner with task.inner(), task.raw with
task.raw(), and page.tasks with page.tasks(), which returns a copy; replace a struct
literal with the constructors above.
Documentation¶
For more information on PyO3, visit the PyO3 documentation.
Contributing¶
Contributions are welcome! Please open an issue or submit a pull request.