Orion Client Types

Copyright (C) 2023 Cadence Design Systems, Inc. (Cadence)

Storage Types

Datasets

class orionclient.types.Dataset(path: str | None = None, id: int | None = None, name: str = '', description: str = '', dirty: bool = False, created='', count: int = NOTHING, owner: int | None = None, project: int | None = None, systemtags: list = [], deleted='', session=None, buffer=None, written: int = 0, buffer_count=NOTHING)

A set of OERecords stored in Orion that can be queried and viewed quickly.

Suggested to use ShardCollections for large numbers of OERecords (Greater than a million OERecords)

Creating a Dataset:

# Automatically will perform conversion to OERecords
ds = Dataset.upload(
    APISession,
    "Molecular File",
    "/path/to/file.sdf"
)
classmethod create(session, name, project=None, folder_path=None)

Creates an empty dataset

Parameters:

  • session (OrionSession) – Authenticated OrionSession

  • name (string) – Name of the Dataset

  • project (int) – Project the Dataset belongs to

  • folder_path (string) – Path of folder which the Dataset will be assigned to

Returns:

Instance of a Dataset

Return type:

Dataset

Raises:
download(record_ids=None, offset=None, limit=None, format='binary', fieldlist: List[str] | None = None)

Download the dataset, either the entirety or the specified set of records specified as a blob

Parameters:

  • record_ids (list) – List of record ids to download

  • offset (int) – Relative offset into the dataset

  • limit (int) – Limit number of records read

  • format (string) – Download format of records (binary or json)

  • fieldlist (List[string]) – List of fields to be included. Leave None to return all fields.

Returns:

Iterator of bytes

Raises:

ValidationError – If the records list isn’t a list of ids

download_to_file(filename, **kwargs)

Download the complete dataset to a file

Parameters:

  • filename (string) – Location to write the dataset to

  • record_ids (list) – List of record ids to download

finalize()

Finalizes the dataset to update the count and indicate that the dataset is safely consumable

flush()

Flushes the underlying buffer that accumulates records before they are written to the Orion API

records(record_ids=None, offset=None, limit=None, fieldlist: List[str] | None = None)

An iterator of all of the records from the dataset or a subset as specified by record_ids. Returns OEMolRecords

Parameters:

  • record_ids (list) – List of record ids to retrieve from the dataset

  • offset (int) – Relative offset into the dataset

  • limit (int) – Limit number of records read

Returns:

Iterator of Records

classmethod upload(session, name, file_path, project=None, path: str | None = None)

Creates a dataset using a local file, which can be any OEChem readable format, OEDB, OEDU, or CSV.

Parameters:

  • session (OrionSession) – Authenticated OrionSession

  • name (string) – Name of the Dataset

  • file_path (string) – Path of the dataset to upload to Orion

  • project (Project/integer) – Project to associate with the upload

  • path (string) – Path to the folder to which the dataset belongs

Returns:

Instance of Dataset

Return type:

Dataset

Raises:
write(record)

Appends a record to a dataset and mark the dataset as dirty if it isn’t already dirty.

Parameters:

record (OERecord) – Record to append to the dataset

Files

class orionclient.types.File(path: str | None = None, id: int | None = None, session=None, name: str = '', state: str = '', reason: str = '', multipart: bool = False, created='', size: int = NOTHING, owner: int | None = None, project: int | None = None, deleted='')

A file stored by Orion that can be accessed from within floes or locally.

Creating a File:

file = File.upload(
    APISession,
    "Local file",
    "/path/to/file.ext"
)
complete()

Completes a multipart file

Raises:
classmethod create_multipart_file(session, name, project=None, folder_path: str | None = None)

Creates a file that allows for uploading in parts.

Parameters:

  • session (OrionSession) – Authenticated OrionSession

  • name (string) – Name of the File

  • project (Project/integer) – Project to associate with the upload

  • folder_path (string) – Path of the folder which the file will be assigned to

Returns:

Instance of File

Return type:

File

Raises:

BadResponse – Failure to create file

download(chunk_size=1048576)

An iterator that returns chunks of bytes that makes up the file

Parameters:

chunk_size (int) – Size of each chunk

Returns:

Iterator of bytes

Return type:

iter

Raises:

  • AuthorizationRequired – If session doesn’t have valid credentials

  • IOError – If the total amount of data yielded doesn’t match the size of the Content-Length header

download_to_file(filename, chunk_size=1048576)

Download the file to a local file path

Parameters:

  • filename (string) – Path of the file to download the file to

  • chunk_size (int) – Size of each chunk

Return type:

None

Raises:

AuthorizationRequired – If session doesn’t have valid credentials

classmethod upload(session, name, path, project=None, folder_path: str | None = None)

Uploads a file to the Orion API

URLs passed as a path must have a protocol, domain and path to be considered valid.

Example: https://example.com/some/path/file.extension

Parameters:

  • session (OrionSession) – Authenticated OrionSession

  • name (string) – Name of the File

  • path (string) – Either a local file path or a URL to upload to Orion

  • project (Project/integer) – Project to associate with the upload

  • folder_path (string) – Path of the folder which the file will be assigned to

Returns:

Instance of File

Return type:

File

Raises:
upload_part(data, attempts=3)

Uploads a chunk of data as part of a file. Must be called sequentially.

Parameters:

  • data (bytes) – Data to upload as a chunk

  • attempts (int) – Number of attempts to upload the chunk

Raises:

Collections and Shards

class orionclient.types.ShardCollection(path: str | None = None, id: int | None = None, name: str = '', state: str = '', owner: int | None = None, project: int | None = None, size: int = NOTHING, created='', deleted='', reason: str = '', metadata: dict = NOTHING, v2: bool = False, session=None)

Collections are a way of storing arbitrary quantities of Binary blobs, which can be defined as the user sees fit. ShardCollections are particularly useful in Workfloes as they allow for IO to be parallelized to much larger scales. They are handled by creating Shards, which is a reference to an object store that uses signed URLs to provide the user with the ability to upload and retrieve data.

Collection states:

  • “open”: Collection can have shards added, may contain temporary shards

  • “ready”: Collection is closed, contains only ready shards

  • “processing”: Collection is being closed, may contain temporary shards

  • “error”: Collection processing or deletion encountered an error, check reason

  • “deleting”: Collection is being deleted

Creating a ShardCollection:

collection = ShardCollection.create(
    APISession,
    "Corporate Database",
    metadata={
        # Arbitrary JSON as metadata
        "virtual": False,
    }
)
shard = Shard.create(
    collection,
    name="Shard 1",
    metadata={"actives": True}
)
close()

Completes the ShardCollection, making it no longer possible to create new shards associated with the collection

The collection enters the processing state and then ready if successful.

classmethod create(session, name, metadata=None, project=None, folder_path: str | None = None, v2: bool = True)

Creates an empty ShardCollection

Parameters:

  • session (OrionSession) – Authenticated OrionSession

  • name (string) – Name of the ShardCollection

  • metadata (dict) – User-specified metadata

  • project (integer or Project instance) – Project the ShardCollection belongs to

  • folder_path – Path of the folder which the ShardCollection will be assigned to

  • v2 (bool) – Make a v2 collection

Returns:

Instance of ShardCollection

Return type:

ShardCollection

Raises:
get_count()

Returns the number of shards contained within the Collection

Returns:

Number of shards

Return type:

int

list_shards(shard_ids: List[int] | None = None, filters: Dict[str, Any] | None = None, session: OrionSession | None = None) → Iterator[Shard]

An iterator that returns all of the Shards that are associated with the collection.

All shards in a collection are listed regardless of state, but only “ready” shards should be read or downloaded. See Shard documentation for more details.

Returns:

Iterator of Shards

open()

Reopens the ShardCollection, making it possible to create new shards associated with the collection

The collection enters the open state if successful.

update(name=None, metadata=None)

Updates the name or metadata of the ShardCollection

Parameters:

  • name (string) – Name of the ShardCollection

  • metadata (dict) – User-specified metadata

Raises:

ValidationError – If parameters are wrong type

class orionclient.types.Shard(id: int | None = None, uri_prefix: str = '', session=None, collection=None, signed_url: str | None = None, kms_id: str | None = None, response_headers: list = NOTHING, v2: bool = False, name: str | None = None, s3_response: dict = NOTHING, state: str = '', size: int = NOTHING, metadata: dict = NOTHING, is_multipart: bool = False)

A single file that makes up a ShardCollection. Used for parallelizing IO in Parallel Cubes.

Max file size of 5Gb per shard for non-multipart shards (v1 & v2 collections), Max file size of 5Tb per shard for multipart shards (v2 collections only)

Shard states:

  • “open”: Shard is empty and can be modified

  • “temporary”: Shard has data associated with it but is not ready yet

  • “error”: An error has occurred with the shard

  • “ready”: Shard has data associated with it, is safe to use, and

    cannot be modified. See Shard’s close method to mark a shard as “ready”

When you first create a shard, its state is “open”. Any shard not in “ready” is deleted when the associated collection is closed.

Creating a Shard:

# Requires a collection to be created
shard = Shard.create(
    collection,
    name="Shard 1",
    metadata={"actives": True}
)
shard.upload_file("path/to/file.ext")
close()

Closes the shard, indicating that the shard has content that the user wants to persist

Changes state of shard from temporary to ready.

Warning

Shards can be marked ready immediately after upload in a serial cube, However, shards must only be marked ready downstream of the parallel cube or parallel cube group that created them. Otherwise, duplicate shards could be created when work is retried.

classmethod create(collection, name='', metadata=None, multipart: bool = False)

Creates a Shard to be a part of a ShardCollection

Parameters:

  • collection (ShardCollection) – ShardCollection to associate the shard with

  • name (string) – Name of the Shard

  • metadata (dict) – User-specified metadata

  • multipart – Whether or not shard should be uploaded directly or in parts

Returns:

Instance of Shard

Return type:

Shard

Raises:

AuthorizationRequired – If session doesn’t have valid credentials

download(chunk_size=1048576)

An iterator that returns chunks of bytes that makes up the shard

Parameters:

chunk_size (int) – Size of each chunk

Returns:

Iterator of bytes

Return type:

iter

Raises:

  • AuthorizationRequired – If collection session doesn’t have valid credentials

  • IOError – If the total amount of data yielded doesn’t match the size of the Content-Length header

download_to_file(filename, chunk_size=1048576, attempts=10)

Downloads the shard to a local file

Parameters:

  • filename (string) – Local file path to download file to

  • chunk_size (int) – Size of each chunk

  • attempts (int) – Number of create/upload attempts (not request-level retries)

Raises:

AuthorizationRequired – If collection session doesn’t have valid credentials

update(name=None, metadata=None)

Updates the name or metadata of the Shard

Parameters:

  • name (string) – Name of the Shard

  • metadata (dict) – User-specified metadata

Raises:

ValidationError – If parameters are wrong type

upload(shard_data, verbose=False)

Uploads the content of a file handler to the shard. Must be opened with ‘rb’, not ‘r’

Parameters:

shard_data (File) – File handler to file to upload

Raises:

AuthorizationRequired – If collection session doesn’t have valid credentials

upload_file(filename, attempts=10, verbose=False)

Uploads a file to the shard

Warning

This may result in the shard’s id being updated. Do not emit a shard or save its ID until after an upload succeeds.

Parameters:

  • filename (string) – Local file path to upload

  • attempts (int) – Number of create/upload attempts (beyond request-level retries)

Returns:

Instance of Shard

Raises:

Collections V2

As of release 2023.1, Orion includes a new implementation of Collections, called Collections V2. The V2 Collection is designed to support abritrary numbers of concurrent shard operations without impacting other Orion users. It is a drop-in replacement, with one extra parameter to be optionally specified when creating collections. These can only be created with orion platform versions >= 5.*, but old versions of orion platform and orion client will also be able to read these collections.

There are some differences in behavior to be aware of, however:

  • Filtering by name is no longer supported.

  • For an open collection, listing shards will not include the shard’s metadata. A shard’s metadata can be retrieved on request.

  • Shard metadata is limited to 2 KB.

  • Shards will not register as closed until the collection is closed.

  • Updates to shard metadata and name have high latency.

  • Old clients reading V2 Collection shards will be slower.

Aside from accounting for the behavioral changes, the only change required to use V2 Collections is

collection = ShardCollection.create(session, name="mycollection", v2=True)

via the python API, and

ocli collections upload --v2 mycollection somefile

via the command line.

Tags

class orionclient.types.OrionTag(id: int | None = None, name: str = '', parent: int | None = None, owner: int | None = None, project: int | None = None)

Represents a private or system tag. Can be set on most resources for organization or discovery purposes.

Tagging a resource:

APISession.tag_resource(file_obj, "tag 1", "tag 2", "tag 3")

Retrieving tags:

tags = APISession.list_tags(file_obj)

Delete a Tag:

APISession.delete_resource(OrionTag(id=618))

Classifications

class orionclient.types.Classification(id: int | None = None, path: str = '')

The description of a Classification that is on Orion.

List Classifications:

classifications = APISession.list_resources(Classification)
for classification in classifications:
    print(classification)

Projects

class orionclient.types.Project(id: int | None = None, name: str = '', description: str = '', owner: int | None = None, statistics=NOTHING)

An Orion Project resource type. A project is a location in which Datasets and Files can be stored which pertain to a specific goal.

Creating a Project:

project = APISession.create_resource(
    Project,
    params={"name": "My personal project"}
)

Secrets

class orionclient.types.Secret(id: int | None = None, name: str = '', owner: int | None = None, created='', last_updated='', description: str = '', value: str = '', session=None)

Orion Secrets are a way of storing values in an encrypted state that can be retrieved for later use.

Creating a Secret:

secret = APISession.create_resource(
    Secret,
    params={
        "name": "Database Password",
        "description": "The password to the corporate collection database",
        "value": "corporate-password",
    }
)
# Update the secret post-creation
secret.set_value("new-corporate-password")
set_value(value)

Sets the value of the secret.

Parameters:

value (string) – Secret value to store in Orion

Raises:

BadResponse – Fails to set the value

Floe Types

WorkFloePackage

class orionclient.types.WorkFloePackage(id: int | None = None, session=None, uuid: str | None = None, created='', source_code=None, state: str = '', reason: str = '', version='', owner: int | None = None, size: int = NOTHING, specification=NOTHING, processing_detail=NOTHING)

Represents a Floe Package that contains Workfloes (WorkFloeSpec) and Cubes (Cube).

Upload a Workfloe Package:

package = WorkFloePackage.upload(
    APISession,
    "path/to/package.tar.gz"
)
get_environment_log()

Returns an iterator of bytes that make up the environment log

Returns:

Iterator of Log bytes

Raises:

BadResponse – Unable to make request to log stream

get_inspection_result()

Returns dictionary that describes the results of package inspection

Returns:

Dictionary of inspection

Raises:

BadResponse – Unable to retrieve inspection results

classmethod upload(session, path)

Uploads a WorkFloe Package to Orion

Parameters:

  • session (OrionSession) – Orion Session

  • path (string) – Filepath or URL to the workfloe package

Raises:

ValidationError – Invalid file path or URL

Cube

class orionclient.types.Cube(id: int | None = None, name: str = '', uuid: str | None = None, title: str = '', state: str = '', version: str = '', cube_class: str = '', specification: dict = NOTHING, owner: int | None = None, package: int | None = None)

The description of a Cube that is on Orion.

List Cubes:

cubes = APISession.list_resources(Cube)
for cube in cubes:
    print(cube)

WorkFloeSpec

class orionclient.types.WorkFloeSpec(id: int | None = None, uuid: str | None = None, name: str = '', title: str = '', state: str = '', version: str = '', specification: dict = NOTHING, owner: int | None = None, package: int = NOTHING, cubes: list = NOTHING, systemtags: list = [])

A WorkFloeSpec is the description of a WorkFloe that defines what a job (WorkFloeJob) will perform.

List WorkFloe Specs:

specs = APISession.list_resources(WorkFloeSpec)
for spec in specs:
    print(spec)

WorkFloeJob

class orionclient.types.WorkFloeJob(id: int | None = None, name: str = '', state: str = '', reason: str = '', created='', started='', finished='', job_type: str = '', notify: bool = False, success: bool = False, project: int | None = None, status=NOTHING, parameters=NOTHING, system_parameters=NOTHING, owner: int | None = None, workfloe=None, log_size: int = 0, session=None)

A WorkFloeJob is a specific run a workfloe (WorkFloeSpec).

List WorkFloe Jobs:

jobs = APISession.list_resources(WorkFloeJob)
for job in jobs:
    print(job)

Starting a Job:

# Retrieve a workfloe spec that you wish to run
workfloe_spec = APISession.get_resource(WorkFloeSpec, 814)
job = WorkFloeJob.start(
    APISession,  # Session to start job with
    workfloe_spec,
    "Name of My Workfloe",  # Name of the floe
    {
        "promoted": {"param": 618},
        "cube": {
            "cube1": {"param": "value"},
            "cube2": {"param": "value"}
        },
        "floe": {},
    }
)
cancel()

Cancels a job if it is running at the time

delete_logs()

Sends delete HTTP request to delete the job logs

get_costs()
Returns a dictionary containing cost information for the workfloe and

constituent cubes. Cost is cached server-side for 30 seconds, so call this function every 30 seconds at most.

Example response:

{
  "total": 0.03,
  "costs": [
    {
      "total": 0,
      "name": "Log processor",
      "usage": [],
      "rank": null
    },
    {
      "total": 0.01,
      "name": "Sink",
      "usage": [
        {
          "hours": 0.17,
          "hourly_rate": 0.06,
          "group_name": ...
        }
      ],
      "rank": 1
    },
    {
      "total": 0.01,
      "name": "Source",
      "usage": [
        {
          "hours": 0.17,
          "hourly_rate": 0.06,
          "group_name": ...
        }
      ],
      "rank": 0
    },
    {
      "total": 0.01,
      "name": "Job Controller",
      "usage": [
        {
          "hours": 0.17,
          "hourly_rate": 0.06,
          "group_name": ...
        }
      ],
      "rank": null
    }
  ]
}
logs()

Returns an iterator of bytes that make up the job logs

Returns:

Iterator of bytes

classmethod start(session: OrionSession, workfloe_spec: WorkFloeSpec, name: str, parameters: dict, system_parameters: dict | None = None, project: Project | None = None, notify: bool | None = False, args: list | None = None) → WorkFloeJob

Start will trigger a Job using the spec and parameters that are provided using the credentials that the session is configured for.

Parameters:

  • session (OrionSession) – Authenticated OrionSession

  • workfloe_spec (WorkFloeSpec) – WorkFloeSpec to run job from

  • name (string) – Name of the job

  • parameters (dict) – Parameters for the job to be run with

  • system_parameters (dict) – System parameters for the job to be run with

  • notify (bool) – Send a notification email when job completes

  • args (list<string>) – Command line arguments to be parsed and added to parameters

Returns:

Instance of WorkFloeJob

Return type:

WorkFloeJob

Raises:

AuthorizationRequired – If session doesn’t have valid credentials

An example of starting a Job with a non-default output location, and a $1.00 job execution cost threshold, using system_parameters:

# Retrieve a workfloe spec that you wish to run
workfloe_spec = APISession.get_resource(WorkFloeSpec, 814)
# set system_parameters
out_path = f"/project/3/user/test_modeler/orion-tests-out/"
system_parameters={
    "path": out_path,
    "cost_termination_threshold": 1.00,
}
job = WorkFloeJob.start(
    APISession,  # Session to start job with
    workfloe_spec,
    "Name of My Workfloe",  # Name of the floe
    {
        "promoted": {"param": 618},
        "cube": {
            "cube1": {"param": "value"},
            "cube2": {"param": "value"}
        },
        "floe": {},
    },
    system_parameters=system_parameters
)
trigger_debug_export()

Requests that Orion export debug data for this job

An email will be sent to the email registered to the user profile that triggers the export

Service Types

Services

class orionclient.types.Service(id: int | None = None, owner: int | None = None, name: str = '', description: str = '', url: str = '', metadata: dict = NOTHING, accepts_orion_tokens: bool = False, created='', last_updated='', session=None)

Orion Services is a registry of internal or external services integrated with the larger Orion system.

Creating a Service:

service = APISession.create_resource(
    Service,
    params={
        "name": "Internal Cluster",
        "description": "Internal service that provides API access",
        "url": "special-sauce.company.com/api/",
        "metadata": {
            # Arbitrary JSON as metadata
            "contact": "help@company.com",
        }
    }
)
check(delete_tokens_first: bool = False) → str

Check that Orion and a registered service (specified by id or name) are configured with the same shared secret.

Warning

Do not set delete_tokens_first to True in parallel.

Parameters:

delete_tokens_first (bool) – Delete any existing tokens for this service before checking the service

Returns:

Redirect URL

Return type:

string

Raises:

OrionServiceError – If service appears to be misconfigured

  1. Optionally delete the user’s token(s) that had been created for this service.

  2. Follow the service redirect. The redirect includes a signed payload for the service which it uses to call back and create a token, if it one doesn’t already exist.

  3. Verify that the user has a usable Orion token created for the service.

delete_tokens(delete_timeout: int = 10)

Delete the tokens created for this service

Parameters:

delete_timeout (int) – How long in seconds to wait for each token to be deleted.

Raises:

OrionServiceError – If service appears to be misconfigured

classmethod from_name(session: OrionSession, name: str)

Get service by name

get_redirect(follow: bool = False) → str | None

Gets the redirect URL for this service. If a secret was set on the service, then the redirect URL gets modified by Orion to include a signed querystring payload that is used to authenticate to the service.

Parameters:

follow (bool) – Follow the redirect

Raises:

BadResponse – Fails to get the redirect

list_tokens() → List[Token]

Get a list of the tokens created for this service

set_secret(value: str)

Sets the value of the secret the service shares with Orion.

Parameters:

value (string) – Secret value to store in Orion

Raises:

BadResponse – Fails to set the value

Management Types

Tokens

class orionclient.types.Token(id: int | None = None, description: str = '', value: str = '', owner: int | None = None, created='')

A token that is used for authentication to communicate with the APIs of Orion

List tokens:

for token in APISession.list_resources(Token):
    print(token)

Create token:

token = APISession.create_resource(Token, params={"description": "new token"})

Users

class orionclient.types.OrionUser(id: int | None = None, email: str = '', username: str = '', first_name: str = '', last_name: str = '', administrator: bool = False)

An Orion User.

List Users:

users = APISession.list_resources(OrionUser)
for user in users:
    print(user)
class orionclient.types.UserProfile(id: int | None = None, email='', username: str = '', first_name: str = '', last_name: str = '', organization: int | None = None, staff=False, remote=False, administrator=False, organization_feedback_email: str = '')

The profile that Orion Client is authenticated against

Get current profile:

profile = APISession.get_user_profile()

Instance Groups

class orionclient.types.InstanceGroup(id: int | None = None, instance_detail=NOTHING, usage=0.0, pool='', name='', size=0, min_size=0, max_size=0, desired_size=0, affinity=0.0, cost=0.0, spot=False, num_healthy=0, min_reserve=0.0, state='')

A group of Instances that is maintained by Orion that Cubes run on.

Warning

Does not support get_resource() as groups do not have identifiers. Use list_resources() to access the groups.

List Instance Groups:

groups = APISession.list_resources(InstanceGroup)
for group in groups:
    print(group)

Costs

class orionclient.types.LedgerEntry(id: int | None = None, created='', comment: str = '', creator: int | None = None, owner: int | None = None, entry_type: str = '', amount=Decimal('0.00'), project: int | None = None, job: int | None = None)

Entries in Orion that relate to the costs that users have.

Can only see other user’s entries if authenticated as an Orion Admin.

List Ledger Entries:

entries = APISession.list_resources(LedgerEntry)
for entry in entries:
    print(entry)
class orionclient.types.Balance(id: int | None = None, balance='', timestamp='')

Balances are snapshots of spending on Orion.

List Balances:

for balance in APISession.list_resources(Balance):
    print(balance)
class orionclient.types.CurrentBalance(id: int | None = None, balance=Decimal('0.0'), ytd=Decimal('0.0'), mtd=Decimal('0.0'), last_updated='')

The current balance for the organization.

Get current balance:

balance = APISession.get_current_balance()