Orion Client Types¶
Storage Types¶
Datasets¶
- class orionclient.types.Dataset(path: Optional[str] = None, id: Optional[int] = None, name: str = '', description: str = '', dirty: bool = False, created='', count: int = NOTHING, owner: Optional[int] = None, project: Optional[int] = None, systemtags: list = [], deleted='', session=None, buffer=None, written: int = 0, buffer_count=NOTHING)¶
A set of OERecords stored in Orion that can be queried and viewed quickly.
Suggested to use ShardCollections for large numbers of OERecords (Greater than a million OERecords)
Creating a Dataset:
# Automatically will perform conversion to OERecords ds = Dataset.upload( APISession, "Molecular File", "/path/to/file.sdf" )
- classmethod create(session, name, project=None, folder_path=None)¶
Creates an empty dataset
- Parameters
session (OrionSession) – Authenticated OrionSession
name (string) – Name of the Dataset
project (int) – Project the Dataset belongs to
folder_path (string) – Path of folder which the Dataset will be assigned to
- Returns
Instance of a Dataset
- Return type
- Raises
AuthorizationRequired – If session doesn’t have valid credentials
ValidationError – If project is not an integer or Project
- download(record_ids=None, offset=None, limit=None, format='binary')¶
Download the dataset, either the entirety or the specified set of records specified as a blob
- Parameters
record_ids (list) – List of record ids to download
offset (int) – Relative offset into the dataset
limit (int) – Limit number of records read
format (string) – Download format of records (binary or json)
- Returns
Iterator of bytes
- Raises
ValidationError – If the records list isn’t a list of ids
- download_to_file(filename, **kwargs)¶
Download the complete dataset to a file
- Parameters
filename (string) – Location to write the dataset to
record_ids (list) – List of record ids to download
- finalize()¶
Finalizes the dataset to update the count and indicate that the dataset is safely consumable
- flush()¶
Flushes the underlying buffer that accumulates records before they are written to the Orion API
- records(record_ids=None, offset=None, limit=None)¶
An iterator of all of the records from the dataset or a subset as specified by record_ids. Returns OEMolRecords
- Parameters
record_ids (list) – List of record ids to retrieve from the dataset
offset (int) – Relative offset into the dataset
limit (int) – Limit number of records read
- Returns
Iterator of Records
- classmethod upload(session, name, file_path, project=None, path: Optional[str] = None)¶
Creates a dataset using a local file, which can be any OEChem readable format, OEDB, OEDU, or CSV.
- Parameters
session (OrionSession) – Authenticated OrionSession
name (string) – Name of the Dataset
file_path (string) – Path of the dataset to upload to Orion
project (Project/integer) – Project to associate with the upload
path (string) – Path to the folder to which the dataset belongs
- Returns
Instance of Dataset
- Return type
- Raises
AuthorizationRequired – If session doesn’t have valid credentials
ValidationError – If name isn’t a string or is empty
ValidationError – If project is not a Project or integer
Files¶
- class orionclient.types.File(path: Optional[str] = None, id: Optional[int] = None, session=None, name: str = '', state: str = '', reason: str = '', multipart: bool = False, created='', size: int = NOTHING, owner: Optional[int] = None, project: Optional[int] = None, deleted='')¶
A file stored by Orion that can be accessed from within floes or locally.
Creating a File:
file = File.upload( APISession, "Local file", "/path/to/file.ext" )
- complete()¶
Completes a multipart file
- Raises
ValidationError – If the file is not multipart
BadResponse – Failure to complete
- classmethod create_multipart_file(session, name, project=None, folder_path: Optional[str] = None)¶
Creates a file that allows for uploading in parts.
- Parameters
session (OrionSession) – Authenticated OrionSession
name (string) – Name of the File
project (Project/integer) – Project to associate with the upload
folder_path (string) – Path of the folder which the file will be assigned to
- Returns
Instance of File
- Return type
- Raises
BadResponse – Failure to create file
- download(chunk_size=1048576)¶
An iterator that returns chunks of bytes that makes up the file
- Parameters
chunk_size (int) – Size of each chunk
- Returns
Iterator of bytes
- Return type
iter
- Raises
AuthorizationRequired – If session doesn’t have valid credentials
- download_to_file(filename, chunk_size=1048576)¶
Download the file to a local file path
- Parameters
filename (string) – Path of the file to download the file to
chunk_size (int) – Size of each chunk
- Return type
None
- Raises
AuthorizationRequired – If session doesn’t have valid credentials
- classmethod upload(session, name, path, project=None, folder_path: Optional[str] = None)¶
Uploads a file to the Orion API
URLs passed as a path must have a protocol, domain and path to be considered valid.
Example: https://example.com/some/path/file.extension
- Parameters
session (OrionSession) – Authenticated OrionSession
name (string) – Name of the File
path (string) – Either a local file path or a URL to upload to Orion
project (Project/integer) – Project to associate with the upload
folder_path (string) – Path of the folder which the file will be assigned to
- Returns
Instance of File
- Return type
- Raises
AuthorizationRequired – If session doesn’t have valid credentials
ValidationError – If name isn’t a string or is empty
ValidationError – If name is longer than 255 characters
ValidationError – If project is not an integer or Project
ValidationError – If filename is not a valid path
- upload_part(data, attempts=3)¶
Uploads a chunk of data as part of a file. Must be called sequentially.
- Parameters
data (bytes) – Data to upload as a chunk
attempts (int) – Number of attempts to upload the chunk
- Raises
ValidationError – If the file is not multipart
BadResponse – Failure to upload
Collections and Shards¶
- class orionclient.types.ShardCollection(path: Optional[str] = None, id: Optional[int] = None, name: str = '', state: str = '', owner: Optional[int] = None, project: Optional[int] = None, size: int = NOTHING, created='', deleted='', reason: str = '', metadata: dict = NOTHING, v2: bool = False, session=None)¶
Collections are a way of storing arbitrary quantities of Binary blobs, which can be defined as the user sees fit. ShardCollections are particularly useful in Workfloes as they allow for IO to be parallelized to much larger scales. They are handled by creating Shards, which is a reference to an object store that uses signed URLs to provide the user with the ability to upload and retrieve data.
Collection states:
“open”: Collection can have shards added, may contain temporary shards
“ready”: Collection is closed, contains only ready shards
“processing”: Collection is being closed, may contain temporary shards
“error”: Collection processing or deletion encountered an error, check reason
“deleting”: Collection is being deleted
Creating a ShardCollection:
collection = ShardCollection.create( APISession, "Corporate Database", metadata={ # Arbitrary JSON as metadata "virtual": False, } ) shard = Shard.create( collection, name="Shard 1", metadata={"actives": True} )
- close()¶
Completes the ShardCollection, making it no longer possible to create new shards associated with the collection
The collection enters the processing state and then ready if successful.
- classmethod create(session, name, metadata=None, project=None, folder_path: Optional[str] = None, v2: bool = False)¶
Creates an empty ShardCollection
- Parameters
session (OrionSession) – Authenticated OrionSession
name (string) – Name of the ShardCollection
metadata (dict) – User-specified metadata
project (integer or Project instance) – Project the ShardCollection belongs to
folder_path – Path of the folder which the ShardCollection will be assigned to
v2 (bool) – Make a v2 collection
- Returns
Instance of ShardCollection
- Return type
- Raises
AuthorizationRequired – If session doesn’t have valid credentials
ValidationError – If project is not an integer or Project
- get_count()¶
Returns the number of shards contained within the Collection
- Returns
Number of shards
- Return type
int
- list_shards(shard_ids: Optional[List[int]] = None, filters: Optional[Dict[str, Any]] = None)¶
An iterator that returns all of the Shards that are associated with the collection.
All shards in a collection are listed regardless of state, but only “ready” shards should be read or downloaded. See Shard documentation for more details.
- Returns
Iterator of Shards
- open()¶
Reopens the ShardCollection, making it possible to create new shards associated with the collection
The collection enters the open state if successful.
- update(name=None, metadata=None)¶
Updates the name or metadata of the ShardCollection
- Parameters
name (string) – Name of the ShardCollection
metadata (dict) – User-specified metadata
- Raises
ValidationError – If parameters are wrong type
- class orionclient.types.Shard(id: Optional[int] = None, uri_prefix: str = '', session=None, collection=None, signed_url: str = '', kms_id: str = '', response_headers: list = NOTHING, v2: bool = False, name: str = '', state: str = '', size: int = NOTHING, metadata: dict = NOTHING)¶
A single file that makes up a ShardCollection. Used for parallelizing IO in Parallel Cubes. Max file size of 5Gb per shard
Shard states:
“open”: Shard is empty and can be modified
“temporary”: Shard has data associated with it but is not ready yet
“error”: An error has occurred with the shard
- “ready”: Shard has data associated with it, is safe to use, and
cannot be modified. See Shard’s close method to mark a shard as “ready”
When you first create a shard, its state is “open”. Any shard not in “ready” is deleted when the associated collection is closed.
Creating a Shard:
# Requires a collection to be created shard = Shard.create( collection, name="Shard 1", metadata={"actives": True} ) shard.upload_file("path/to/file.ext")
- close()¶
Closes the shard, indicating that the shard has content that the user wants to persist
Changes state of shard from temporary to ready.
Warning
Shards can be marked ready immediately after upload in a serial cube, However, shards must only be marked ready downstream of the parallel cube or parallel cube group that created them. Otherwise, duplicate shards could be created when work is retried.
- classmethod create(collection, name='', metadata=None)¶
Creates a Shard to be a part of a ShardCollection
- Parameters
collection (ShardCollection) – ShardCollection to associate the shard with
name (string) – Name of the Shard
metadata (dict) – User-specified metadata
- Returns
Instance of Shard
- Return type
- Raises
AuthorizationRequired – If session doesn’t have valid credentials
- download(chunk_size=1048576)¶
An iterator that returns chunks of bytes that makes up the shard
- Parameters
chunk_size (int) – Size of each chunk
- Returns
Iterator of bytes
- Return type
iter
- Raises
AuthorizationRequired – If collection session doesn’t have valid credentials
IOError – If the total amount of data yielded doesn’t match the size of the Content-Length header
- download_to_file(filename, chunk_size=1048576, attempts=10)¶
Downloads the shard to a local file
- Parameters
filename (string) – Local file path to download file to
chunk_size (int) – Size of each chunk
attempts (int) – Number of create/upload attempts (not request-level retries)
- Raises
AuthorizationRequired – If collection session doesn’t have valid credentials
- update(name=None, metadata=None)¶
Updates the name or metadata of the Shard
- Parameters
name (string) – Name of the Shard
metadata (dict) – User-specified metadata
- Raises
ValidationError – If parameters are wrong type
- upload(shard_data)¶
Uploads the content of a file handler to the shard. Must be opened with ‘rb’, not ‘r’
- Parameters
shard_data (File) – File handler to file to upload
- Raises
AuthorizationRequired – If collection session doesn’t have valid credentials
- upload_file(filename, attempts=10)¶
Uploads a file to the shard
Warning
This may result in the shard’s id being updated. Do not emit a shard or save its ID until after an upload succeeds.
- Parameters
filename (string) – Local file path to upload
attempts (int) – Number of create/upload attempts (beyond request-level retries)
- Returns
Instance of Shard
- Raises
AuthorizationRequired – If collection session doesn’t have valid credentials
ValidationError – If file is empty or too large
Collections V2¶
As of release 2023.1, Orion includes a new implementation of Collections, called Collections V2. The V2 Collection is designed to support abritrary numbers of concurrent shard operations without impacting other Orion users. It is a drop-in replacement, with one extra parameter to be optionally specified when creating collections. These can only be created with orion platform versions >= 5.*, but old versions of orion platform and orion client will also be able to read these collections.
There are some differences in behavior to be aware of, however:
Filtering by name is no longer supported.
For an open collection, listing shards will not include the shard’s metadata. A shard’s metadata can be retrieved on request.
Shard metadata is limited to 2 KB.
Shards will not register as closed until the collection is closed.
Updates to shard metadata and name have high latency.
Old clients reading V2 Collection shards will be slower.
Aside from accounting for the behavioral changes, the only change required to use V2 Collections is
collection = ShardCollection.create(session, name="mycollection", v2=True)
via the python API, and
ocli collections upload --v2 mycollection somefile
via the command line.
Tags¶
- class orionclient.types.OrionTag(id: Optional[int] = None, name: str = '', parent: Optional[int] = None, owner: Optional[int] = None, project: Optional[int] = None)¶
Represents a private or system tag. Can be set on most resources for organization or discovery purposes.
Tagging a resource:
APISession.tag_resource(file_obj, "tag 1", "tag 2", "tag 3")
Retrieving tags:
tags = APISession.list_tags(file_obj)
Delete a Tag:
APISession.delete_resource(OrionTag(id=618))
Classifications¶
- class orionclient.types.Classification(id: Optional[int] = None, path: str = '')¶
The description of a Classification that is on Orion.
List Classifications:
classifications = APISession.list_resources(Classification) for classification in classifications: print(classification)
Projects¶
- class orionclient.types.Project(id: Optional[int] = None, name: str = '', description: str = '', owner: Optional[int] = None, statistics=NOTHING)¶
An Orion Project resource type. A project is a location in which Datasets and Files can be stored which pertain to a specific goal.
Creating a Project:
project = APISession.create_resource( Project, params={"name": "My personal project"} )
Secrets¶
- class orionclient.types.Secret(id: Optional[int] = None, name: str = '', owner: Optional[int] = None, created='', last_updated='', description: str = '', value: str = '', session=None)¶
Orion Secrets are a way of storing values in an encrypted state that can be retrieved for later use.
Creating a Secret:
secret = APISession.create_resource( Secret, params={ "name": "Database Password", "description": "The password to the corporate collection database", "value": "corporate-password", } ) # Update the secret post-creation secret.set_value("new-corporate-password")
- set_value(value)¶
Sets the value of the secret.
- Parameters
value (string) – Secret value to store in Orion
- Raises
BadResponse – Fails to set the value
Floe Types¶
WorkFloePackage¶
- class orionclient.types.WorkFloePackage(id: Optional[int] = None, session=None, uuid: Optional[str] = None, created='', source_code=None, state: str = '', reason: str = '', version='', owner: Optional[int] = None, size: int = NOTHING, specification=NOTHING, processing_detail=NOTHING)¶
Represents a Floe Package that contains Workfloes (WorkFloeSpec) and Cubes (Cube).
Upload a Workfloe Package:
package = WorkFloePackage.upload( APISession, "path/to/package.tar.gz" )
- get_environment_log()¶
Returns an iterator of bytes that make up the environment log
- Returns
Iterator of Log bytes
- Raises
BadResponse – Unable to make request to log stream
- get_inspection_result()¶
Returns dictionary that describes the results of package inspection
- Returns
Dictionary of inspection
- Raises
BadResponse – Unable to retrieve inspection results
- classmethod upload(session, path)¶
Uploads a WorkFloe Package to Orion
- Parameters
session (
OrionSession
) – Orion Sessionpath (string) – Filepath or URL to the workfloe package
- Raises
ValidationError – Invalid file path or URL
Cube¶
- class orionclient.types.Cube(id: Optional[int] = None, name: str = '', uuid: Optional[str] = None, title: str = '', state: str = '', version: str = '', cube_class: str = '', specification: dict = NOTHING, owner: Optional[int] = None, package: Optional[int] = None)¶
The description of a Cube that is on Orion.
List Cubes:
cubes = APISession.list_resources(Cube) for cube in cubes: print(cube)
WorkFloeSpec¶
- class orionclient.types.WorkFloeSpec(id: Optional[int] = None, uuid: Optional[str] = None, name: str = '', title: str = '', state: str = '', version: str = '', specification: dict = NOTHING, owner: Optional[int] = None, package: int = NOTHING, cubes: list = NOTHING, systemtags: list = [])¶
A WorkFloeSpec is the the description of a WorkFloe that defines what a job (WorkFloeJob) will perform.
List WorkFloe Specs:
specs = APISession.list_resources(WorkFloeSpec) for spec in specs: print(spec)
WorkFloeJob¶
- class orionclient.types.WorkFloeJob(id: Optional[int] = None, name: str = '', state: str = '', reason: str = '', created='', started='', finished='', job_type: str = '', notify: bool = False, success: bool = False, project: Optional[int] = None, status=NOTHING, parameters=NOTHING, system_parameters=NOTHING, owner: Optional[int] = None, workfloe=None, log_size: int = 0, session=None)¶
A WorkFloeJob is a specific run a workfloe (WorkFloeSpec).
List WorkFloe Jobs:
jobs = APISession.list_resources(WorkFloeJob) for job in jobs: print(job)
Starting a Job:
# Retrieve a workfloe spec that you wish to run workfloe_spec = APISession.get_resource(WorkFloeSpec, 814) job = WorkFloeJob.start( APISession, # Session to start job with workfloe_spec, "Name of My Workfloe", # Name of the floe { "promoted": {"param": 618}, "cube": { "cube1": {"param": "value"}, "cube2": {"param": "value"} }, "floe": {}, } )
- cancel()¶
Cancels a job if it is running at the time
- delete_logs()¶
Sends delete HTTP request to delete the job logs
- get_costs()¶
- Returns a dictionary containing cost information for the workfloe and
constituent cubes. Cost is cached server-side for 30 seconds, so call this function at most every 30 seconds.
Example response:
{ "total": 0.03, "costs": [ { "total": 0, "name": "Log processor", "usage": [], "rank": null }, { "total": 0.01, "name": "Sink", "usage": [ { "hours": 0.17, "hourly_rate": 0.06, "group_name": ... } ], "rank": 1 }, { "total": 0.01, "name": "Source", "usage": [ { "hours": 0.17, "hourly_rate": 0.06, "group_name": ... } ], "rank": 0 }, { "total": 0.01, "name": "Job Controller", "usage": [ { "hours": 0.17, "hourly_rate": 0.06, "group_name": ... } ], "rank": null } ] }
- logs()¶
Returns an iterator of bytes that make up the job logs
- Returns
Iterator of bytes
- classmethod start(session: orionclient.session.OrionSession, workfloe_spec: orionclient.types.floes.WorkFloeSpec, name: str, parameters: dict, system_parameters: Optional[dict] = None, project: Optional[orionclient.types.base.Project] = None, args: Optional[list] = None) orionclient.types.floes.WorkFloeJob ¶
Start will trigger a Job using the spec and parameters that are provided using the credentials that the session is configured for.
- Parameters
session (OrionSession) – Authenticated OrionSession
workfloe_spec (WorkFloeSpec) – WorkFloeSpec to run job from
name (string) – Name of the job
parameters (dict) – Parameters for the job to be run with
system_parameters (dict) – System parameters for the job to be run with
args (list<string>) – Command line arguments to be parsed and added to parameters
- Returns
Instance of WorkFloeJob
- Return type
- Raises
AuthorizationRequired – If session doesn’t have valid credentials
An example of starting a Job with a non-default output location, and a $1.00 job execution cost threshold, using system_parameters:
# Retrieve a workfloe spec that you wish to run workfloe_spec = APISession.get_resource(WorkFloeSpec, 814) # set system_parameters out_path = f"/project/3/user/test_modeler/orion-tests-out/" system_parameters={ "path": out_path, "cost_termination_threshold": 1.00, } job = WorkFloeJob.start( APISession, # Session to start job with workfloe_spec, "Name of My Workfloe", # Name of the floe { "promoted": {"param": 618}, "cube": { "cube1": {"param": "value"}, "cube2": {"param": "value"} }, "floe": {}, }, system_parameters=system_parameters )
- trigger_debug_export()¶
Requests that Orion export debug data for this job
An email will be sent to the email registered to the user profile that triggers the export
Service Types¶
Services¶
- class orionclient.types.Service(id: Optional[int] = None, owner: Optional[int] = None, name: str = '', description: str = '', url: str = '', metadata: dict = NOTHING, accepts_orion_tokens: bool = False, created='', last_updated='', session=None)¶
Orion Services is a registry of internal or external services integrated with the larger Orion system.
Creating a Service:
service = APISession.create_resource( Service, params={ "name": "Internal Cluster", "description": "Internal service that provides API access", "url": "special-sauce.company.com/api/", "metadata": { # Arbitrary JSON as metadata "contact": "help@company.com", } } )
- check(delete_tokens_first: bool = False) str ¶
Check that Orion and a registered service (specified by id or name) are configured with the same shared secret.
Warning
Do not set delete_tokens_first to True in parallel.
- Parameters
delete_tokens_first (bool) – Delete any existing tokens for this service before checking the service
- Returns
Redirect URL
- Return type
string
- Raises
OrionServiceError – If service appears to be misconfigured
Optionally delete the user’s token(s) that had been created for this service.
Follow the service redirect. The redirect includes a signed payload for the service which it uses to call back and create a token, if it one doesn’t already exist.
Verify that the user has a usable Orion token created for the service.
- delete_tokens(delete_timeout: int = 10)¶
Delete the tokens created for this service
- Parameters
delete_timeout (int) – How long in seconds to wait for each token to be deleted.
- Raises
OrionServiceError – If service appears to be misconfigured
- classmethod from_name(session: orionclient.session.OrionSession, name: str)¶
Get service by name
- get_redirect(follow: bool = False) Optional[str] ¶
Gets the redirect URL for this service. If a secret was set on the service, then the redirect URL gets modified by Orion to include a signed querystring payload that is used to authenticate to the service.
- Parameters
follow (bool) – Follow the redirect
- Raises
BadResponse – Fails to get the redirect
- list_tokens() List[orionclient.types.users.Token] ¶
Get a list of the tokens created for this service
- set_secret(value: str)¶
Sets the value of the secret the service shares with Orion.
- Parameters
value (
string
) – Secret value to store in Orion- Raises
BadResponse – Fails to set the value
Management Types¶
Tokens¶
- class orionclient.types.Token(id: Optional[int] = None, description: str = '', value: str = '', owner: Optional[int] = None, created='')¶
A token that is used for authentication to communicate with the APIs of Orion
List tokens:
for token in APISession.list_resources(Token): print(token)
Create token:
token = APISession.create_resource(Token, params={"description": "new token"})
Users¶
- class orionclient.types.OrionUser(id: Optional[int] = None, email: str = '', username: str = '', first_name: str = '', last_name: str = '', administrator: bool = False)¶
An Orion User.
List Users:
users = APISession.list_resources(OrionUser) for user in users: print(user)
- class orionclient.types.UserProfile(id: Optional[int] = None, email='', username: str = '', first_name: str = '', last_name: str = '', organization: Optional[int] = None, staff=False, remote=False, administrator=False, organization_feedback_email: str = '')¶
The profile that Orion Client is authenticated against
Get current profile:
profile = APISession.get_user_profile()
Instance Groups¶
- class orionclient.types.InstanceGroup(id: Optional[int] = None, instance_detail=NOTHING, usage=0.0, pool='', name='', size=0, min_size=0, max_size=0, desired_size=0, affinity=0.0, cost=0.0, spot=False, num_healthy=0, min_reserve=0.0, state='')¶
A group of Instances that is maintained by Orion that Cubes run on.
Warning
Does not support
get_resource()
as groups do not have identifiers. Uselist_resources()
to access the groups.List Instance Groups:
groups = APISession.list_resources(InstanceGroup) for group in groups: print(group)
Costs¶
- class orionclient.types.LedgerEntry(id: Optional[int] = None, created='', comment: str = '', creator: Optional[int] = None, owner: Optional[int] = None, entry_type: str = '', amount=Decimal('0.00'), project: Optional[int] = None, job: Optional[int] = None)¶
Entries in Orion that relate to the costs that users have.
Can only see other user’s entries if authenticated as an Orion Admin.
List Ledger Entries:
entries = APISession.list_resources(LedgerEntry) for entry in entries: print(entry)
- class orionclient.types.Balance(id: Optional[int] = None, balance='', timestamp='')¶
Balances are snapshots of spending on Orion.
List Balances:
for balance in APISession.list_resources(Balance): print(balance)
- class orionclient.types.CurrentBalance(id: Optional[int] = None, balance=Decimal('0.0'), ytd=Decimal('0.0'), mtd=Decimal('0.0'), last_updated='')¶
The current balance for the organization.
Get current balance:
balance = APISession.get_current_balance()