Orion CLI¶
Options¶
The Orion CLI has several top level options that modify the behavior of the commands that are run.
Timeout¶
The --timeout
flag can be used to set the request timeout in seconds, i.e.
if the server takes longer to respond to a request than the specified number of
seconds, the client disconnects. The default value is 10, but that may be too
short for some cases:
Slower or inconsistent internet connections
Sharing a package with many cubes to many users
Uploading large files
JSON¶
The --json
flag can be used to have the Orion CLI return JSON
rather than plaintext to make it easier parse the results
ocli --json <command>
Profile¶
The --profile
flag specifies which profile configuration to use while
making requests to the Orion API. The default is the default
profile
ocli --profile=testing <command>
ocli
config directory and file permissions are now forced to be read/write for the owner
Max Retries¶
The --max-retries
flag specifies the number of times that the client
will retry a connection if it fails or gets a 404 error. This is set to 2
by default. There is back off implemented with the retries, so the time will
increase exponentially (to a certain point) with the number of retries.
Shard uploads and downloads are unaffected by --max-retries
.
ocli --max-retries=5 <command>
Configuration Commands (ocli config)¶
Configure an Orion Profile¶
ocli config profile --url <hostname> --username <username> [--password <text>] [--with-token] [--project <project_id>]
Configure an Orion Profile for Use with SAML¶
To configure an Orion profile for use with SAML requires first creating a token in the web interface’s user profile page.
ocli config profile --url <hostname> --username <username> --with-token [--project <project_id>]
Change the Default Project of an Orion Profile¶
ocli config project
Delete an Orion Profile¶
ocli config delete <profile_name>
Retrieve Information about the Current Profile¶
ocli config info
List the Configured Profiles¶
ocli config list
Refresh Your Orion Profile token¶
ocli config token
Storage Commands¶
Folders¶
Commands to interact with Orion’s virtual filesystem
Note that folder commands are TOP LEVEL - e.g. ocli ls <path>
instead of ocli folders ls <path>
.
Folders contain Files, Datasets, ShardCollections, and other folders. They can take the following forms:
/project/<project-id>/user/<username>/<path>
Project folders, which are private to user with the given username.
/project/<project-id>/team/<path>
Project shared folders. All project members have access to these folders and their contents.
/project/<project-id>/workspace/<workspace-id>/<path>
Project shared workspace folders. Project members are given permission by sharing the workspace with them.
/organization/<path>
Organization shared folders.
List Contents of a Folder, or Details of an Item¶
Setting --recurse=True
will list all subdirectories and contents of the provided path(s). Items can be specified by path or
file://<file_id>|dataset://<dataset_id>
.
ocli ls <path(s)|item(s)> [--recurse=bool]
Create a Folder¶
ocli mkdir <path(s)>
Copy to/from Orion¶
This works similarly to scp, but dest must be a folder or local directory.
Currently, only file and folder upload is supported. Setting --recurse=True
will copy children
recursively. Setting --dry-run=True
will list the items that would be copied and then quit.
ocli cp <source> <dest> [--recurse=<bool>] [--dry-run=<bool>]
Peek Inside a File or Dataset in Orion¶
Items can be specified by
path or by file://<file_id>|dataset://<dataset_id>
. --number=
or -n
will optionally specify
the number of lines for a file or the number of records for a dataset.
ocli head <path(s)|item(s)> [-n, --number=<integer>]
Move a Folder or an Item¶
This will move all the children if the path refers to a Folder. Setting
--dry-run=True
will list the move(s) and then quit.
Note: items will inherit the destination folder’s permissions. Therefore, mv
can be used to share
across teams, workspaces, or organization.
ocli mv <source> <dest> [--dry-run=<bool>]
Remove a Folder or Item¶
Setting --dry-run=True
will list the item(s) to be removed and then quit.
Note: currently will not remove a folder with children. In the future this will use the trash-can api.
ocli rm <path(s)> [--dry-run=<bool>]
Workspaces¶
The CLI provides a number of commands to interact with workspaces, shared areas for Orion files, datasets, and collections organized into folders.
Create a Workspace¶
To create a workspace, use the create command and give it a slug, a name unique to this project. The backend will automatically give the workspace a unique id, which can be used in subsequent calls to add or remove users by sharing the workspace.
ocli workspaces create <workspace_slug>
Datasets¶
The CLI provides several ways to interact with datasets
Get Information about a Dataset¶
Using the id of a dataset it is possible to access all of the information associated with the dataset
ocli datasets info <resource_id>
List Datasets¶
To list the datasets the current Orion profile has access to. The arguments
passed after list are used as query parameters. For example
tags=2D,3D name=foo
or name=aspirin
.
ocli datasets list [--project=<integer>] [--name=<string>] \
[--limit=<integer>] \
[--offset=<integer>] \
[owner=<integer>] \
[deleted=<boolean>] \
[expired=<boolean>] \
[tags=TAG1,...] \
[tag_ids=<integer>,...] \
[systags=<tag>,...] \
[exclude_tags=<tag>,...] \
[tag_match=[default: exact](exact | iexact | contains | icontains | startswith | istartswith | endswith | iendswith)] \
[order_by=[-](created | name | count | owner)] \
[<key>=<value>,...]
Example:
ocli datasets list
ocli datasets list --name=aspirin_ --limit=30
ocli datasets list \
--project=12 \
--limit=20 \
tag_match=exact \
order_by=-size \
tags=3D,foo_jobs \
exclude_tags=2D,bar_jobs \
Upload a Dataset¶
To upload a dataset, pass the local path to the file and the client will create a new dataset. Handles any OEChem readable format, except JSON.
ocli datasets upload <file> [--project=<identifier>]
Download a Dataset¶
The following command will download a dataset, with any OEChem writable format except JSON, to your local file system.
ocli datasets download <resource_id> <file>
Finalize a Dataset¶
If the creation of a dataset is interrupted (for example, a job fails or is canceled as
it is writing records), then it will be left in a dirty state. The following
command will finalize a dataset, which sets dirty
to false and provides
an accurate count of records contained in the dataset.
ocli datasets finalize <resource_id>
Files¶
The Orion CLI provides several ways to interact with Files.
Retrieve Information about an Orion File¶
ocli files info <resource_id>
Delete an Orion File¶
ocli files delete <resource_id>
List Orion Files¶
Possible to filter using query parameters.
For example tags=foo
will filter Orion Files that have the tag foo
.
ocli files list [--project=<integer>] [--name=<string>] \
[--limit=<integer>] \
[--offset=<integer>] \
[owner=<integer>] \
[deleted=<boolean>] \
[expired=<boolean>] \
[tags=TAG1,...] \
[tag_ids=<integer>,...] \
[tag_match=(exact | iexact | contains | icontains | startswith | istartswith | endswith | iendswith)] \
[order_by=[-](created | name | size | owner | id)] \
[<key>=<value>,...]
Examples:
ocli files list
ocli files list --name=foo_ --limit=10
ocli files list \
--project=12 \
--limit=20 \
tag_match=exact \
order_by=-size \
tags=mol_jobs,foo_jobs \
exclude_tags=bar_jobs \
Uploading a File from the Local File System to Orion¶
ocli files upload <file> [--project=<identifier>]
Downloading an Orion File to the Local File System¶
ocli files download <resource_id> <file>
Shard Collections¶
Commands for managing collections.
List the Collections to Which You Have Access¶
ocli collections list [--limit=<integer>] [--offset=<integer>] \
[owner=<integer>] [project=<integer>] \
[deleted=<boolean>] [expired=<boolean>] \
[tags=<tag>,...] [tag_ids=<integer>,...] \
[tag_match=(exact | iexact | contains | icontains | startswith | istartswith | endswith | iendswith)] \
[order_by=[-](created | name | size | owner | id)] \
[<key>=<value>,...]
Examples:
ocli collections list
ocli collections list --name=foo_
ocli collections list \
--project=12 \
--limit=20 \
tag_match=exact \
order_by=-size \
tags=mol_jobs,foo_jobs \
exclude_tags=bar_jobs \
Get Information on a Specific Collection¶
ocli collections info <resource_id>
List Shards in a Collection¶
ocli collections list-shards <resource_id> \
[--limit=<integer>] [--offset=<integer>] \
[name=<name>] [after=<integer>]
ex:
ocli collections list-shards 344923
ocli collections list-shards 344923 --limit=5
ocli collections list-shards 344923 name=foo_12
Get Information on a Specific Shard in a Collection¶
ocli collections info-shard <resource_id> <shard_id>
Upload a List of Files to a New Collection¶
ocli collections upload <name> <file> [<file> ...]
Upload a List of Files to an Existing Collection¶
ocli collections upload <resource_id> <file> [<file> ...]
Open an Existing, Closed Collection in Order to Add to It¶
ocli collections open <resource_id>
Close an Existing, Open Collection in Order to Prevent Adding to It¶
ocli collections close <resource_id>
Download All Shards in an Existing Collection to a Directory¶
ocli collections download <resource_id> <destination>
Download a List of Shards in an Existing Collection to a Directory¶
ocli collections download-shards <resource_id> [<shard_id> ...] <destination>
Delete an Existing Collection¶
ocli collections delete <resource_id>
Delete Individual Shards in an Existing Collection¶
ocli collections delete-shards <resource_id> <shard_id> ...
Secrets¶
Secrets are a way of storing tokens in Orion. All values are encrypted upon upload and are decrypted when request through the API
Get Information on a Specific Secret¶
ocli secrets info <resource_id>
Delete an Existing Secret¶
ocli secrets delete <resource_id>
List the Secrets to Which You Have Access¶
ocli secrets list [--name=<string>] \
[--limit=<integer>] [--offset=<integer>] \
[description=<description>] [owner=<integer>] \
[<key>=<value>,...]
ex:
ocli secrets list
ocli secrets list --name=secret_foo
Create a New Secret¶
If --value
is not provided, a prompt will provide the ability to set
the value using a password input to avoid storing the value in the terminal
history.
ocli secrets create <secret_name> [--value=<secret_value>] [--description=<string>]
Services¶
Orion Services is a registry of internal or external services integrated with the larger Orion system
Get Information on a Specific Service¶
ocli services info <resource_id>
Delete an Existing Service¶
ocli services delete <resource_id>
List the Services to Which You Have Access¶
ocli services list [--name=<string>] \
[--limit=<integer>] [--offset=<integer>] \
[owner=<integer>] [<key>=<value>,...]
ex:
ocli services list
ocli services list --limit=5 owner=10
Create a New Service¶
ocli services create <service_name> [--description=<string>]
Projects¶
Commands for managing projects.
Get Information on a Specific Project¶
ocli projects info <resource_id>
Delete an Existing Project¶
ocli projects delete <resource_id>
List the Projects to Which You Have Access¶
ocli projects list [--limit=<integer>] [--offset=<integer>] \
[name=<name>] [as_admin=<boolean>] \
[deleted=<boolean>] [expired=<boolean>] \
[<key>=<value>,...]
ex:
ocli projects list
ocli projects list name=foo_
ocli projects list deleted=true --limit=5
Create a New Project¶
ocli projects create <project_name>
Leave a Project¶
ocli projects leave <project_id> [--new-owner=<user_id>]
Any member of a project can use this command to leave a project.
If
--new-owner
isn’t specified, the project assets belonging to the leaving user will be transferred to the project owner.If the user specified as
--new-owner
isn’t already a member of the project, they will automatically join the project.If the leaving user is the project owner
--new-owner
must be specified.
Floe Commands¶
The following commands are a set of resources in Orion that relate to Floe which includes Packages, WorkFloes, and Jobs.
Packages¶
Commands related to Floe Packages
Initializing a Package¶
ocli packages init
The ocli packages init
command:
Sets up a skeleton of an Orion package containing cubes and floes.
Provides simple example cube and floe.
Supports testing setup using PyTest including working tests for the example cube and floe.
Provides a
setup.py
file with commands to run tests or build the package.Provides version configuration via storing the version only in the module’s
__init__.py
file.
Retrieving the Information about a Floe Package¶
ocli packages info <resource_id>
Retrieving the Environment Build Log for a Floe Package¶
ocli packages environment <resource_id>
Retrieving the Inspection Information for a Floe Package¶
ocli packages inspection <resource_id>
Deleting a Floe Package¶
ocli packages delete <resource_id>
Listing Floe Packages¶
ocli packages list [--limit=<integer>] [--offset=<integer>] \
[states=<state>,... (ready, deprecated, queued, error)] \
[name=<name>] [UUID=<uuid>] [owner=<integer>] \
[<key>=<value>,...]
ex:
ocli packages list
ocli packages list states=error
ocli packages list --limit=20 states=ready,queued
Upload a Floe Package¶
ocli packages upload <file_or_url>
WorkFloes¶
Commands for accessing Workfloes in Orion. Workfloes are a collection of cubes that defines the flow of data.
Retrieving the information about a WorkFloe¶
ocli workfloes info <resource_id>
Deleting a Workfloe¶
ocli workfloes delete <resource_id>
Listing Workfloes¶
ocli workfloes list [--package=<integer>] [--name=<string>] \
[--limit=<integer>] [--offset=<integer>] \
[states=<state>,... (ready, deprecated, hidden, invalid)] \
[systags=<tag>,...] [tags=<tag>,...] [owner=<integer>] \
[classification=<string> [classification_depth=<integer>]
[order_by=(recently_used | name)]
[<key>=<value>,...]
ex:
ocli workfloes list
ocli workfloes list tags=tag1,tag2
ocli workfloes list --name=collection_to_ --limit=10 order_by=recently_used
ocli workfloes list --package=10 states=ready,deprecated
Listing Classifications¶
ocli workfloes list-classifications [--name=<string>] [--path=<string>] \
[--depth=<integer>] [--limit=<integer>] [--offset=<integer>] \
[<key>=<value>,...]
ex:
ocli workfloes list-classifications
ocli workfloes list-classifications --path=Example/Test
ocli workfloes list-classifications --name=exam --depth=1 --limit=10
Jobs¶
Commands for interacting with Jobs in Orion. Jobs are an instantiation of a Workfloe that performs the computation defined by the Workfloe.
Retrieving the Information about a Job¶
ocli jobs info <resource_id>
Retrieve the Logs of a Job¶
ocli jobs logs <resource_id>
Canceling a Running Job¶
ocli jobs cancel <resource_id>
Deleting a Job¶
ocli jobs delete <resource_id>
Displaying Size of Logs¶
Retrieve the size of the logs of an Orion job.
ocli jobs log_size [OPTIONS] JOB_ID
Arguments
- JOB_ID
Required argument identifying the job.
Deleting Job Logs¶
Delete the logs of an Orion job.
ocli jobs delete_logs [OPTIONS] JOB_ID
Arguments
- JOB_ID
Required argument
Listing Existing Jobs¶
ocli jobs list [--limit=<integer>] [--offset=<integer>] \
[name=<name>] [project=<integer>] [workfloe=<integer>] \
[job_type=(system | user)] [owner=<integer>] \
[states=<state>,... (queued, running, complete)] \
[tags=<tag>,...] [<key>=<value>,...]
ex:
ocli jobs list
ocli jobs list workfloe=20
ocli jobs list tags=tag1,tag2 states=complete
ocli jobs list job_type=user name=aspirin_ states=queued,running
Tag¶
Tag a specific job.
ocli jobs tag [OPTIONS] IDENTIFIER [TAGS]...
Options
–project <project>
Arguments
- IDENTIFIER
Required argument identifying the job.
- TAGS
Optional string argument(s) to be used as tags.
The –project` option applies the tags using project ID.
For example:
(ocli_dev) ocli jobs tag --project 25412 203761 candidates
Tagged WorkFloeJob 203761 with ('candidates',)
Project tags are displayed in pink in the user interface. Job tags are yellow.
Both kinds of tag can be used in searches in the user interface.
List Tags¶
List tags associated with a job
ocli jobs list-tags [OPTIONS] IDENTIFIER
Arguments
- IDENTIFIER
Required argument identifying the job.
Starting a Job¶
ocli jobs start <workfloe_id> <job_name> [--wait] \
[FLOE_PARAMETERS]
ex:
ocli jobs start 6836 job_name0 --in 601 --out run_output.oedb
Note
Adding -- --help
to the ocli jobs start
command will print the
parameters that are expected by the Workfloe.
Watching a Job¶
ocli jobs watch <workfloe_id> <job_name> [CUBES] [OPTIONS]
Collect Debug Information¶
ocli jobs debug-export <job_id>
Retrieving Summary of Jobs Within an Organization¶
ocli jobs summary-export [--package-uuids=<str>] [--workfloe-uuid=<str>] \
[--job_ids=<str>] [--start-date=<str>] [--end-date=<str>] \
[--output-path=<str>] [--download=(--download | --no-download)] \
[--notify=(--notify | --no-notify)]
Examples:
ocli jobs summary-export
ocli jobs summary-export --notify --download
ocli jobs summary-export --start-date 2023-01-01 --end-date "2023-01-01 18:00:00"
ocli jobs summary-export --output-path "/project/<project-id>/users/<user-id>/folder-name/"
ocli jobs summary-export --package-uuids "uuid_A, uuid_B" --workfloe-uuids "uuid_C" --job-ids "6, 19, 47"
This functionality is available only to stack admins, org admins and mt-org admins.
Cost Exporting¶
Generating Storage Use Report¶
The report generated by the following commands contains estimates based on the amount of storage used. They do not reflect the exact billed cost.
The results do not take into account the AWS tiered storage system.
Generating the report does not move anything from a lower cost tier into a higher cost tier.
ocli cost export --report-type=storage_use [USER IDS]
ex:
ocli cost export --report-type=storage_use 10 11 20
Generating Spend Summary Report¶
ocli cost export --report-type=spend_summary [USER IDS]
ex:
ocli cost export --report-type=spend_summary 10 11 20