Creating a New Orion Package

The easiest way to start a new package containing cubes and floes is to generate a basic template using the ocli ocli packages init command. This template provides a consistent starting point for creating cubes and floes. It also sets up helpful tools for running tests.

After you have followed the directions in this section, see Orion Integration for more details on packaging, linting and detection, hardware requirements, parallelism, scheduling, and other runtime considerations.

Features

The ocli packages init command:

  • Sets up a skeleton of an Orion package containing cubes and floes.

  • Provides simple examples of cube and floe.

  • Supports a testing setup using PyTest, including working tests for the example cube and floe.

  • Provides a setup.py file with commands to run tests or build the package.

  • Provides version configuration via storing the version only in the module’s __init__.py file.

Requirements

  • Python 3.9 (For your specific platform, see Prerequisites.) We recommend starting with a clean conda environment.

  • orionplatform_library version 4.2 or higher.

  • Access to OpenEye’s Python package server, Magpie. If you are a licensed Orion user and don’t have access, please contact OpenEye Support.

Follow the instructions there to configure access to Orion python packages via your pip.conf file.

Setup

  1. Create and activate a new Anaconda environment:

    conda create -n ocli_dev python=3.9
.
.
conda activate ocli_dev

  2. Using the new conda environment, install the Orion Platform package, including the command line interface ocli, by running:

    pip install openeye-orionplatform

  3. Run:

    ocli packages init

  4. This will generate a directory with the name you provided as the project_slug when invoking the command. Switch into the directory:

    cd <project_slug>

  5. Next install the package and the development requirements:

    pip install -e .
pip install -r requirements_dev.txt

Commands

Once all dependencies are installed, you can build the package for upload to Orion (the tar.gz will be in the dist directory):

python setup.py package
ocli packages upload dist/<package-filename>.tar.gz

Tests are set up for each of the floes included; they can be run locally:

python setup.py test-all

Command to just test cubes:

python setup.py test-cubes

Command to just test floes:

python setup.py test-floes

To clean up generated documentation and packaging files, run:

python setup.py clean

Output Skeleton

The following directory structure will be created by the template generator; the items marked in {{ }} will be replaced by your choices upon completion:

{{project_slug}}/       <-- Top directory for your Project.
├── MANIFEST.in
├── README.md                        <-- README with your Project Name and description.
├── docs/                            <-- Docs subdirectory set up for automatic documentation of cubes and floes.
│    ├── Makefile
│    ├── make.bat
│    └── source
│        ├── _static
│        ├── _templates
│        ├── conf.py
│        └── index.rst
├── floes/                           <-- Subdirectory where all floes should be placed.
│    └── myfloe.py                   <-- An example floe.
├── manifest.json                    <-- Manifest for Orion.
├── requirements.txt                 <-- Requirements file for developers of this package.
├── setup.py                         <-- Python file for creating a python package
├── tasks.py                         <-- Python file with defined tasks for building docs, running tests, and building the package.
├── tests/                           <-- Subdirectory for testing of cubes and floes.
│    ├── test_mycube.py              <-- An example unit test for the included cube.
│    └── floe_tests/                 <-- Subdirectory for floe tests.
│        └── test_myfloe.py          <-- An example unit test for the included floe.
└── {{ module_name}}/    <-- Subdirectory of the package for the python module. All cubes should go in here.
    ├── __init__.py
    └── mycube.py                    <-- An example cube.

Note

For more details on package structure and linting, see the floe documentation.

Package Documentation

The provided documentation files are included as a convenience, to provide users with a starting point. They are incomplete, however, and will not build correctly as-is. If you wish to create full Sphinx documentation, including autogenerated references for cubes and floes, please consider using the floe-pkg-tools library

In the package’s manifest.json, specify: documentation_root: <RELATIVE_PATH_TO_DOCS>. The root directory specified must contain an index.html file.

Example manifest.json contents:

{
    "name": "package_name",
    "requirements": "requirements.txt",
    "python_version": "3.7",
    "version": "0.1.0",
    "documentation_root": "docs"
}

Where docs is the relative path to a directory named docs/, which must contain an index.html file that acts as the starting point for the package’s documentation alongside the rest of your documentation files.

Directions for Sphinx are beyond the scope of this section, but the system is well designed for Python programmers and is adequately documented. For an introduction, see Sphinx.

Utilizing Private Python Packages

There are two ways to include a private dependency, such as a package hosted in a non-public repository. The most flexible is to create and upload a complete docker image as described in the following tutorial. The second way is for the repository URL and access token to be included in your organization’s pip.conf file so that any package uploaded to Orion can include it. To enable a private Python package for your organization in this way, please provide ops or your account representative with an access token and the corresponding non-firewalled URL.

Creating a Package Image (Advanced)

If using the template (manifest-based) package is not flexible enough for your needs, the alternative, for those with Docker experience, is to create a Docker image.

The following commands assume that you are in an existing package directory (for example, the package you created in the previous section “Creating a New Orion Package”).

Prerequisites

To get started, you’ll need the following:

  • Docker requirements:

    • Docker is (https://docs.docker.com/desktop/) installed.

    • The Docker group must be created (Linux Only)

    • The user must be part of the Docker group (Linux Only).

    • The Docker Daemon must be running.

  • If using GPUs:
  • ORION requirements:

    • orionclient credentials (openeye-orionplatform>=4.3.1)

    • ocli configured

  • If using a CONDA token (Optional, Recommended):
  • If using OpenEye toolkit libraries:

    • OpenEye Licence file

  • If using Docker-hub as an image repository:

    • Docker-hub account

    • Docker-hub credentials (username and password)

  • If using AWS ECR as an image repository:

  • magpie credentials added to your pip conf

  • pyyaml (https://pypi.org/project/PyYAML/) installed in your Python environment e.g. pip install pyyaml

  • a working floe package

Overview

The following diagram illustrates the environment:

  Package Image Layers

+--------------------------------+
|         Cubes and Floes        |
+--------------------------------+
|       Conda Environment        |
+--------------------------------+
|        Orion Base Image        |
+--------------------------------+
|   Linux Distribution (Amazon   |
|      Linux, Ubuntu, etc.)      |
+--------------------------------+

A Docker image consists of file system layers. When a container runs the image, the layers become merged into a single view of the filesystem. Orion package images are structured in layers as shown above.

Images are portable. They can be imported and exported as binary files from Docker (see docker save, docker load). Those files can also be uploaded and downloaded from Orion. An image prepared on a developer’s machine and containing cubes and floes can be uploaded to Orion, where the image can be used to execute those cubes and floes without any modification.

Uploading Docker images to Orion instead of floe packages has several benefits: * The package environment and code is executed without modification. What you test and upload is what you ship. * Private packages can be used without uploading them to magpie or making them visible to users. * Compiled binaries, system packages, binary data, and such can be bundled into the image. * Package ingestion is faster for packages with complex dependencies. * Package portability is improved, as fully prepared images cannot fail to process in the future due to changes in conda/pip/etc.

Creating an image from an existing floe

1. Download a base image

  • First, select a base image os from the available choices. To see the available choices:

    $ ocli packages list-base-images
ubuntu-14.04
ubuntu-18.04
ubuntu-20.04
amazonlinux1
amazonlinux2

  • Trigger an image export job in Orion and download the resulting file:

    $ ocli packages export-base-image amazonlinux2 --download

2. Import base image file into Docker

  • Load the image into Docker:

    $ docker load -i orion-base-amazonlinux2-<VERSION>
Loaded image ID: sha256:01da4f8f9748b3ac6cf5d265152fb80b9d7545075be8aa0a3d60770a98db9768

  • Tag the image using the SHA from the previous step:

    $ docker tag sha256:01da4f8f9748b3ac6cf5d265152fb80b9d7545075be8aa0a3d60770a98db9768 orion-base-amazonlinux2-<VERSION>

Note

The version included in the orion-base image name is used to control how Orion expects the image to be structured. This version may change after an Orion release. Export jobs always export the current/latest version, so tag the image accordingly.

3. Create a conda environment.yml

A command for converting a manifest.json into a conda environment.yml file is provided for convenience. The environment.yml file is required by the included Dockerfile and used to construct a Conda environment. Running this requires pyyaml be added to your environment as a dependency (https://pypi.org/project/PyYAML/):

$ ocli packages create-environment-yaml -o environment.yml ./manifest.json

Sample environment.yml file:

dependencies:
  - python=3.9
  - pip:
    - '# Developer should handle tightening pinning of OpenEye-snowball version'
    - OpenEye-orionplatform<5.0.0,>=4.0.0
    - OpenEye-snowball
name: user_env

Note

For details about Conda environment files, see: https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file

4. Create Dockerfile and .dockerignore

The Dockerfile contains all of the commands necessary to build a working Conda environment and add the package source code to the image.

For convenience, a Dockerfile can be generated:

$ ocli packages show-dockerfile -o Dockerfile

Note

This dockerfile is provided for reference, but using it is not required. The version of the base image specified in the generated dockerfile must be changed to match the image loaded into Docker. If the check-image command succeeds, then the image should work in Orion. For further detail on package requirements, run ocli packages show-packaging-help.

Customize as necessary, then build the dockerfile. Typical customizations might include installing package/compilation dependencies or installing binary executables not available from conda/pip. The generated dockerfile includes comments describing common changes such as those needed to use the OpenEye toolkits.

The package process for a normal Orion floe package excludes tests and other development files from the tarfile that gets uploaded. For details, see MANIFEST.in, or try building and examining the contents of the package created by the python setup.py package. Similarly, a .dockerignore file is the typical way of ensuring that unneeded files are not included in an image when using Docker.

Create a file named .dockerignore. In the file, insert these lines:

*.pyc
requirements_dev.txt
dist
tests
docs

That will exclude the listed items from the Docker image.

Note

Since MANIFEST.in is not being used during the Docker build process, it may be deleted to avoid confusion. Alternatively, the existing Python build process could be utilized as the first of two steps in building an image, for example, by first running python setup.py package && cd dist && tar xf ./<my-package>.tar.gz and then copying from the dist directory rather than the source directory.

5. Build the Docker Image

The basic command for making a docker image without a CONDA token and without using any OpenEye licenced code is:

$ DOCKER_BUILDKIT=1 docker build --secret id=pip.conf,src=<path_to_pop.conf> -t mypackage-mypackageversion .

If you are using a CONDA token, you need to add the following argument:

$ --secret id=condatoken,src=<path_to_condatoken>

If you are using any OpenEye Licensed code, you also need to add the following argument:

$  --secret id=oe_license,src=<path_to_oe_license_file>

Enabling Docker BuildKit provides support for the --secret flag. The -t argument will apply a tag to the resulting image. The tag is not required, but it’s a Docker image best practice.

Note

Depending on the operating system, a file holding the conda token might need to be created manually. To check if this file already exists, use the following command find $HOME -name condatoken. To create this file (assuming the token is configured in conda), use conda token list | sed "s/.*\ //" > $HOME/condatoken.

6. Export the image and upload to Orion

Save the image as a file:

$ docker save mypackage-mypackageversion -o mypackage-mypackageversion

Upload the image file to Orion and trigger inspection:

$ ocli packages upload-package-image mypackage-mypackageversion

7. Upload the image using Docker-hub as a docker image registry

As an alternative to locally saving and uploading the image, Orion also supports using a docker registry. This could, for example, be Docker-hub or AWS ECR, but other registries are also possible.

Before you can push images to a registry, you need to authenticate yourself to the registry.

If you are using Docker-hub, you can authenticate yourself to the repository using:

$ docker login --username <my_dockerhub_username> --password <my_dockerhub_password>

Now, you need to tag your image so that Docker knows where to push it:

$ docker tag mypackage-mypackageversion <my-dockerhub-username>/<my-dockerhub-repository-name>:mypackage-mypackageversion

You can then push the image to your docker-hub repository using:

$ docker image push <my-dockerhub-username>/<my-dockerhub-repository-name>:mypackage-mypackageversion

Since your image should now exist in Docker-hub, you can remove the tagged image from Docker using:

$ docker rmi <my-dockerhub-username>/<my-dockerhub-repository-name>:mypackage-mypackageversion

Before you can pull your image from Docker-hub into ORION, you need to authenticate yourself to Docker-hub. You do this by storing your Docker-hub password in an ORION secret using:

$ ocli secrets create hubsecret --value <my-dockerhub-password>

Note

ocli secrets are unrelated to Docker secrets that were introduced when building the Docker image.

This yields a secret ID which you can now use to pull your image into ORION using:

$ ocli packages import-from-registry --registry-url <my-dockerhub-username>/<my-dockerhub-repository-name> -t mypackage-mypackageversion -u <my-dockerhub-username> -s <my-hubsecret-id>

If all goes well, this should pull your image to ORION from Docker-hub. If it succeeds, you can now remove your ocli secret using:

$ ocli secrets delete <my-hubsecret-id>

Note

There are Docker special cases registries on Docker Hub. Instead of using the full URL for the registry, only use the suffix. In the example above, my-registry/my-package-repo gets expanded (in Docker) to https://hub.docker.com/repository/docker/my-registry/my-package-repo. For any non-dockerhub registry, the full URL must be used.

8. Upload the image using AWS ECR as a docker image registry

As an alternative to locally saving and uploading the image, Orion also supports using a Docker registry. This could for example be Docker-hub or AWS ECR but other registries are also possible.

Before you can push images to a registry, you need to authenticate yourself to the registry.

If you are using AWS ECR you should verify that your AWS CLI is properly configured by checking if you have access to your basic AWS credentials.

Note

Unlike Docker-hub, AWS ECR by default comes with very restricted user permissions. For example, you HAVE to know the name of your repository up front, as you will not by default have permissions to list repositories available to you in order to find the repository name. By default, you will also not have permissions to create new repositories.

Your AWS access key:

$ aws configure get aws_access_key_id

Your AWS secret access key:

$ aws configure get aws_secret_access_key

Your AWS account ID:

$ aws sts get-caller-identity --query=Account

Your AWS region:

$ aws configure get region

If any of these commands fail, you need to first configure your AWS CLI.

You then have have to generate a login token using:

$ aws ecr get-login-password --region <my-aws-region>

You store this token and use it for pushing/pulling your images to/from AWS ECR. You authenticate yourself to AWC ECR by using:

$ docker login <my-aws-account-id>.dkr.ecr.<my-aws-region>.amazonaws.com --username AWS --password <my-aws-ecr-token>

Now, you need to tag your image so that Docker knows where to push it:

$ docker tag mypackage-mypackageversion <my-aws-account-id>.dkr.ecr.<my-aws-region>.amazonaws.com/<my-aws-ecr-repository-name>:mypackage-mypackageversion

You can then push the image to your AWS ECR repository using:

$ docker image push <my-aws-account-id>.dkr.ecr.<my-aws-region>.amazonaws.com/<my-aws-ecr-repository-name>:mypackage-mypackageversion

Since your image should now exist in AWS ECR, you can remove the tagged image from Docker using:

$ docker rmi <my-aws-account-id>.dkr.ecr.<my-aws-region>.amazonaws.com/<my-aws-ecr-repository-name>:mypackage-mypackageversion

Note

AWS ECR does not permit two repositories with the same tag. Unlike Docker-hub, which will merge changes if you push a repository with the same tag as an existing repository, AWS ECR will untag the existing image when you push an image with the same tag as an existing image. This means that if you push images with the same tag to AWS ECR multiple times, you will find many untagged images in your repository.

Before you can pull your image from AWS ECR into ORION, you need to authenticate yourself to AWS ECR. You do this by storing your AWS ECR login token in an ORION secret using:

$ ocli secrets create ecrsecret --value <my-aws-ecr-token>

This yields a secret ID which you can now use to pull your image into ORION using:

$ ocli packages import-from-registry --registry-url <my-aws-account-id>.dkr.ecr.<my-aws-region>.amazonaws.com/<my-aws-ecr-repository-name> -t mypackage-mypackageversion -u AWS -s <my-ecrsecret-id>

If all goes well, this should pull your image to ORION from AWS ECR. If it succeeds, you can now remove your ocli secret using:

$ ocli secrets delete <my-ecrsecret-id>