Merge Records

../../../../../_images/MergeCubeIcon.svg

This serial cube merges records from two ports, using a specified field as the merge key.

The input ports are labeled A and B. Records with matching identifiers are combined into a single record and output on the success port.



Note

This cube is identical to Join Records cube, only using different nomenclatures.

The ID A field parameter specifies which field in “A” records contains the unique identifier used to merge the records. The ID B field parameter specifies which field in the “B” records contains the identifier. ID B is null by default, meaning the ID A field will be used for both “A” and “B” records. Successfully merged records are emitted to the success port.

The Merge Type parameter determines what happens to records that cannot be merged with a matching record:

  • “A” means that unmatched records from the A input port are sent to the success port, while unmatched records from the B input port will be sent to the failure port.
  • “B” means that unmatched records from the B input port are sent to the success port, while unmatched records from the A input port will be sent to the failure port.
  • “A∩B” means only successfully merged records will be emitted from the success port, with all unmatched records going to the failure port.
  • “A∪B” means all merged plus unmatched records will be emitted from the success port, and no records will be emitted on the failure port.

If the cube encounters identically named fields on two records being merged, the fields on the “B” record will be ignored, renamed, or appended, depending on the value of the Duplicate Fields parameter.

Main Parameters

Parameter Name Associated Port Port Type
Duplicate Fields    
Merge Type    
ID A    

Related Cubes

Calculation Parameters

  • CPUs (integer) : The number of CPUs to run this cube with
    Default: 1 Min: 1 Max: 128
  • Cube Metrics (string) : Set of metrics to be collected

    Choices: cpu, disk, memory, network
  • Temporary Disk Space (MiB) (decimal) : The minimum amount of disk space in MiB (1048576 B) this cube requires. Due to overhead, request a couple hundred MiB more than required.
    Default: 5120.0 Min: 128.0 Max: 8589934592
  • Duplicate Fields (string) : This parameter specifies how to handle duplicate fields when merging records.
    Default: Rename
    Choices: Ignore, Rename, Append
  • GPUs (integer) : The number of GPUs to run this cube with
    Default: 0 Max: 16
  • Instance Tags (string) : Only run on machines with matching tags (comma separated)
    Default: “”
  • Instance Type (string) : The type of instance that this cube needs to be run on
  • Merge Type (string) : This parameter determines what happens to unmatched records. See the cube description for explanation.
    Default: A∪B
    Choices: A, B, A∩B, A∪B
  • ID A (string) : Field containing a unique identifier for records on the ‘A’ port.
  • Memory (MiB) (decimal) : The minimum amount of memory in MiBs (1048576 B) this cube requires. Due to overhead, request a couple hundred MiB more than required.
    Default: 1800 Min: 256.0 Max: 8589934592
  • Metric Period (decimal) : How often to sample metrics, in seconds
    Default: 60 Min: 1 Max: 300
  • ID B (string) : Field containing a unique identifier for records on the ‘B’ port.
  • Spot policy (string) : Control cube placement on spot market instances
    Default: Prohibited
    Choices: Allowed, Preferred, NotPreferred, Prohibited, Required
  • Match molecules by title (boolean) : If set, and a molecule field is chosen for merging, the molecule titles will be used as unique identifiers. Otherwise, the molecule SMILES will be used.
    Default: False

Hardware Parameters

Machine hardware requirements

  • Memory (MiB) (decimal) : The minimum amount of memory in MiBs (1048576 B) this cube requires. Due to overhead, request a couple hundred MiB more than required.
    Default: 1800 Min: 256.0 Max: 8589934592
  • Temporary Disk Space (MiB) (decimal) : The minimum amount of disk space in MiB (1048576 B) this cube requires. Due to overhead, request a couple hundred MiB more than required.
    Default: 5120.0 Min: 128.0 Max: 8589934592
  • GPUs (integer) : The number of GPUs to run this cube with
    Default: 0 Max: 16
  • CPUs (integer) : The number of CPUs to run this cube with
    Default: 1 Min: 1 Max: 128
  • Instance Type (string) : The type of instance that this cube needs to be run on
  • Spot policy (string) : Control cube placement on spot market instances
    Default: Prohibited
    Choices: Allowed, Preferred, NotPreferred, Prohibited, Required
  • Instance Tags (string) : Only run on machines with matching tags (comma separated)
    Default: “”

Metrics Parameters

Cube Metric Parameters

  • Metric Period (decimal) : How often to sample metrics, in seconds
    Default: 60 Min: 1 Max: 300
  • Cube Metrics (string) : Set of metrics to be collected

    Choices: cpu, disk, memory, network

Tip

filename: snowball/logic/join.py