Molecule Reading and Writing

Molecule Streams

The previous section demonstrated reading and writing SMILES strings from a file. This required the programmer to perform the file I/O explicitly. Whilst this may be reasonable for SMILES strings that may be read via STL‘s getline, it isn’t suitable for more complex file formats. To ease this task, OEChem provides the oemolstream abstraction. The classes oemolistream and oemolostream allow input and output of molecules using the OEReadMolecule and OEWriteMolecule functions, respectively.

Listing 1: High-level Molecule I/O using molstreams

#!/usr/bin/env python
from openeye.oechem import *

ifs = oemolistream()
ofs = oemolostream()

mol = OEGraphMol()

while OEReadMolecule(ifs, mol):
    OEWriteMolecule(ofs, mol)

Listing 1 will read molecules from stdin in SMILES format and write them to stdout in absolute SMILES format. Notice that in this example, the following is done automatically for you by OEReadMolecule:

Note

OEReadMolecule will skip any invalid molecules automatically.

OEChem provides the following more Python-like generator methods to read molecules out of a oemolistream if the user prefers. It is important to realize that the same OEGraphMol instance is reused for every molecule being read from the oemolistream.

Generator methods for reading molecules

#!/usr/bin/env python
from openeye.oechem import *

ifs = oemolistream()
ofs = oemolostream()

for mol in ifs.GetOEGraphMols():
    OEWriteMolecule(ofs, mol)

Since the same OEGraphMol instance is being reused for every molecule in the stream, special precautions must be taken to load up an entire file into memory. Listing 2 will read molecules into memory by making sure there is a copy made when inserting the molecule into the container. This is useful if the molecules need to be repeatedly accessed.

Listing 2: Reading molecules into memory

#!/usr/bin/env python
from openeye.oechem import *

ifs = oemolistream()
mollist = []

for mol in ifs.GetOEGraphMols():
    mollist.append(OEGraphMol(mol))

File Formats

In addition to SMILES strings, OEChem is able to read numerous other molecular file formats, including MDL SD files, Tripos Tripos MOL2 files and PDB files. The format of an input file or stream may be associated with an oemolistream using the SetFormat method, and may be retrieved with GetFormat. The file format is represented by an unsigned integer, which should be one of the constants defined in the OEFormat namespace. A value of OEFormat_UNDEFINED means that there is no file format associated with the oemolstream.

Note

The default format associated with an oemolstream is OEFormat_SMI.

Listing 3 demonstrates how to use oemolstreams to convert MDL SD files into Tripos MOL2 files.

Listing 3: Explicitly setting file formats

#!/usr/bin/env python
from openeye.oechem import *

ifs = oemolistream()
ofs = oemolostream()

ifs.SetFormat(OEFormat_SDF)
ofs.SetFormat(OEFormat_MOL2)

for mol in ifs.GetOEGraphMols():
    OEWriteMolecule(ofs, mol)
OEChem file format support
File Format Description Read Write
OEFormat_CAN Canonical SMILES Yes Yes
OEFormat_CDX ChemDraw CDX Yes Yes
OEFormat_CSV Comma separated values Yes Yes
OEFormat_FASTA FASTA protein sequence Yes Yes
OEFormat_INCHI IUPAC InChI No Yes
OEFormat_INCHIKEY IUPAC InChI Key No Yes
OEFormat_ISM Canonical isomeric SMILES Yes Yes
OEFormat_MDL MDL Mol File Yes Yes
OEFormat_MF Molecular Formula (Hill order) No Yes
OEFormat_MMOD Macromodel Yes Yes
OEFormat_MOL2 Tripos Sybyl mol2 file Yes Yes
OEFormat_MOL2H Sybyl mol2 with explicit hydrogens Yes Yes
OEFormat_MOPAC MOPAC file format(s) No Yes
OEFormat_OEB OpenEye OEBinary Yes Yes
OEFormat_PDB Protein Databank PDB file Yes Yes
OEFormat_RDF MDL RDF File Yes No
OEFormat_SDF MDL SD File Yes Yes
OEFormat_SKC MDL ISIS Sketch File Yes No
OEFormat_SLN Tripos Sybyl Line Notation No Yes
OEFormat_SMI Canonical isomeric SMILES Yes Yes
OEFormat_USM SMILES Yes Yes
OEFormat_XYZ XMol XYZ format Yes Yes

File Input and Output

In addition to standard in and standard out, OEChem‘s oemolstreams also support reading from files. To open a file, use the open method with the required filename. For input oemolistreams, if the file does not exist the open fails and returns false, for output oemolostreams, the output file is created if it did not previously exist and overwritten if it did. If no arguments are passed to open, an oemolistream will use standard in, and an oemolostream will use standard out. Much like regular file I/O, oemolstreams can be closed after use, using the close method. oemolstreams are automatically closed by their destructors.

Listing 4: Reading and writing molecule files

#!/usr/bin/env python
from openeye.oechem import *

ifs = oemolistream()
ofs = oemolostream()

if ifs.open("input.sdf"):
    if ofs.open("output.mol2"):
        for mol in ifs.GetOEGraphMols():
            OEWriteMolecule(ofs, mol)
    else:
        OEThrow.Fatal("Unable to create 'output.mol2'")
else:
    OEThrow.Fatal("Unable to open 'input.sdf'")

One convenient feature of the open method of oemolstreams is that it sets the file format associated with the stream from the file extension of the filename used as an argument. Listing 4 converts the file ‘input.sdf’ in MDL file format into the file ‘output.mol2’ in Tripos MOL2 file format. This behavior can be overridden by calling SetFormat after the open but before the first molecule is read or written to a stream.

String Input and Output

The oemolistream and oemolostream provide the ability to read and write memory buffers instead of files. This is accomplished through the openstring methods, oemolistream.openstring and oemolostream.openstring.

Compressed Input and Output

For any of the molecular file formats supported by OEChem, it is often convenient to read and write compressed files or strings. Molecule streams support gzipped input and output via the zlib library. The ”.gz” suffix on any filename used to open a stream is recognized and the stream is read or written in compressed format. This mechanism does not interfere with the format perception. For instance, “fn.sdf.gz” is recognized as a gzipped file in MDL’s SD format.

Listing 5: Reading and writing compressed molecule files

#!/usr/bin/env python
from openeye.oechem import *
import sys

ifs = oemolistream("input.sdf.gz")
ofs = oemolostream("output.oeb.gz")

if ifs.IsValid():
    if ofs.IsValid():
        for mol in ifs.GetOEGraphMols():
            OEWriteMolecule(ofs, mol)
    else:
        sys.stderr.write("Error: Unable to create output.oeb.gz\n")
else:
    sys.stderr.write("Error: Unable to open input.sdf.gz\n")

Listing 5 converts all of the molecules in a gzipped SD format file into a gzipped OEBinary version 2 format file.

Command Line Format Control

Using the methods outlined above, it is possible to allow the stream format to be controlled from the command line. OEChem‘s oemolstreams control the format by interpreting the input and output file names.

Listing 6: Controlling File Format from the Command Line

#!/usr/bin/env python
from openeye.oechem import *
import sys

if len(sys.argv) != 3:
    OEThrow.Usage("%s <input> <output>" % sys.argv[0])

ifs = oemolistream()
ofs = oemolostream()

if not ifs.open(sys.argv[1]):
    OEThrow.Fatal("Unable to open %s" % sys.argv[1])

if not ofs.open(sys.argv[2]):
    OEThrow.Fatal("Unable to create %s" % sys.argv[2])

for mol in ifs.GetOEGraphMols():
    OEWriteMolecule(ofs, mol)

Listing 6 is an example of using command-line arguments to allow OEChem programs to support many file formats at run-time. For instance, if Listing 6 is a program called convert:

prompt> convert file1.sdf file1.smi

This command will convert file1.sdf from SD format to SMILES format. A first extension of this idea allows access to standard in and standard out via the “-” filename. For instance:

prompt> convert file2.mol2 -

This command will convert file2.mol2 in MOL2 format and write the molecules to standard out in SMILES, the default format.

Thus if you have another program, GetFromDatabase, which retrieves molecules from a database and writes them in SMILES format, you can chain it with any OEChem program. Using your operating systems redirection commands (e.g. - Unix pipe “|” or redirect “>”) you can move molecules directly from GetFromDatabase to convert without a temporary file:

prompt> GetFromDatabase | convert - file3.sdf

This command will take the SMILES format output from GetFromDatabase and generate an SD format file.

However, to make this concept of using standard in and standard out for piping data really useful, one needs to be able to control the format of standard in and standard out similarly to the way it would be controlled for temporary files. To facilitate this, oemolstreams interpret filenames which are ONLY format extensions to indicate format control for standard in and standard out.

Listing 7: Controlling standard in and standard out File Format

#!/usr/bin/env python
from openeye.oechem import *

ifs = oemolistream(".sdf")
ofs = oemolostream(".mol2")

for mol in ifs.GetOEGraphMols():
    OEWriteMolecule(ofs, mol)

Listing 6 is an example of opening an oemolistream to standard in with the format set to SDF. The output oemolostream is opened to standard out with the format set to MOL2. This is exactly equivalent to Listing 3. However, it demonstrates the extensibility of controlling the format of standard in and standard out from the command line.

Note

This prevents you from naming files as pure extensions: ”.mol2”, ”.sdf”, etc.

Now, using our program convert from Listing 6 it is possible to do the following:

prompt> convert .sdf .mol2

This command opens standard in with the SDF format and opens standard out with the MOL2 format.

Now there is complete format control of standard in and standard out from the command line. If we have a program GenerateStructures, which only writes the MOL2 format and another program GenerateData, which only reads the SD format, we can use them from the command line with any OEChem program which uses command-line arguments for file specification:

prompt> GenerateStructures | convert .mol2 .sdf | GenerateData

This command demonstrates how any OEChem program with command-line file specification can be used to pipe formatted input and output.

Flavored Input and Output

The general goal of the oemolstream input and output classes in OEChem is to provide the user with transparent access to the very complex task of reading and writing molecules in a wide variety of formats. However, occasionally, a programmer may want to tweak the behavior of specific readers or writers without abandoning the oemolstreams to use the low level writers (such as OEWriteMDLFile). For these instances, oemolstreams provide the oemolstreambase.SetFlavor and oemolstreambase.GetFlavor methods.

The oemolstreambase.SetFlavor method takes two unsigned integer arguments, the first is the format for which the flavor is being specified and the second is the flavor itself. The formats are specified as discussed in File Formats. The input flavors are specified in the OEIFlavor namespace and the output flavors are specified in the OEOFlavor namespace. Unlike the formats, the flavors are a bitmask and may be binary OR’d together. Under the OEIFlavor and OEOFlavor namespaces, there is a namespace for each format as well as a OEIFlavor_Generic namespace. The OEOFlavor_Generic namespace is used to control aromaticity perception and other properties common to all of formats. To completely specify a flavor, one would typically binary-OR a OEOFlavor_Generic flag and a format specific flag and pass the resultant value to oemolstreambase.SetFlavor.

The default behavior for the PDB reader is that TER specifies the termination of a disconnected fragment within the same molecule while END specifies the termination of a connection table. However, some users may want to have the reader split PDB input files into different molecules every time a TER appears.

The following code is an example of changing the PDB reader flavor.

Listing 8: Changing oemolistream Reader Flavor

#!/usr/bin/env python
from openeye.oechem import *

ifs = oemolistream('input.pdb')
ofs = oemolostream('output.mol2')

flavor = OEIFlavor_Generic_Default | OEIFlavor_PDB_Default | OEIFlavor_PDB_TER
ifs.SetFlavor(OEFormat_PDB, flavor)

for mol in ifs.GetOEGraphMols():
    OEWriteMolecule(ofs, mol)

Similar low-level control can be exerted over both input and output oemolstreams using the powerful oemolstreambase.SetFlavor method. See the API documentation for the associated low-level routine and namespace for details on the effects of specific flavor flags.

OEChem low-level readers and input flavors
File Format Low-Level Reader Input Flavors
OEFormat_CAN OEParseSmiles OEIFlavor_CAN
OEFormat_CDX OEReadCDXFile OEIFlavor_CDX
OEFormat_CSV OEReadCSVFile OEIFlavor_CSV
OEFormat_FASTA OEReadFASTAFile OEIFlavor_FASTA
OEFormat_ISM OEParseSmiles OEIFlavor_ISM
OEFormat_MDL OEReadMDLFile OEIFlavor_MDL
OEFormat_MF    
OEFormat_MMOD OEReadMacroModelFile OEIFlavor_MMOD
OEFormat_MOL2 OEReadMol2File OEIFlavor_MOL2
OEFormat_MOL2H OEReadMol2File OEIFlavor_MOL2H
OEFormat_MOPAC    
OEFormat_OEB OEReadOEBFile  
OEFormat_PDB OEReadPDBFile OEIFlavor_PDB
OEFormat_SDF OEReadMDLFile OEIFlavor_SDF
OEFormat_SKC OEReadSketchFile OEIFlavor_SKC
OEFormat_SLN    
OEFormat_SMI OEParseSmiles OEIFlavor_SMI
OEFormat_XYZ OEReadXYZFile OEIFlavor_XYZ
OEChem low-level writers and output flavors
File Format Low-Level Writer Output Flavors
OEFormat_CAN OECreateCanSmiString OEOFlavor_CAN
OEFormat_CDX OEWriteCDXFile OEOFlavor_CDX
OEFormat_CSV OEWriteCSVFile OEOFlavor_CSV
OEFormat_FASTA OEWriteFASTAFile OEOFlavor_FASTA
OEFormat_ISM OECreateIsoSmiString OEOFlavor_ISM
OEFormat_MDL OEWriteMDLFile OEOFlavor_MDL
OEFormat_MF OEMolecularFormula OEOFlavor_MF
OEFormat_MMOD OEWriteMacroModelFile OEOFlavor_MMOD
OEFormat_MOL2 OEWriteMol2File OEOFlavor_MOL2
OEFormat_MOL2H OEWriteMol2File OEOFlavor_MOL2H
OEFormat_MOPAC OEWriteMOPACInputFile OEOFlavor_MOPAC
OEFormat_OEB N/A (OEWriteMolecule works)  
OEFormat_PDB OEWritePDBFile OEOFlavor_PDB
OEFormat_SDF OEWriteMDLFile OEOFlavor_SDF
OEFormat_SKC    
OEFormat_SLN OECreateSlnString OEOFlavor_SLN
OEFormat_SMI OECreateSmiString OEOFlavor_SMI
OEFormat_XYZ OEWriteXYZFile OEOFlavor_XYZ
OEFormat_INCHI OECreateInChI OEOFlavor_INCHI
OEFormat_INCHIKEY OECreateInChIKey OEOFlavor_INCHIKEY

Note

Low-level readers will not call Clear on the molecule before reading the molecule into it. This is useful for concatenating multiple molecules together.

Molecular Property Preservation

Warning

OEWriteMolecule will change your molecule!

The high-level OEWriteMolecule function standardizes the molecule according to the output type for uniformity. For writing molecules without changing them, there are two options. If you would like the data to appear in the file exactly as it is in the molecule (perhaps Tripos atom names in a .pdb format), then you should use a low level writer. On the other hand, if you would like to write a standardized molecule (e.g. Tripos atom types in a MOL2 file), but don’t want your molecule changed, you can use OEWriteConstMolecule.