This is an abstract base class used to represent an object that can efficiently split up a molecular file into a stream of tokens, each token representing one molecule in the stream. Implementations of this class are returned by the OEGetTokenizer function. This class is expected to do the minimal amount of string parsing possible to determine the next chunk of bytes representing a molecule.
std::string *GetNextToken(OEPlatform::oeistream &ifs)=0
Returns a chunk of bytes representing the next molecule in the stream ifs. The stream of bytes is expected to be in the same file format as was passed to the OEGetTokenizer factory function used to construct this object.