schrodinger.application.matsci.flywheel.ldmol.tokenizer module

SentencePiece tokenizer wrapper for MolT5 text encoding.

class schrodinger.application.matsci.flywheel.ldmol.tokenizer.SimpleMolT5Tokenizer(logger=None)

Bases: object

Minimal tokenizer for MolT5 using SentencePiece.

Replaces transformers.AutoTokenizer to eliminate the external dependency. Vocab is loaded lazily on first use.

__init__(logger=None)

Parameters:

logger (logging.Logger) – Logger instance for debug output

ensureVocabLoaded()

Load SentencePiece vocab on first use.

encode(texts, max_length)

Tokenize texts with padding and truncation.

Parameters:

• texts (list(str)) – Input text strings

• max_length (int) – Fixed sequence length

Return type:

dict

Returns:

input_ids and attention_mask tensors