schrodinger.application.matsci.flywheel.ldmol.smiles_utils module¶
Regex-based BPE tokenizer for SMILES strings.
- class schrodinger.application.matsci.flywheel.ldmol.smiles_utils.RegexTokenizer¶
Bases:
objectRegex-based BPE tokenizer for SMILES strings.
Uses 300-token vocabulary optimized for chemistry. Replaces transformers tokenizer to eliminate external dependency.
- __init__()¶
Initialize tokenizer with BPE vocabulary.
- ensureVocabLoaded()¶
Load vocab file on first use (lazy loading).
- encode(smiles, add_special_tokens=True, max_length=None)¶
Encode SMILES to token IDs. Always truncates to max_length if tokens exceed limit.
- Parameters:
smiles (str) – SMILES string
add_special_tokens (bool) – Add [CLS] and [SEP] tokens
max_length (int) – Override default max_length
- Returns:
Dictionary with ‘input_ids’ and ‘attention_mask’
- Return type:
dict
- decode(token_ids)¶
Decode token IDs to SMILES.
- Parameters:
token_ids (torch.Tensor or list) – Token IDs (tensor or list)
- Returns:
SMILES string
- Return type:
str