schrodinger.application.transforms.dofn_utils module

class schrodinger.application.transforms.dofn_utils.DataFile(path: Path)

Bases: object

A helper class for use in DoFn instances where an argument is a file path.

Use the writeFile method in the setup of the DoFn to write the data to local (temporary) storage.

__init__(path: Path)

writeFile(folder: Path) → str

class schrodinger.application.transforms.dofn_utils.DefaultMainDoOutputsTuple(wrapped)

Bases: DoOutputsTuple

Wrapper for returned DoOutputsTuple from a PTransform.with_outputs() call that allows defaults to using the main pcollection in pipelines. Exmaple usage:

class MyTransform(beam.PTransform):

    def expand(self, pcoll):
        return DefaultMainDoOutputsTuple(
            pcoll | 'MyTransform' >> beam.ParDo(MyDoFn()).with_outputs('tag1', main="main_tag")
        )

with beam.Pipeline() as p:
    (p
     | 'Create' >> beam.Create([1, 2, 3])
     | MyTransform()
     | beam.Map(print)) # by default this will use the pcoll from the main tag

__init__(wrapped)