schrodinger.seam.testing.benchmarks module¶

Benchmarks for different parts of the SeamRunner or Seam transforms.

Usage:

$SCHRODINGER/run seam_benchmarks.py

class schrodinger.seam.testing.benchmarks.BenchmarkResult(value: float, units: Optional[str] = None)¶

Bases: object

Variables:

value – Current value of the metric
units – (optional) units of the value.

value: float¶

units: Optional[str] = None¶

__init__(value: float, units: Optional[str] = None) → None¶

class schrodinger.seam.testing.benchmarks.Benchmark¶

Bases: object

UNIT_TEST_N = 1¶

PERFORMANCE_TEST_N = 500¶

SKIP = False¶

setup()¶

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

schrodinger.seam.testing.benchmarks.AlkaneStructure(n: int) → Structure¶

schrodinger.seam.testing.benchmarks.tmp_cwd()¶

class schrodinger.seam.testing.benchmarks.WriteStructuresToFile_MaxSizeOfDirectory¶

Bases: Benchmark

This benchmark is to measure WriteStructuresToFile’s max shard directory size.

UNIT_TEST_N = 5¶

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.WriteStructuresToFile_Runtime¶

Bases: Benchmark

This benchmark is to measure WriteStructuresToFile’s runtime.

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.GroupByKeyWithSlowCoder¶

Bases: Benchmark

This benchmark is to test the performance of GroupByKey when using keys and values that are slow to deserialize.

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.ExecStagesWithSlowCoder¶

Bases: Benchmark

This benchmark is to test the performance of ExecutableStages which have outputs that are slow to deserialize.

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.FlattenWithSlowCoder¶

Bases: Benchmark

This benchmark is to test the performance of Flatten when using elements that are slow to deserialize.

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.FlattenBenchmark¶

Bases: Benchmark

UNIT_TEST_N = 3¶

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.GroupByKey¶

Bases: Benchmark

property gbk_executor¶

Set up a GroupByKeyExecutor with an input and output EmbeddedPCollManager

Groups input type of (int, int) to output type of (int, Iterable[int])

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.MaxBundleMemUsage¶

Bases: Benchmark

Test that that bundles that are loaded directly into memory to process never exceed the max bundle memory limit.

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.WriteStructuresToFileMemoryBenchmark¶

Bases: Benchmark

Test that that bundles that are loaded directly into memory to process never exceed the max bundle memory limit.

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.LargeOutputPerBundleWithInProcessWorkerBenchmark¶

Bases: Benchmark

Test that running a pipeline where the size of outputs generated per bundle is large doesn’t result in large memory consumption.

UNIT_TEST_N = 5¶

PERFORMANCE_TEST_N = 2000¶

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.LargeOutputPerBundleWithSubprocessWorkerBenchmark¶

Bases: Benchmark

Test that running a pipeline where the size of outputs generated per bundle is large doesn’t result in large memory consumption.

UNIT_TEST_N = 5¶

PERFORMANCE_TEST_N = 8000¶

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.LargeOutputPerBundleWithLocalWorkerBenchmark¶

Bases: Benchmark

Test that running a pipeline where the size of outputs generated per bundle is large doesn’t result in large memory consumption even when using a DoFn marked with @requires_local_execution.

UNIT_TEST_N = 5¶

PERFORMANCE_TEST_N = 8000¶

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.WorkerServerWithSortingBundleProcessingTime¶

Bases: Benchmark

Test that running a pipeline where the size of outputs generated per bundle is large doesn’t result in large memory consumption even when using a DoFn marked with @requires_local_execution.

UNIT_TEST_N = 1000¶

PERFORMANCE_TEST_N = 1000000¶

SKIP = True¶

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.WorkerServerBundleProcessingTime¶

Bases: Benchmark

Test that running a pipeline where the size of outputs generated per bundle is large doesn’t result in large memory consumption even when using a DoFn marked with @requires_local_execution.

UNIT_TEST_N = 1000¶

PERFORMANCE_TEST_N = 100000¶

SKIP = True¶

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.PCollFileManager_CreateReadersWithManyDataFiles¶

Bases: Benchmark

static pcoll_mngr()¶

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.GroupByKeyLargeValues_runtime¶

Bases: Benchmark

UNIT_TEST_N = 5¶

static generate_1M_random_string()¶

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.GroupByKeyDiskSpaceWithLargeValues¶

Bases: Benchmark

UNIT_TEST_N = 100¶

static generate_1M_random_string()¶

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.GroupByKeyLargeElements_memory¶

Bases: Benchmark

UNIT_TEST_N = 5¶

PERFORMANCE_TEST_N = 1000¶

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.GBKCombineLargeCompElements_memory¶

Bases: Benchmark

Measure GBK combine memory while merging large compressed elements.

Each partition starts with a distinct key so the merge heap must retain one element from every partition without creating a large hot-key group.

UNIT_TEST_N = 2¶

PERFORMANCE_TEST_N = 100¶

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.GroupByKeyWithManyDataChunkFiles¶

Bases: Benchmark

Test that running GroupByKey with many data chunk files that exceed the _MAX_OPEN_FILES limit does not lead to excessive memory usage

UNIT_TEST_N = 5¶

PERFORMANCE_TEST_N = 1000¶

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.RedistributingWithLargeElements¶

Bases: Benchmark

Test that running a pipeline where the size of outputs generated per bundle is large doesn’t result in large memory consumption even when using a DoFn marked with @requires_local_execution.

UNIT_TEST_N = 100¶

PERFORMANCE_TEST_N = 1000¶

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.RedistributingRuntime¶

Bases: Benchmark

UNIT_TEST_N = 1¶

PERFORMANCE_TEST_N = 1000¶

BUFFER_BYTES_THRESHOLD = 1024¶

BLOB_SIZE = 256¶

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

runRedistribute(pcoll_mngr: FilePCollManager, n: int)¶

class schrodinger.seam.testing.benchmarks.ParallelRedistributingRuntime¶

Bases: RedistributingRuntime

runRedistribute(pcoll_mngr: FilePCollManager, n: int)¶

class schrodinger.seam.testing.benchmarks.SeamRunnerOverheadBenchmark¶

Bases: Benchmark

Test a basic Map transform with the SeamRunner to test the overhead of the SeamRunner itself.

UNIT_TEST_N = 1¶

PERFORMANCE_TEST_N = 1¶

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.WriteRandomLogs(message_size: int)¶

Bases: DoFn

Write one fixed-size random message for each input element.

__init__(message_size: int)¶

process(log_number: int) → Iterable[int]¶

Method to use for processing elements.

This is invoked by DoFnRunner for each element of a input PCollection.

The following parameters can be used as default values on process arguments to indicate that a DoFn accepts the corresponding parameters. For example, a DoFn might accept the element and its timestamp with the following signature:

def process(element=DoFn.ElementParam, timestamp=DoFn.TimestampParam):
  ...

The full set of parameters is:

DoFn.ElementParam: element to be processed, should not be mutated.
DoFn.SideInputParam: a side input that may be used when processing.
DoFn.TimestampParam: timestamp of the input element.
DoFn.WindowParam: Window the input element belongs to.
DoFn.TimerParam: a userstate.RuntimeTimer object defined by the spec of the parameter.
DoFn.StateParam: a userstate.RuntimeState object defined by the spec of the parameter.
DoFn.KeyParam: key associated with the element.
DoFn.RestrictionParam: an iobase.RestrictionTracker will be provided here to allow treatment as a Splittable DoFn. The restriction tracker will be derived from the restriction provider in the parameter.
DoFn.WatermarkEstimatorParam: a function that can be used to track output watermark of Splittable DoFn implementations.
DoFn.BundleContextParam: allows a shared context manager to be used per bundle
DoFn.SetupContextParam: allows a shared context manager to be used per DoFn

Args:: element: The element to be processed *args: side inputs **kwargs: other keyword arguments.
Returns:: An Iterable of output elements or None.

class schrodinger.seam.testing.benchmarks.SeamRunnerLogDiskUsageWithDBLoggingBenchmark¶

Bases: _SeamRunnerLogDiskUsageBenchmarkBase

Measure log disk usage with Database logging enabled.

USE_DB_LOGGING = True¶

class schrodinger.seam.testing.benchmarks.SeamRunnerLogDiskUsageWithoutDBLoggingBenchmark¶

Bases: _SeamRunnerLogDiskUsageBenchmarkBase

Measure log disk usage with file-only logging.

class schrodinger.seam.testing.benchmarks.BatchElementsStreamingBenchmark¶

Bases: Benchmark

Benchmark BatchElements performance with bundle-streaming enabled.

This benchmark processes elements through BatchElements with a 1-second sleep per batch. With bundle-streaming enabled, BatchElements accumulates elements into full-sized batches. Without bundle-streaming, it flushes at each bundle boundary, creating many small batches.

UNIT_TEST_N = 128¶

PERFORMANCE_TEST_N = 1024¶

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.StructureCompressionRatio¶

Bases: _CompressionBenchmarkBase

Compression ratio for bare Structure objects.

class schrodinger.seam.testing.benchmarks.AnnotatedStructureCompressionRatio¶

Bases: _CompressionBenchmarkBase

Compression ratio for a dataclass with Structure + float fields.

class schrodinger.seam.testing.benchmarks.PackingResultCompressionRatio¶

Bases: _CompressionBenchmarkBase

Compression ratio for a dataclass with Structure + mixed metadata.

class schrodinger.seam.testing.benchmarks.StructureEncodingSpeed¶

Bases: StructureCompressionRatio, _EncodingSpeedBenchmarkBase

Encoding speed ratio for bare Structure objects.

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.AnnotatedStructureEncodingSpeed¶

Bases: AnnotatedStructureCompressionRatio, _EncodingSpeedBenchmarkBase

Encoding speed ratio for AnnotatedStructure.

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

class schrodinger.seam.testing.benchmarks.PackingResultEncodingSpeed¶

Bases: PackingResultCompressionRatio, _EncodingSpeedBenchmarkBase

Encoding speed ratio for PackingResult.

run(n: int) → BenchmarkResult¶

Parameters:: n – The number of elements to process

schrodinger.seam.testing.benchmarks.get_benchmarks()¶