schrodinger.job.launchapi module¶

Provide an API that allows a definition of a job.

A job is composed of two pieces:

A task specification that defines a generic command-line invocation and defines what input files are needed and what output files are produced.

Runtime parameters that include specific input and output filenames.

To launch a job, you must also provide a set of launch parameters, which specify things like the job host, number of threads per process, etc.

To construct a job specification for a script that you want to run under job control, use the JobSpecificationArgsBuilder. This creates a JobSpecification from a list of command-line arguments. You must indicate the specific input and output files in the command-line arguments so they can be transferred and retrieved.

exception schrodinger.job.launchapi.TaskSpecificationError¶: Bases: RuntimeError

exception schrodinger.job.launchapi.RuntimeParametersError¶: Bases: RuntimeError

exception schrodinger.job.launchapi.SpecificationKeyError(key, context=None)¶

Bases: KeyError

An error thrown when a key is missing from a serialized specification.

spec_description = 'specification'¶

__init__(key, context=None)¶

exception schrodinger.job.launchapi.TaskSpecificationKeyError(key, context=None)¶

Bases: schrodinger.job.launchapi.SpecificationKeyError

spec_description = 'task specification'¶

exception schrodinger.job.launchapi.RuntimeParametersKeyError(key, context=None)¶

Bases: schrodinger.job.launchapi.SpecificationKeyError

spec_description = 'runtime parameters'¶

exception schrodinger.job.launchapi.JobSpecificationKeyError(key, context=None)¶

Bases: schrodinger.job.launchapi.SpecificationKeyError

spec_description = 'job specification'¶

schrodinger.job.launchapi.check_valid_key(key)¶: Make sure the provided key is valid. Raise a RuntimeError if it’s not.

class schrodinger.job.launchapi.TaskSpecification¶

Bases: object

A class that holds the information necessary to run a task on a compute resource.

It holds a templated shell command along with info about required input, output created, and information about how to generate a jobname. It doesn’t hold any information specific to a job run - e.g. hostname, input filenames, output filenames.

This class has no methods that modify instance state. Use a builder class (TaskSpecificationBuilder or JobSpecificationBuilder) to set up a TaskSpecification.

__init__()¶

validate()¶: Make sure that the variables used in the command-line template are defined with input or output specifications. Raise a TaskSpecificationError if not.

static fromJSON(json_spec)¶

asJSON(**kwargs)¶

getCommandTemplate()¶: Return the command-line as a list of arguments with keys in the form of <KEY> that will be replaced with runtime parameters.

needsExplicitJobname()¶: Return True if a jobname must be specified as part of the runtime parameters.

getInputKeys()¶: Return a list of keys that define inputs.

getJobnameSource()¶: Return the input key that should be used to derive the jobname.

getProgramName()¶

isStoppable()¶: If True, a job is stoppable by sending ‘halt’ message to the backend. And transfer output files if any. Otherwise, job supervisor kills the backend and transfer output files.

getLicenses()¶: Returns licenses as a list of tuples of ( license_name, num_tokens)

driverReservesCores()¶

If True, the driver will allocate N slots matching -HOST hostname: N processes. This is useful when the driver allocates parallel slots on one node so it can resubmit jobs to localhost. If no N is specified, assume to be 1.

This is an optimization to avoid resubmission back to the queue if we know we are running a set of parallel jobs with the same CPU requirements.

jobUsesTPP()¶

If True, this job will allocate N slots on one node matching -TPP N.

If False, this job only passes TPP on as an option for subjobs.

TPP stands for Threads Per Process.

getOutputKeys()¶: Return a list of keys that define outputs.

useSchrodingerRun()¶: Return whether to run the command under $SCHRODINGER/run.

useSchrodingerProduct()¶: Return whether to run the command under $SCHRODINGER/run.

schrodinger.job.launchapi.var(string)¶: Format the provided string to be used as a variable in the TaskSpecification command-line template.

class schrodinger.job.launchapi.TaskSpecificationBuilder(data=None)¶

Bases: object

A class for building up a TaskSpecification from a specific list of command-line arguments.

__init__(data=None)¶

getTaskSpec()¶: Return a TaskSpecification from this builder.

setCommandLine(args, use_schrodinger_run=True, schrodinger_product=None)¶

Set the command line as provided. Add a SCHRODINGER/run if requested.

Parameters

args (list of str) – The command-line script and arguments.
use_schrodinger_run – If True, run the command under $SCHRODINGER/run.
schrodinger_product (str) – A product directory to search for the script/executable. This should be the name of a directory under SCHRODINGER without the trailing version (i.e. the “-v*” part).

setInputFile(key, jobname_source=False)¶: Set an input file for the task specification.

addLicense(license_name, license_tokens)¶

Parameters

license_name (schrodinger.infra.licensing.MMLIC3*) – Name of a license token that is checked out.
license_tokens (str) – The number of tokens required by this job type.

setOutputFile(key, runtime_path, stream=False, incorporate=False)¶

Parameters

runtime_path (str) – The path of an output file that will be generated. May contain input keys in the form <KEY> or <JOBNAME> (if a jobname source has been specified).
key (str) – A key to be used to reference this output file.
stream (bool) – If True and if possible, the output file will be streamed back while the job is running. Defaults to False. Optional.

setStderr(runtime_path, stream=False)¶

Parameters

runtime_path (str) – A runtime_path template with input keys in the form <KEY>.
stream (bool) – If True and if possible, the file will be streamed back while the job is running. Defaults to False. Optional.

setStdout(runtime_path, stream=False)¶

Parameters

runtime_path (str) – A runtime_path template with input keys in the form <KEY>.
stream (bool) – If True and if possible, the file will be streamed back while the job is running. Defaults to False. Optional.

setStoppable(stoppable)¶: If passed True, the job supervisor sends the ‘halt’ message to the backend to stop. And transfer output files if any. Otherwise, the supervisor kills the backend and transfer output files.

setProgramName(program_name)¶

setDriverReservesCores(reserved)¶

If passed True, the driver will allocate N slots matching -HOST hostname: N processes. This is useful when the driver allocates parallel slots on one node so it can resubmit jobs to localhost. If no N is specified, assume to be 1.

This is an optimization to avoid resubmission back to the queue if we know we are running a set of parallel jobs with the same CPU requirements.

setJobUsesTPP(uses_tpp)¶

If passed True, this job will reserve the number of cores specified from -TPP on the cmdline. Unfortunately, we use -TPP N to mean two things

this job will reserve N processors on one node directly
this job takes -TPP as a cmdline argument, but only to pass along to subjobs

In case 1, the uses_tpp should be set to True.

schrodinger.job.launchapi.get_file_url(path)¶: Get a file:// URL for a file path.

class schrodinger.job.launchapi.JobSpecification(task_spec, runtime_params)¶

Bases: object

This class provides a serializable job definiton and consists of a task specification along with runtime parameters.

Currently unsupported features:

Runtime requirements (e.g. GPU, memory requirements).

Features that will probably never be supported:

The equivalent of job control’s ‘-LOCAL’ option.
Full shell commands. (Arguments must be provided as a list of strings. Chaining commands together is not supported. Redirection is provided only through APIs.)

__init__(task_spec, runtime_params)¶

asJSON(**kwargs)¶

static fromJSON(json_job_spec)¶: Return a JobSpecification instance created from the provided JSON.

validate()¶: A validation method that makes sure the JobSpecification has all the data it needs and is self-consistent.

getCommand()¶

Return the shell command that will run the job.

Returns: list of str

getOutputFile(key, stream=None, incorporate=None)¶

Get the output file corresponding to the given key from the job specification.

Parameters: stream – If stream=True, the file is returned only if it is a log file. If stream=False, the file is returned only if it is an output file. If stream=None, the file is returned.

getOutputFiles(stream=None, incorporate=None)¶

Get all the output files from the job specification.

Parameters

stream (True, False or None(Default)) – If stream=None, both output and log files are returned. If stream=True, only log files are returned. If stream=False, only output files are returned. Defaults to None. Optional.
incorporate – If incorporate=None, incorporatable and other files are returned. If incorporate=True, only files to be incorporated are returned. If incoporate=False, only output files not incorporated are returned.

getStderr()¶

getStdout()¶

getDefaultStderr()¶

getDefaultStdout()¶

useSchrodingerRun()¶

getInputFiles()¶

getHost()¶

debugEnabled()¶

getJobname()¶

setJobname(jobname)¶

getProgramName()¶

isStoppable()¶: If True, a job is stoppable by sending ‘halt’ message to the backend. And transfer output files if any. Otherwise, job supervisor kills the backend and transfer output files.

driverReservesCores()¶

If True, the driver will allocate N slots matching -HOST hostname: N processes. This is useful when the driver allocates parallel slots on one node so it can resubmit jobs to localhost. If no N is specified, assume to be 1.

This is an optimization to avoid resubmission back to the queue if we know we are running a set of parallel jobs with the same CPU requirements.

jobUsesTPP()¶

If True, this job will allocate N slots on one node matching -TPP N.

If False, this job uses TPP as an option to pass along to subjobs

addLicense(license_name, license_tokens)¶

Add license to the task spec.

Parameters

license_name (schrodinger.infra.licensing.MMLIC3*) – Name of a license token that is checked out.
license_tokens (str) – The number of tokens required by this job type.

getLicenses()¶: Returns licenses as a list of tuples of ( license_name, num_tokens)

class schrodinger.job.launchapi.JobSpecificationBuilder(task_spec)¶

Bases: object

A helper class to create a JobSpecification from an existing TaskSpecification.

__init__(task_spec)¶

getJobSpec()¶

setJobname(jobname)¶

setInputFiles(**input_files)¶

Indicate the filenames to be used for the input keys.

Parameters: input_files – Keyword arguments with the key as INPUT_KEY and the value as the associated filename.

setOutputFiles(**output_files)¶

Indicate the filenames to be used for the output keys.

Parameters: output_files – Keyword arguments with the key as OUTPUT_KEY and the value as the associated filename.

class schrodinger.job.launchapi.JobSpecificationArgsBuilder(args, use_schrodinger_run=True, schrodinger_product=None, program_name=None, default_jobname=None, use_jobname_log=False)¶

Bases: object

A helper class to create a JobSpecification from a specific (i.e. non-generic) set of command-line arguments.

__init__(args, use_schrodinger_run=True, schrodinger_product=None, program_name=None, default_jobname=None, use_jobname_log=False)¶

See TaskSpecificationBuilder.setCommandLine for argument descriptions.

Parameters: use_jobname_log – If True, set the STDOUT, STDERR as <JOBNAME>.log and stream it. Default is False.

getJobSpec()¶

setJobname(jobname)¶

getJobname()¶

Get the job name set from runtime parameters if there is one

This will return any jobname set via command line or the setJobname function. It will not return any jobname set using jobname_source=True when setting an input/output file.

Return type: str or None
Returns: The current set job name, or None if no name is set

setProgramName(program_name)¶

setStoppable(stoppable)¶

Mark the job as stoppable, meaning the backend has been designed to listen for and respond to graceful stop requests through the jobcontrol API.

Jobs that are not marked stoppable will be sent a SIGKILL and immediately terminated upon a user stop request.

Please be aware that jobs that are marked stoppable but do not actually implement graceful stops are bad actors and will never be stopped by a user request.

setInputDirectory(path, runtime_path=None)¶

Register a directory as input for the job. All files found under it recursively are registered as input.

Please avoid using this method unless you know you want all the files in the directory.

If the path given is absolute or a relative path not in a subdirectory, the default runtime path will be in the job directory. If the path is relative and in a subdirectory, it will be used as is.

setInputFile(path: str, key: Optional[str] = None, runtime_path: Optional[str] = None) → None¶

Register a file as input for the job.

Parameters

runtime_path –

The path that will be used for this input file at runtime. If not specified, the behavior is as follows:

If the path given is absolute or is a relative path not in a subdirectory, the default runtime_path will always be the basename of the file. Absolute paths always default to the basename, even if they refer to a path in a subdirectory. For example:

path=”/scr/input_dir/input.txt” will have a runtime_path of “input.txt”.

path=”../input_dir/input.txt” will have a runtime_path of “input.txt”

If the path given is relative and within a subdirectory, the default runtime_path will use the path as is. For example:

path=”input_dir/input.txt” will have a runtime_path of “input_dir/input.txt”

setStderr(runtime_path, stream=False)¶

setStdout(runtime_path, stream=False)¶

setOutputFile(runtime_path, key=None, stream=False, incorporate=False)¶

Parameters

runtime_path (str) – The path of an output file that will be created by the job.
stream (bool) – If True and if possible, the output file will be streamed back while the job is running. Defaults to False. Optional.
incoporate – If True, mark this file for incorporation in maestro. NOTE: Only one file can be declared incorporatable. Defaults to False. Optional.

addLicense(license_string, license_tokens)¶

setDriverReservesCores(reserved)¶

If passed True, the driver will allocate N slots matching -HOST hostname: N processes. This is useful when the driver allocates parallel slots on one node so it can resubmit jobs to localhost. If no N is specified, assume to be 1.

This is an optimization to avoid resubmission back to the queue if we know we are running a set of parallel jobs with the same CPU requirements.

setJobUsesTPP(uses_tpp)¶

If uses_tpp is True, this job will reserve the number of cores specified from -TPP (Threads Per Process) on the command line.

Currently we use -TPP N to mean two different things: 1) Reserve N processors for the job being launched. 2) Reserve N processors for each subjob of the job being launched.

Case 1 is the case that needs uses_tpp=True.

schrodinger.job.launchapi.is_file_not_found_exception(exc: OSError) → bool¶: Return True if the exception indicates that a file does not exist.

schrodinger.job.launchapi.are_same_file(path1: pathlib.Path, path2: pathlib.Path) → bool¶: Compare if two file paths are the same, by examining the filesystem. This does not raise an exception if either path does not exist.

schrodinger.job.launchapi.resolve_to_abspath(path: pathlib.Path) → pathlib.Path¶: Attempt resolution to absolute path. If the provided path does not exist, return original path.