PyDAG

Simple Python Interface to HTCondor and DAGMan

PyDAG allows you to create HTCondor submit descriptions, DAGMan nodes, and DAGs via Python. Any Python object that has a string representation that corresponds to a valid HTCondor command or DAGMan keyword can be added to the submit description. You can easily extend PyDAG by defining your own command/keyword classes; classes for pre/post-scripts and macros are already provided.

class pydag.htcondor.HTCondorSubmit(filename, executable, queue=1, **kwargs)

HTCondor submit description

Represents a submit description of a HTCondor job.

Parameters:
  • filename (str) – HTCondor submit description file
  • executable (str) – HTCondor executable file
  • queue (int, optional) – Number of jobs to queue
  • **kwargs – Submit description file commands
commands

Mapping of describing submit description file commands to objects representing values

Type:dict(str, object)

Examples

Create HTCondor submit description for a Python script.

>>> import pydag
>>> job = pydag.htcondor.HTCondorSubmit("example.submit", "example.py")
>>> job.commands["initialdir"] = "$ENV(HOME)"
>>> print(job)
universe = vanilla
executable = example.py
initialdir = $ENV(HOME)
queue 1
>>> job.dump()
>>> print(job.written_to_disk)
True
dump()

Write HTCondor submit description to filename.

written_to_disk

If True submit description was written to disk.

Type:bool
class pydag.dagman.DAGManJob(filename, nodes)

DAGMan job

Specify a DAG’s input.

Parameters:
  • filename (str) – DAGMan input file
  • nodes (list(DAGManNode)) – Sequence of DAGMan nodes

Examples

Create a DAG with one node; a macro inputfile is defined; the macro’s value is passed as an argument to the Python script that is specified as the executable in the node’s HTCondor submit description.

>>> import pydag
>>> job = pydag.htcondor.HTCondorSubmit("example.submit", "example.py")
>>> job.commands["arguments"] = "$(inputfile)"
>>> node = pydag.dagman.DAGManNode("example", job)
>>> node["VARS"] = pydag.dagman.Macros(inputfile="example.txt")
>>> dag = pydag.dagman.DAGManJob("example.dag", [node])
>>> print(dag)
JOB example example.submit
VARS example inputfile="example.txt"
>>> dag.dump()
>>> print(dag.written_to_disk)
True
add_dependency(parents, children)

Add dependency within the DAG.

Nodes are parents and/or children within the DAG. A parent node must be completed successfully before any of its children may be started. A child node may only be started once all its parents have successfully completed.

Parameters:
  • parents (tuple(str)) – Parent node names
  • children (tuple(str)) – Children node names
Raises:

ValueError – If parents or children contains an unknown DAGMan node.

dependencies

Dependencies within the DAG

Type:str
dump()

Write DAGMan input file to filename.

written_to_disk

If True DAGMan input file was written to disk.

Type:bool
class pydag.dagman.DAGManNode(name, submit_description, **kwargs)

DAGMan node

Specify a DAG’s node.

Parameters:
  • name (str) – Uniquely identifies nodes within the DAGMan input file and in output messages
  • submit_description (HTCondorSubmit, str) – HTCondor submit description; either a string specifying a submit description file or an HTCondorSubmit instance
  • **kwargs – Node keywords
keywords

Mapping of node keywords to objects representing values

Type:dict(str, object)
class pydag.dagman.DAGManScript(executable, *args)

Pre-processing or post-processing

Specify a shell script/batch file to be executed either before a job within a node is submitted or after a job within a node completes its execution.

Parameters:
  • executable (str) – Specify the shell script/batch file to be executed.
  • *args – Script/batch file arguments
arguments

Sequence of objects representing script/batch file arguments

Type:list(object)
class pydag.dagman.Macros

Define macros for DAGMan node.

Derived from dict; only __str__ is re-implemented to meet the DAGMan macro syntax.

Indices and tables