expyre.schedulers package

expyre.schedulers package#

Submodules#

expyre.schedulers.base module#

class expyre.schedulers.base.Scheduler(host, remsh_cmd=None)[source]#

Bases: object

cancel(remote_ids, verbose=False)[source]#

cancel remote job

Parameters

remote_ids (str, list(str)) – remote ids of jobs to hold

hold(remote_ids, verbose=False)[source]#

hold remote job

Parameters

remote_ids (str, list(str)) – remote ids of jobs to hold

static node_dict_env_var_commands(node_dict)[source]#
release(remote_ids, verbose=False)[source]#

release remote job

Parameters

remote_ids (str, list(str)) – remote ids of jobs to hold

status(remote_ids, verbose=False)[source]#
submit(id, remote_dir, partition, commands, max_time, header, node_dict, no_default_header=False, script_exec='/bin/bash', pre_submit_cmds=[], verbose=False)[source]#
static unset_scheduler_env_vars(prefix)[source]#

expyre.schedulers.local module#

class expyre.schedulers.local.Local(host, remsh_cmd=None)[source]#

Bases: Scheduler

expyre.schedulers.pbs module#

class expyre.schedulers.pbs.PBS(host, remsh_cmd=None)[source]#

Bases: Scheduler

Create PBS object

Parameters

  • host (str) – username and host for ssh/rsync username@machine.fqdn

  • remsh_cmd (str, default EXPYRE_RSH env var or 'ssh') – remote shell command to use

status(remote_ids, verbose=False)[source]#

determine status of remote jobs

Parameters

remote_ids (str, list(str)) – list of remote ids to check

Returns

dict { str remote_id – “queued”, “held”, “running”, “done”, “failed”, “timeout”, “other” all remote ids passed in are guaranteed to be keys in dict, specific jobs that are not listed by queueing system have status “done”

Return type

str status}, status is one of :

submit(id, remote_dir, partition, commands, max_time, header, node_dict, no_default_header=False, script_exec='/bin/bash', pre_submit_cmds=[], verbose=False)[source]#

Submit a job on a remote machine

Parameters

  • id (str) – unique job id (local)

  • remote_dir (str) – remote directory where files have already been prepared and job will run

  • partition (str) – partition (or queue or node type)

  • commands (list(str)) – list of commands to run in script

  • max_time (int) – time in seconds to run

  • header (list(str)) – list of header directives, not including walltime specific directive

  • node_dict (dict) – properties related to node selection. Fields: num_nodes, num_cores, num_cores_per_node, ppn, id, max_time, partition (and its synonum queue)

  • no_default_header (bool, default False) – do not add normal header fields, only use what’s passed in in “header”

  • script_exec (str, default '/bin/bash') – executable for first line of job script

  • pre_submit_cmds (list(str), default []) – command to run in the remote process that does the submission before the actual submission, e.g. to fix the environment

Return type

str remote job id

expyre.schedulers.sge module#

class expyre.schedulers.sge.SGE(host, remsh_cmd=None)[source]#

Bases: Scheduler

Create SGE object

Parameters

  • host (str) – username and host for ssh/rsync username@machine.fqdn

  • remsh_cmd (str, default EXPYRE_RSH env var or 'ssh') – remote shell command to use

status(remote_ids, verbose=False)[source]#

determine status of remote jobs

Parameters

remote_ids (str, list(str)) – list of remote ids to check

Returns

dict { str remote_id – “queued”, “held”, “running”, “done”, “failed”, “timeout”, “other” all remote ids passed in are guaranteed to be keys in dict, specific jobs that are not listed by queueing system have status “done”

Return type

str status}, status is one of :

submit(id, remote_dir, partition, commands, max_time, header, node_dict, no_default_header=False, script_exec='/bin/bash', pre_submit_cmds=[], verbose=False)[source]#

Submit a job on a remote machine

Parameters

  • id (str) – unique job id (local)

  • remote_dir (str) – remote directory where files have already been prepared and job will run

  • partition (str) – partition (or queue or node type)

  • commands (list(str)) – list of commands to run in script

  • max_time (int) – time in seconds to run

  • header (list(str)) – list of header directives, not including walltime specific directive

  • node_dict (dict) – properties related to node selection. Fields: num_nodes, num_cores, num_cores_per_node, ppn, id, max_time, partition (and its synonum queue)

  • no_default_header (bool, default False) – do not add normal header fields, only use what’s passed in in “header”

  • script_exec (str, default '/bin/bash') – executable for first line of job script

  • pre_submit_cmds (list(str), default []) – command to run in the remote process that does the submission before the actual submission, e.g. to fix the environment

Return type

str remote job id

expyre.schedulers.slurm module#

class expyre.schedulers.slurm.Slurm(host, remsh_cmd=None)[source]#

Bases: Scheduler

Create Slurm object

Parameters

  • host (str) – username and host for ssh/rsync username@machine.fqdn

  • remsh_cmd (str, default EXPYRE_RSH env var or 'ssh') – remote shell command to use

status(remote_ids, verbose=False)[source]#

determine status of remote jobs

Parameters

remote_ids (str, list(str)) – list of remote ids to check

Returns

dict { str remote_id – “queued”, “held”, “running”, “done”, “failed”, “timeout”, “other” all remote ids passed in are guaranteed to be keys in dict, specific jobs that are not listed by queueing system have status “done”

Return type

str status}, status is one of :

submit(id, remote_dir, partition, commands, max_time, header, node_dict, no_default_header=False, script_exec='/bin/bash', pre_submit_cmds=[], verbose=False)[source]#

Submit a job on a remote machine

Parameters

  • id (str) – unique job id (local)

  • remote_dir (str) – remote directory where files have already been prepared and job will run

  • partition (str) – partition (or queue or node type)

  • commands (list(str)) – list of commands to run in script

  • max_time (int) – time in seconds to run

  • header (list(str)) – list of header directives, not including walltime specific directive

  • node_dict (dict) – properties related to node selection. Fields: num_nodes, num_cores, num_cores_per_node, ppn, id, max_time, partition (and its synonum queue)

  • no_default_header (bool, default False) – do not add normal header fields, only use what’s passed in in “header”

  • script_exec (str, default '/bin/bash') – executable for first line of job script

  • pre_submit_cmds (list(str), default []) – command to run in the remote process that does the submission before the actual submission, e.g. to fix the environment

Return type

str remote job id

Module contents#

Top level interface to all the implemented schedulers in expyre.schedulers_impl

Contents