Skip to content

added parallelization options of qsub command #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
MiqG wants to merge 2 commits into
base: dev
Choose a base branch
from
Open

added parallelization options of qsub command #3

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2e21f8d
added parallelization options to GridEngineScheduler
Oct 22, 2020
457c34b
updated GridEngineScheduler description
MiqG Oct 22, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 29 additions & 12 deletions gridengine/schedulers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,12 +108,17 @@ def __init__(self, **resources):

Only one instance may run per Python process, since the underlying drmaa
layer is a singleton.

Keyword Args:
Resources to be passed to the -l command of qsub. e.g.
h_cpu: maximum time expressed in format '02:00:00' (2 hours)
h_vmem: maximum memory allocation before job is killed in format '10G' (10GB)
virtual_free: memory free on host BEFORE job can be allocated
Resources to be passed to qsub commands. These override any
arguments that were given to the constructor:
`-l` command:
h_cpu: maximum time expressed in format '02:00:00' (2 hours)
h_vmem: maximum memory allocation before job is killed in format '10G' (10GB)
virtual_free: memory free on host BEFORE job can be allocated
`-pe` command:
pe_type: either 'smp' or 'ompi' for shared memory or distributed memory.
n_slots: the number of slots to request to the grid engine.
"""
import drmaa
self.drmaa = drmaa
Expand Down Expand Up @@ -141,18 +146,22 @@ def schedule(self, submission_host, job_queue, **resources):
job_queue: the dict of {jobid, job.Job} items to run

Keyword Args:
Resources to be passed to the -l command of qsub. These override any
arguments that were given to the constructor. e.g.
h_cpu: maximum time expressed in format '02:00:00' (2 hours)
h_vmem: maximum memory allocation before job is killed in format '10G' (10GB)
virtual_free: memory free on host BEFORE job can be allocated
Resources to be passed to qsub commands. These override any
arguments that were given to the constructor:
`-l` command:
h_cpu: maximum time expressed in format '02:00:00' (2 hours)
h_vmem: maximum memory allocation before job is killed in format '10G' (10GB)
virtual_free: memory free on host BEFORE job can be allocated
`-pe` command:
pe_type: either 'smp' or 'ompi' for shared memory or distributed memory.
n_slots: the number of slots to request to the grid engine.
"""

# dont spin up the scheduler if there's nothing to do
if not job_queue: return

# update the keyword resources
resources = dict(self.resources.items() + resources.items())
resources = {**self.resources, **resources}

# retrieve the job target
target = job_queue[0].target
Expand All @@ -166,9 +175,17 @@ def schedule(self, submission_host, job_queue, **resources):
jt.args = [submission_host]
jt.jobName = resources.pop('name',target)
jt.jobName = ''.join(jt.jobName.split())[:15]
jt.nativeSpecification = '-l ' + ','.join(
## prepare -pe
if 'pe_type' in resources.keys() and 'n_slots' in resources.keys():
jt.nativeSpecification = '-pe '+' '.join([resources.pop('pe_type'),
resources.pop('n_slots')])+' '
else:
jt.nativeSpecification = ''
## prepare -l
jt.nativeSpecification += '-l ' + ','.join(
resource + '=' + str(value) for resource,value in resources.items()
) if resources else ''

jt.joinFiles = True
jt.outputPath = ':'+os.path.expanduser(settings.TEMPDIR)
jt.errorPath = ':'+os.path.expanduser(settings.TEMPDIR)
Expand Down