LSF scheduler integration

[robertsawko]

Hi,

My colleague and I are working on LSF integration. We have created and adapted relevant files: lsf.py and lsf.sh and LSF is displaying as "Scheduler" in cluster settings. Unfortunately we are still getting "Unsupported scheduler" when we press "Save" button.

Looking at the code we can see that queue in cluster.py still doesn't find the type even though we added relevant content in dictionary type_to_adapter in queue/__init__.py.

Any advice would welcome, thanks.

[carpemonf-zz]

Only add that we also modified the constants.py file for the cumulus plugin, and the hpccloud files (LSF.js, index.js and RunCluster.js) to see the LSF scheduler.

[cjh1]

The message will be produced if this is False. Are you using the same string to identify the queue as the one registered?

[robertsawko]

Can we just double check with you: where do we put the string to identify the queue and and wehere do we register it?

[cjh1]

Its the key used when you add to the type_to_adapter dict.

[carpemonf-zz]

I think we have the same string for the LSF queue. We have the following definition for /opt/hpccloud/cumulus/cumulus/constants.py:

class QueueType:
    SGE = 'sge'
    PBS = 'pbs'
    SLURM = 'slurm'
    LSF = 'lsf'
    NEWT = 'newt'

and for /opt/hpccloud/hpccloud/src/panels/SchedulerConfig/index.js:

            <option value="sge">Sun Grid Engin</option>
            <option value="pbs">PBS</option>
            <option value="slurm">SLURM</option>
            <option value="lsf">LSF</option>

[cjh1]

@carpemonf In that case are you sure your are running the updated code? May be add a quick print statement to ensure your changes are being loaded on the server.

[robertsawko]

@cjh1 , sorry we are a bit new to Girder and python->WebUI conversion. Could you tell us how to print or or that data and where to find the standard output or log file? Thanks.

[carpemonf-zz]

Thanks, I have just added a modified statement through ValidationException message in /opt/hpccloud/cumulus/girder/cumulus/server/models/cluster.py. It also includes, for testing, a message when the queue is supported.

        if not queue.is_valid_type(scheduler_type):
            raise ValidationException('Unsupported scheduler: %s.' % scheduler_type, 'type')
        else:
            raise ValidationException('Supported scheduler: %s.' % scheduler_type, 'type')

The new messages appear, so at least this piece of code is updated.

Next, I modified the is_valid_type(type) function in /opt/hpccloud/cumulus/cumulus/queue/__init__.py to fix the return value to False to see if the function was checking the LSF queue properly, but It always give me a True statement for the default queues.

    valid = False
    return valid
    #return type in type_to_adapter

[robertsawko]

Sorry for bumping this thread, but we are still struggling with this! So what we did right now was to comment out the exception throwing and we actually managed to move forward. This kind of shows that the code is compiling but maybe not all of it? Any advice would be welcome.

[cjh1]

@robertsawko Sorry I have away a conference. Will try to take a look today.

[cjh1]

@robertsawko Are your code changes pushed somewhere so I could take look?

[carpemonf-zz]

@cjh1 We had some problems with the cumulus dependencies compiling HPCCloud for development, we are playing in the meanwhile with the prebuilt VMs on HPCCloud-deploy/prebuilt/hpccloud-server/. I have attached a patch: lsf.txt. This should work for this VM.

Please let me know if it works for you.

[cjh1]

So I was able take your server changes ( the new adapter ) and apply it locally and was then able to create a LSF cluster using following POST ( outside the web app ):

{
  "config": {
    "scheduler": {"type": "lsf"},
    "ssh": {"user": "test"},
"host": "test"
  },
  "name": "test3",
  "type": "trad"
}

@jourdain Can you check that client side is constructing the appropriate JSON.

[carpemonf-zz]

@cjh1 Could you please confirm me that applying the following to cumulus/cumulus/queue/__init__.py, a cluster with PBS, SGE, SLURM or LSF can not be created? I can still create them with these changes:

-    return type in type_to_adapter
+    valid = False
+    return valid
+    #return type in type_to_adapter

[cjh1]

@carpemonf If I return False from is_valid_type, I am unable to create a cluster, I get "Unsupported scheduler." as expected.