Conversation
- move `Scheduler.options` to `QueuedScheduler` and rename (with deprecation warning) as `directives` since it only applies to schedulers that use a job submission system - remove `Scheduler` attributes `shell_args` and `shebang_args` (and class variables `DEFAULT_SHELL_ARGS` and `DEFAULT_SHEBANG_ARGS`) and replace with `Scheduler.shebang_executable`, which will typically not be used, and is a way to override the shell's executable and executable arguments in the jobscript shebang line. - remove unused `DEFAULT_SHELL_EXECUTABLE` class variable from `Scheduler` sub-classes - add `Shell.executable_args` and include this in `Shell.executable`
dkfellows
approved these changes
Apr 23, 2025
Collaborator
dkfellows
left a comment
dkfellows
left a comment
There was a problem hiding this comment.
Subject to one question, this is fine.
The question: what's the way of handling adding extra bits to a shebang line? The classic example of that is running a script interpreter with PATH searching using: #!/usr/bin/env interpreter though I've also seen an occasional need to pass other options there too. It's more common with Perl scripts I suppose...
In any case, as long as there is a documented answer, whatever it is, I'm happy with this PR.
Collaborator
|
Thanks Adam. I made a new nenv and installed hpcflow using On CSF3 using SGE it has worked :) Using SLURM, it doesn't. I updated the config like this: But the workflow wouldn't submit. |
Contributor
Author
Thanks @dkfellows, I'll add some documentation. |
Contributor
Author
|
@gcapes glad this fixes the issue on SGE! Could you check your workflow template file and any task schema files in a yaml linter? |
Collaborator
|
Yep, the workflow contains the task schema definition, and is valid yaml: |
Contributor
Author
Could you also check your environment definintion file(s) and any other YAML files pointed to in your config (and the built in template component files, if you've made any changes to them)? I will merge this PR soon. I think your issue above is unrelated, so could you open a new issue if you are still having issues (unless you think it is related to this PR?). |
Contributor
Author
Collaborator
|
Thanks @aplowman I'm not really sure the right place for this, so just adding it here. I still have the problem, and this is the config error. Could you advise how to fix it? |
Contributor
Author
No problem. The error is saying that |
Collaborator
|
That's correct. I'm using the branch from PR #819 in an attempt to find which file contains the yaml error. I'm confused what changes would be needed to my config so maybe we can go through this in today's meeting? |
Collaborator
|
After a couple of hours, I finally got some output However, my config doesn't contain this keyword. My history suggests I installed this branch, but I'm going to do a local repo install so it's easier to figure out which branch I'm using for hpcflow. I have Submits but errors. I think I need to try this again with a local install so I know for sure which branch I'm using. I guess that should just be develop/latest release? I only need feat/yaml-error-path if the main release version doesn't work for me? |
Contributor
Author
|
I would expect that error if 1. the I have just merged One thing to check is |
Collaborator
|
Ok, making some progress.
$ matflow config get --all
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────── Config 'default' ─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ machine 'DEFAULT_MACHINE' │
│ log_file_path PosixPath('/mnt/iusers01/support/mbexegc2/.matflow-new/logs/MatFlow_v0.3.0a159.log') │
│ task_schema_sources [] │
│ parameter_sources [] │
│ command_file_sources [] │
│ environment_sources [] │
│ default_scheduler 'direct' │
│ default_shell 'bash' │
│ schedulers {'direct': {'defaults': {}}} │
│ shells {'bash': {'defaults': {}}} │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
I think I have now fixed that too $ matflow config get --all
╭────────────────────────────────────────────────────────────────────────────────────────────────────── Config 'manchester-CSF3-slurm' ──────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ machine 'manchester-CSF3-new' │
│ log_file_path PosixPath('/mnt/iusers01/support/mbexegc2/.matflow-new/logs/MatFlow_v0.3.0a159.log') │
│ task_schema_sources [] │
│ parameter_sources [] │
│ command_file_sources [] │
│ environment_sources [PosixPath('/mnt/iusers01/support/mbexegc2/.matflow-new/envs-slurm.yaml')] │
│ default_scheduler 'slurm' │
│ default_shell 'bash' │
│ schedulers { │
│ 'direct': {'defaults': {}}, │
│ 'slurm': { │
│ 'defaults': {}, │
│ 'partitions': { │
│ 'serial': {'num_cores': [1, 1, 1]}, │
│ 'multicore': { │
│ 'num_nodes': [1, 1, 1], │
│ 'num_cores': [2, 1, 168], │
│ 'num_cores_per_node': [2, 1, 168], │
│ 'parallel_modes': ['distributed', 'shared', 'hybrid'] │
│ }, │
│ 'multinode': {'num_nodes': [2, 1, None], 'num_cores': [80, 40, None], 'num_cores_per_node': [40, 40, 40], 'parallel_modes': ['distributed', 'hybrid']} │
│ } │
│ } │
│ } │
│ shells {'bash': {'defaults': {'executable_args': ['--login']}}} │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
After deleting two duplicate environment executables .... SUCCESS!! It's now my priority to write this all up in hpcflow/matflow#376 Thanks for bearing with my stream-of-consciousness-debugging spam at the end of this PR :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scheduler.optionstoQueuedSchedulerand rename (with deprecation warning) asdirectivessince it only applies to schedulers that use a job submission systemSchedulerattributesshell_argsandshebang_args(and class variablesDEFAULT_SHELL_ARGSandDEFAULT_SHEBANG_ARGS) and replace withScheduler.shebang_executable, which will typically not be used, and is a way to override the shell's_executableandexecutable_argsin the jobscript shebang line.DEFAULT_SHELL_EXECUTABLEclass variable fromSchedulersub-classesShell.executable_args(e.g. for using bash in--loginmode) and include this inShell.executable@gcapes please could you test this PR on CSF3 and check if it resolves your issue?
Recommended changes to configuration files
If the
schedulers.[scheduler-name].defaults.shebang_argsis used in a configuration file, this will need to be removed and added to the shell's newexecutable_argsconfiguration, e.g.shells.bash.defaults.executable_args, which should be a list of strings. An example change is shown here.Recommended changes to workflow template files
The
Schedulerargumentoptions(for verbatim scheduler directives) has been renameddirectives. Theoptionskey can still be used, but a deprecation warning will be printed.