Python ValueError in add_userid.py

[abrown41]

When I attempt to run scua with a broader date range than just a few days or a single month, I get the following error.

>> ./bin/scua -S 2024-01-01T00:00 -E 2024-06-31T23:59
Running: sacct -S 2024-01-01T00:00 -E 2024-06-31T23:59  -a -n --format JobIDRaw,JobName%30,User,Account,NNodes,NTasks,ElapsedRaw,State,ConsumedEnergyRaw,MaxRSS,AveRSS,ReqCPUFreq -P --delimiter :: | egrep "^[0-9]+\.[0-9]" | egrep -v "RUNNING|PENDING|REQUEUED"
Running: sacct -S 2024-01-01T00:00 -E 2024-06-31T23:59  -a -n -X --format JobIDRaw,JobName%30,User,Account,NNodes,NCPUS,ElapsedRaw,State,ConsumedEnergyRaw,MaxRSS,AveRSS,ReqCPUFreq -P --delimiter :: | egrep -v "RUNNING|PENDING|REQUEUED"
Running: /home/e813/e813/abrown_e813/usage-analysis/bin/add_userid.py scua_sacct_step.dat scua_sacct_job.dat scua_sacct_users.dat scua_sacct.dat
Traceback (most recent call last):
  File "/home/e813/e813/abrown_e813/usage-analysis/bin/add_userid.py", line 45, in <module>
    df_step[['JobID','SubJobID']] = df_step['JobID'].str.split(pat='.', n=1, expand=True)
  File "/home2/home/e813/e813/abrown_e813/usage-analysis/venv/lib/python3.9/site-packages/pandas/core/frame.py", line 4299, in __setitem__
    self._setitem_array(key, value)
  File "/home2/home/e813/e813/abrown_e813/usage-analysis/venv/lib/python3.9/site-packages/pandas/core/frame.py", line 4341, in _setitem_array
    check_key_length(self.columns, key, value)
  File "/home2/home/e813/e813/abrown_e813/usage-analysis/venv/lib/python3.9/site-packages/pandas/core/indexers/utils.py", line 390, in check_key_length
    raise ValueError("Columns must be same length as key")
ValueError: Columns must be same length as key
No jobs found
rm: cannot remove 'scua_sacct.dat': No such file or directory

[aturner-epcc]

This is because the Slurm query is too big for the accounting database to handle when you specify extended reporting periods so no job information is returned by sacct. We can look at catching this more gracefully and printing a more useful error message.

The only solution at the moment is to split the sacct query up into shorter periods and then combine the data before running it through the Python analysis tool.

[abrown41]

I thought it was probably something like that. Thanks for looking into it!