uqstat broken: missing qtime field

[aidanheerdegen]

Running uqstat from /g/data/hh5/public/apps/nci_scripts/uqstat gives an error:

$ uqstat --project tm70 
Traceback (most recent call last):
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'qtime'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/g/data/hh5/public/apps/nci_scripts/uqstat", line 189, in <module>
    main()
  File "/g/data/hh5/public/apps/nci_scripts/uqstat", line 132, in main
    df = nqstat_df(args.project)
  File "/g/data/hh5/public/apps/nci_scripts/uqstat", line 96, in nqstat_df
    df['qtime'] = pandas.to_datetime(df['qtime'], unit='s')
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3/lib/python3.9/site-packages/pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err
KeyError: 'qtime'

It seems the qtime field is no longer always returned by the PBS server, e.g.

(Pdb) p list(df.columns.values)
['Job_Name', 'Job_Owner', 'resources_used.cput', 'resources_used.jobfs', 'resources_used.mem', 'resources_used.ncpus', 'resources_used.walltime', 'job_state', 'queue', 'Resource_List.jobfs', 'Resource_List.mem', 'Resource_List.ncpus', 'Resource_List.storage', 'Resource_List.walltime', 'project', 'stime']

Supposition: qtime is returned for a queued job, stime for a running job.