[Bug] SED fitting task fails with "No valid data to fit"

[manning-ncsa]

🐞 Bug Report

Describe the bug

Some SED fitting tasks are failing with error:

AssertionError('No valid data to fit: check the sign of the masks.')

where the line line triggering the error from the trace below is here.

Trace:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 479, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 779, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/host/transient_tasks.py", line 1664, in local_host_sed_fitting
    LocalHostSEDFitting(transient_name).run_process()
  File "/app/host/transient_tasks.py", line 263, in run_process
    status_message = self._run_process(transient)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/host/transient_tasks.py", line 1203, in _run_process
    status_message = super()._run_process(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/app/host/transient_tasks.py", line 1082, in _run_process
    observations = build_obs(transient, aperture_type)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/host/prospector.py", line 196, in build_obs
    return fix_obs(obs_data)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prospect/utils/obsutils.py", line 80, in fix_obs
    assert obs["ndof"] > 0, "No valid data to fit: check the sign of the masks."
           ^^^^^^^^^^^^^^^
AssertionError: No valid data to fit: check the sign of the masks.

[djones1040]

just thinking about this -- any simple way to figure out which transient this failed on?

[manning-ncsa]

I stupidly neglected to note which transient failed and now the logs are gone. I do not know how to determine which one failed.

[djones1040]

ahh ok, I think maybe we should close this until it pops up again as it'll be hard to track this down as is. Most likely it was just a transient with no useful data anyways, and just a non-optimal error message for us to fix

[manning-ncsa]

Even if this is low priority and if we cannot identify a real-world trigger for the code path, we should be able to handle the error. I'll leave it open as a "good first issue" bug.

[manning-ncsa]

@djones1040 Coincidentally, I just saw another of these failure in the logs and have the offending transient name: 2017ejb. The "Local Host SED Fitting" task failed with traceback:

  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 479, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 779, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/host/transient_tasks.py", line 1710, in local_host_sed_fitting
    LocalHostSEDFitting(transient_name).run_process()
  File "/app/host/transient_tasks.py", line 265, in run_process
    status_message = self._run_process(transient)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/host/transient_tasks.py", line 1232, in _run_process
    status_message = super()._run_process(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/app/host/transient_tasks.py", line 1111, in _run_process
    observations = build_obs(transient, aperture_type)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/host/prospector.py", line 197, in build_obs
    return fix_obs(obs_data)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prospect/utils/obsutils.py", line 80, in fix_obs
    assert obs["ndof"] > 0, "No valid data to fit: check the sign of the masks."
           ^^^^^^^^^^^^^^^
AssertionError: No valid data to fit: check the sign of the masks.