allow manual flagging of individual scans, feeds, etc.

[aakepley]

I would like to be able to manually flag fdnums and scans for a particular observing session in the degas pipeline. For truly general implementation, you could also implement spw and polarization flagging as well, but degas only has one spw and argus only has one polarization, so this functionality isn't strictly necessary at present.

I'm creating the ticket here since I believe that the underlying gbtpipe code needs to be changed (although some changes will likely be necessary in the degas pipeline).

The above is implementation agnostic. As I see it, there are two possible appropriate ways to get the info into the DEGAS pipeline: we can use the observing logs creating an extra column for manual flags or we can use a simple text file that we feed into the data reduction. My slight preference to use the observing logs since that's were all the information is aggregated anyway, but I could be persuaded to go the text file route. I'm not sure what's the best route for the best reproducibility.

[aakepley]

Decision based on discussion with Sam and Erik is to have this noted in the observing log and then the pipeline processes this. The column will be named "flagFeeds" and it will have a comma-separated list of feeds to flag. The numbering will be "software-feed", i.e., 0-15.

[aakepley]

Column added to DEGAS observing logs as column "O" (to the right of the Bad Scans column).

[aakepley]

After the DEGAS meeting today, Andy and Adam both suggested that we include the capability to flag feeds not only per map, but also per scan within the map. The idea is that you're already touching that bit of code, so might as well make it fully generalizable now in case we need it in the future. How easy would this be to do?

[low-sky]

I think just entering the Bad Feeds for that particular scan in the Logs should do per-scan bad feed flagging as things are currently set up?

[aakepley]

But we're processing set of scans from the log. Each map is one row in the observing log, specified with a start and end scan.

[low-sky]

Ah, right you are. Okay, that can be another column in the Logs... I'll chew on whether there's a clever way to do it, but by default, there would be something like a BadScan_BadFeed column populated by pairs like (74, 3), (75, 3) etc. Or...?

[aakepley]

How about FlagData column that consolidates the current Bad Scans and Bad Feeds column and gives list of (feed, scan) tuples. I've put feed first since we are more likely to have bad feeds than bad scans.

If the column has no data (empty), then flag nothing. If it has data in it flag based on the list of tuples.

Ranges would be indicated for each value in the tuple by 37~39 (i.e., CASA style).

Examples:

• Flag feed 3, scans 37-43: (3,37~43)
• Flag all feeds, scan 45: (0~15, 45)
• Flag multiple feeds, multiple scans: (3~6,37~43), (7, 45), (8,38~43)

[low-sky]

That shouldn't be too bad of a syntax to write a parser for.

[aakepley]

The range thing is a nice to have, but I think it will make it easier to deal cases where one scan is bad for all feeds (and vice versa).

If we go this route, I do want to modify the timeseries plots to show the scans so we can just read off the numbers. I think this is something I can do since I"m probably going to be staring at those plots for a while.

[aakepley]

BadFeeds and BadScans is working pretty well. Do we need to make the functionality more complicated? I.e., do we need to flag a scan and feed or can treat them independently (i.e., scan is bad for everyfeed and feed is bad for every scan). I'd say that given the granularity of the data, we typically have scans that are bad for all feeds and feed that are bad for all scans.