Ingest CATWISE #629

kaseyLee123 · 2025-06-18T17:18:00Z

Short description: Include what type of data being ingested and appropriate references.

Link to relevant issue: Closes #614

For data ingests:

includes script used for ingest
includes modified JSON files
Add new tests
Update the Versions table

kelle · 2025-06-24T14:54:52Z

scripts/ingests/catwise/create_matches_csv.py

+        try:
+            filtered_results = results[(results["ab_flags"] == '00') & (results["cc_flags"] == '0000')][0]
+            writer.writerow([
+                filtered_results[0],


include the db source name in the CSV file. I think it's source["source"]

kelle · 2025-06-24T14:55:27Z

scripts/ingests/catwise/create_matches_csv.py

+            source_num+=1
+            print(source_num)
+            logger.warning("no source match found")


also make a CSV file of the sources with no match.

also, add counters. Count how many sources have multiple matches, 1 match, and no match.

kelle · 2025-06-24T15:00:09Z

scripts/ingests/catwise/create_matches_csv.py

+                filtered_results[1],
+                filtered_results[2],
+                filtered_results[3],
+                filtered_results[4],
+                filtered_results[7],
+                filtered_results[8],
+                filtered_results[9],
+                filtered_results[10],
+                filtered_results[11],
+                filtered_results[12]


I'm sure there's a better way to convert this list into a comma delimited string.

kelle · 2025-06-24T15:03:09Z

scripts/ingests/catwise/create_matches_csv.py

+        results = Irsa.query_region(coordinates=coord, spatial='Cone', catalog='catwise_2020', radius=0.5 * u.arcmin, columns="source_name,PMRA,sigPMRA,PMDec,sigPMDec,ab_flags,cc_flags,w1mpro,w1sigmpro,w2mpro,w2sigmpro,ra,dec")
+
+        try:
+            filtered_results = results[(results["ab_flags"] == '00') & (results["cc_flags"] == '0000')][0]


there is no particular motivation for picking the first one. If there are multiple matches which pass these flags, you should keep them. Might as well save them to the CSV file. I would save all matches that pass these filters. If there are sources with multiple matches, save them to a DIFFERENT CSV file and we can look at them individually.

scripts/ingests/catwise/create_matches_csv.py

kelle · 2025-06-24T20:02:31Z

scripts/ingests/catwise/create_matches_csv.py

+    for source in sources:
+        #create skycoord object because one of the parameters for query region for position
+        coord = SkyCoord(ra = source["ra"], dec = source["dec"], unit = "deg", frame = "icrs")
+
+        # generates a list of objects from the catwise2020 catalogs that are within this radius of a certain position/coordinate
+        results = Irsa.query_region(coordinates=coord, spatial='Cone', catalog='catwise_2020', radius=0.5 * u.arcmin, columns="source_name,PMRA,sigPMRA,PMDec,sigPMDec,ab_flags,cc_flags,w1mpro,w1sigmpro,w2mpro,w2sigmpro,ra,dec")


Loop over results instead of sources.

scripts/ingests/catwise/ingest_CATWISE.py

Co-authored-by: Kelle Cruz <kellecruz@gmail.com>

into ingestCATWISE

kaseyLee123 · 2025-06-26T18:28:24Z

I keep getting this error that I think means the query_region doesn't take in multiple values/whole columns (?)
Traceback (most recent call last): File "/Users/kasey/Documents/GitHub/SIMPLE-db/scripts/ingests/catwise/create_matches_csv.py", line 29, in <module> results = Irsa.query_region(coordinates=coord_vector, spatial='Cone', catalog='catwise_2020', radius=0.5 * u.arcmin, columns="source_name,PMRA,sigPMRA,PMDec,sigPMDec,ab_flags,cc_flags,w1mpro,w1sigmpro,w2mpro,w2sigmpro,ra,dec") File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/astropy/utils/decorators.py", line 618, in wrapper return function(*args, **kwargs) File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/astroquery/ipac/irsa/core.py", line 219, in query_region response = self.query_tap(query=adql) File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/astroquery/ipac/irsa/core.py", line 73, in query_tap return self.tap.search(query, language='ADQL', maxrec=maxrec) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/pyvo/dal/tap.py", line 282, in run_sync **keywords).execute() ~~~~~~~^^ File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/pyvo/dal/tap.py", line 1121, in execute return TAPResults(self.execute_votable(), url=self.queryurl, session=self._session) File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/pyvo/dal/adhoc.py", line 111, in __init__ super().__init__(votable, url=url, session=session) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/pyvo/dal/query.py", line 338, in __init__ raise DALQueryError(self._status[1], self._status[0], url) pyvo.dal.exceptions.DALQueryError: UsageFault: BAD_REQUEST: Invalid or unsupported ADQL query string. See TAP documentation here: https://irsa.ipac.caltech.edu/docs/program_interface/TAP.html

kelle · 2025-06-26T20:17:24Z

scripts/ingests/catwise/create_matches_csv.py

+results = Irsa.query_region(coordinates=coord_vector, spatial='Cone', catalog='catwise_2020', radius=0.5 * u.arcmin, 
+columns="source_name,PMRA,sigPMRA,PMDec,sigPMDec,ab_flags,cc_flags,w1mpro,w1sigmpro,w2mpro,w2sigmpro,ra,dec")


try running this with verbose=True. Documentation is here: https://astroquery.readthedocs.io/en/latest/api/astroquery.ipac.irsa.IrsaClass.html#astroquery.ipac.irsa.IrsaClass.query_region

It gives me a warning saying this:
WARNING: AstropyDeprecationWarning: "verbose" was deprecated in version 4 and will be removed in a future version. [main]

The deprecation warning is ok. Does it provide any additional output when you set it to True?

Traceback (most recent call last): File "/Users/kasey/Documents/GitHub/SIMPLE-db/scripts/ingests/catwise/create_matches_csv.py", line 29, in <module> results = Irsa.query_region(coordinates=coord_vector, spatial='Cone', catalog='catwise_2020', radius=0.5 * u.arcmin, columns="source_name,PMRA,sigPMRA,PMDec,sigPMDec,ab_flags,cc_flags,w1mpro,w1sigmpro,w2mpro,w2sigmpro,ra,dec", verbose=True) File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/astropy/utils/decorators.py", line 618, in wrapper return function(*args, **kwargs) File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/astroquery/ipac/irsa/core.py", line 219, in query_region response = self.query_tap(query=adql) File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/astroquery/ipac/irsa/core.py", line 73, in query_tap return self.tap.search(query, language='ADQL', maxrec=maxrec) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/pyvo/dal/tap.py", line 282, in run_sync **keywords).execute() ~~~~~~~^^ File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/pyvo/dal/tap.py", line 1121, in execute return TAPResults(self.execute_votable(), url=self.queryurl, session=self._session) File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/pyvo/dal/adhoc.py", line 111, in __init__ super().__init__(votable, url=url, session=session) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/pyvo/dal/query.py", line 338, in __init__ raise DALQueryError(self._status[1], self._status[0], url) pyvo.dal.exceptions.DALQueryError: UsageFault: BAD_REQUEST: Invalid or unsupported ADQL query string. See TAP documentation here: https://irsa.ipac.caltech.edu/docs/program_interface/TAP.html

I think this means its still doesn't take in whatever I passed?

And it's not giving us any more useful information. Let me try to get the astroquery folks to help us....

I've chatted with astroquery developers and It seems like this is a real bug. I've opened an issue here: astropy/astroquery#3360

I think we want to use astroquery.vizier instead : https://astroquery.readthedocs.io/en/latest/vizier/vizier.html

https://vizier.cds.unistra.fr/viz-bin/VizieR-3?-source=II/365/catwise

I notice it says "A Table can also be used to specify the coordinates in a region query if it contains the columns _RAJ2000 and _DEJ2000" but do these columns exist in sources? Or are these just the equivalent of RA and DEC

kaseyLee123 · 2025-06-27T18:16:45Z

When I add the db source in the file, since I wouldn't be able to do source["source"] anymore I would have to use find_source_in_db right? I remember doing this for my original script where everything was in one file and there were a few sources that didn't exist. In this case do I ingest it? or collect these sources somewhere else

kelle · 2025-06-27T18:36:01Z

When I add the db source in the file, since I wouldn't be able to do source["source"] anymore I would have to use find_source_in_db right? I remember doing this for my original script where everything was in one file and there were a few sources that didn't exist. In this case do I ingest it? or collect these sources somewhere else

I see what you're saying - if you query IRSA with a vector of coords, you don't have easy access to the SIMPLE source name. I would just ignore this for now.

In this case, you are comparing everything in SIMPLE to CatWISE. There should be no cases where you need to ingest a new source into SIMPLE. BUT, there will be cases where there is no match in CatWISE to the SIMPLE source. And yes, we'd like to keep track of those. But it depends on how we end up doing the search....

kelle · 2025-06-27T19:29:31Z

scripts/ingests/catwise/create_matches_csv.py

+results = Irsa.query_region(coordinates=coord_vector, spatial='Cone', catalog='catwise_2020', radius=0.5 * u.arcmin, 
+columns="source_name,PMRA,sigPMRA,PMDec,sigPMDec,ab_flags,cc_flags,w1mpro,w1sigmpro,w2mpro,w2sigmpro,ra,dec", verbose=True)


change this to using vizier instead of IRSA.

kaseyLee123 · 2025-07-05T20:15:02Z

I just pushed my code that ran. Numbers seem a little off for what we expected (I think like 890 for sources not matched).

kaseyLee123 · 2025-07-06T00:09:19Z

I'm also realizing I probably shouldve made a csv for the skipped sources since that can include the case where the photometry error is not availiable or its a duplicate. I also did not ingest propermotions yet because I wasn't sure what to leave for the propermotions source since there is no column for that but in the commented out one I just did Maro21 for everything since that is the paper from catwise 2020 I think

This reverts commit b74a480.

created script

c2d31a8

kaseyLee123 self-assigned this Jun 18, 2025

kaseyLee123 added 4 commits June 18, 2025 15:40

good basis

6c89c32

need to find references for proper motions somehow...

3ebba40

runtime changes

a0f7044

updates

2f9d85e

kelle reviewed Jun 24, 2025

View reviewed changes

?

ad69255

kelle reviewed Jun 24, 2025

View reviewed changes

scripts/ingests/catwise/create_matches_csv.py Outdated Show resolved Hide resolved

kelle reviewed Jun 24, 2025

View reviewed changes

worked on dr cruz's suggestions

ca7d3be

kelle reviewed Jun 26, 2025

View reviewed changes

scripts/ingests/catwise/ingest_CATWISE.py Outdated Show resolved Hide resolved

kaseyLee123 and others added 3 commits June 26, 2025 14:16

Update scripts/ingests/catwise/ingest_CATWISE.py

f1fcf61

Co-authored-by: Kelle Cruz <kellecruz@gmail.com>

changed tuple --> astropy table

7537a8e

Merge branch 'ingestCATWISE' of https://github.com/kaseyLee123/SIMPLE-db

c3f267e

into ingestCATWISE

kelle reviewed Jun 26, 2025

View reviewed changes

tried using verbose

be06ee8

kaseyLee123 added 2 commits June 27, 2025 15:12

shorten code

d4d4978

attempted to do dr.cruz's method...

0fc15aa

kelle reviewed Jun 27, 2025

View reviewed changes

updated ingest catwise using ultra cool sheet

d61cb24

added skipped table

3f59f26

kaseyLee123 added 9 commits July 7, 2025 22:03

updated

f03d160

more changes

307955e

ran w save_db

28f9318

json files

b74a480

Revert "json files"

f78332a

This reverts commit b74a480.

updated versions table + propermotions

4dcc48a

Update ingest_CATWISE.py

3754b95

json files

4ac46f5

pytest

64c9aa8

kaseyLee123 closed this Jul 21, 2025

		results = Irsa.query_region(coordinates=coord_vector, spatial='Cone', catalog='catwise_2020', radius=0.5 * u.arcmin,
		columns="source_name,PMRA,sigPMRA,PMDec,sigPMDec,ab_flags,cc_flags,w1mpro,w1sigmpro,w2mpro,w2sigmpro,ra,dec")

Ingest CATWISE #629

Ingest CATWISE #629

Uh oh!

Conversation

kaseyLee123 commented Jun 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kaseyLee123 commented Jun 26, 2025

Uh oh!

kelle Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kaseyLee123 commented Jun 27, 2025

Uh oh!

kelle commented Jun 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kaseyLee123 commented Jul 5, 2025

Uh oh!

kaseyLee123 commented Jul 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kelle Jun 26, 2025 •

edited

Loading