WIP: Adds ability to precompute Contam database and bypass calculation #660

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open

hover2pi wants to merge 95 commits into main from precompute

.github/workflows/test.yml

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -16,7 +16,7 @@ jobs:
  
          max-parallel: 5

          matrix:

            os: [ubuntu-latest, macos-latest]

            python-version: ['3.9', '3.10', '3.11']

            python-version: ['3.10', '3.11', '3.12']

          fail-fast: false

        steps:

.gitignore

-Original file line number
+Diff line change
@@ Expand Up / @@ -6,6 +6,9 @@ @@
     *.so
     *.x
+    # tmp files for LDC
+    *.ecsv
     # Directories
     __pycache__
     exoctk/__pycache__
@@ Expand All / @@ -15,12 +18,13 @@ exoctk/__pycache__ @@
     *.c
     # Other generated files
-    */_version.py
     */cython_version.py
     htmlcov
     .coverage
     MANIFEST
     .ipynb_checkpoints
+    contam_tool.log
+    soss_contam.h5
     # Sphinx
     docs/api
@@ Expand Down @@

README.rst

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -80,15 +80,15 @@ Next, one can create the ``exoctk`` ``conda`` environment via the appropriate ``
  
      conda env create -f environment-<PYTHON_VERSION>.yml

    where ``<PYTHON_VERSION>`` is the version of python you are using (e.g. ``environment-3.8.yml``)

    where ``<PYTHON_VERSION>`` is the version of python you are using (e.g. ``environment-3.11.yml``)

    Lastly, one can activate the newly-created environment with:

    ::

      conda activate exoctk-<PYTHON_VERSION>

    where again, ``<PYTHON_VERSION>`` is the version of python you are using (e.g. ``exoctk-3.8``)

    where again, ``<PYTHON_VERSION>`` is the version of python you are using (e.g. ``exoctk-3.11``)

    Package Installation

docker/.env

-Original file line number
+Diff line change
@@ -0,0 +1,12 @@
+    # set the HAS_EXO variable depending on whether EXOCTK_DATA is defined.
+    # HAS_EXO=${${EXOCTK_DATA+copy}:-download} (doesn't work?)
+    HAS_EXO=${EXOCTK_DATA+copy}
+    HAS_EXO=${HAS_EXO:-download}
+    # set the HAS_PANDEIA variable depending on whether pandeia_refdata is defined.
+    HAS_PANDEIA=${pandeia_refdata+copy}
+    HAS_PANDEIA=${HAS_PANDEIA:-download}
+    # set the HAS_SYN variable depending on whether PYSYN_CDBS is defined.
+    HAS_SYN=${PYSYN_CDBS+copy}
+    HAS_SYN=${HAS_SYN:-download}

docker/celery.yaml

Empty file.

docker/compose.yaml

-Original file line number
+Diff line change
@@ -0,0 +1,56 @@
+    include:
+       - redis.yaml
+    #   - celery.yaml
+    services:
+      # Data Store Images
+      exoctk:
+        build:
+          context: ./exoctk
+          dockerfile: Dockerfile
+          args:
+            - exoctk_version=v1.2.6.3
+            - python_version=3.11
+            - exoctk_data=$EXOCTK_DATA
+            - has_exo=$HAS_EXO
+        volumes:
+          - exoctk_data:/exoctk/exoctk_data
+        ports:
+          - "5000:5000"
+        restart: unless-stopped
+      pandexo:
+        build:
+          context: ./pandexo
+          dockerfile: Dockerfile
+          args:
+            - pandexo_version=3.0.0
+            - python_version=3.11
+            - pandeia_version=3.2
+            - has_exo=$HAS_EXO
+            - has_pandeia=$HAS_PANDEIA
+            - has_syn=$HAS_SYN
+        volumes:
+          - exoctk_data:/data/exoctk
+          - pandeia_data:/data/pandeia
+          - synphot_data:/data/trds
+        ports:
+          - "1111:1111"
+        restart: unless-stopped
+    volumes:
+      exoctk_data:
+          driver: local
+          driver_opts:
+             type: none
+             o: bind
+             device: $EXOCTK_DATA
+      pandeia_data:
+          driver: local
+          driver_opts:
+             type: none
+             o: bind
+             device: $pandeia_refdata
+      synphot_data:
+          driver: local
+          driver_opts:
+             type: none
+             o: bind
+             device: $PYSYN_CDBS

docker/dockerfiles/DataDockerfileExoctk

-Original file line number
+Diff line change
@@ -0,0 +1,39 @@
+    # Create Container
+    ARG has_exo
+    FROM alpine AS base
+    # Define how to load exoctk data if a copy is already present locally
+    FROM base AS copy
+    WORKDIR /tmp/data
+    ADD . /tmp/data/
+    # Define how to load exoctk data by download
+    FROM base AS download
+    WORKDIR /tmp/data
+    ADD https://data.science.stsci.edu/redirect/JWST/ExoCTK/compressed/modelgrid_ACES_1.tar.gz modelgrid_ACES_1.tar.gz
+    RUN tar zxf modelgrid_ACES_1.tar.gz
+    RUN mv modelgrid.ACES_1 modelgrid
+    ADD https://data.science.stsci.edu/redirect/JWST/ExoCTK/compressed/modelgrid_ACES_2.tar.gz  modelgrid_ACES_2.tar.gz
+    RUN tar zxf modelgrid_ACES_2.tar.gz
+    RUN mv modelgrid.ACES_2/* ./modelgrid/
+    WORKDIR /tmp/data/modelgrid
+    ADD https://data.science.stsci.edu/redirect/JWST/ExoCTK/compressed/modelgrid_ATLAS9.tar.gz modelgrid_ATLAS9.tar.gz
+    RUN tar zxf modelgrid_ATLAS9.tar.gz
+    RUN mv modelgrid.ATLAS9 ATLAS9
+    WORKDIR /tmp/data
+    ADD https://data.science.stsci.edu/redirect/JWST/ExoCTK/compressed/generic.tar.gz generic.tar.gz
+    RUN tar zxf generic.tar.gz
+    ADD https://data.science.stsci.edu/redirect/JWST/ExoCTK/compressed/fortney.tar.gz fortney.tar.gz
+    RUN tar zxf fortney.tar.gz
+    ADD https://data.science.stsci.edu/redirect/JWST/ExoCTK/compressed/groups_integrations.tar.gz groups_integrations.tar.gz
+    RUN tar zxf groups_integrations.tar.gz
+    ADD https://data.science.stsci.edu/redirect/JWST/ExoCTK/compressed/exoctk_contam.tar.gz exoctk_contam.tar.gz
+    RUN tar zxf exoctk_contam.tar.gz
+    # And now combine together and do whichever one the environment tells us to
+    FROM ${has_exo} AS final
+    RUN mv /tmp/data /data
+    WORKDIR /data
+    CMD ["/bin/sh", "echo", "'Built from ${has_exo}'"]
+    # CMD ["tail", "-f", "/dev/null"]

docker/dockerfiles/DataDockerfilePandeia

-Original file line number
+Diff line change
@@ -0,0 +1,22 @@
+    # Create Container
+    ARG has_pandeia
+    FROM alpine AS base
+    # Define how to load exoctk data if a copy is already present locally
+    FROM base AS copy
+    WORKDIR /tmp/data
+    ADD . /tmp/data/
+    # Define how to load pandeia data by download
+    FROM base AS download
+    WORKDIR /tmp/data
+    # ADD https://data.science.stsci.edu/redirect/JWST/ExoCTK/compressed/fortney.tar.gz fortney.tar.gz
+    # RUN tar zxf fortney.tar.gz
+    # And now combine together and do whichever one the environment tells us to
+    FROM ${has_pandeia} AS final
+    RUN mv /tmp/data /data
+    WORKDIR /data
+    CMD ["/bin/sh", "echo", "'Built from ${has_pandeia}'"]
+    # CMD ["tail", "-f", "/dev/null"]

docker/dockerfiles/DataDockerfileSynphot

-Original file line number
+Diff line change
@@ -0,0 +1,25 @@
+    # Create Container
+    ARG has_syn
+    FROM alpine AS base
+    # Define how to load synphot data if a copy is already present locally
+    FROM base AS copy
+    WORKDIR /tmp/data
+    ADD . /tmp/data/
+    # Define how to load synphot data by download
+    FROM base AS download
+    WORKDIR /tmp/data
+    ADD https://archive.stsci.edu/hlsps/reference-atlases/hlsp_reference-atlases_hst_multi_pheonix-models_multi_v3_synphot5.tar pysyn_data.tar
+    RUN tar xf pysyn_data.tar
+    ADD https://archive.stsci.edu/hlsps/reference-atlases/hlsp_reference-atlases_hst_multi_everything_multi_v11_sed.tar pysyn_norm.tar
+    RUN tar xf pysyn_norm.tar
+    RUN cp -r /tmp/data/grp/redcat/trds /tmp/data
+    # And now combine together and do whichever one the environment tells us to
+    FROM ${has_syn} AS final
+    RUN mv /tmp/data /data
+    WORKDIR /data
+    CMD ["/bin/sh", "echo", "'Built from ${has_syn}'"]
+    # CMD ["tail", "-f", "/dev/null"]

docker/exoctk/Dockerfile

-Original file line number
+Diff line change
@@ -0,0 +1,49 @@
+    # syntax=docker/dockerfile:1
+    # Get python version
+    ARG python_version
+    # Create Container
+    FROM python:${python_version}-alpine
+    # Get remaining arguments
+    ARG exoctk_version
+    # set up working directory
+    WORKDIR /exoctk
+    # Set up container environment
+    ENV EXOCTK_DATA="/exoctk/exoctk_data/"
+    ENV MODELGRID_DIR="/exoctk/exoctk_data/modelgrid/"
+    ENV EXOCTK_VERSION=${exoctk_version}
+    # Install necessary base software
+    RUN apk update
+    RUN echo "http://dl-8.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories
+    RUN apk --no-cache --update-cache add gcc gfortran build-base wget freetype-dev libpng-dev openblas-dev git hdf5-dev
+    # Install numpy with custom batman
+    WORKDIR /exoctk
+    RUN pip install --force-reinstall --no-cache-dir numpy==2.2.6
+    RUN pip uninstall batman-package
+    RUN git clone https://github.com/lkreidberg/batman.git
+    WORKDIR /exoctk/batman
+    RUN pip install .
+    WORKDIR /exoctk
+    # Install very specific versions of packages
+    RUN pip install svo-filters==0.4.4
+    # Clone exoctk repository
+    RUN git clone https://github.com/ExoCTK/exoctk.git
+    WORKDIR /exoctk/exoctk
+    RUN git checkout $EXOCTK_VERSION
+    RUN pip install -e .
+    # Set up the port that exoctk will listen on
+    EXPOSE 5000
+    # Run exoctk
+    WORKDIR /exoctk
+    # CMD ["tail", "-f", "/dev/null"]
+    CMD ["python", "/exoctk/exoctk/exoctk/exoctk_app/app_exoctk.py"]

docker/pandexo/Dockerfile

-Original file line number
+Diff line change
@@ -0,0 +1,49 @@
+    # syntax=docker/dockerfile:1
+    # Get python version and create final container
+    ARG python_version
+    FROM python:${python_version}-alpine
+    ENV FORTGRID_DIR=/data/exoctk/fortney
+    ENV pandeia_refdata=/data/pandeia
+    ENV PYSYN_CDBS=/data/trds
+    # Get remaining arguments
+    ARG pandexo_version
+    ARG pandeia_version
+    # set up working directory
+    WORKDIR /pandexo
+    # Set up container environment
+    ENV PANDEXO_VERSION=${pandexo_version}
+    ENV PANDEIA_VERSION=${pandeia_version}
+    # Install necessary base software
+    RUN apk update
+    RUN echo "http://dl-8.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories
+    RUN apk --no-cache --update-cache add gcc gfortran build-base wget freetype-dev libpng-dev openblas-dev hdf5-dev git
+    RUN apk --no-cache --update-cache add linux-headers
+    # Install numpy with custom batman
+    RUN pip install --force-reinstall --no-cache-dir numpy==2.2.6
+    RUN pip uninstall batman-package
+    RUN git clone https://github.com/lkreidberg/batman
+    WORKDIR /pandexo/batman
+    RUN pip install .
+    WORKDIR /pandexo
+    # Install pandexo
+    RUN pip install numpy==1.26.4
+    RUN pip install scipy==1.14.1
+    # ***** NOTE *****
+    # In the future, it would be useful to be able to either directly install this
+    # with pip (i.e. pip install pandexo==VERSION), or do a git checkout of a given
+    # version (i.e. git checkout VERSION)
+    RUN git clone --recursive https://github.com/natashabatalha/PandExo.git
+    WORKDIR /pandexo/PandExo
+    RUN pip install .
+    RUN pip install pandeia.engine==${pandeia_version}
+    WORKDIR /pandexo
+    CMD ["start_pandexo", "--workers=10"]

docker/redis.yaml

-Original file line number
+Diff line change
@@ -0,0 +1,4 @@
+    services:
+      redis:
+        image: "redis:alpine"
+        restart: unless-stopped

docs/index.rst

-Original file line number
+Diff line change
@@ Expand Up / @@ -18,7 +18,6 @@ The subpackages currently included are: @@
     Contamination and Visibility Calculators
     Groups and Intrgrations Calculator
-    Transit Light-Curve Fitting Tools
     Limb-Darkening Calculator
     Phase Constraint Calculator
     Atmospheric Forward Modeling - Currently only available through the `website <https://exoctk.stsci.edu/fortney>`_.
@@ Expand Down Expand Up @@
       source/exoctk.groups_integrations
-    **Lightcurve Fitting Tool**
-    The light-curve fitting tool fits large numbers of spectroscopic light curves simultaneously while sharing model parameters across wavelengths and visits. It includes multiple uncertainty estimation algorithms and a comprehensive library of physical and systematic model components that are fully customizable.
-    .. toctree::
-      :maxdepth: 1
-      source/exoctk.lightcurve_fitting
     **Limb Darkening Calculator**
@@ Expand Down @@

docs/source/exoctk.lightcurve_fitting.rst

This file was deleted.

env/environment-3.11.yml

-Original file line number
+Diff line change
@@ Expand Up / @@ -32,7 +32,7 @@ dependencies: @@
         - bibtexparser>=1.4.0
         - boto3>=1.28.5
         - corner>=2.2.2
-        - docutils>=0.16ssss
+        - docutils>=0.16
         - flask_wtf>=1.1.1
         - h5py>=3.9.0
         - gunicorn>=21.1.0
@@ Expand Down @@

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Adds ability to precompute Contam database and bypass calculation #660

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

WIP: Adds ability to precompute Contam database and bypass calculation #660

Are you sure you want to change the base?

Uh oh!

WIP: Adds ability to precompute Contam database and bypass calculation #660

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!