diff --git a/release_notes.md b/release_notes.md new file mode 100644 index 00000000..95ab0ee9 --- /dev/null +++ b/release_notes.md @@ -0,0 +1,652 @@ +# v0.10.0 + +This release brings a new functionality to astrohack, the ability to +analyze beam cuts. Beam cuts are extremely useful as they are in a +sense a type of very poor resolution holography while but also are +very cheap to obtain in terms of time. + +## General: + +- Documentation has been slightly reorganized. + - Tutorials are now found in a dedicated section of the read the + docs pages. + - Remaining pages are now all found in one directory inside the docs + root. + +- A script for cloudflare file uploading has been added to + astrohack/etc/test_data_upload, it relies on the user setting + cloudflare account info and tokens as environment variables + +- Added a deep checking tool to verify if 2 mdses are equal, it works + with both regular dict based mdses and newer data tree based mdses. + +- Python files containing IO, and in memory data storage functions and + classes that are user facing have been moved to a sub directory + called IO. + +- Fixed numpy version to <= 2.2 as Numba does not support numpy 2.3 or + 2.4. + +- Tutorial notebooks now close their clients at the end of execution. + + +## Beam cuts: + +- `beamcut` is a new astohack function to process .holog.zarr files + that contain beam cut measurements. + +- The newly introduced `AstrohackBeamcutFile` class uses Xarray Data + Trees to store that in memory and uses Xarray interfaces to save it + to disk. + +- `open_beamcut` is a new astrohack function to process .beamcut.zarr + files created by `beamcut`. + +- A beam cut calibration pipeline can be found at + astrohack/etc/beamcuts. + +- A beam cut tutorial has been included and can be found on the new + tutorial section of astrohack.readthedocs.io. + + +## Holography: + +- `extract_holog` now stores scan information in the output + .holog.zarr files. This change was made to simplify processing of + beam cuts. + +# v0.9.4 + +This release contains only fixes to the readthedocs pages. + +# v0.9.3 + +This release contains no changes to the code itself, but it contains +needed changes to address the change in the URL for the Astrohack +repo. + +# v0.9.0 + +This releases brings quite a few major changes, with the biggest +change being the tentative full support for the ngVLA prototype +holography commissioning tests. A summary of the changes can be found +below. + +## General: + +- Telescope objects: + + - The telescope class has been split up. + + - Cassegrain telescopes are now supported by the `RingedCassegrain` + class. + + - The ngvla prototype is supported by the `NgvlaPrototype` class. + + - Telescope objects may now be initialized with the + global function `get_proper_telescope`. + +- Changed import scopes: + + - Now only user intended functions are available at the package + level. + +- Increased test coverage: + + - Now 2D gridding and 1D interpolation functions are tested during + continuous integration. + + - Opening routines for all `AstrohackDataFiles` are now tested during + continuous integration. + +- Bug fixes: + + - The creation of file names based on input files now uses + str.removesuffix as .rstrip has behaviors that were leading to + weirdness in output file names. + + - `check_if_file_can_be_opened` now checks if path exists before doing + anything. + +## Holography: + +- `panel`: + + - part of the work in processing the apertures done by the + AntennaSurface class has now be ported to the new telescope + classes as some of the processing differs between Cassegrain and + the ngVLA due to their different optics. + + - Improved `PolygonPanel` class has been brought to production, this + class is used to represent panels that are represented by + polygons, such as the ngVLA panels. + + - Phase to deviation correction for the ngVLA is computed using a + Quintic Pseudo Spline (QPS) representation of the surface to + compute the cosine of the surface normal to the direction of + boresight. + + - Aperture plot orientation and panel sample coordinates reference + frames have been reconciled yeilding cohesive coordinate frames + between panel points and the points in aperture plots. + +- `extract_holog`: + + - Found a bug in the interpolation of pointings onto visibility + times that allowed for NaNs in the interpolated pointings. + + - Added a new method to interpolate pointings onto visibility + times using a gaussian convolution, this is also the fallback + behavior when NaNs are detected in the linear interpolation of + the pointings. + +- `extract_pointing`: + + - `extract_pointing` now raises a warning when the pointing tables + have irregular sampling times. + + +# v0.8.0 + +This release includes the following changes: + +## Holography: + +- There is now a observation summary attached to each holography + xarray dataset inside a mds (holog.zarr, image.zarr and panel.zarr). + +- This summary can be printed and exported to an ASCII file through a + method called .observation_summary of the respective mdses. + +- This allows for the decoupling of automatically defined beam + gridding characteristics between DDIs. + +- This change breaks backward compatibility with files from versions + before this one. + +- Checking if a file can be opened is now performed at the start of + processing and when opening files. + +## General: + +- Added a Z scale control to the `image_compare_tool`. + +# v0.7.1 + +This release addresses a bug that disabled the 'AIPS' colormap. + +# v0.7.0 + +This release includes some features for astrohack. + +## General: +- Astrohack is now fully Black compliant. + +- Astrohack now supports python versions 3.11, 3.12 and 3.13. + +- Astrohack now uses the toolviper set of configurations for testing + and code coverage. + +- Astrohack now uses Numpy V>2. + +- Astrohack is now compatible with the latest releases of Dask. + +## Holography + +- `holog`: + - `holog` now fits Zernike polynomials to the apertures before + converting them to Stokes. + + - The Zernike polynomial fitting can be controlled by chosen the + highest order of polynomials to be fitted using parameter + `zernike_n_order`. + + - The highest order of Zernike polynomials that can be fitted is N = + 10 (66 polynomial coefficients in total). + + - The Zernike polynomial fit can also be used for phase fitting, + this new feature can be chosen by using the parameter + `phase_fit_engine`. + + - Zernike polynomial phase fitting should be used with orders 2-4 as + higher orders may start fitting structures in the aperture plane + that are caused by the panels. + + - It is possible to retrieve the values of the fitted Zernike + polynomial coefficients by using the `image_mds` method + `export_zernike_fit_results`. + + - It is possible to plot the Zernike polynomial model along with the + fitting residuals by using the `image_mds` method `plot_zernike_model`. + + - Both phase fitting engines are now almost fully covered by tests. + + - Perturbation phase fitting (classic AIPS like algorithm) can now + handle non-square pixels or images. + +# v0.6.2 + +## General: + +- Fixed issues with problematic scape sequences. + +- Moved user documentation pages from the wiki to the readthedocs. + +- Misc improvements to readthedocs page. + +## FITS Comparison tool: + +- Fixed bug caused by a typo. + +- Added new types of plot: Reference image plot and unresampled data + plot. + +- Added statistics to the headers of all plots. + +- Created a function to extract RMS values from an FITSImage obj: + `rms_table_from_zarr_datatree`. + +## Holography: + +- Panel: + - Fixed a bug that prevented old files with no polarization state + information from being opened. + +# v0.6.1 + +## General: + +- Dropped support for Python 3.9 as the latest release of xarray drops + support for this python version + +- Added support for Python 3.12, this python version was not supported + by python-casacore for a long time but that is no longer the case. + +## Holography: + +- Panel: + + - Introduced a new parameter called `exclude_shadows`. + + - This parameter activates using a mask which also excludes the + shadows caused by the secondary support structures. + +## Image Comparison Tool: + +- This release introduces a simple FITS comparison tool to gauge the + differences between apertures taken at different epochs and softwares + such as AIPS. + +# v0.6.0 + +This release includes the release of the Cassegrain ray Tracing tool. +More information about this tool can be seen at our +[astrohack.readthedocs.io](readthedocs page). + +# v0.5.11 + +## Bug fixes: + +- `locit`: + - `locit` was crashing when a delay fit failed, there is now error + trapping around the fitting routines giving out a warning when a + fit fails. + +# v0.5.10 + +This is a patch release, that brings a bug fix to `holog`. + +## Bug fixes: + +- `holog`: + + - After the phase fitting process the phase corrections were + subtracted from the phase image without wrapping the phase to + the - $\pi$ to $\pi$ interval, this has now been fixed by the + addition of phase wrapping at the end of the phase fitting + process. + + - A test has been added to panel to detect images from `holog` that + have been produced with astrohack versions previous to this one to + trigger a phase wrapping of the input phase image before computing + deviations. + + +# v0.5.9 + +This release brings some improvements to `locit`. + +## Antenna position corrections: +- `locit`: + - It is now possible to use 'RL', 'LR', 'XY' or 'YX' to describe + the polarization to be used, 'both' is still accepted as well. + + - `locit` no longer crashes in the event of missing data, it will + produce a warning instead. + + - If all data for an antenna DDI combination is missing during + processing due to filtering (e.g. low elevation data) `locit` will + now produce a warning. + +# v0.5.8 + +This release contains a few changes to locit, plus a new method for +installing astrohack in NRAO machines. + +## Antenna position corrections: +- `locit`: + + - Delay plots with the fitted delay model now display the delay + model residuals. + + - Improved selection of X and Y limits on delay, and sky coverage + plots. + + - Table that contains position corrections now present station + numbers and fit RMS in degrees. + + - Added a method to the `position_mds` called `export_to_parminator` to + export `locit` results to a parminator file. + + - Changed formatting of values in the table that contains position + corrections. + +## Installation under a VENV: +- Installation script at etc/installation/. + +- CASA fringe fit and phase gain script for pre `locit` reduction. + +- `locit` execution script. + +- post locit parminator export script + +- installation instructions on the wiki + +- Execution instructions on the wiki + +# v0.5.7 + +This release contains a miscellany of minor changes and a few bug fixes. + +## Bug fixes: + +- `extract_holog`: + + - Corrected the time interpolation of pointing for the case in which + the pointing table is sampled less frequently than the + visibilities (pre 2021 VLA). + + - Changed test on needing to fix the pointing table to when there + are no mapping antennas rather than using a date. + +- `locit`: + - Fixed the case where field_ids were taken from the negative phases + but assumed to be the same as the positive phases which caused an + access error due to array length. + + - Reference antenna is now always marked on the output table with + antenna position corrections. + + - The antenna position corrections table is now printed when + `position_mds.export_locit_fit_results` is called. + +- Miscellanious: + + - Fixed the formatting of the API documentation in readthedocs for a + few tasks where line breaks were causing a formatting breakdown. + +## Holography: + +- `extract_holog`: + + - Changed the default of `baseline_average_nearest` parameter from + 'all' to 1. + + +## General: + +- Updated github workflows to use V4 artefacts rather than V3 + artefacts. + +# v0.5.6 + +This release contains a series of minor improvements. + +## Holography: + +- `panel`: + - New amplitude cutoff method based on noise maximum and a threshold + for excluding data. + + - RMS in screws file is now always quoted in mm as well as the unit + chosen. + + - Antenna gain is now computed using the expected FWHM of the + primary beam. + +- `extract_holog`: + - It is now possible to exclude antennas that have bad data. + +## General: + +- Fixed a typo in notebooks referencing `apply_mask`, a deprecated + parameter of holog, that caused the notebooks to not complete + execution. + +# v0.5.5 + +This release contains the following changes. + +## Hologaphy: + +- `panel`: + - It is now possible to specify a specific amplitude cut off for any + antenna and ddi combination. + + - Aperture amplitude plots now display simple amplitude and noise + statistics. + +- `holog`: + - Apertures are no longer clipped to allow for the analysis of the + noise outside the dish image. + + +# v0.5.4 + +## Holography: + +- `panel`: + - Added antenna surface RMS estimates before and after screw + adjustments to Gain table and screw adjustment files. + +# v0.5.3 + +## Bug fixes: + +- `holog`: + - The U and V axes of the aperture FITS files produced by + `image_mds.export_to_fits` had the wrong values, this is now fixed. + +- `extract_holog`: + - `extract_holog` now works with `int` as inputs for `ddi`. + + +## Holography: + +- `panel`: + - Improved resolution of the display of frequency in the file + containing the screw adjustments + + - Improved header definition of the exported ASCII screw adjustment + file + + - Panel fitting and correction have been optimized with execution + times being 10-50% faster depending on model. + + - New method for panel_mds file that export estimated antenna gains + at multiple frequencies: `export_gains_table`. + + - New model for panel fitting, flexible, based on the model of the + same name found in AIPS. + + +- `holog`: + - gaussian convolution is now the default gridding method in `holog`. + + - New method for `image_mds` file that export phase fitting results: + `export_phase_fit_results`. + + + +- `extract_holog`: + + - `holog_obs_dict` json files are no longer created as hidden files + (. file name). + + - `inspect_holog_obs_dict` now also opens the `holog_obs_dict` inside + of `holog_mds` files. + + +## General: + +- Decreasead overall verbosity of functions. + +- Replaced dependency of graphviper with toolviper + +# v0.5.1 + +## Holography: + +-`holog`: + - New method of gridding the visibilities onto a beam. This method + uses the convolution of the visibilities with a gaussian kernel + with a size similar to the primary beam. This is a fast release to + attend to VLA validation needs, no documentation on the gaussian + convolution gridding is provided at this point. + +# v0.5.0 + +## Bug fixes: +- `holog`: + - Calling holog with a `grid_size` and `cell_size` no longer overwrite + these values in the input .holog.zarr file + +## General + - Dropped support for python 3.8, prompted by drop of support for + 3.8 in `graphviper`. + + - Pixel coordinates now reflect the middle of the pixel rather than + its left edge. + + - U and V axes are now always in meters in data files rather than in + kilo lambda. + +## Holography: + +- `combine`: + + - Fixed weighted combining of DDIs + +- `holog`: + - Refactoring for better separation between beam gridding and + aperture generation. + + - Padding with zeros now produces images that have a size of + `2^(ceiling(log2((original_size*padding_factor)))`. + + - `image_mds.plot_apertures` can now choose polarization state. + + - `image_mds.plot_beams` plots are now split by polarization state. + + - Beam gridding using a gaussian kernel. + +- ALMA Near field support (Alpha): + - ALMA NF ASDM filler to the .holog.zarr format at etc/alma-nf-filler/. + + - Selection of ALMA OSF pad in holog to determine distance and + defocusing. + + - Beam apodization. + + - Non-fresnel terms in aperture generation (Not fully correct). + + - Phase near field corrections (Not fully correct). + + - GILDAS\CLIC like phase fitting including astigmatism. + + +- `panel`: + + - Screw adjustments file now contains the method used to fit the + panel and if it was a fallback. + + - Polarization state selection with I being the default. + + + + +# v0.4.3 + +## Bug Fixes: + +- `extract_holog`: + - Fixed issue with dropping antenna messing with antenna indexing. + + - Updated default calculation for grid and cell size. + + - Filter out SYSTEM_CONFIGURATION scans. + +## Holography: + +- `extract_holog`: + - Added time smoothing of the visibilities and pointing data. + +# v0.4.2 + +## Bug fixes: + +- `extract_holog`: + + - Fixed missing import of extract_pointing in memory test. + +- Miscellanious: + - Fix bug where client didn't load worker logging plugin when logger + parameters were not specifically passed to `local_client(...)` + function. + +## Holography: + +- `panel`: + - `panel` can now select a polarization state. + - Previously `panel` picked the 0th element in the polarization axis + of the aperture for doing the work, but this was not robust if + the data is not on the stokes order I, Q, U, V. + - `panel` is now allowed to choose which polarization state to pick + from the data, for example running on RR or XX when stokes + parameters are not available. + +- `extract_point`: + - `extract_point(...)` now includes an option to drop a given + antenna(s). + - The changes have been propagated into `extract_holog(...)` as + well. The new input parameter to `extract_point(...)` is `exclude + = []` and takes a single or list of antennas. + + - A warning has been added to `extract_point(...)` to notify the + user when the antenna data length ensemble has a fractional error + of more than 1%, ie. this should be essentially zero. + + - Dropped antenna info in recorded in the pointing dataset object. + + +## General: + +- Generalize creation of default file name. + - When output file names were not given the code created a default + name based on the input name. This was repeated code among the + different modules and has now been generalized to a single + agnostic function. + +- Parameter checking no longer fails in Github Actions in MacOS. + +- Parameter checking validation added to plotting API.