Docs, tests, and CI: API reference, unit test coverage, and documentation updates

CONTRIBUTING.md

@@ -11,9 +11,9 @@ If you encounter a bug, have a question about usage, or have a feature request:
 
 ## Contributing code or documentation
 
-1. **Fork the repository** and create a branch from the default branch.
-2. **Make your changes** and add or update tests if applicable. Run the test suite with `pytest tests` (use `pytest tests -m "not integration"` for offline runs).
-3. **Open a pull request** against the main repository. Describe your changes clearly and reference any related issues.
+1. **Fork the repository** and create a branch (often from `dev` for ongoing development; the default branch may be `main`).
+2. **Make your changes** and add or update tests if applicable. Run the test suite with `pytest tests` (use `pytest tests -m "not integration"` for offline runs without fixture downloads).
+3. **Open a pull request** against the main repository (or against `dev` when that branch is used for integration). Describe your changes clearly and reference any related issues.
 4. **Code style:** Follow the existing style in the codebase. The project uses standard Python packaging and type hints where appropriate.
 
 Instrument-specific changes (e.g. new instruments or header/sorting logic) should include a brief justification and, if possible, a note in the PR description on how the change was tested.

README.md

@@ -76,8 +76,12 @@ From the project root (with test deps installed: `pip install -e ".[test]"`):
 pytest tests
 ```
 
-- **Offline (no network):**  
-  `pytest tests -m "not integration"`
+- **Unit tests only (no network):**  
+  `pytest tests -m "not integration"`  
+  This skips the five integration tests that download fixtures from external storage. CI runs this by default.
+- **All tests (including integration):**  
+  `pytest tests`  
+  Requires network access for fixture downloads.
 - **Verbose:**  
   `pytest tests -v`
 
@@ -227,7 +231,9 @@ POTPyRI can stack data from multiple nights, although it will not use unique cal
 
 ## Documentation
 
-Full API documentation (generated from docstrings) is available at [https://CIERA-Transients.github.io/POTPyRI/](https://CIERA-Transients.github.io/POTPyRI/). To build the docs locally, install with `pip install -e ".[docs]"` and run `cd docs && make html` (or use the Sphinx `make.bat` on Windows).
+- **Online:** Full API documentation (generated from docstrings) is at [https://ciera-transients.github.io/POTPyRI/](https://ciera-transients.github.io/POTPyRI/).
+- **Local build:** Install with `pip install -e ".[docs]"` and run `cd docs && make html` (on Windows use `make.bat html`). Output is in `docs/build/html/`.
+- **Deployment:** Docs are built and deployed to GitHub Pages on pushes to `main`; see `docs/DEPLOY.md` for setup details.
 
 ## Contributing and support
 
@@ -237,7 +243,7 @@ We welcome contributions and feedback. See [CONTRIBUTING.md](CONTRIBUTING.md) fo
 - How to **contribute** code or documentation (pull requests)
 - Where to **seek support** (issues or developer contact)
 
-If you encounter an error in the pipeline, have a special data setup that does not run through the pipeline, or wish to add an instrument, please open an issue on [GitHub](https://github.com/CIERA-Transients/POTPyRI/issues) or contact the developers at `ckilpatrick@northwestern.edu`.
+Development often uses the `dev` branch; pull requests may target `main` or `dev` as noted in the repository. If you encounter an error in the pipeline, have a special data setup that does not run through the pipeline, or wish to add an instrument, please open an issue on [GitHub](https://github.com/CIERA-Transients/POTPyRI/issues) or contact the developers at `ckilpatrick@northwestern.edu`.
 
 ## License
 

docs/source/index.rst

@@ -7,6 +7,8 @@ It provides instrument-specific reduction workflows for bias, dark, and flat
 calibration; image alignment and stacking; WCS solving with astrometry.net;
 photometry with photutils; and flux calibration with astroquery.
 
+The package includes nine instrument modules (BINOSPEC, DEIMOS, F2, FOURSTAR, GMOS, IMACS, LRIS, MMIRS, MOSFIRE), primitives for calibration, image processing, file sorting, WCS solving, photometry, and absolute photometry, plus CLI scripts for the main pipeline and archive data download. For installation, usage, and contributing, see the repository `README <https://github.com/CIERA-Transients/POTPyRI>`_ and `CONTRIBUTING.md <https://github.com/CIERA-Transients/POTPyRI/blob/main/CONTRIBUTING.md>`_.
+
 Contents
 ========
 

docs/source/installation.rst

@@ -1,7 +1,7 @@
 Installation
 ============
 
-POTPyRI requires **Python 3.11 or later**.
+POTPyRI requires **Python 3.11 or later**. For full platform notes (conda, Apple Silicon, etc.) and the ``environment.yml``-based workflow, see the main **README** in the repository root.
 
 From PyPI
 ---------
@@ -43,7 +43,15 @@ The ``-e`` (editable) flag uses the current directory as the live source, so cha
 Non-Python dependencies
 ------------------------
 
-The pipeline uses **astrometry.net** and **Source Extractor** (SExtractor). Install them via your system package manager or conda (see the main README). Index files for astrometry.net can be installed with the ``download_anet_index`` script after astrometry.net is on your path.
+The pipeline uses **astrometry.net** and **Source Extractor** (SExtractor). Install them via your system package manager or conda (see the main README for a platform table). Index files for astrometry.net (~59 GB) can be installed with the ``download_anet_index`` script after astrometry.net is on your path.
+
+Testing
+-------
+
+From the project root with ``.[test]`` installed:
+
+- **Unit tests only (no network):** ``pytest tests -m "not integration"`` — skips the five integration tests that download fixtures. This is what CI runs.
+- **All tests:** ``pytest tests`` — requires network access for fixture downloads.
 
 Building the documentation
 ---------------------------
@@ -54,4 +62,4 @@ With ``.[docs]`` installed, from the project root:
 
     cd docs && make html
 
-Output is in ``docs/build/html/``. On Windows use ``make.bat html``.
+Output is in ``docs/build/html/``. On Windows use ``make.bat html``. The live API docs are published at https://ciera-transients.github.io/POTPyRI/; see ``docs/DEPLOY.md`` for deployment setup.

potpyri/instruments/instrument.py

@@ -25,6 +25,49 @@
 from astropy.stats import SigmaClip
 from astropy.time import Time
 
+# WCS-related keywords to remove from calibration headers so saved files never
+# trigger InvalidTransformError (e.g. DEC=-100, ill-conditioned CD matrix).
+_CAL_HEADER_WCS_KEYS = (
+    'CTYPE1', 'CTYPE2', 'CRVAL1', 'CRVAL2', 'CRPIX1', 'CRPIX2',
+    'CD1_1', 'CD1_2', 'CD2_1', 'CD2_2', 'CDELT1', 'CDELT2', 'CROTA2',
+    'RADECSYS', 'RADESYS', 'EQUINOX', 'RA', 'DEC', 'LONPOLE', 'LATPOLE',
+    'CUNIT1', 'CUNIT2', 'MJD-OBS',
+    'LTM1_1', 'LTM1_2', 'LTM2_1', 'LTM2_2', 'LTV1', 'LTV2',
+    'DTV1', 'DTV2', 'DTM1_1', 'DTM2_2',
+)
+
+
+def _sanitize_calibration_header(header):
+    """Remove WCS and coordinate keywords from a header so it can be written safely.
+
+    Calibration frames (bias, dark, flat, sky) inherit headers from input frames
+    that may contain invalid or ill-conditioned WCS (e.g. DEC=-100). Removing
+    these keywords ensures saved calibration files do not cause
+    InvalidTransformError when later loaded by code that parses WCS.
+    Modifies header in place.
+    """
+    for key in _CAL_HEADER_WCS_KEYS:
+        if key in header:
+            del header[key]
+    # Remove PV* distortion keywords (can be ill-conditioned)
+    to_del = [k for k in header.keys() if k.startswith('PV')]
+    for k in to_del:
+        del header[k]
+
+
+def _read_calibration_ccd(path, unit, hdu_index=0):
+    """Read a calibration FITS into CCDData without parsing WCS.
+
+    Master bias/dark/flat frames often have WCS keywords that are ill-conditioned
+    or copied from templates. Parsing them can raise InvalidTransformError and
+    log 'Ill-conditioned coordinate transformation parameter'. Calibration
+    frames do not need WCS, so we load data and header explicitly with wcs=None.
+    """
+    with fits.open(path) as hdu_list:
+        hdu = hdu_list[hdu_index]
+        return CCDData(hdu.data, meta=hdu.header.copy(), unit=unit, wcs=None)
+
+
 class Instrument(object):
     """Base class for all POTPyRI instruments.
 
@@ -384,10 +427,9 @@ def load_bias(self, paths, amp, binn):
         """
         bias = self.get_mbias_name(paths, amp, binn)
         if os.path.exists(bias):
-            mbias = CCDData.read(bias)
+            mbias = _read_calibration_ccd(bias, u.electron, hdu_index=0)
         elif os.path.exists(bias+'.fz'):
-            hdu = fits.open(bias+'.fz')
-            mbias = CCDData(hdu[1].data, header=hdu[1].header, unit=u.electron)
+            mbias = _read_calibration_ccd(bias+'.fz', u.electron, hdu_index=1)
         else:
             raise Exception(f'Could not find bias: {bias}')
         return(mbias)
@@ -416,10 +458,9 @@ def load_dark(self, paths, amp, binn):
         """
         dark = self.get_mdark_name(paths, amp, binn)
         if os.path.exists(dark):
-            mdark = CCDData.read(dark)
+            mdark = _read_calibration_ccd(dark, u.electron, hdu_index=0)
         elif os.path.exists(dark+'.fz'):
-            hdu = fits.open(dark+'.fz')
-            mdark = CCDData(hdu[1].data, header=hdu[1].header, unit=u.electron)
+            mdark = _read_calibration_ccd(dark+'.fz', u.electron, hdu_index=1)
         else:
             raise Exception(f'Could not find dark: {dark}')
         return(mdark)
@@ -450,10 +491,9 @@ def load_flat(self, paths, fil, amp, binn):
         """
         flat = self.get_mflat_name(paths, fil, amp, binn)
         if os.path.exists(flat):
-            mflat = CCDData.read(flat)
+            mflat = _read_calibration_ccd(flat, u.dimensionless_unscaled, hdu_index=0)
         elif os.path.exists(flat+'.fz'):
-            hdu = fits.open(flat+'.fz')
-            mflat = CCDData(hdu[1].data, header=hdu[1].header, unit=u.electron)
+            mflat = _read_calibration_ccd(flat+'.fz', u.dimensionless_unscaled, hdu_index=1)
         else:
             raise Exception(f'Could not find flat: {flat}')
         return(mflat)
@@ -484,10 +524,9 @@ def load_sky(self, paths, fil, amp, binn):
         """
         sky = self.get_msky_name(paths, fil, amp, binn)
         if os.path.exists(sky):
-            msky = CCDData.read(sky)
+            msky = _read_calibration_ccd(sky, u.dimensionless_unscaled, hdu_index=0)
         elif os.path.exists(sky+'.fz'):
-            hdu = fits.open(sky+'.fz')
-            msky = CCDData(hdu[1].data, header=hdu[1].header, unit=u.electron)
+            msky = _read_calibration_ccd(sky+'.fz', u.dimensionless_unscaled, hdu_index=1)
         else:
             raise Exception(f'Could not find sky frame: {sky}')
         return(msky)
@@ -615,8 +654,9 @@ def create_bias(self, bias_list, amp, binn, paths,
             'Version of telescope parameter file used.')
 
         bias_filename = self.get_mbias_name(paths, amp, binn)
+        _sanitize_calibration_header(mbias.header)
         mbias.write(bias_filename, overwrite=True, output_verify='silentfix')
-        log.info(f'Master bias written to {bias_filename}')
+        if log: log.info(f'Master bias written to {bias_filename}')
 
         return
 
@@ -686,6 +726,7 @@ def create_dark(self, dark_list, amp, binn, paths, mbias=None, log=None):
 
         darkname = self.get_mdark_name(paths, amp, binn)
         if log: log.info(f'Writing master dark to {darkname}')
+        _sanitize_calibration_header(mdark.header)
         mdark.write(darkname, overwrite=True, output_verify='silentfix')
 
         return
@@ -798,8 +839,9 @@ def create_flat(self, flat_list, fil, amp, binn, paths, mbias=None,
             'Version of telescope parameter file used.')
 
         flat_filename = self.get_mflat_name(paths, fil, amp, binn)
+        _sanitize_calibration_header(mflat.header)
         mflat.write(flat_filename, overwrite=True, output_verify='silentfix')
-        log.info(f'Master flat written to {flat_filename}')
+        if log: log.info(f'Master flat written to {flat_filename}')
 
         return
 
@@ -881,8 +923,9 @@ def create_sky(self, sky_list, fil, amp, binn, paths, log=None, **kwargs):
             'Version of telescope parameter file used.')
 
         sky_filename = self.get_msky_name(paths, fil, amp, binn)
+        _sanitize_calibration_header(msky.header)
         msky.write(sky_filename, overwrite=True, output_verify='silentfix')
-        log.info(f'Master sky written to {sky_filename}')
+        if log: log.info(f'Master sky written to {sky_filename}')
 
         return
 
@@ -1121,10 +1164,12 @@ def process_science(self, sci_list, fil, amp, binn, paths, mbias=None,
             for i,frame in enumerate(processed):
 
                 mean, med, stddev = sigma_clipped_stats(frame.data)
-                frame_sky = sky_frame.multiply(med, 
+                # Scale normalized sky to same units as science (electrons) so subtract is valid
+                science_unit = frame.unit if frame.unit is not None else u.electron
+                frame_sky = sky_frame.multiply(med * science_unit,
                     propagate_uncertainties=True, handle_meta='first_found')
 
-                processed[i] = frame.subtract(frame_sky, 
+                processed[i] = frame.subtract(frame_sky,
                     propagate_uncertainties=True, handle_meta='first_found')
                 processed[i].header['SKYBKG']=med
                 processed[i].header['SATURATE']-=med

potpyri/primitives/absphot.py

@@ -463,6 +463,9 @@ def convert_filter_name(self, filt):
             return 'z'
         if filt=='Y':
             return 'J'
+        # K, Ks, Kspec all use 2MASS K-band for calibration
+        if filt in ('K', 'Ks', 'Kspec'):
+            return 'K'
         else:
             return filt
 

potpyri/utils/utilities.py

@@ -71,19 +71,24 @@ def find_catalog(catalog, fil, coord_ra, coord_dec):
     # If these catalogs are to be updated in the future, select mag columns that correspond to the
     # PSF mags.
     if catalog.upper() == 'SDSS':
-        if fil.lower() not in ['u','g','r','i','z']: return(catalog_ID, ra, dec, mag, err)
+        if fil.lower() not in ['u','g','r','i','z']: return(catalog, catalog_ID, ra, dec, mag, err)
         catalog_ID, ra, dec, mag, err = 'V/154', 'RA_ICRS', 'DE_ICRS', fil.lower()+'mag', 'e_'+fil.lower()+'mag'
     elif catalog.upper() == '2MASS':
-        if fil.upper() not in ['J','H','K']: return(catalog_ID, ra, dec, mag, err)
-        catalog_ID, ra, dec, mag, err = 'II/246', 'RAJ2000', 'DEJ2000', fil.upper()+'mag', 'e_'+fil.upper()+'mag'
+        # K, Ks, Kspec all use 2MASS K-band columns (Kmag, e_Kmag)
+        fil_2mass = fil.upper()
+        if fil_2mass in ('KS', 'KSPEC'):
+            fil_2mass = 'K'
+        if fil_2mass not in ['J', 'H', 'K']:
+            return(catalog, catalog_ID, ra, dec, mag, err)
+        catalog_ID, ra, dec, mag, err = 'II/246', 'RAJ2000', 'DEJ2000', fil_2mass+'mag', 'e_'+fil_2mass+'mag'
     elif catalog.upper() == 'UKIRT':
-        if fil.upper() not in ['Y','J','H','K']: return(catalog_ID, ra, dec, mag, err)
+        if fil.upper() not in ['Y','J','H','K']: return(catalog, catalog_ID, ra, dec, mag, err)
         catalog_ID, ra, dec, mag, err = 'II/319', 'ra', 'dec', fil.upper() + 'mag', 'e_'+fil.upper()+'mag'
     elif catalog.upper() == 'PS1':
-        if fil.lower() not in ['g','r','i','z','y']: return(catalog_ID, ra, dec, mag, err)
+        if fil.lower() not in ['g','r','i','z','y']: return(catalog, catalog_ID, ra, dec, mag, err)
         catalog_ID, ra, dec, mag, err = 'II/349', 'RAJ2000', 'DEJ2000', fil.lower()+'mag', 'e_'+fil.lower()+'mag'
     elif catalog.upper() == 'SKYMAPPER':
-        if fil.lower() not in ['u','v','g','r','i','z']: return(catalog_ID, ra, dec, mag, err)
+        if fil.lower() not in ['u','v','g','r','i','z']: return(catalog, catalog_ID, ra, dec, mag, err)
         catalog_ID, ra, dec, mag, err = 'II/379/smssdr4', 'RAICRS', 'DEICRS', fil.lower()+'PSF', 'e_'+fil.lower()+'PSF'
 
     return(catalog, catalog_ID, ra, dec, mag, err)

tests/test_absphot.py

@@ -91,6 +91,10 @@ def test_convert_filter_name():
     assert cal.convert_filter_name('Y') == 'J'
     assert cal.convert_filter_name('V') == 'g'
     assert cal.convert_filter_name('I') == 'i'
+    # K, Ks, Kspec all map to 2MASS K-band
+    assert cal.convert_filter_name('K') == 'K'
+    assert cal.convert_filter_name('Ks') == 'K'
+    assert cal.convert_filter_name('Kspec') == 'K'
 
 
 def test_get_minmag():

tests/test_image_procs.py

@@ -103,3 +103,22 @@ def test_add_stack_mask(tmp_path):
         assert np.any(stack[1].data >= 0)
     finally:
         log.close()
+
+
+def test_create_error(tmp_path):
+    """create_error returns error HDU from science path, mask HDU, and rdnoise."""
+    science_path = os.path.join(tmp_path, "sci.fits")
+    data = np.ones((16, 16), dtype=np.float32) * 100.0
+    hdu_sci = fits.PrimaryHDU(data)
+    hdu_sci.header["SATURATE"] = 50000.0
+    hdu_sci.writeto(science_path, overwrite=True)
+
+    mask_data = fits.ImageHDU(np.zeros((16, 16), dtype=np.uint8))
+    rdnoise = 4.0
+
+    err_hdu = image_procs.create_error(science_path, mask_data, rdnoise)
+    assert err_hdu is not None
+    assert err_hdu.data.shape == (16, 16)
+    assert err_hdu.header.get("BUNIT") == "ELECTRONS"
+    assert np.all(err_hdu.data > 0)
+    assert np.all(np.isfinite(err_hdu.data))