Exposure Pipeline

This article describes the Exposure Pipeline, which processes uncalibrated data from the Roman Space Telescope Wide Field Instrument (WFI) into calibrated rate images. It also provides basic instructions for running the Exposure Pipeline.





Introduction to the Exposure Pipeline

This article references technical information from the STScI data calibration pipeline suite RomanCal. Throughout the text, references to the STScI pipeline suite as a whole are referred to as “RomanCal,” the name of the Python package is stylized as romancal , and individual pipelines within RomanCal are named (e.g., Exposure Pipeline). Technical information sources for RomanCal include:

The Exposure Pipeline applies detector-level calibrations that process Level 1 uncalibrated Roman WFI data into Level 2 calibrated rate images (see Data Level and Products for additional details). Specifically, it corrects raw, uncalibrated detector ramp data, measured in instrumental units of data numbers (DN; also known as analog-to-digital units or ADU), into calibrated rate images measured in DN per second (DN/s), following the steps described in the sections below. Most steps use data from corresponding calibration reference files hosted in the Calibration Reference Data System (CRDS). For more information about calibration reference files used from CRDS, see the CRDS for Reference Files article.

While we provide brief summaries in each section, detailed descriptions of the algorithms and the arguments accepted by each step can be found in the romancal documentation. See the Table of Steps in the Exposure Pipeline for links to the ReadTheDocs documentation pages for those steps.

This article corresponds to romancal version 1.2.0. This software is under development and subject to change; the documentation on this page refers to the most recently validated version of romancal and newer, unvalidated versions may exist.


Before a pipeline run, all status indicators in the file metadata that correspond to a pipeline step are set to "INCOMPLETE" . Once a step is successfully completed, the entry changes to "COMPLETE" . If the logic within the  romancal code results in a step being skipped (e.g., the flatfield reference file is not applied to data obtained in spectroscopic mode) then the entry is set to "SKIPPED" . Pipeline steps perform operations on the input data arrays, producing updated data arrays as output. The variance and total error arrays are created and updated by certain steps. Additionally, each step in the pipeline tracks and updates the data quality (DQ) flags for each pixel. Pixels may be flagged for various reasons, and descriptions of the DQ flags are available on readthedocs.

The definition of the DQ flags is under active development and expected to change based on characterization of the WFI detectors from ground-based testing and commissioning activities.




Pipeline Step Descriptions

Data Quality Initialization 

The DQ initialization step, romancal.dq_init.DQInitStep() , reads the MASK reference file from CRDS that is associated to the input data. This reference file contains a mask array where problematic pixels (e.g., dead pixels) are flagged. The mask is copied into the pixeldq array. Note that the  pixeldq array is later replaced by the  dq array after the completion of the ramp fitting step. The DQ initialization step also creates a temporary groupdq array, which allows the pipeline to flag pixels in specific resultants up-the-ramp (e.g., to flag the onset of saturation). After ramp fitting, the groupdq array is flattened with bitwise addition to indicate if a pixel had a jump at any point in the exposure to the  pixeldq array, resulting in the dq array.

Please see the  romancal documentation page on the  MASK reference file for a comprehensive list of DQ values and their meanings. Note that subsequent steps in the Exposure Pipeline may flag additional pixels.


Saturation Detection

The saturation step, romancal.saturation.SaturationStep() , identifies pixels with values in DN that are either less than or equal to zero or above a defined saturation threshold. This threshold does not correspond to saturation in the same sense as in charge-coupled devices (CCDs), where saturation refers to the point at which a pixel can no longer accumulate additional charge. Instead, it should be understood as a significant deviation from linearity in the detector response. Pixels with values above the thresholds specified in the SATURATION reference file are considered to lack accurate calibration levels. The results from this step are included in the three-dimensional groupdq array.

Since L1 input files are data cubes with one dimension corresponding to the resultants throughout exposure, the saturation check is repeated for all pixels in each resultant. Note that the values in the SATURATION reference file apply to individual reads of the WFI detectors. If a resultant consists of multiple reads that have been averaged together, the saturation step scales the threshold accordingly to flag pixels that saturated in any of the contributing reads. A pixel that exceeds the saturation threshold in one resultant will automatically be flagged in all subsequent resultants.

Pixels that either lack a defined saturation threshold or are flagged to skip this step in the  SATURATION reference file, will only be flagged in groupdq if their values reach the 16-bit analog-to-digital converter limit of 65,535 DN.


Reference Pixel Correction 

The Improved Roman Reference pixel Correction (IRRC) step, romancal.refpix.RefPixStep() , uses values from the four rows and columns of physical reference pixels along the detector's borders, as well as from the additional 4096x128 array of virtual pixels read out by the detector's 33rd amplifier. For more details on reference pixels, see the Description of WFI article. Since none of these reference pixels is exposed to light during science observations, they can be used to estimate the electronic noise added to each pixel as the signal is read out from the detector. 

This step subtracts from each resultant a time-dependent linear combination of the reference pixels optimally computed for each science pixel and the data are corrected in pixel space. The weights of the three reference pixel streams (left, right, and 33rd amplifier) are stored in the  REFPIX reference file. Detailed information on the reference pixel correction theory and application to WFI ground test data can be found in Rauscher, Fixsen, and Mosby (2022) and Betti et al. (2024), respectively.


Dark Decay Correction

The dark decay correction step, romancal.dark_decay.DarkDecayStep() , corrects for an anomalous, detector-wide signal present in the first few resultants of WFI exposures. This electronic effect is well described by an exponential decay whose amplitude and time constant vary from detector to detector but are uniform across all pixels on a given detector. As the detector is read out, the magnitude of the dark decay signal decreases with time, producing a characteristic offset in the early resultants. 

This step uses the DARK DECAY  reference file from CRDS to obtain the detector-specific decay amplitude and time constant. For each resultant, the pipeline computes the expected dark decay signal based on the readout time of every pixel, averages the correction over the frames contributing to each resultant, and subtracts the resulting correction image from the science data. The correction is applied before the linearity correction because dark decay is a readout electronics effect rather than a property of the detector response.


WFI18 Transient Correction 

The WFI18 correction step, romancal.wfi18_transient.WFI18TransientStep() , corrects a transient detector artifact observed in the WFI18 detector. The artifact appears as a time-dependent electronic signal that affects the early resultants of an exposure and is unique to this detector and mostly constrained to the bottom 256 rows. If left uncorrected, the transient would introduce a systematic offset into the measured signal during the initial stages of the exposure pipeline. 


Linearity Correction

The linearity correction step, romancal.linearity.LinearityStep() , is responsible for addressing non-linear responses from the detector's electronics. This effect is also referred to as classic non-linearity, and this step does not apply to other sources of non-linear behavior such as count rate non-linearity or the brighter-fatter effect. To perform this correction, the pipeline reads the corresponding LINEARITY reference file, which contains coefficients for an n-order polynomial. The correction is adapted from the approach used to address classic non-linearity for the Hubble Space Telescope's WFC3/IR mode (Hilbert 2009). This process corrects for the detector electronics’ non-linear response to signal and is applied to every pixel in every resultant of the science data.

This step assumes that the saturation step has already been run, and skips all pixels flagged as saturated in the groupdq array. Pixels that have at least one undefined polynomial coefficient or are flagged to skip this step in the LINEARITY reference file are not modified, and are flagged in the pixeldq array. Additionally, the size of the data array is trimmed from (N, 4096, 4096) pixels to (N, 4088, 4088) pixels, where N is the number of resultants, since the 4-pixel reference pixel border is no longer needed after the reference pixel correction has been applied.


Dark Current Subtraction

The dark current subtraction step, romancal.dark_current.DarkCurrentStep() , corrects for noise caused by dark current in the detector. This step retrieves the corresponding DARK reference file, which contains data representing the current recorded by the detector's photo-diode arrays in the absence of any scientific signal - essentially, when the detectors are expected to be "dark."

The DARK reference file is a pixel-by-pixel rate image that is used to subtract the dark signal from the science data. Pixels that are undefined in the dark reference file will not be subtracted from the science data.


Ramp Fitting and Jump Detection

The combined ramp fitting and jump detection step, romancal.ramp_fitting.RampFitStep() , calculates each pixel's best-fit count rate in DN/s and identifies discontinuities, or jumps, in the flux received by each pixel in each resultant over the course of an integration (e.g., due to cosmic rays). The jump detection and ramp fitting algorithms are implemented jointly in romancal , rather than as separate steps; however, they are described separately below.

Slopes are calculated by conducting an optimally-weighted least-squares fit to the slopes between consecutive resultants of the cumulative flux received by each science pixel. For example, a pixel in an integration that includes a ramp of n resultants may have as many as n-1 segment slopes that are incorporated into the fit. By default these weightings are calculated by minimizing \chi^2 using the inverse of the covariance matrix appropriate for each individual pixel; see Brandt (2024a) for more information. The covariance matrix depends on the readout pattern, the read noise (from the READNOISE reference file), the count rate, and the electronic gain.  A different, optional algorithm for segment weighting provides similar results; it is described in Casertano (2022).

An array with a slope for each pixel effectively collapses the input L1 3-D ramp cube along the resultant axis into a 2-D rate image. This portion of the step also calculates variances due to Poisson and read noise, saving them in the var_poisson and  var_rnoise arrays of the data product. The square root of the sum of these two variances is stored in the err array.

The jump detection step aims to identify resultants where a cosmic ray deposited an abnormally large amount of charge on a pixel. The default Brandt (2024a) and optional Casertano (2022) algorithms are each paired with a different jump detection algorithm. The likelihood-based approach paired with Brandt (2024a) is described in Brandt (2024b). It works by computing the best \chi^2 for each pixel using all ramps and comparing it to the \chi^2 value excluding either one or two pairs of resultant differences. A likelihood ratio test is then used to flag and possibly reject the pair(s) of differences whose omission most improves \chi^2. The default likelihood ratio corresponds to 4.5\sigma. The jump detection algorithm paired with the Casertano (2022) ramp fitting is described in Sharma and Casertano (2024). In that case, the differences between the flux in a given resultant (n) and the two successive ones (n+1 and n+2) are calculated. In addition, the slope computed in the ramp fitting step is used to set a statistical threshold. For a given pixel, if the difference of (n+1)-th resultant with respect to the n-th resultant contains a jump, then both the n-th and (n + 1)-th resultants are considered to be contaminated. This is because if the n-th resultant is the first to be contaminated, then both the forward and backward differences appear abnormal. Once a jump is identified, the ramp is split into sub ramps, each of them fitted independently. These sub ramps are combined later on to give a single estimate of the overall slope.

Jumps identified by whichever of the two algorithms is used are flagged in the  groupdq array. Once these jumps are detected, a snowball detection algorithm may optionally be run. The snowball algorithm searches for clusters of jumps and expands the jump flag spatially.

The ramp fitting and jump detection step assumes that saturation flagging, linearity correction, and dark current subtraction (which itself requires reference pixel correction) have already been performed. Note that a data product must contain at least two resultants after each detected jump for the slope to be measured.


WCS Assignment

The World Coordinate System (WCS) assignment step, romancal.assign_wcs.AssignWcsStep() , reads the correct DISTORTION reference file from CRDS in order to associate the image with a WCS object based on the coarse pointing information in the file metadata. The WCS object converts image pixel coordinates from the science coordinate system to the International Celestial Reference System's (ICRS) frame. The WCS object is stored in the meta.wcs attribute of the ASDF file.

This step assumes that ramp fitting has already taken place

WCS Assignment is the last step of the exposure pipeline that is applied to WFI prism and grism data. Subsequent steps are only relevant for WFI image data.


Flatfield Correction

This correction compensates for pixel-to-pixel sensitivity and variations at larger scales, effectively 'flattening' the measured differences and aiming for a uniform surface brightness response in the data. The flatfield correction step, romancal.flatfield.FlatFieldStep() , retrieves the corresponding FLAT reference file from CRDS, containing the flatfield data. The data array is then divided by the FLAT reference file data array, while the var_poisson and  var_rnoise arrays are divided by the square of the flatfield's science array.

The err array is updated to include the variance due to the flatfield uncertainty as well as the updated var_poisson and var_rnoise arrays. Pixels in the  FLAT reference file that either have non-positive values or are flagged to skip this step will not be updated.


Photometric Calibration

The photometric calibration step, romancal.photom.PhotomStep() , reads an image's corresponding PHOTOM reference file from CRDS in order to update a data product's  photometry metadata fields. These updates include information on how to convert instrumental units of DN/s into physical surface brightness units of megaJanskys per steradian (MJy/sr).


Source Catalog

The source catalog step, romancal.source_catalog.SourceCatalogStep() , uses the photutils package to detect, separate (deblend), and perform photometric and morphological measurements of astronomical sources in an image. Within this step, if the option psf_fit = True  is selected, PSF models generated by STPSF are fit to stars detected via  DAOStarFinder . 

The source catalog step produces two output data products: 1) a source catalog and 2) a segmentation map. Both products are stored as ASDF files (see WFI Data Levels and Products for more information). Note that prompt L4 source catalogs made from L2 inputs are used primarily for the alignment of the L2 data to the Gaia astrometric catalog, and should not be considered a replacement for L4 source catalogs made from L3 inputs. The source catalog contains photometric and morphology information. The morphology measurements are based on 2-D image moments within the source segment. Additional details about the source catalog can be found in the readthedocs. This step can be executed on L2 or co-added mosaic Level 3 (L3) images, and the output data products are considered Level 4 (L4) products.


TweakReg

The TweakReg step, romancal.tweakreg.TweakRegStep() , compares observed point source locations in an image to coordinates from the Gaia Data Release 3 catalog and, if necessary, corrects the image WCS to achieve astrometric alignment. It generates a fit to the reference catalog that is used to apply a linear coordinate transformation to the image's WCS. Although this step uses by default the input catalog generated by the source detection step stored in the meta.source_catalog.tweakreg_catalog metadata field, users may also provide a custom source catalog.


Reference Pixels in the Exposure Pipeline

Reference pixels are preserved through the early stages of the exposure pipeline and copied into dedicated arrays during DQInit step. After they are used for reference-pixel-based corrections (e.g., IRRC), they are removed from the science data array at the end of the RampFit step but remain available in their dedicated arrays within the data model. To summarize below.

  • L1 raw images contain reference pixels in the main data arrays.
  • romancal.dq_init.DQInitStep() copies reference pixels into dedicated arrays.
  • romancal.refpix.RefPixStep() uses the reference pixels to mitigate horizontal banding.
  • romancal.rampfit.RampFitStep() trims the reference pixels from the science data array.


Running the Exposure Pipeline

Users may run the Exposure Pipeline from the command line or as part of a Python script. To run the full Exposure Pipeline on the command line, use the strun command:

Full Pipeline Run
strun romancal.pipeline.ExposurePipeline r0008308002010007027_0019_wfi01_f106_uncal.asdf

The command's first argument is the pipeline's full class name (as above) or, for convenience, a shorter alias. The Exposure Pipeline alias is roman_elp . See the romancal documentation for more on the ExposurePipeline class. The second argument is the name of an existing, uncalibrated L1 file for the pipeline to process to L2. The command returns a calibrated 2-D image, a source catalog file, and a segmentation map (a map where each pixel contains an integer value that corresponds to a source label number from the source catalog file).

The strun command can take both global and step-specific arguments that modify its output. For example, the following code runs the full exposure pipeline with a global flag that saves all outputs to a specific directory and a step-specific flag that skips the saturation detection step.

Full Pipeline Run
strun roman_elp r0008308002010007027_0019_wfi01_f106_uncal.asdf --output_dir='elp_results' --steps.saturation.skip=True

It is also possible to run a single pipeline step at a time. To run the dark current subtraction step with a custom  DARK reference file named custom_dark.asdf , use its specific class name in the place of the exposure pipeline's alias.

Full Pipeline Run
strun romancal.step.DarkCurrentStep r0008308002010007027_0019_wfi01_f106_uncal.asdf --override='custom_dark.asdf'

Running a single step produces an intermediate file with a name whose suffix corresponds with the step that produced it. For example, the file created by previous code snippet would be named r0008308002010007027_0019_wfi01_f106_darkcurrent.asdf . See the romancal documentation for a full list of these suffixes.

For a full accounting of available parameters, use strun 's --help flag in combination with the class name or alias of interest.




For additional questions not answered in this article, please contact the Roman Help Desk.




References




Latest Update

 

Updated for clarity and accuracy.
Publication

 

Initial publication of the article.