This article describes the exposure-level pipeline, one of the three STScI data pipelines developed for Roman, and gives details on each of its steps. It also provides basic instructions on how to run the exposure level pipeline via the command line.

Overview

The exposure-level pipeline, called exposure_pipeline , applies detector-level calibration that processes images from the Roman Wide Field Instrument (WFI) from Level 1 to Level 2. Specifically, it corrects raw, uncalibrated detector ramp data measured in instrumental units of data numbers (DNs) into calibrated rate images measured in DNs per second following a series of steps described in the sections below. Most steps use data from corresponding reference files hosted in Roman Calibration Reference Data System (CRDS). These reference data are pre-cached based on the input image file upon initiating  Romancal so they're ready for immediate use in a pipeline run. Please see the "CRDS for Reference Files" article for more information on CRDS.

While we provide short summaries in each section, please refer to the table below for links to the readthedocs for each step in the  Romancal documentation. These provide more detailed descriptions of the algorithms at work and the arguments accepted by each step.

Before a pipeline run, all status indicators in the image file metadata that correspond to a pipeline step are initially set to "INCOMPLETE" . Once a step is successfully completed, the entry changes to "COMPLETE" . If a step is voluntarily skipped by the user during a full pipeline run, the entry changes to  "SKIPPED" . A description of the Data quality flags in reference files is available on readthedocs.

Pipeline Step Descriptions

Data Quality Initialization

The data quality initialization step ( romancal.dq_init.DQInitStep() ) reads the  MASK reference file from CRDS associated to the input data. The reference file contains a mask array where problematic pixels are flagged. This mask is copied into the data product's PIXELDQ array.

Pixels may be flagged for a number of reasons; please see the  Romancal documentation page on the  MASK reference file to see a comprehensive list of DQ values and their meanings. Subsequent steps in the exposure-level pipeline may also flag more pixels and update the  PIXELDQ array.

Saturation Detection

The saturation step ( romancal.saturation.SaturationStep() ) identifies pixels with values, in counts ($//<![CDATA[ \begin{array}{l}\mathrm{DN}\end{array} //]]>$), that are either non-positive or above a defined saturation threshold. Users can find the results from this step in the three-dimensional GROUPDQ array.

Since Level 1 input files are data cubes with one dimension corresponding to the resultants in the exposure, the check is repeated for all pixels in each resultant. A pixel that exceeds the saturation threshold in one resultant will automatically be flagged in all subsequent resultants.

Pixels that either lack a defined saturation threshold or are flagged to skip this step in the relative  SATURATION reference file from CRDS will only be flagged in  GROUPDQ if their value reaches the 16-bit analog-to-digital converter limit of 65535 counts ($//<![CDATA[ \begin{array}{l}\mathrm{DN}\end{array} //]]>$).

Reference Pixel Correction 

The reference pixel correction step ( romancal.refpix.RefPixStep() ) utilizes the values from the four rows and columns of physical reference pixels along the detector's borders, as well as the extra 4096 by 128 column of virtual pixels read out by the detector's 33rd amplifier. See the "Description of WFI" article for more details on reference pixels. Since none of these pixels are exposed to light during science observations, they are used to estimate the amount of noise added to each by electronics as the signal is read out from the detector.

This step subtracts (from each resultant) a time dependent linear combination of the reference pixels optimally computed for each science pixel.

Linearity Correction

The linearity correction step ( romancal.linearity.LinearityStep() ) is responsible for addressing non-linear responses from the detector's electronics. To perform this correction, it reads the corresponding LINEARITY reference file or the Level 1 data product from CRDS. This reference file contains all the coefficients for an n-order polynomial, adapted from the "classic" linearity correction approach used in the Hubble Space Telescope's WFC3/IR mode. The process is meant to correct for non-linear responses to signal from the detector's electronics and is applied to every pixel in every resultant of the science data.

This step assumes that the saturation step has already been run and skips all pixels flagged as saturated in the GROUPDQ array. Pixels that have at least one undefined polynomial coefficient or flagged to skip this step in the  LINEARITY reference file are not modified and are flagged in the PIXELDQ array. 

Dark Current Subtraction

The dark current subtraction step ( romancal.dark_current.DarkCurrentStep() ) orrects for noise caused by dark current in the detector. This step retrieves the corresponding DARK reference file for the Level 1 data product from CRDS. The  DARK reference file contains data representing the current recorded by the detector's photo-diode arrays in the absence of any scientific signal—essentially, when the detectors are expected to be "dark."

The DARK reference file is selected based on the Multi-accumulation table used acquire the science data. The dark reference files have the same three-dimensional shape as the L1 science data and are subtracted on a pixel-by-pixel and resultant-by-resultant basis.

This step assumes that both the L1 product's science data and the DARK reference file's dark data are bias subtracted. This is currently performed during the reference pixel correction step, while investigations are ongoing to determine if further removal of instrument bias is necessary. Pixels that are undefined in the dark reference file will not be subtracted from the science data.

Ramp Fitting and Jump Detection

The combined ramp fitting and jump detection step ( romancal.ramp_fitting.RampFitStep() ) calculates each pixel's mean count rate in counts per second ($//<![CDATA[ \begin{array}{l}\mathrm{DN/s}\end{array} //]]>$) and identifies cosmic-ray induced discontinuities, or jumps, in the flux received by each pixel in each resultant over the course of an integration.

Ramp slopes are calculated by conducting an optimally-weighted least-squares fit to the resultant-to-resultant slopes of cumulative flux received by each science pixel. For example, a pixel in an integration that includes a ramp of $//<![CDATA[ \begin{array}{l}n\end{array} //]]>$ resultants may have as many as $//<![CDATA[ \begin{array}{l}n-1\end{array} //]]>$ segment slopes that are incorporated into the fit. Segment weighting depends on many factors, including read noise (from the READNOISE reference file), a resultant's distance from the midpoint in integration time, and the data's signal-to-noise ratio. See the Romancal documentation for more information on the optimal weighting algorithm.

The resulting array of ramp slopes is collapsed along the resultant axis to become two-dimensional. Additionally, its size is trimmed to 4,088 by 4,088 pixels since ramp slopes are not measured for reference pixels. This portion of the step also calculates variances due to Poisson and read noise, saving them in the data product's VAR_POISSON and  VAR_RNOISE arrays. The combined variance is saved in the data product's ERR array.

The jump detection step aims to identify resultants where a cosmic ray deposited an abnormally large amount of charge on a pixel. For a given pixel, the resultant containing the jump as well as the following one are considered contaminated. To identify the jumps, the differences between the flux in a given resultant ($//<![CDATA[ \begin{array}{l}n\end{array} //]]>$) and the two consecutive ones ($//<![CDATA[ \begin{array}{l}n+1\end{array} //]]>$ and $//<![CDATA[ \begin{array}{l}n+2\end{array} //]]>$) are calculated. In addition, the slope computed in the ramp fitting step is used to set a statistical threshold. For more information, see the Romancal documentation. Jumps are flagged in the GROUPDQ array wherever this threshold is surpassed. 

This step assumes that saturation flagging, linearity correction, and dark current subtraction (which requires reference pixel correction) have already taken place. Note that a data product must contain at least two resultants in order for the algorithms to work properly.

WCS Assignment

The WCS assignment step ( romancal.assign_wcs.AssignWcsStep() ) reads an image's corresponding  DISTORTION reference file from CRDS in order to associate the image with a World Coordinate System object. The WCS object converts the image's coordinates from the detector frame to the International Celestial Reference System's (ICRS)  world coordinate frame and saves the information in the data product's meta.wcs attribute of the L2 ASDF DataModel .

This step assumes that ramp fitting has already taken place. This is the last step of the exposure-level pipeline that is applied to WFI prism and grism data. Subsequent steps are only relevant for WFI image data.

Flatfield Correction

This correction compensates for pixel-to-pixel sensitivity variations, effectively 'flattening' the measured differences and ensuring uniform data. The flatfield correction step ( romancal.flatfield.FlatFieldStep() ) retrieves the corresponding  FLAT reference file from CRDS to obtain flatfield data. This data is then used to divide the image's SCI

, ERR , and variance-associated arrays. Specifically, the  SCI array is divided directly by the flatfield's science array, while the VAR_POISSON and  VAR_RNOISE arrays are divided by the square of the flatfield's science array.

The data product's VAR_FLAT array is calculated based on both the science and error arrays from the FLAT reference file, and the data product's ERR array is updated as the square of the sum of VAR_FLAT and the newly-updated VAR_POISSON and  VAR_RNOISE arrays. Pixels in the FLAT reference file that either have non-positive values or are flagged to skip this step will not be updated.

Photometric Calibration

The photometric calibration step ( romancal.photom.PhotomStep() ) reads an image's corresponding  PHOTOM reference file from CRDS in order to update a data product's  photometry metadata fields. These updates include information about how to convert observed counts per second to physical flux densities and pixel area to square arcseconds for the observation.

Source Detection

The source detection step ( romancal.source_detection.SourceDetectionStep() ) utilizes the photutils   DAOStarFinder class to find, catalog, and fit PSFs to point sources in an image. The resulting catalog comes ready for use in the upcoming TweakReg step and is saved to the data product's  meta.source_catalog.tweakreg_catalog metadata field as a  numpy array.

TweakReg

The TweakReg step ( romancal.tweakreg.TweakRegStep() ) compares observed point source locations in an image to coordinates from any Gaia Data Release catalog and, if necessary, corrects the image's WCS to bring it into astrometric alignment. The step generates a fit to the reference catalog that is used to apply a linear coordinate transformation to the image's WCS. Though this step can take the input catalog generated by the source detection step in  meta.source_catalog.tweakreg_catalog metadata field, users can provide a custom source catalog as well.

Running the Exposure Level Pipeline

Users may run the exposure-level pipeline from the command line or as part of a Python script. To run the full exposure-level pipeline on the command line, use the  strun command:

strun romancal.pipeline.ExposurePipeline r0008308002010007027_06311_0019_WFI01_uncal.asdf

The command's first argument is the pipeline's full class name (as above) or, for convenience, a shorter alias. The exposure-level pipeline's alias is  roman_elp . See the Romancal documentation for more on the ExposurePipeline class. The second argument is the name of an existing, uncalibrated ramp file for the pipeline to process to Level 2. The command returns a calibrated 2-D image, a source catalog file, and a segmentation map (a map where each pixel contains an integer value that corresponds to a source label number from the source catalog file).

The strun command can take both global and step-specific arguments that modify its output. For example, the following code runs the full exposure-level pipeline with a global flag that saves all outputs to a specific directory and a step-specific flag that and a step-specific flag that skips the saturation detection step.

strun roman_elp r0008308002010007027_06311_0019_WFI01_uncal.asdf --output_dir='elp_results' --steps.saturation.skip=True

It is also possible to run a single pipeline step at a time. To run the dark current subtraction step with a custom DARK reference file named custom_dark.asdf , use its specific class name in the place of the exposure-level pipeline's alias.

strun romancal.step.DarkCurrentStep r0008308002010007027_06311_0019_WFI01_uncal.asdf --override='custom_dark.asdf'

Running a single step produces an intermediate file with a name whose suffix corresponds with the step that produced it. For example, the file created by previous code snippet would be named r0008308002010007027_06311_0019_WFI01_darkcurrent.asdf . See  Romancal the documentation for a full list of these suffixes.

For a full accounting of available parameters, use strun 's "--help" flag in combination with the class name or alias of interest.

For additional questions not answered in this article, please contact the Roman Help Desk at STScI.

References

• The Roman Space Telescope Calibration Pipeline, readthedocs, maintained by Roman calibration pipeline developers. Revision bc1d143a.  Last updated on Aug 29, 2024.
  
• Romancal Exposure Pipeline code on GitHub, romancal maintained by Roman calibration pipeline developers at STScI. Romancal release 0.16.3 on Aug 29, 2024. 
• CRDS for Reference Files, RDox
• Description of WFI: Readout and Reference Pixels, RDox 
• Appendix: WFI MultiAccum Tables, RDox (Draft)
• Sanjib Sharma and Stefano Casertano, 2024 "Cosmic Ray Jump Detection for the Roman Wide Field Instrument", PASP 136 054504
• Casertano et al 2022, "Determining the best-fitting slope and its uncertainty for up-the-ramp sampled images with unevenly distributed resultants", STScI Technical Report