This article describes the Exposure Level Pipeline, which processes uncalibrated Roman Space Telescope Wide Field Instrument data into calibrated rate images. In addition, basic instructions on how to run the Exposure Level Pipeline are provided.

Overview

The Exposure Level Pipeline applies detector-level calibrations that process uncalibrated (Level 1; L1) Roman Space Telescope (Roman) Wide Field Instrument (WFI) data into calibrated rate images (Level 2; L2). Specifically, it corrects raw, uncalibrated detector ramp data, measured in instrumental units of data numbers (DN – also known as ADU), into calibrated rate images measured in data numbers per second (DN/s), following a series of steps described in the sections below. Most steps use data from corresponding reference files hosted in Roman Calibration Reference Data System (CRDS). For more information, see the CRDS for Reference Files article.

While we provide brief summaries in each section, detailed descriptions of the algorithms and the arguments accepted by each step can be found in the romancal documentation. See the table below for links to the readthedocs site.

REFPIX

Before a pipeline run, all status indicators in the file metadata that correspond to a pipeline step are set to "INCOMPLETE" . Once a step is successfully completed, the entry changes to "COMPLETE" . If the logic within the  romancal code results in a step being skipped (e.g., the flatfield reference file is not applied to data obtained in spectroscopic mode) then the entry is set to "SKIPPED" . Pipeline steps perform operations on the input data arrays, producing updated data arrays as output. Variance and total error information are also updated by certain steps. Additionally, each step in the pipeline tracks and updates the data quality (DQ) flags for each pixel. Pixels may be flagged for various reasons, and descriptions of the DQ flags are available on readthedocs.

Pipeline Step Descriptions

Data Quality Initialization

The DQ initialization step ( romancal.dq_init.DQInitStep() ) reads the  MASK reference file from CRDS that is associated to the input data. This reference file contains a mask array where problematic pixels (e.g., dead pixels) are flagged. The mask is copied into the pixeldq array. Note that the  pixeldq array is later replaced by the  dq array after the completion of the ramp fitting step. The DQ initialization step also creates a temporary groupdq array, which allows the pipeline to flag pixels in specific resultants up-the-ramp (e.g., to flag the onset of saturation). After ramp fitting, the groupdq array is flattened and added bitwise to the pixeldq array, resulting in the dq array.

Please see the  romancal documentation page on the  MASK reference file for a comprehensive list of DQ values and their meanings. Note that subsequent steps in the Exposure Level Pipeline may flag additional pixels.

Saturation Detection

The saturation step ( romancal.saturation.SaturationStep() ) identifies pixels with values in DN that are either less than or equal to zero or above a defined saturation threshold. This threshold does not correspond to saturation in the same sense as in charge-coupled devices (CCDs), where saturation refers to the point at which a pixel can no longer accumulate additional charge. Instead, it should be understood as a significant deviation from linearity in the detector response. Pixels with values above the thresholds specified in the SATURATION reference file are considered to lack accurate calibration levels. The results from this step are included in the three-dimensional groupdq array.

Since L1 input files are data cubes with one dimension corresponding to the resultants throughout exposure, the saturation check is repeated for all pixels in each resultant. Note that the values in the SATURATION reference file apply to individual reads of the WFI detectors. If a resultant consists of multiple reads that have been averaged together, the saturation step scales the threshold accordingly to flag pixels that saturated in any of the contributing reads. A pixel that exceeds the saturation threshold in one resultant will automatically be flagged in all subsequent resultants.

Pixels that either lack a defined saturation threshold or are flagged to skip this step in the  SATURATION reference file, will only be flagged in groupdq if their values reach the 16-bit analog-to-digital converter limit of 65,535 DN.

Reference Pixel Correction 

The reference pixel correction step ( romancal.refpix.RefPixStep() ) uses values from the four rows and columns of physical reference pixels along the detector's borders, as well as from the additional 4096x128 column of virtual pixels read out by the detector's 33rd amplifier. For more details on reference pixels, see the Description of WFI article. Since none of these reference pixels is exposed to light during science observations, they can be used to estimate the electronic noise added to each pixel as the signal is read out from the detector.

This step subtracts from each resultant a time-dependent linear combination of the reference pixels optimally computed for each science pixel. Data are corrected in Fourier space. The weights of the three reference pixel streams (left, right, and 33rd amplifier) are stored in the  REFPIX reference file. Detailed information on the reference pixel correction theory and application to WFI ground test data can be found in Rauscher, Fixsen, and Mosby (2022) and Betti et al. (2024), respectively.

Linearity Correction

The linearity correction step ( romancal.linearity.LinearityStep() ) is responsible for addressing non-linear responses from the detector's electronics. This effect is also referred to as classic non-linearity, and this step does not apply to other sources of non-linear behavior such as count rate non-linearity or the "brighter-fatter" effect. To perform this correction, the pipeline reads the corresponding LINEARITY reference file, which contains coefficients for an n-order polynomial. The correction is adapted from the approach used to address classic non-linearity for the Hubble Space Telescope's WFC3/IR mode (Hilbert 2009). This process corrects for the detector electronics’ non-linear response to signal and is applied to every pixel in every resultant of the science data.

This step assumes that the saturation step has already been run, and skips all pixels flagged as saturated in the groupdq array. Pixels that have at least one undefined polynomial coefficient or are flagged to skip this step in the LINEARITY reference file are not modified, and are flagged in the pixeldq array. Additionally, the size of the data array is trimmed from (N, 4096, 4096) pixels to (N, 4088, 4088) pixels, where N is the number of resultants, since the 4-pixel reference pixel border is no longer needed after the reference pixel correction has been applied.

Dark Current Subtraction

The dark current subtraction step ( romancal.dark_current.DarkCurrentStep() ) corrects for noise caused by dark current in the detector. This step retrieves the corresponding DARK reference file, which contains data representing the current recorded by the detector's photo-diode arrays in the absence of any scientific signal - essentially, when the detectors are expected to be "dark."

The DARK reference file is selected based on the multi-accumulation table (see WFI MultiAccum Tables article for more information about MA Tables) used to obtain the science data. The dark reference files have the same three-dimensional shape as the L1 science data and are subtracted on a pixel-by-pixel and resultant-by-resultant basis. Pixels that are undefined in the dark reference file will not be subtracted from the science data.

Ramp Fitting and Jump Detection

The combined ramp fitting and jump detection step ( romancal.ramp_fitting.RampFitStep() ) calculates each pixel's mean count rate in DN/s and identifies discontinuities, or jumps, in the flux received by each pixel in each resultant over the course of an integration (e.g., due to cosmic rays). The jump detection and ramp fitting algorithms are implemented jointly in  romancal , rather than as separate steps; however, they are described separately below.

Slopes are calculated by conducting an optimally-weighted least-squares fit to the slopes between consecutive resultants of the cumulative flux received by each science pixel. For example, a pixel in an integration that includes a ramp of $//<![CDATA[ \begin{array}{l}n\end{array} //]]>$ resultants may have as many as $//<![CDATA[ \begin{array}{l}n-1\end{array} //]]>$ segment slopes that are incorporated into the fit. Segment weighting depends on many factors, including read noise (from the READNOISE reference file), a resultant's distance from the midpoint in integration time, and the data's signal-to-noise ratio. See Casertano (2022) for more information on the optimal weighting algorithm.

An array of slopes for each pixel effectively collapses the input L1 3-D ramp cube along the resultant axis into a 2-D rate image. This portion of the step also calculates variances due to Poisson and read noise, saving them in the  var_poisson and  var_rnoise arrays of the data product. The square root of the sum of the variances is stored in the err array.

The jump detection step aims to identify resultants where a cosmic ray deposited an abnormally large amount of charge on a pixel. To identify the jumps, the differences between the flux in a given resultant ($//<![CDATA[ \begin{array}{l}n\end{array} //]]>$) and the two consecutive ones ($//<![CDATA[ \begin{array}{l}n+1\end{array} //]]>$ and $//<![CDATA[ \begin{array}{l}n+2\end{array} //]]>$) are calculated. In addition, the slope computed in the ramp fitting step is used to set a statistical threshold. For a given pixel, if the difference of ($//<![CDATA[ \begin{array}{l}n+1\end{array} //]]>$)-th resultant with respect to the $//<![CDATA[ \begin{array}{l}n\end{array} //]]>$-th resultant contains a jump, then both the $//<![CDATA[ \begin{array}{l}n\end{array} //]]>$-th and ($//<![CDATA[ \begin{array}{l}n + 1\end{array} //]]>$)-th resultants are considered to be contaminated. This is because if the $//<![CDATA[ \begin{array}{l}n\end{array} //]]>$-th resultant is the first to be contaminated, then both the forward and backward differences appear abnormal. Once a jump is identified, the ramp is split into sub ramps, each of them fitted independently. These sub ramps are combined later on to give a single estimate of the overall slope. For more information, see Sharma and Casertano (2022) for the theory of the algorithm and the romancal documentation for the implementation. Jumps are flagged in the groupdq array wherever this threshold is surpassed. 

This step assumes that saturation flagging, linearity correction, and dark current subtraction (which itself requires reference pixel correction) have already been performed. Note that a data product must contain at least two resultants after each detected jump for the slope to be measured.

WCS Assignment

The World Coordinate System (WCS) assignment step ( romancal.assign_wcs.AssignWcsStep() ) reads the correct DISTORTION reference file from CRDS in order to associate the image with a WCS object based on the coarse pointing information in the file metadata. The WCS object converts image pixel coordinates from the science coordinate system to the International Celestial Reference System's (ICRS) frame. The WCS object is stored in the meta.wcs attribute of the ASDF file.

This step assumes that ramp fitting has already taken place. This is the last step of the exposure-level pipeline that is applied to WFI prism and grism data. Subsequent steps are only relevant for WFI image data.

Flatfield Correction

This correction compensates for pixel-to-pixel sensitivity variations, effectively 'flattening' the measured differences and ensuring uniform data. The flatfield correction step ( romancal.flatfield.FlatFieldStep() ) retrieves the corresponding  FLAT reference file from CRDS, containing the flatfield data. The data array is then divided by the FLAT reference file data array, while the var_poisson and  var_rnoise arrays are divided by the square of the flatfield's science array.

The err array is updated to include the variance due to the flatfield as well as the updated var_poisson and var_rnoise arrays. Pixels in the  FLAT reference file that either have non-positive values or are flagged to skip this step will not be updated.

Photometric Calibration

The photometric calibration step ( romancal.photom.PhotomStep() ) reads an image's corresponding  PHOTOM reference file from CRDS in order to update a data product's  photometry metadata fields. These updates include information on how to convert instrumental units of DN/s into physical surface brightness units of megaJanskys per steradian (MJy/sr).

Source Catalog

The source catalog step ( romancal.source_catalog.SourceCatalogStep() ) uses the photutils package to detect, separate (deblend), and perform photometric and morphological measurements of astronomical sources in an image. Within this step, if the option psf_fit = True  is selected, PSF models generated by STPSF are fit to stars detected via  DAOStarFinder . 

The source catalog step produces two output data products: 1) a source catalog and 2) a segmentation map. Both products are stored as ASDF files (see Data Levels and Products for more information). The source catalog contains photometric and morphology information. The morphology measurements are based on 2-D image moments within the source segment. Additional details about the source catalog can be found in the readthedocs. This step can be executed on L2 or co-added mosaic Level 3 (L3) images, and the output data products are considered Level 4 (L4) products.

TweakReg

The TweakReg step ( romancal.tweakreg.TweakRegStep() ) compares observed point source locations in an image to coordinates from the Gaia Data Release 3 catalog and, if necessary, corrects the image WCS to achieve astrometric alignment. It generates a fit to the reference catalog that is used to apply a linear coordinate transformation to the image's WCS. Although this step can use the input catalog generated by the source detection step stored in the  meta.source_catalog.tweakreg_catalog metadata field, users may also provide a custom source catalog.

Running the Exposure Level Pipeline

Users may run the Exposure Level Pipeline from the command line or as part of a Python script. To run the full Exposure Level Pipeline on the command line, use the strun command:

strun romancal.pipeline.ExposurePipeline r0008308002010007027_0019_wfi01_f106_uncal.asdf

The command's first argument is the pipeline's full class name (as above) or, for convenience, a shorter alias. The Exposure Level Pipeline alias is roman_elp . See the romancal documentation for more on the ExposurePipeline class. The second argument is the name of an existing, uncalibrated L1 file for the pipeline to process to L2. The command returns a calibrated 2-D image, a source catalog file, and a segmentation map (a map where each pixel contains an integer value that corresponds to a source label number from the source catalog file).

The strun command can take both global and step-specific arguments that modify its output. For example, the following code runs the full exposure-level pipeline with a global flag that saves all outputs to a specific directory and a step-specific flag that skips the saturation detection step.

strun roman_elp r0008308002010007027_0019_wfi01_f106_uncal.asdf --output_dir='elp_results' --steps.saturation.skip=True

It is also possible to run a single pipeline step at a time. To run the dark current subtraction step with a custom  DARK reference file named custom_dark.asdf , use its specific class name in the place of the exposure-level pipeline's alias.

strun romancal.step.DarkCurrentStep r0008308002010007027_0019_wfi01_f106_uncal.asdf --override='custom_dark.asdf'

Running a single step produces an intermediate file with a name whose suffix corresponds with the step that produced it. For example, the file created by previous code snippet would be named r0008308002010007027_0019_wfi01_f106_darkcurrent.asdf . See the romancal documentation for a full list of these suffixes.

For a full accounting of available parameters, use strun 's --help flag in combination with the class name or alias of interest.

For additional questions not answered in this article, please contact the Roman Help Desk.

References

• The Roman Space Telescope Calibration Pipeline, readthedocs, maintained by Roman calibration pipeline developers. Revision bc1d143a.  Last updated on Aug 29, 2024.
• Romancal Exposure Pipeline code on GitHub, romancal maintained by Roman calibration pipeline developers at STScI. Romancal release 0.16.3 on Aug 29, 2024. 
• Sanjib Sharma and Stefano Casertano, 2024 "Cosmic Ray Jump Detection for the Roman Wide Field Instrument", PASP 136 054504
• Casertano et al 2022, "Determining the best-fitting slope and its uncertainty for up-the-ramp sampled images with unevenly distributed resultants", STScI Technical Report