This article contains a high-level overview of the science data pipelines for the Roman Wide Field Instrument (WFI) imaging data processing at STScI including design, philosophy, and installation instructions.  

Overview of WFI Pipelines at STScI

WFI imaging observations are processed through several pipelines to create different data products. At STScI, we are developing three pipelines:

• Exposure Level Pipeline: Performs detector-level calibration of Level 1 to Level 2 WFI data products.
• Mosaic Level Pipeline: Handles re-pixelation and mosaicking of Level 2 WFI imaging data into Level 3 products.
• Catalog-Level Pipeline: Generates Level 4 catalogs from Level 2 and Level 3 products (currently under development).

The romancal repository undergoes continuous integration testing using both unit tests and larger regression test suites to ensure that changes to the code do not result in unexpected changes of the products. Furthermore, before WFI pipeline steps are released, they are rigorously tested and validated by the engineers and instrument scientists at STScI.

Installation Instructions

All of the STScI pipelines for Roman are contained in a single Python package called romancal that is publicly developed on GitHub with released versions available via the Python Package Index (PyPI). 

Additional information on how to install specific versions, including the latest development version, can be found on the pipeline installation page of the romancal readthedocs documentation. Basic installation on a Unix-based operating system using a conda environment manager can be accomplished in a bash terminal by typing the following:

$ conda create -n <environment_name> python
$ conda activate <environment_name>
$ pip install romancal 

Note that the $ symbol indicates the bash prompt. The variable environment_name is at the discretion of the user. By indicating the argument "python" during the environment creation, the latest available version of Python will be installed in the environment along with other necessary tools such as pip.

Installing  romancal will install several other dependency packages including but not limited to:

• roman_datamodels
• asdf
• crds

Users will also need access to calibration reference files for some pipeline steps and should see the CRDS for Reference Files article for additional information, including how to set up necessary environment variables.

Pipeline Descriptions

Here, we provide a high-level description of the individual pipelines used to produce Roman WFI data products. Detailed information about each pipeline is provided in separate articles. Users are also advised to consult the WFI Data Levels and Products page for information on the formats and contents of the different WFI data products.

The WFI detectors are an updated version of the detectors used in JWST instruments; therefore, the philosophical starting point for the development of the WFI data pipelines is the JWST science data pipeline. Deviations from the JWST pipelines occur when either the JWST pipeline steps are not appropriate or insufficient for WFI data, or when Roman mission science accuracy requirements necessitate changes to the underlying algorithms.

Level 2 Exposure Pipeline

The Level 2 exposure pipeline contains the algorithms necessary to correct raw WFI ramps for instrumental effects, and collapse the ramps along the time axis into rate images suitable for scientific analysis. The exposure pipeline corrects for the following instrumental effects:

• Signal induced by the readout electronics (e.g., 1/f noise) using reference pixels
• Dark current
• Classic non-linearity
• Flat-field (variations in quantum efficiency)

In addition, static bad pixels and pixels with poor calibration from the calibration reference files are flagged in the data quality arrays. Rows that intersect the guide window on each detector are flagged in the data quality arrays due to changes in the noise properties of the intersecting rows. Finally, the following steps are performed:

• A slope per pixel is fit up the ramp to produce a count rate per pixel
• A WCS model, including the geometric distortion, for transformation from pixels to sky coordinates (and the inverse) is added to the metadata
• Photometric calibration information including zeropoints and nominal pixel area are added to the metadata
• Alignment to Gaia astrometric sources is performed to update the WCS model

Note that the input Level 1 files to the exposure pipeline are separated per WFI detector (i.e., there are 18 files for a full WFI exposure), and, similarly, the output Level 2 files from the exposure pipeline are also separated per detector.

Please see the Exposure Level Pipeline article for more information.

Level 3 Mosaic Pipeline

The Level 3 mosaic pipelines contains the modules necessary to combine the calibrated level 2 data into mosaic images. This pipeline will:

• Convert the data into MegaJansky/steradian (MJy/sr);
• Apply a background correction so that contiguous mosaics have matching backgrounds;
• Perform outlier detection;
• Combine and resample the input images into a mosaic Level 3 output. 

This pipeline is where the 18 WFI detectors are combined into a single image, as well as where multiple WFI FOVs are combined for greater depth and/or size. 

Please see the Mosaic Level Pipeline article for more information.

Level 4 Catalog Pipeline

Overview of WFI Pipelines at IPAC

WFI imaging observations are processed through several pipelines to create different data products. At IPAC, we are developing five sets of pipelines:

Wide Field Spectroscopy Pipelines

• Science Data Pipelines: Process all WFI spectroscopic data, produce Level 4 grism and prism science data products, and deliver these data products, along with information on data quality, to the Roman archive.
• Calibration Data Pipeline: Produces the spectroscopic calibration reference files necessary for the spectroscopic Science Data Pipeline to process the Level 2 grism/prism data into the Level 4 spectroscopy data products.

Galactic Bulge Survey Pipelines

• MSOS Photometry Pipelines: Generate a Level 3 Reference Frame Product consisting of supersampled images, together with three distinct types of Level 4 catalogs: Fiducial, Periodic, and Light Curve catalogs.
• MSOS Event Pipelines: Generate identification and characterization of microlensing events via event identification, lensing model classification and physical parameter determination.
• MSOS Detection Efficiency Pipeline: Evaluate the completeness (false negative rate or FN rate) and reliability (false positive rate or FP rate) of microlensing events, in particular of planetary events.

Pipeline Descriptions

Here, we provide a high-level description of the individual pipelines used to produce higher-level products from basic Roman WFI data. Detailed information about each pipeline is provided in separate articles. Users are also advised to consult the WFI Data Levels and Products page for information on the formats and contents of the different WFI data products.

Spectroscopic Science Data Pipeline

The Science Data Pipelines (SDP) are made up of two parts. The first part, the G2DP, is designed to calibrate the 2D grism and prism science data and produce 1D spectra for each identified source. The second part, the G1DP, is designed to analyze and extract basic information from the 1D spectra, namely, redshifts and emission/absorption line parameters, for each target.

The basic steps of the SDP are:

• Identification and position measurement of all spectra
• Spectral flat fielding
• Background identification and subtraction
• Identification and removal of contaminating sources, producing decontaminated 2D spectral cutouts
• Extraction of 1D spectra from the 2D cutouts
• Relative and absolute flux calibration of the 1D spectra
• Combination of 1D spectra from different exposures, dithers, and/or rolls
• Fitting of the 1D spectra to produce redshifts and other spectral feature parameters.

Level 3 and Level 4 image data products are inputs to the SDP and are used in conjunction with the Level 2 spectroscopic data to identify and extract sources from the grism and prism data. Calibration reference files produced in the Calibration Data Pipeline (CDP) are used as inputs to the SDP to calibrate the grism and prism science data.

Please see the Wide Field Spectroscopy Science Data Pipelines article for more information.

Spectroscopic Calibration Data Pipeline

The Calibration Data Pipeline (CDP) produces the spectroscopic calibration reference files necessary for the spectroscopic Science Data Pipelines (SDP) to process the Level 2 grism/prism data into the Level 4 spectroscopy data products.

The spectroscopic calibration reference files produced by the CDP are:

• The optical model (which includes direct-to-dispersed mapping, trace & wavelength solution)
• The pixel-level ("small-scale") flat-field
• Relative flux calibration
• Absolute flux calibration
• Spectral point-spread function (PSF)

Please see the Wide Field Spectroscopy Calibration Data Pipeline article for more information.

MSOS Photometry Pipelines

The Microlensing Science Operations System (MSOS) Photometry Pipelines will generate three distinct types of Level 4 catalogs (Fiducial, Periodic, and Light Curve catalogs), together with a Level 3 Reference Frame Product consisting of supersampled images.

Brief descriptions of each are provided below:

• Reference Frame Product – supersampled image product used to establish the common astrometric and photometric reference frame for downstream processing and catalog generation.
• Fiducial Catalog – master catalog containing all detected objects, with reference positions standardized to a common epoch; provides aggregate positional and photometric properties for each object and serves as the basis for cross-object comparisons; updated periodically using data from the corresponding Periodic Catalog.
• Periodic Catalog – contains per-object measurements based on approximately a week worth of data; derived from reference frame information constructed over each period, and used to monitor changes in position and brightness; averaging is applied to achieve sufficient signal-to-noise for reliable measurements.
• Lightcurve Catalog – contains individual, unaveraged photometric measurements; enables reconstruction of object light curves and analysis of variability on short timescales.

Please see the MSOS Photometry Pipelines article for more information.

MSOS Event Pipelines

The purpose of this MSOS pipeline component is the identification and parameter characterization of microlensing events.

The MSOS Event Pipeline consists of three main components:

• Microlensing Event Identification, using data from the Light Curve and Object Catalogs.
• Lensing Model Classification, into single-lens or binary-lens systems.
• Physical Parameter Determination, based on model fits and external datasets.

Please see the Galactic Bulge Survey MSOS Event Pipelines article for more information.

MSOS Detection Efficiency Pipeline

This automated MSOS pipeline is for the image analysis and the subsequent identification and characterization of microlensing events, in particular of planetary events.

• Simulated microlensing events are injected into Level 2 images or into light curves and processed through the MSOS photometry and modeling pipelines.
• The output of the pipeline is compared with the input and the false positive (FP) and false negative (FN) rates are recorded; all input and output properties are saved into a catalog for the community to access.
• A catalog containing all input and output parameters of the Rhie method analysis will be provided to the community after the end of the Galactic Bulge Time Domain Survey (GBTDS).

Please see the Galactic Bulge Survey MSOS Detection Efficiency Pipeline article for more information.

Automatic Data Processing

As WFI data are downlinked from the Roman spacecraft, they are automatically processed through several data pipelines with only a few variations depending on the observation type. For example, the exposure level pipeline does not apply a flat-field correction to WFI spectroscopic observations, as the Science Support Center applies a wavelength-dependent flat-field correction in the spectroscopic pipeline. Wherever present, these special cases are described in detail in the articles dedicated to each of the pipelines. After the data products are generated, they are ingested into MAST and immediately made available to the community with no proprietary exclusive access period. See Accessing WFI Data for more information on how to retrieve WFI data products.

For additional questions not answered in this article, please contact the Roman Help Desk.

References

1. The Roman Space Telescope Calibration Pipeline, Readthedocs maintained by STScI 2023, Latest version
2. JWST Science Calibration Pipeline Overview, Last Update 29 Nov 2022, JWST User Documentation (JDox)