Estimating Resource Usage and Costs
This page provides guidance on estimating the computing and storage resources required for scientific analyses on the Roman Research Nexus and translating these requirements into estimated Nexus credit consumption. It includes practical examples illustrating how to account for workflow complexity, repeated analyses, data storage, and data egress when applicable. Users are encouraged to consult the Computational Benchmarks page for measured CPU and memory usage for representative scientific workflows, as well as the Compute Resources and Storage Resources pages for current resource configurations and credit consumption rates.
Designing Resource-Efficient Workflows
When planning scientific analyses on the Roman Research Nexus, users are encouraged to make use of existing SOC- and SSC-produced data products available through the MAST archive, whenever these meet their scientific requirements. This can significantly reduce compute and storage requirements by avoiding unnecessary data reprocessing.
Data products can also be accessed directly from MAST as needed, without downloading and storing complete datasets within the Nexus. This approach, known as data streaming, is particularly useful when working with large datasets.
Users should also consider the expected scale of their analyses, opportunities to reuse intermediate results, and potential data egress costs when estimating their resource needs.
Example: Estimating Resources for a Galaxy Analysis Workflow
The following example illustrates how the results presented in Computational Benchmarks can be used to estimate the resources required for a scientific analysis. It considers a representative catalog-based workflow, from source selection to detailed galaxy characterization, and explores how different analysis methods affect computational requirements.
The example also accounts for repeated analyses, storage requirements, and credit consumption over the lifetime of a scientific project.
Scientific Workflow
Consider a scientific program that uses existing Roman data products to identify and characterize a population of galaxies. The analysis begins with multiple single-band source catalogs produced as prompt products and identifies approximately 3,000 candidate galaxies for further investigation.
The workflow includes:
Catalog cross-matching: Cross-match single-band source catalogs from different filters to construct a multi-band catalog and identify candidate galaxies for further analysis.
Image cutouts: Generate cutouts from Level 2 (L2) rate image and Level 3 (L3) mosaic imaging data products for the selected candidates.
Photometric verification: Perform additional photometric measurements to verify detections and identify potential contamination.
Structural measurements: Fit Sérsic profiles to characterize galaxy morphology.
SED analysis: Fit multi-band photometry to estimate redshifts and other source properties, using either template-based or Bayesian methods.
For this example, we assume eight filters, three L2 exposures and one L3 mosaic per filter, and one Sérsic fit per candidate per filter.
Estimating Computational Requirements
The computational requirements of each task can be estimated by scaling the measured CPU-hours reported in the Computational Benchmarks to the expected number of executions.
For this example, we adopt the following assumptions:
Catalog cross-matching: We assume eight single-band source catalogs, each containing approximately 10,000 sources. Using one catalog as the reference and matching it against the remaining seven requires 7 cross-matching operations involving 10,000 sources each. The resulting multi-band catalog is used to identify 3,000 candidate galaxies for further analysis.
Image cutouts: For each of the 3,000 selected galaxies, we generate one 100 × 100 pixel cutout from each of three L2 exposures and one L3 mosaic, across eight filters. This corresponds to 3,000 × 8 × (3 + 1) = 96,000 cutouts.
Aperture photometry: We assume one additional photometric measurement per galaxy per filter, corresponding to 3,000 × 8 = 24,000 measurements.
Sérsic fitting: We perform one structural fit per galaxy per filter, corresponding to 3,000 × 8 = 24,000 fits.
SED fitting: We construct one multi-band SED per galaxy and consider two fitting methods:
Template-based fitting: One fit per galaxy, assuming 10 CPU-seconds per fit, for 3,000 fits.
Bayesian fitting: One fit per galaxy, assuming 5 CPU-minutes per fit, for 3,000 fits.
The SED-fitting execution times are illustrative assumptions rather than measured Nexus benchmarks. Such assumptions are sometimes necessary and can be informed by experience or by testing on small datasets.
CPU-Hour Estimates by Analysis Task
| Analysis task | Number of operations | Estimated Total CPU-hours |
|---|---|---|
| Catalog cross-matching | 7 × 10,000-source matches | 0.0243 |
| Image cutouts | 96,000 cutouts | 0.9571 |
| Aperture photometry | 24,000 measurements | 0.0006 |
| Sérsic fitting | 24,000 fits | 17.0160 |
| Template-based SED fitting | 3,000 fits | 8.3333 |
| Bayesian SED fitting | 3,000 fits | 250.0000 |
| Sum | 276.33 |
Note: These estimates are extrapolations from individual computational benchmarks and assume that resource usage scales linearly with the number of operations. Actual performance may vary depending on workflow implementation, parallelization, data access patterns, and server configuration.
For a more accurate estimate, users are encouraged to develop and execute a representative sandbox workflow on the Roman Research Nexus, measure its resource consumption and wall-clock execution time, and extrapolate those results to the full analysis.
Estimating Compute Credit Consumption
Nexus compute credits are consumed based on the time a server remains active, rather than the total CPU-hours used by an analysis. Consequently, the same scientific workflow may have different credit costs depending on how its operations are executed. To estimate credit consumption, we first translate the computational requirements into wall-clock execution time using the measurements reported in the Computational Benchmarks.
For each task, we scale the measured wall-clock time to the expected number of operations:
| T_{\mathrm{task}} = T_{\mathrm{benchmark}} \times \frac{N_{\mathrm{task}}}{N_{\mathrm{benchmark}}} |
The benchmarks were performed on Small Nexus servers (2 vCPUs). For this example, we consider executing the workflow on a Medium server (8 vCPUs, 64 GB RAM), consuming 0.504 credits per hour, and compare two scenarios:
Serial execution: Operations are executed sequentially, assuming execution times comparable to those measured on the Small server.
Parallel execution: Independent operations are parallelized, assuming ideal scaling with the increase in available vCPUs from 2 to 8, corresponding to a 4× reduction in wall-clock time for parallelizable tasks.
These scenarios illustrate how execution strategy can affect credit consumption; actual performance will depend on the workflow implementation.
Scenario 1: Serial Execution
In this scenario, all operations are executed sequentially, without taking advantage of the additional vCPUs available on the Medium server. We assume that individual execution times remain comparable to those measured on the Small server.
Wall-Clock Estimates — Serial Execution
| Analysis task | Number of operations | Benchmark wall-clock time | Estimated total wall-clock time |
|---|---|---|---|
| Catalog cross-matching | 7 × 10,000-source matches | 18.25 s / 10,000 matches | 0.035 h |
| Image cutouts | 96,000 cutouts | 5.11 s / 100 cutouts | 1.363 h |
| Aperture photometry | 24,000 measurements | 0.96 s / 10,000 measurements | 0.00064 h |
| Sérsic fitting | 24,000 fits | 2.5 s / fit | 16.667 h |
| Template-based SED fitting | 3,000 fits | 10 s / fit* | 8.333 h |
| Bayesian SED fitting | 3,000 fits | 300 s / fit* | 250.000 h |
| Total | 276.40 h |
*SED-fitting execution times are illustrative assumptions, not measured Nexus benchmarks
The estimated credit consumption for one complete execution is calculated by multiplying the total wall-clock time by the Medium server credit rate:
276.40 hours × 0.504 credits/hour = 139.3 credits
This represents a conservative estimate in which the workflow is executed sequentially, without exploiting the additional parallel processing capacity of the Medium server.
Scenario 2: Parallel Execution
In this scenario, independent operations are distributed across multiple CPUs on the Medium server. We assume ideal parallel scaling relative to the Small-server benchmarks, corresponding to a 4× reduction in wall-clock time for parallelizable tasks, reflecting the increase from 2 to 8 vCPUs. For illustration, we assume that image cutout generation, aperture photometry, Sérsic fitting, and SED fitting can be parallelized, while catalog cross-matching remains sequential.
Wall-Clock Estimates — Parallel Execution
| Analysis task | Estimated wall-clock time |
|---|---|
| Catalog cross-matching | 0.035 h |
| Image cutouts | 0.341 h |
| Aperture photometry | 0.00016 h |
| Sérsic fitting | 4.167 h |
| Template-based SED fitting | 2.083 h |
| Bayesian SED fitting | 62.500 h |
| Total | 69.13 h |
The estimated credit consumption for one complete execution is:
69.13 hours × 0.504 credits/hour = 34.8 credits
This represents an idealized parallel-execution scenario rather than measured Medium-server performance. Actual execution times will depend on workflow implementation, parallelization efficiency, memory requirements, and data access patterns.
Estimating Compute Costs Over the Project Lifetime
Over a three-year project, compute consumption includes both interactive analysis and production processing. Interactive usage covers workflow development, testing, exploratory analysis, and partial processing of the dataset.
For this example, we assume:
Production processing: Two complete workflow executions, plus an additional 0.25 workflow equivalent for repeated analyses.
Interactive usage: 200 hours in Year 1, 100 hours in Year 2, and 50 hours in Year 3, for a total of 350 hours on a Medium server.
The interactive usage corresponds to 176.4 credits, while the 2.25 production workflow equivalents require approximately 313.4 credits for serial execution or 78.4 credits for parallel execution.
The estimated total compute consumption over three years is therefore approximately 490 credits for serial execution or 255 credits for parallel execution.
These estimates assume that servers are active only during processing or interactive use and that partial analyses performed during development are not counted again as production executions.
Estimating Storage Costs
In addition to compute resources, scientific workflows may require persistent storage for catalogs, intermediate data products, analysis results, and files used during interactive development.
The Roman Research Nexus provides EFS storage for shared working files and S3 object storage for larger datasets and retained analysis products. Since the Roman data products used in this example are already available through MAST, the workflow does not require maintaining local copies of the original imaging dataset.
For this three-year project, we assume average storage requirements of:
50 GB of EFS Standard for notebooks, scripts, catalogs, and working files.
0.5 TB of S3 Standard for generated cutouts, derived data products, and retained analysis results.
Using the approximate Nexus storage rates:
EFS: 0.05 TB × 279 credits/TB-month × 36 months = 502.2 credits
S3: 0.5 TB × 10 credits/TB-month × 36 months = 180 credits
The estimated total storage consumption is therefore approximately 682 credits over three years.
These estimates assume constant average storage usage throughout the project. Actual costs will depend on the volume of generated products, how storage requirements evolve, and how long intermediate files are retained.
Estimating EFS I/O Costs
In addition to storage capacity, EFS incurs costs associated with reading and writing data. For this example, we assume that the scientific workflow generates EFS I/O during interactive analysis, catalog processing, and the creation of intermediate data products.
We adopt the following illustrative assumptions over the three-year project:
1 TB of EFS reads, including repeated access to catalogs and intermediate analysis products.
0.5 TB of EFS writes, including temporary files, catalogs, and analysis outputs.
Using the EFS Elastic Throughput rates of 0.03 credits/GB for reads and 0.06 credits/GB for writes:
Reads: 1,000 GB × 0.03 credits/GB = 30 credits
Writes: 500 GB × 0.06 credits/GB = 30 credits
The estimated EFS I/O consumption is therefore 60 credits over three years.
These volumes are illustrative assumptions rather than measurements of the complete workflow. Actual consumption will depend on data access patterns, intermediate processing, and the number of file operations performed.
Estimating Data Egress Costs
Data egress refers to transferring data from the Nexus to external locations, such as a user's local computer, while data ingress refers to transferring data from an external location to the Nexus. Transferring data into the Nexus does not incur charges, while exporting data is charged at 84 credits/TB.
For this example, we assume that all 96,000 generated image cutouts are exported once. Each cutout contains 100 × 100 pixels, stored using 4 bytes per pixel.
96,000 × 100 × 100 × 4 bytes = 3.84 GB
The corresponding egress cost is:
0.00384 TB × 84 credits/TB = 0.32 credits
This estimate considers only raw image pixel data and excludes additional file overhead, metadata, and other exported products. Actual egress costs will depend on the volume of data transferred outside the Nexus.
Three-Year Resource Cost Estimate
Combining compute, storage, EFS I/O, and egress, we can estimate the total Nexus credits required for the three-year scientific project.
Three-Year Nexus Credit Estimate
| Resource | Serial execution | Parallel execution |
|---|---|---|
| Compute (production and interactive) | 489.8 | 254.8 |
| EFS Standard (50 GB) | 502.2 | 502.2 |
| S3 Standard (0.5 TB) | 180.0 | 180.0 |
| EFS I/O (1 TB reads, 0.5 TB writes) | 60.0 | 60.0 |
| Data egress (3.84 GB) | 0.3 | 0.3 |
| Total credits | 1,232.3 | 997.3 |
The estimated resource consumption is therefore approximately 1,000–1,230 credits over three years, depending on the degree of parallelization achieved. These estimates illustrate how compute, storage, EFS I/O, and data egress contribute to the overall Nexus credit budget. In this example, persistent storage represents a substantial fraction of the total cost, while EFS I/O and data egress contribute relatively little.
The estimates are based on representative benchmarks and illustrative assumptions rather than measurements of the complete scientific workflow. Actual resource consumption will depend on workflow implementation, data access patterns, storage retention, and parallelization efficiency.
Users are encouraged to execute a representative analysis on the Roman Research Nexus, measure its resource consumption, and refine their estimates accordingly.
For additional questions not answered in this article, please contact the Roman Help Desk.
