Abstract¶

LSST Data Management’s verification program ensures that code meets performance specifications during construction. During operations, verification continues with an additional focus towards ensuring that data releases meet science requirements. To facilitate verification activities, we are introducing the LSST Verification Framework. This framework is implemented in the lsst.verify Python package, available at https://github.com/lsst/verify.

This technical note introduces the framework’s concepts and usage patterns through a working tutorial. First, this tutorial demonstrates how new metrics (observable concepts) and specifications (requirements and milestones that metric measurements should meet) are created. Then we measure metrics, using both a lightweight approach that is easy to retrofit into LSST Science Pipelines Tasks and a second more rigorous measurement approach that enables detailed diagnostics. Finally, this tutorial shows how metric measurements can be analyzed in a Jupyter notebook environment.

Set up¶

This technical note is available as a Jupyter notebook from its GitHub repository: https://github.com/lsst-sqre/sqr-019. You are encouraged to run and modify this notebook to help you learn about the lsst.verify package. This section covers the dependencies needed to run this notebook.

First, install the LSST Science Pipelines with lsstsw. Specifically, build and setup the verify package:

rebuild verify
setup verify

The ``verify`` package is not yet distributed with the LSST Science Pipelines Stack as of this writing. lsstsw is the most convenient means of installing ``verify`` from scratch.

Next, install these packages with Anaconda:

conda install jupyter pandas bokeh

These additional packages are used for this technote note, but are not required by lsst.verify itself.

These are the Python imports needed for this technical note:

In [1]:

# Standard library and third party packages used by this notebook
import json
import os
from tempfile import TemporaryDirectory

import astropy.units as u
import numpy as np
import yaml

# For demonstration  plots
from bokeh.charts import Scatter
from bokeh.models import Range1d, Span
from bokeh.layouts import row
from bokeh.io import show, output_notebook
from bokeh.plotting import figure
import pandas

# Load Bokeh
output_notebook()

In [2]:

# The Verification Framework itself
import lsst.verify

Introduction¶

lsst.verify is the new framework for making verification measurements in the LSST Science Pipelines. Verification is an activity where well-defined quantities, called metrics, are measured to ensure that LSST’s data and pipelines meet requirements, which we call specifications.

You might be familar with validate_drp [DMTN-008]. That package currently measures metrics of ProcessCcdTask outputs and posts results to SQUASH [SQR-009]. By tracking metric measurements we are able to understand trends in the algorithmic performance of the LSST Science Pipelines, and ultimately verify that we will meet our requirements.

With lsst.verify we sought to generalize the process of defining metrics, measuring those metrics, and tracking those measurements in SQUASH. Rather than supporting only specially-designed verification afterburner Tasks, our goal is to empower developers to track performance metrics of their own specific pipeline Tasks. By defining metrics relevant to specific Tasks, verification becomes a highly relevant integration testing activity for day-to-day pipelines development. The lsst.verify design is described in SQR-017.

This tutorial demonstrates key features and patterns in the lsst.verify framework, from defining metrics and specifications, to making measurements, to analyzing and summarizing performance.

Defining metrics¶

Metrics are definitions of measurable things that you want to track. A measureable thing could be anything: the \(\chi^2\) of a fit, the number of sources identified and measured, or even the latency or memory usage of a function.

In the verification framework, all metrics are centrally defined in the verify_metrics package. To define a new metric, simply add or modify a YAML file in the /metrics directory of verify_metrics. Each Stack package that measures metrics has its own YAML definitions file (jointcal.yaml, validate_drp.yaml, and so on).

SQUASH watches verify_metrics so that when a metric is committed to the GitHub repo is it also known to the SQUASH dashboard.

For this tutorial, we will create metrics for hypothetical demo1 and demo2 packages.

First, content for a hypothetical /metrics/demo1.yaml file:

In [3]:

demo1_metrics_yaml = """
ZeropointRMS:
  unit: mmag
  description: >
    Photometric calibration RMS.
  reference:
    url: https://example.com/PhotRMS
  tags:
    - photometry
    - demo

Completeness:
  unit: mag
  description: >
    Magnitude of the catalog's 50% completeness limit.
  reference:
    url: https://example.com/Complete
  tags:
    - photometry
    - demo
"""

This YAML defines two metrics: demo1.ZeropointRMS and demo1.Completeness. A metric consists of:

• A name. Names are prefixed by name of the package that defines them.
• A description. This helps to document metrics, even if they are more thoroughly defined in other documentation (see the reference field).
• A unit. Units are astropy.units-compatible strings. Metrics of unitless quantities should use the dimensionless_unscaled unit, an empty string.
• References. References can be made to URLs, or even to document handles and page numbers. We may expand the reference field’s schema to accomodate formalize reference identifiers in the future.
• Tags. These help us group metrics together in reports.

For the purposes of this demo, we’ll parse this YAML object into a lsst.verify.MetricSet collection. Normally this doesn’t need to be done since metrics should be pre-defined in verify_metrics, which are automatically loaded as we’ll see later.

In [4]:

with TemporaryDirectory() as temp_dir:
    demo1_metrics_path = os.path.join(temp_dir, 'demo1.yaml')
    with open(demo1_metrics_path, mode='w') as f:
        f.write(demo1_metrics_yaml)
    demo_metrics = lsst.verify.MetricSet.load_single_package(demo1_metrics_path)
demo_metrics

Out[4]:

Defining metric specifications¶

Specifications are tests of metric measurements. A specification can be thought of as a milestone; if a measurement passes a specification then data and code are working as expected.

Like metrics, specifications are usually defined centrally in the verify_metrics repository. Specifications for each package are defined in one or more YAML files in the specs subdirectory of verify_metrics. See the validate_drp directory for an example.

Here is a typical specification written in YAML; in this case for the demo1.ZeropointRMS metric:

name: "minimum"
metric: "ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"
  value: 20.0
tags:
  - "minimum"

The fully-qualified name for this specification is demo1.ZeropointRMS.minimum, following a {package}.{metric}.{spec_name} format. Specification names should be unique, but otherwise can be anything. The Verification Framework does not place special meaning on “minimum,” “design,” and “stretch” specifications. Instead, we recommend that you use tags to designate specifications with operational meaning.

The core of a specification is its test. The demo1.ZeropointRMS.minimum specification defines its test in the threshold YAML field. Here, a measurement passes the specification if \(\mathrm{measurement} \leq 20.0~\mathrm{mmag}\).

We envision other types of specifications beyond thresholds (binary comparisions). Possibilities include ranges and tolerances.

name: "minimum_megacam_r"
metric: "ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"
  value: 20.0
tags:
  - "minimum"
metadata_query:
  camera: "megacam"
  filter_name: "r"

{
  'camera': 'megacam',
  'filter_name': 'r'
  'dataset_repo': 'https://github.com/lsst/ci_cfht.git'
}

{
  'filter_name': 'i'
  'dataset_repo': 'https://github.com/lsst/ci_cfht.git'
}

In [5]:

zeropointrms_specs_yaml = """
---
name: "minimum"
metric: "ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"
  value: 20.0
tags:
  - "minimum"

---
name: "minimum_megacam_r"
base: ["ZeropointRMS.minimum"]
threshold:
  value: 15.0
metadata_query:
  camera: "megacam"
  filter_name: "r"
"""

with TemporaryDirectory() as temp_dir:
    # Write YAML to disk, emulating the verify_metrics package for this demo
    specs_dirname = os.path.join(temp_dir, 'demo1')
    os.makedirs(specs_dirname)
    demo1_specs_path = os.path.join(specs_dirname, 'zeropointRMS.yaml')
    with open(demo1_specs_path, mode='w') as f:
        f.write(zeropointrms_specs_yaml)

    # Parse the YAML into a set of Specification objects
    demo1_specs = lsst.verify.SpecificationSet.load_single_package(specs_dirname)

demo1_specs

Out[5]:

---
# Base specification: demo1.ZeropointRMS.minimum
name: "minimum"
metric: "ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"
  value: 20.0
tags:
  - "minimum"

---
# Base specification: demo1.Completeness.minimum
name: "minimum"
metric: "Completeness"
threshold:
  operator: ">="
  unit: "mag"
  value: 20.0
tags:
  - "minimum"

---
# A demo1.ZeropointRMS specification targetting MegaCam r-band
name: "minimum_megacam_r"
base: ["ZeropointRMS.minimum"]
threshold:
  value: 15.0
metadata_query:
  camera: "megacam"
  filter_name: "r"

---
# A demo1.CompletenessRMS specification targetting MegaCam r-band
name: "minimum_megacam_r"
base: ["Completeness.minimum"]
threshold:
  value: 24.0
metadata_query:
  camera: "megacam"
  filter_name: "r"

---
# Partial for MegaCam r-band specifications
id: "megacam-r"
metadata_query:
  camera: "megacam"
  filter_name: "r"

---
# Base specification: demo1.ZeropointRMS.minimum
name: "minimum"
metric: "ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"
  value: 20.0
tags:
  - "minimum"

---
# Base specification: demo1.Completeness.minimum
name: "minimum"
metric: "Completeness"
threshold:
  operator: ">="
  unit: "mag"
  value: 20.0
tags:
  - "minimum"

---
# A demo1.ZeropointRMS specification targetting MegaCam r-band
name: "minimum_megacam_r"
base: ["ZeropointRMS.minimum", "#megacam-r"]
threshold:
  value: 15.0


---
# A demo1.Completeness specification targetting MegaCam r-band
name: "minimum_megacam_r"
base: ["Completeness.minimum", "#megacam-r]
threshold:
  value: 24.0

In [6]:

demo1_specs_yaml = """
# Partials that define metadata queries
# for pipeline execution contexts with
# MegaCam r and u-band data, or HSC r-band.

---
id: "megacam-r"
metadata_query:
  camera: "megacam"
  filter_name: "r"

---
id: "megacam-u"
metadata_query:
  camera: "megacam"
  filter_name: "u"

---
id: "hsc-r"
metadata_query:
  camera: "hsc"
  filter_name: "r"

# We'll also write partials for each metric,
# that set up the basic test. Alternatively
# we could create full specifications to
# inherit from for each camera.

---
id: "ZeropointRMS"
metric: "demo1.ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"

---
id: "Completeness"
metric: "demo1.Completeness"
threshold:
  operator: ">="
  unit: "mag"

# Partials to tag specifications as
# "minimum" requirements or "stretch
# goals"
---
id: "tag-minimum"
tags:
  - "minimum"

---
id: "tag-stretch"
tags:
  - "stretch"

# ZeropointRMS specifications
# tailored for each camera, in
# minimum and stretch goal variants.

---
name: "minimum_megacam_r"
base: ["#ZeropointRMS", "#megacam-r", "#tag-minimum"]
threshold:
  value: 15.0

---
name: "stretch_megacam_r"
base: ["#ZeropointRMS", "#megacam-r", "#tag-stretch"]
threshold:
  value: 10.0

---
name: "minimum_megacam_u"
base: ["#ZeropointRMS", "#megacam-u", "#tag-minimum"]
threshold:
  value: 30.0

---
name: "stretch_megacam_u"
base: ["#ZeropointRMS", "#megacam-u", "#tag-stretch"]
threshold:
  value: 20.0

---
name: "minimum_hsc_r"
base: ["#ZeropointRMS", "#hsc-r", "#tag-minimum"]
threshold:
  value: 12.0

---
name: "stretch_hsc_r"
base: ["#ZeropointRMS", "#hsc-r", "#tag-stretch"]
threshold:
  value: 6.0

# Competeness specifications,
# tailored for each camera in
# minimum and stretch goal variants

---
name: "minimum_megacam_r"
base: ["#Completeness", "#megacam-r", "#tag-minimum"]
threshold:
  value: 24.0

---
name: "stretch_megacam_r"
base: ["#Completeness", "#megacam-r", "#tag-stretch"]
threshold:
  value: 26.0

---
name: "minimum_megacam_u"
base: ["#Completeness", "#megacam-u", "#tag-minimum"]
threshold:
  value: 20.0

---
name: "stretch_megacam_u"
base: ["#Completeness", "#megacam-u", "#tag-stretch"]
threshold:
  value: 24.0

---
name: "minimum_hsc_r"
base: ["#Completeness", "#hsc-r", "#tag-minimum"]
threshold:
  value: 20.0

---
name: "stretch_hsc_r"
base: ["#Completeness", "#hsc-r", "#tag-stretch"]
threshold:
  value: 28.0
"""

with TemporaryDirectory() as temp_dir:
    # Write YAML to disk, emulating the verify_metrics package for this demo
    specs_dirname = os.path.join(temp_dir, 'demo1')
    os.makedirs(specs_dirname)
    demo1_specs_path = os.path.join(specs_dirname, 'demo1.yaml')
    with open(demo1_specs_path, mode='w') as f:
        f.write(demo1_specs_yaml)

    # Parse the YAML into a set of Specification objects
    demo_specs = lsst.verify.SpecificationSet.load_single_package(specs_dirname)

demo_specs

Out[6]:

In [7]:

sourcecount_metric = lsst.verify.Metric(
    'demo2.SourceCount',
    "Number of matched sources.",
    unit=u.dimensionless_unscaled,
    tags=['demo'])
demo_metrics.insert(sourcecount_metric)

print(demo_metrics['demo2.SourceCount'])

demo2.SourceCount (dimensionless_unscaled): Number of matched sources.

In [8]:

u.dimensionless_unscaled == u.Unit('')

Out[8]:

True

In [9]:

sourcecount_minimum_spec = lsst.verify.ThresholdSpecification(
    'demo2.SourceCount.minimum_cfht_r',
    250 * u.dimensionless_unscaled,
    '>=',
    tags=['minimum'],
    metadata_query={'camera': 'megacam', 'filter_name': 'r'}
)
demo_specs.insert(sourcecount_minimum_spec)

sourcecount_stretch_spec = lsst.verify.ThresholdSpecification(
    'demo2.SourceCount.stretch_cfht_r',
    500 * u.dimensionless_unscaled,
    '>=',
    tags=['stretch'],
    metadata_query={'camera': 'megacam', 'filter_name': 'r'}
)
demo_specs.insert(sourcecount_stretch_spec)

In [10]:

demo_metrics

Out[10]:

In [11]:

demo_specs

Out[11]:

Making measurements¶

Now that we’ve defined metrics, we can measure them. Measurements happen in Pipelines code, either within regular Tasks, or in dedicated afterburner Tasks.

The Verification Framework provides two patterns for making measurements: either using the full measurement API, or a more lightweight capture of measurement quantities. For the demo1 package we’ll use the more comprehensive approach, and then make lightweight measurements for the demo2 package.

In [12]:

catalog_mags = np.random.uniform(18, 26, size=100)*u.mag

obs_mags = catalog_mags - 25*u.mag + np.random.normal(scale=12.0, size=100)*u.mmag

In [13]:

zp = np.median(catalog_mags - obs_mags)

In [14]:

zp_rms = np.std(catalog_mags - obs_mags)

In [15]:

zp_meas = lsst.verify.Measurement('demo1.ZeropointRMS', zp_rms)

In [16]:

zp_meas.extras['zp'] = lsst.verify.Datum(zp, label="m_0",
                                         description="Estimated zeropoint.")
zp_meas.extras['catalog_mags'] = lsst.verify.Datum(catalog_mags, label="m_cat",
                                                   description="Catalog magnitudes.")
zp_meas.extras['obs_mags'] = lsst.verify.Datum(obs_mags, label="m_obs",
                                               description="Instrument magnitudes.")

In [17]:

zp_meas.notes['estimator'] = 'numpy.std'

In [18]:

# Here's a mock dataset
mag_grid = np.linspace(22, 28, num=50, endpoint=True)
c_percent = 1. / np.cosh((mag_grid - mag_grid.min()) / 2.) * 100.

# Make the measurement
completeness_mag = np.interp(50.0, c_percent[::-1], mag_grid[::-1]) * u.mag

# Package the measurement
completeness_meas = lsst.verify.Measurement(
    'demo1.Completeness',
    completeness_mag,
)
completeness_meas.extras['mag_grid'] = lsst.verify.Datum(
    mag_grid*u.mag,
    label="m",
    description="Magnitude")
completeness_meas.extras['c_frac'] = lsst.verify.Datum(
    c_percent*u.percent,
    label="C",
    description="Photometric catalog completeness.")

Packaging measurements in a Verification Job¶

In the Verification Framework, a “job” is a pipeline run that produces metric measurements. The lsst.verify.Job class allows us to package several measurements from the pipeline run. With a Job object, we can then analyze the measurements, save verification datasets to disk, and dispatch datasets to the SQUASH database.

Normally when we create a Job object from scratch we seed it with the metrics and specifications defined in the verify_metrics repo:

In [19]:

job = lsst.verify.Job.load_metrics_package()

Of course, we created ad hoc metrics and specifications outside of verify_metrics. We can add those to the job:

In [20]:

job.metrics.update(demo_metrics)
job.specs.update(demo_specs)

Now add the measurements:

In [21]:

job.measurements.insert(zp_meas)
job.measurements.insert(completeness_meas)

The pipeline Tasks that is making this Job knows about the camera and filter of the dataset. The Task code can record this metadata:

In [22]:

job.meta.update({'camera': 'megacam', 'filter_name': 'r'})

Job metadata is a dict-like mapping. Here’s the full set of metadata recorded for the job:

In [23]:

print(job.meta)

{
    "camera": "megacam",
    "demo1.ZeropointRMS.estimator": "numpy.std",
    "filter_name": "r"
}

As expected, the camera and filter_name is present, but so is the estimator annotation that we attached to the demo1.ZeropointRMS measurement. Measurement annotations are automatically included in a Job’s metadata, but keys are prefixed with the measurement’s metric name. Specification metadata_query definitions can act on both job and measurement-level metadata.

Before a Task exits, it should write the verification Job dataset to disk. Serialization to disk is a temporary shim until Job datasets can be persisted through the Butler.

The native serialization format of the Verification Framework is JSON:

In [24]:

job.write('demo1.verify.json')

Making lightweight quantity-only measurements with output_quantities()¶

lsst.verify.Measurement and lsst.verify.Job classes are necessary for producing rich job datasets (for example, associating extras with measurements). Many Tasks, though, won’t need this functionality. A Task might record a measurement as an Astropy quantity and persist that measurement with as little overhead as possible. The lsst.verify.output_quantities function enables this use case.

First, a Task will create a dictionary to collect measurements throughout the lifetime of the Task’s execution:

In [25]:

demo2_measurements = {}

Then the task measures the demo2.SourceCount metric:

In [26]:

demo2_measurements['demo2.SourceCount'] = 350*u.dimensionless_unscaled

Measurements are always Astropy quantities.

Finally, before the Task returns, it can output measurements to disk. The default filename format for the Verification job dataset file is {package}.verify.json.

In [27]:

lsst.verify.output_quantities('demo2', demo2_measurements)

Out[27]:

'demo2.verify.json'

Post processing verification jobs¶

Our hypothetical pipeline has produced measurements for two packages: demo1 and demo2. These measurements are persisted to demo1.verify.json and demo2.verify.json files on disk. Now we’d like to gather these measurements and either submit them to the SQUASH dashboard, or collate the measurements for local analysis.

The dispatch_verify.py tool lets us do this. For this demo we won’t upload measurements to SQUASH. Instead we will combine the mesurements into one JSON file.

In [28]:

%%bash
export DYLD_LIBRARY_PATH=$LSST_LIBRARY_PATH
dispatch_verify.py --test --ignore-lsstsw --write demo.verify.json demo1.verify.json demo2.verify.json

verify.bin.dispatchverify.main INFO: Loading demo1.verify.json
verify.bin.dispatchverify.main INFO: Loading demo2.verify.json
verify.bin.dispatchverify.main INFO: Merging verification Job JSON.
verify.bin.dispatchverify.main INFO: Refreshing metric definitions from verify_metrics
verify.bin.dispatchverify.main INFO: Writing Job JSON to demo.verify.json.

The flags used here are:

• --test: prevents dispatch_verify.py from attempting to upload to the SQUASH service.
• --ignore-lsstsw: since the $LSSTSW environment variable may not be available in this notebook context, we’ll avoid scraping it for information (such as Git commits and branches of packages included in the Pipeline stack).
• --write demo.verify.json: Write the merged job dataset to demo.verify.json.
• demo1.verify.json and demo2.verify.json are inputs, as positional arguments, pointing to the job JSON files that we created earlier with metric measurements.

See dispatch_verify.py --help for more information.

Analyze verification results locally¶

For code development, it’s convenient to look at the results of verification measurements locally, rather than in SQUASH. The Verification Framework is designed for this workflow, with special affordances for Jupyter Notebook users.

The collated measurement dataset produced by dispatch_verify.py earlier is in the file demo.verify.json. Let’s open this dataset using the Job.deserialize class method:

In [29]:

with open('demo.verify.json') as f:
    job = lsst.verify.Job.deserialize(**json.load(f))

In [30]:

job.report().show()

Out[30]:

In [31]:

print(job.meta)

{
    "camera": "megacam",
    "demo1.ZeropointRMS.estimator": "numpy.std",
    "filter_name": "r",
    "packages": {}
}

In [32]:

job.report(name='demo1').show()

Out[32]:

In [33]:

job.report(name='demo1.ZeropointRMS').show()

Out[33]:

In [34]:

job.specs['demo1.ZeropointRMS.minimum_megacam_r'].tags

Out[34]:

{'minimum'}

In [35]:

job.report(spec_tags=['minimum']).show()

Out[35]:

In [36]:

job.report(spec_tags=['minimum', 'stretch']).show()

Out[36]:

In [37]:

job.report(name='demo1', spec_tags=['minimum']).show()

Out[37]:

In [38]:

m = job.measurements['demo1.ZeropointRMS']

In [39]:

list(m.extras.keys())

Out[39]:

['zp', 'obs_mags', 'catalog_mags']

In [40]:

df = pandas.DataFrame({"obs_mags": m.extras['obs_mags'].quantity,
                       "catalog_mags": m.extras['catalog_mags'].quantity,
                       "delta_mags": m.extras['catalog_mags'].quantity - m.extras['obs_mags'].quantity})

In [41]:

# Scatter plot of observed vs. catalog stellar photometry
p = Scatter(df, x='obs_mags', y='catalog_mags',
            title="Zeropoint stellar sample",
            xlabel="{0.label} [{0.unit}]".format(m.extras['obs_mags']),
            ylabel="{0.label} [{0.unit}]".format(m.extras['catalog_mags']),
            plot_width=350, plot_height=350)

In [42]:

# Histogram of zeropoint estimates from individual matched stars.
# We're not using the Histogram Bokeh chart for some extra control.
hist_counts, hist_edges = np.histogram(df['delta_mags'], bins=10)
h = figure(tools="xpan, xwheel_zoom, reset",
           active_scroll="xwheel_zoom",
           y_range=(0, hist_counts.max()+2),
           y_axis_label="Count",
           x_axis_label="{0.label} - {1.label} [{0.unit}]".format(m.extras['obs_mags'], m.extras['catalog_mags']),
           plot_width=350, plot_height=350)
# Draw histogram edges on the figure
h.quad(bottom=0,
       left=hist_edges[:-1],
       right=hist_edges[1:],
       top=hist_counts,
       color="lightblue",
       line_color="#3A5785")
# Line at zeropoint estimate
span = Span(location=m.extras['zp'].quantity.value,
            dimension='height', line_color="black",
            line_dash='dashed', line_width=3)
h.add_layout(span)

In [43]:

# Plot side-by-side
show(row(p, h), notebook_handle=True)

Out[43]:

Summary and outlook¶

This technical note has demonstrated the full usage cycle of lsst.verify:

• Defining metrics.
• Defining specifications of metrics.
• Measuring metrics.
• Associating extra datasets with measurements.
• Analyzing verification pipeline jobs, including building pass/fail reports and making plots.

We encourage Data Management engineers and scientists to consider how you might instrument your own code, particularly pipeline Tasks, with verification measurements. By systematically monitoring performance metrics in your code, you will gain a clearer picture of how code development is affecting your systems.

This technical note has only shown local usage patterns with the lsst.verify framework. We are integrating lsst.verify with the SQUASH dashboard application. With SQUASH, your metric measurements are centrally available to the whole organization. We believe that lsst.verify and SQUASH will become an everday service for DM developers to ensure that code contributions do not introduce adverse performance side-effects across the Stack.

References¶

Astropy Collaboration et al (2013). Astropy: A community Python package for astronomy. A&A, 558, A33, 10.1051/0004-6361/201322068.

Fausti, Angelo (2016). SQUASH dashboard prototype. SQuaRE Technical Note SQR-009. https://sqr-009.lsst.io.

Ivezić, Željko, and The LSST Science Collaboration (2011). LSST Science Requirements Document. LPM-17. https://ls.st/LPM-17.

Parejko, John and Sick, Jonathan (2017). Validation Metrics Framework. SQuaRE Technical Note SQR-017. https://sqr-017.lsst.io.

Wood-Vasey, Michael (2016). Introducing validate_drp: Calculate SRD Key Performance Metrics for an output repository. Data Management Technical Note DMTN-008. https://dmtn-008.lsst.io.

In [ ]: