Introduction¶

lsst.verify is the new framework for making verification measurements in the LSST Science Pipelines. Verification is an activity where well-defined quantities, called metrics, are measured to ensure that LSST’s data and pipelines meet requirements, which we call specifications.

You might be familar with validate_drp. That package that currently measures metrics of ProcessCcdTask outputs and posts results to SQUASH. By tracking metric measurements we are able to understand trends in the algorithmic performance of the LSST Science Pipelines, and ultimately verify that we will meet our requirements.

With lsst.verify we sought to generalize the process of defining metrics, measuring those metrics, and tracking those measurements in SQUASH. Rather than supporting only specially-design verification afterburner tasks, our goal is to empower developers to track performance metrics of their own specific pipeline Tasks. By defining metrics relevant to specific Tasks, verification becomes a highly relevant integration testing activity for day-to-day pipelines development.

This tutorial demonstrates key features and patterns in the lsst.verify framework, from defining metrics and specifications, to making measurements, to analyzing and summarizing performance.

In [1]:

import astropy.units as u
import numpy as np
import yaml

import lsst.verify

Defining metrics¶

Metrics are definitions of measurable things that you want to track. A measureable thing could be anything: the \(\chi^2\) of a fit, the number of sources identified and measured, or even the latency or memory usage of a function.

In the verification framework, all metrics are centrally defined in the verify_metrics package. To define a new metric, simply add or modify a YAML file in the /metrics directory of verify_metrics. Each Stack package that measures metrics has its own YAML definitions file (jointcal.yaml, validate_drp.yaml, and so on).

SQUASH watches verify_metrics so that when a metric is committed to the GitHub repo is it also known to the SQUASH dashboard.

For this tutorial, we will create metrics for hypothetical demo1 and demo2 packages. Here are some metric definitions.

First, content for a hypothetical /metrics/demo1.yaml file:

In [2]:

demo1_metrics_yaml = """
ZeropointRMS:
  unit: mmag
  description: >
    Photometric calibration RMS.
  reference:
    url: https://example.com/PhotRMS
  tags:
    - photometry

Completeness:
  unit: mag
  description: >
    Magnitude of the catalog's 50% completeness limit.
  reference:
    url: https://example.com/Complete
  tags:
    - photometry

SourceCount:
  unit: ''
  description: >
    Number of objects in the output catalog.
  tags:
    - photometry
"""

This YAML defines three metrics: demo1.ZeropointRMS, demo1.Complete and demo1.SourceCount. A metric consists of:

• A name. Names are prefixed by name of the package that defines them.
• A description. This helps to document metrics, even if they are more thoroughly defined in other documentation (see the reference field).
• A unit. Units are astropy.units-compatibble strings. Metrics of unitless quantities should use the [dimensionless_unscaled}(http://docs.astropy.org/en/stable/units/standard_units.html#the-dimensionless-unit) unit, an empty string.
• References. References can be made to URLs, or even to document handles and page numbers. We can expand the reference field’s schema to accomodate formalize reference identifiers in the future.
• Tags. These help us group metrics together in reports.

For the purposes of this demo, we’ll parse this YAML object into a lsst.verify.MetricSet collection. Normally this doesn’t need to be done since metrics should be pre-defined in verify_metrics, which are automatically loaded as we’ll see later.

In [3]:

from tempfile import TemporaryDirectory
import os

with TemporaryDirectory() as temp_dir:
    demo1_metrics_path = os.path.join(temp_dir, 'demo1.yaml')
    with open(demo1_metrics_path, mode='w') as f:
        f.write(demo1_metrics_yaml)
    demo_metrics = lsst.verify.MetricSet.load_single_package(demo1_metrics_path)
print(demo_metrics)

<MetricSet: 3 Metrics>

Defining metric specifications¶

Specifications are tests of metric measurements. A specification can be thought of as a milestone; if a measurement passes a specification then data and code are working as expected.

Like metrics, specifications can be centrally defined in the verify_metrics repository. Specifications for each package are defined in one or more YAML files in the specs subdirectory of verify_metrics. See the validate_drp directory for an example.

Here is a typical specification, in this case for the demo1.ZeropointRMS metric:

name: "minimum"
metric: "ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"
  value: 20.0
tags:
  - "demo-minimum"

The fully-qualified name for this specification is demo1.ZeropointRMS.minimum, following a {package}.{metric}.{spec_name} format. Specifications names should be unique, but otherwise can be anything. The Verification Framework does not place special meaning on “minimum,” “design,” and “stretch” specifications. Instead, we recommend that you use tags to designate specifiations with operational meaning.

The core of a specification is its test, and the demo1.ZeropointRMS.minimum specification defines its test in the threshold YAML field. Here, a measurement passes the specification if \(\mathrm{measurement} \leq 20.0~\mathrm{mmag}\).

We envision other types of specifications beyond thresholds (binary comparisions). Possible specification types include ranges and tolerances.

name: "minimum_megacam_r"
metric: "ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"
  value: 20.0
tags:
  - "demo-minimum"
metadata_query:
  camera: "megacam"
  filter_name: "r"

{
  'camera': 'megacam',
  'filter_name': 'r'
  'dataset_repo': 'https://github.com/lsst/ci_cfht.git'
}

{
  'filter_name': 'i'
  'dataset_repo': 'https://github.com/lsst/ci_cfht.git'
}

In [4]:

zeropointrms_specs_yaml = """
---
name: "minimum"
metric: "ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"
  value: 20.0
tags:
  - "demo-minimum"

---
name: "minimum_megacam_r"
base: ["ZeropointRMS.minimum"]
threshold:
  value: 15.0
metadata_query:
  camera: "megacam"
  filter_name: "r"
"""

with TemporaryDirectory() as temp_dir:
    # Write YAML to disk, emulating the verify_metrics package for this demo
    specs_dirname = os.path.join(temp_dir, 'demo1')
    os.makedirs(specs_dirname)
    demo1_specs_path = os.path.join(specs_dirname, 'zeropointRMS.yaml')
    with open(demo1_specs_path, mode='w') as f:
        f.write(zeropointrms_specs_yaml)

    # Parse the YAML into a set of Specification objects
    demo1_specs = lsst.verify.SpecificationSet.load_single_package(specs_dirname)

print(demo1_specs)

<SpecificationSet: 2 Specifications>

In [5]:

demo1_specs['demo1.ZeropointRMS.minimum_megacam_r'].json

Out[5]:

{'metadata_query': {'camera': 'megacam', 'filter_name': 'r'},
 'name': 'demo1.ZeropointRMS.minimum_megacam_r',
 'tags': ['demo-minimum'],
 'threshold': {'operator': '<=', 'unit': 'mmag', 'value': 15.0},
 'type': 'threshold'}

---
# Base specification: demo1.ZeropointRMS.minimum
name: "minimum"
metric: "ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"
  value: 20.0
tags:
  - "demo-minimum"

---
# Base specification: demo1.Completeness.minimum
name: "minimum"
metric: "Completeness"
threshold:
  operator: ">="
  unit: "mag"
  value: 20.0
tags:
  - "demo-minimum"

---
# A demo1.ZeropointRMS specification targetting MegaCam r-band
name: "minimum_megacam_r"
base: ["ZeropointRMS.minimum"]
threshold:
  value: 15.0
metadata_query:
  camera: "megacam"
  filter_name: "r"

---
# A demo1.CompletenessRMS specification targetting MegaCam r-band
name: "minimum_megacam_r"
base: ["Completeness.minimum"]
threshold:
  value: 24.0
metadata_query:
  camera: "megacam"
  filter_name: "r"

---
# Partial for MegaCam r-band specifications
id: "megacam-r"
metadata_query:
  camera: "megacam"
  filter_name: "r"

---
# Base specification: demo1.ZeropointRMS.minimum
name: "minimum"
metric: "ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"
  value: 20.0
tags:
  - "demo-minimum"

---
# Base specification: demo1.Completeness.minimum
name: "minimum"
metric: "Completeness"
threshold:
  operator: ">="
  unit: "mag"
  value: 20.0
tags:
  - "demo-minimum"

---
# A demo1.ZeropointRMS specification targetting MegaCam r-band
name: "minimum_megacam_r"
base: ["ZeropointRMS.minimum", "#megacam-r"]
threshold:
  value: 15.0


---
# A demo1.Completeness specification targetting MegaCam r-band
name: "minimum_megacam_r"
base: ["Completeness.minimum", "#megacam-r]
threshold:
  value: 24.0

In [6]:

demo1_specs_yaml = """
# Partials that define metadata queries
# for pipeline execution contexts with
# MegaCam r and u-band data, or HSC r-band.

---
id: "megacam-r"
metadata_query:
  camera: "megacam"
  filter_name: "r"

---
id: "megacam-u"
metadata_query:
  camera: "megacam"
  filter_name: "u"

---
id: "hsc-r"
metadata_query:
  camera: "hsc"
  filter_name: "r"

# We'll also write partials for each metric,
# that set up the basic test. Alternatively
# we could create full specifications to
# inherit from for each camera.

---
id: "ZeropointRMS"
metric: "demo1.ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"

---
id: "Completeness"
metric: "demo1.Completeness"
threshold:
  operator: ">="
  unit: "mag"

# Partials to tag specifications as
# "minimum" requirements or "stretch
# goals"
---
id: "tag-minimum"
tags:
  - "minimum"

---
id: "tag-stretch"
tags:
  - "stretch"

# ZeropointRMS specifications
# tailored for each camera, in
# minimum and stretch goal variants.

---
name: "minimum_megacam_r"
base: ["#ZeropointRMS", "#megacam-r", "#tag-minimum"]
threshold:
  value: 15.0

---
name: "stretch_megacam_r"
base: ["#ZeropointRMS", "#megacam-r", "#tag-stretch"]
threshold:
  value: 10.0
tags:
  - "stretch"

---
name: "minimum_megacam_u"
base: ["#ZeropointRMS", "#megacam-u", "#tag-minimum"]
threshold:
  value: 30.0
tags:
  - "minimum"

---
name: "stretch_megacam_u"
base: ["#ZeropointRMS", "#megacam-u", "#tag-stretch"]
threshold:
  value: 20.0
tags:
  - "stretch"

---
name: "minimum_hsc_r"
base: ["#ZeropointRMS", "#hsc-r", "#tag-minimum"]
threshold:
  value: 12.0
tags:
  - "minimum"

---
name: "stretch_hsc_r"
base: ["#ZeropointRMS", "#hsc-r", "#tag-stretch"]
threshold:
  value: 6.0
tags:
  - "stretch"

# Competeness specifications,
# tailored for each camera in
# minimum and stretch goal variants

---
name: "minimum_megacam_r"
base: ["#Completeness", "#megacam-r", "#tag-minimum"]
threshold:
  value: 24.0
tags:
  - "minimum"

---
name: "stretch_megacam_r"
base: ["#Completeness", "#megacam-r", "#tag-stretch"]
threshold:
  value: 26.0
tags:
  - "stretch"

---
name: "minimum_megacam_u"
base: ["#Completeness", "#megacam-u", "#tag-minimum"]
threshold:
  value: 20.0
tags:
  - "minimum"

---
name: "stretch_megacam_u"
base: ["#Completeness", "#megacam-u", "#tag-stretch"]
threshold:
  value: 24.0
tags:
  - "stretch"

---
name: "minimum_hsc_r"
base: ["#Completeness", "#hsc-r", "#tag-minimum"]
threshold:
  value: 20.0
tags:
  - "minimum"

---
name: "stretch_hsc_r"
base: ["#Completeness", "#hsc-r", "#tag-stretch"]
threshold:
  value: 28.0
tags:
  - "stretch"
"""

with TemporaryDirectory() as temp_dir:
    # Write YAML to disk, emulating the verify_metrics package for this demo
    specs_dirname = os.path.join(temp_dir, 'demo1')
    os.makedirs(specs_dirname)
    demo1_specs_path = os.path.join(specs_dirname, 'demo1.yaml')
    with open(demo1_specs_path, mode='w') as f:
        f.write(demo1_specs_yaml)

    # Parse the YAML into a set of Specification objects
    demo_specs = lsst.verify.SpecificationSet.load_single_package(specs_dirname)

print(demo1_specs)

<SpecificationSet: 2 Specifications>

In [7]:

sourcecount_metric = lsst.verify.Metric(
    'demo2.SourceCount',
    "Number of matched sources.",
    unit=u.dimensionless_unscaled,
    tags=['demo'])
demo_metrics.insert(sourcecount_metric)

demo_metrics['demo2.SourceCount']

Out[7]:

<lsst.verify.metric.Metric at 0x10e936dd8>

In [8]:

u.dimensionless_unscaled == u.Unit('')

Out[8]:

True

In [9]:

sourcecount_minimum_spec = lsst.verify.ThresholdSpecification(
    'demo2.SourceCount.minimum_cfht_r',
    250 * u.dimensionless_unscaled,
    '>=',
    tags=['minimum'],
    metadata_query={'camera': 'megacam', 'filter_name': 'r'}
)
demo_specs.insert(sourcecount_minimum_spec)

sourcecount_stretch_spec = lsst.verify.ThresholdSpecification(
    'demo2.SourceCount.stretch_cfht_r',
    500 * u.dimensionless_unscaled,
    '>=',
    tags=['stretch'],
    metadata_query={'camera': 'megacam', 'filter_name': 'r'}
)
demo_specs.insert(sourcecount_stretch_spec)

Making measurements¶

Now that we’ve defined metrics, we can measure them. Measurements happen in Pipelines code, either within regular Tasks, or in dedicated afterburner Tasks.

The Verification Framework provides two patterns for making measurements: either using the full measurement API, or a more lightweight capture of measurement quantities. For the demo1 package we’ll use the more comprehensive approach, and make lightweight measurements for the demo2 package.

In [10]:

catalog_mags = np.random.uniform(18, 26, size=100)*u.mag

obs_mags = catalog_mags - 25*u.mag + np.random.normal(scale=12.0, size=100)*u.mmag

In [11]:

zp = np.median(catalog_mags - obs_mags)

In [12]:

zp_rms = np.std(catalog_mags - obs_mags)

In [13]:

zp_meas = lsst.verify.Measurement('demo1.ZeropointRMS', zp_rms)

In [14]:

zp_meas.extras['zp'] = lsst.verify.Datum(zp, label="$m_0$",
                                         description="Estimated zeropoint.")
zp_meas.extras['catalog_mags'] = lsst.verify.Datum(catalog_mags, label="$m_\mathrm{cat}$",
                                                   description="Catalog magnitudes.")
zp_meas.extras['obs_mags'] = lsst.verify.Datum(obs_mags, label="$m_\mathrm{obs}",
                                               description="Instrument magnitudes.")

In [15]:

zp_meas.notes['estimator'] = 'numpy.std'

In [16]:

# Here's a mock dataset
mag_grid = np.linspace(22, 28, num=50, endpoint=True)
c_percent = 1. / np.cosh((mag_grid - mag_grid.min()) / 2.) * 100.

# Make the measurement
completeness_mag = np.interp(50.0, c_percent[::-1], mag_grid[::-1]) * u.mag

# Package the measurement
completeness_meas = lsst.verify.Measurement(
    'demo1.Completeness',
    completeness_mag,
)
completeness_meas.extras['mag_grid'] = lsst.verify.Datum(
    mag_grid*u.mag,
    label="$m$",
    description="Magnitude")
completeness_meas.extras['c_frac'] = lsst.verify.Datum(
    c_percent*u.percent,
    label="$C$",
    description="Photometric catalog completeness.")

Packaging measurements in a Verification Job¶

In the Verification Framework, a “job” is a run of pipeline that produces metric measurements. The lsst.verify.Job class allows us to package several measurements from the pipeline run. With a Job object, we can then analyze the measurements, save verification datasets to disk, and dispatch datasets to the SQUASH database.

Normally when we create a Job object from scratch we seed it with the metrics and specifications defined in the verify_metrics repo:

In [17]:

job = lsst.verify.Job.load_metrics_package()

Of course, we created ad hoc metrics and specifications outside of verify_metrics. We can add those to the job:

In [18]:

job.metrics.update(demo_metrics)
job.specs.update(demo_specs)

Now add the measurements:

In [19]:

job.measurements.insert(zp_meas)
job.measurements.insert(completeness_meas)

The pipeline Tasks that is making this Job knows about the camera and filter of the dataset. The Task code can record this metadata:

In [20]:

job.meta.update({'camera': 'megacam', 'filter_name': 'r'})

Job metadata is a dict-like mapping. Here’s the full set of metadata recorded for the job:

In [21]:

job.meta

Out[21]:

ChainMap({'filter_name': 'r', 'camera': 'megacam'})

As expected, the camera and filter_name is present, but so is the estimator annotation that we attached to the demo1.ZeropointRMS measurement. Measurement annotations are automatically included in a Job’s metadata, but keys are prefixed with the measurement’s metric name. Specification metadata_query definitions can act on both job and measurement-level metadata.

Before a Task exits, it should write the verification Job dataset to disk. Serialization to disk is a temporary shim until Job datasets can be persisted through the Butler.

The native serialization format of the Verification Framework is JSON:

In [22]:

job.write('demo1.verify.json')

Making lightweight quantity-only measurements with output_quantities()¶

lsst.verify.Measurement and lsst.verify.Job classes are necessary for producing rich job datasets (for example, associating extras with measurements. Many Tasks, though, won’t need this functionality. A Task might record a measurement as an astropy quantity and persist that measurement with as little overhead as possible. The lsst.verify.output_quantities function enables this usecase.

First, a Task will create a dictionary to collect measurements throughout the lifetime of the Task’s execution:

In [23]:

demo2_measurements = {}

Then the task measures the demo2.SourceCount metric:

In [24]:

demo2_measurements['demo2.SourceCount'] = 350*u.dimensionless_unscaled

Measurements are always Astropy quantities.

Finally, before the Task returns, it can output measurements to disk. The default filename format for the Verification job dataset file is {package}.verify.json.

In [25]:

lsst.verify.output_quantities('demo2', demo2_measurements)

Out[25]:

'demo2.verify.json'

Post processing verification jobs¶

Our hypothetical pipeline has produced measurements for two packages: demo1 and demo2. These measurements are persisted to the demo1.verify.json and demo2.verify.json files on disk. Now we’d like to gather these measurements and either submit them to the SQUASH dashboard, or otherwise collate the measurements for convenient local analysis.

The dispatch_verify.py tool lets us do this. For this demo we won’t upload measurements to squash, but instead we will combine the mesurements into one JSON file, and also add metadata about the versions of the pipeline packages that produced the measurements.

In [26]:

%%bash
echo $LSSTSW
export DYLD_LIBRARY_PATH=$LSST_LIBRARY_PATH
dispatch_verify.py --test --ignore-lsstsw --write demo.verify.json demo1.verify.json demo2.verify.json

verify.bin.dispatchverify.main INFO: Loading demo1.verify.json
verify.bin.dispatchverify.main INFO: Loading demo2.verify.json
verify.bin.dispatchverify.main INFO: Merging verification Job JSON.
verify.bin.dispatchverify.main INFO: Refreshing metric definitions from verify_metrics
verify.bin.dispatchverify.main INFO: Writing Job JSON to demo.verify.json.

The flags used here are:

• --test: prevents dispatch_verify.py from attempting to upload to the SQUASH service.
• --ignore-lsstsw: since the $LSSTSW environment variable may not be available in this notebook context, we’ll avoid scraping it for information (such as Git commits and branches of packages included in the Pipeline stack).
• --write demo.verify.json: Write the merged job dataset to demo.verify.json.
• demo1.verify.json and demo2.verify.json are inputs, as position arguments, pointing to the job JSON files that we created earlier with metric measurements.

See dispatch_verify.py for help.

In [ ]: