SQR-019: LSST Verification Framework API Demonstration

  • Jonathan Sick and
  • Angelo Fausti

Latest Revision: 2018-04-26

Abstract

LSST Data Management’s verification program ensures that code meets performance specifications during construction. During operations, verification continues with an additional focus towards ensuring that data releases meet science requirements. To facilitate verification activities, we are introducing the LSST Verification Framework. This framework is implemented in the lsst.verify Python package, available at https://github.com/lsst/verify.

This technical note introduces the framework’s concepts and usage patterns through a working tutorial. First, this tutorial demonstrates how new metrics (observable concepts) and specifications (requirements and milestones that metric measurements should meet) are created. Then we measure metrics, using both a lightweight approach that is easy to retrofit into LSST Science Pipelines Tasks and a second more rigorous measurement approach that enables detailed diagnostics. Finally, this tutorial shows how metric measurements can be analyzed in a Jupyter notebook environment.

Set up

This technical note is available as a Jupyter notebook from its GitHub repository: https://github.com/lsst-sqre/sqr-019. You are encouraged to run and modify this notebook to help you learn about the lsst.verify package. This section covers the dependencies needed to run this notebook.

First, install the LSST Science Pipelines with lsstsw. Specifically, build and setup the verify package:

rebuild verify
setup verify

The ``verify`` package is not yet distributed with the LSST Science Pipelines Stack as of this writing. lsstsw is the most convenient means of installing ``verify`` from scratch.

Next, install the following packages

pip install bokeh pandas

These additional packages are used for this technote note, but are not required by lsst.verify itself.

These are the Python imports needed for this technical note:

In [1]:
# Standard library and third party packages used by this notebook
import json
import os
from tempfile import TemporaryDirectory

import astropy.units as u
import numpy as np
import yaml

# For demonstration  plots
from bokeh.io import output_notebook
from bokeh.plotting import figure, show
from bokeh.models import Range1d, Span
from bokeh.layouts import row

import pandas

# Load Bokeh
output_notebook()
Loading BokehJS ...
In [2]:
# The Verification Framework itself
import lsst.verify

Introduction

lsst.verify is the new framework for making verification measurements in the LSST Science Pipelines. Verification is an activity where well-defined quantities, called metrics, are measured to ensure that LSST’s data and pipelines meet requirements, which we call specifications.

You might be familar with validate_drp [DMTN-008]. That package currently measures metrics of ProcessCcdTask outputs and posts results to SQUASH [SQR-009]. By tracking metric measurements we are able to understand trends in the algorithmic performance of the LSST Science Pipelines, and ultimately verify that we will meet our requirements.

With lsst.verify we sought to generalize the process of defining metrics, measuring those metrics, and tracking those measurements in SQUASH. Rather than supporting only specially-designed verification afterburner Tasks, our goal is to empower developers to track performance metrics of their own specific pipeline Tasks. By defining metrics relevant to specific Tasks, verification becomes a highly relevant integration testing activity for day-to-day pipelines development. The lsst.verify design is described in SQR-017.

This tutorial demonstrates key features and patterns in the lsst.verify framework, from defining metrics and specifications, to making measurements, to analyzing and summarizing performance.

Defining metrics

Metrics are definitions of measurable things that you want to track. A measureable thing could be anything: the \(\chi^2\) of a fit, the number of sources identified and measured, or even the latency or memory usage of a function.

In the verification framework, all metrics are centrally defined in the verify_metrics package. To define a new metric, simply add or modify a YAML file in the /metrics directory of verify_metrics. Each Stack package that measures metrics has its own YAML definitions file (jointcal.yaml, validate_drp.yaml, and so on).

SQUASH watches verify_metrics so that when a metric is committed to the GitHub repo is it also known to the SQUASH dashboard.

For this tutorial, we will create metrics for hypothetical demo1 and demo2 packages.

First, content for a hypothetical /metrics/demo1.yaml file:

In [3]:
demo1_metrics_yaml = """
ZeropointRMS:
  unit: mmag
  description: >
    Photometric calibration RMS.
  reference:
    url: https://example.com/PhotRMS
  tags:
    - photometry
    - demo

Completeness:
  unit: mag
  description: >
    Magnitude of the catalog's 50% completeness limit.
  reference:
    url: https://example.com/Complete
  tags:
    - photometry
    - demo
"""

This YAML defines two metrics: demo1.ZeropointRMS and demo1.Completeness. A metric consists of:

  • A name. Names are prefixed by name of the package that defines them.
  • A description. This helps to document metrics, even if they are more thoroughly defined in other documentation (see the reference field).
  • A unit. Units are astropy.units-compatible strings. Metrics of unitless quantities should use the dimensionless_unscaled unit, an empty string.
  • References. References can be made to URLs, or even to document handles and page numbers. We may expand the reference field’s schema to accomodate formalize reference identifiers in the future.
  • Tags. These help us group metrics together in reports.

For the purposes of this demo, we’ll parse this YAML object into a lsst.verify.MetricSet collection. Normally this doesn’t need to be done since metrics should be pre-defined in verify_metrics, which are automatically loaded as we’ll see later.

In [5]:
with TemporaryDirectory() as temp_dir:
    demo1_metrics_path = os.path.join(temp_dir, 'demo1.yaml')
    with open(demo1_metrics_path, mode='w') as f:
        f.write(demo1_metrics_yaml)
    demo_metrics = lsst.verify.MetricSet.load_single_package(
        demo1_metrics_path)
demo_metrics
Out[5]:
<Table length=2>
NameDescriptionUnitsReferenceTags
str18str50str15str28str16
demo1.CompletenessMagnitude of the catalog's 50% completeness limit.$\mathrm{mag}$https://example.com/Completedemo, photometry
demo1.ZeropointRMSPhotometric calibration RMS.$\mathrm{mmag}$https://example.com/PhotRMSdemo, photometry

Defining metric specifications

Specifications are tests of metric measurements. A specification can be thought of as a milestone; if a measurement passes a specification then data and code are working as expected.

Like metrics, specifications are usually defined centrally in the verify_metrics repository. Specifications for each package are defined in one or more YAML files in the specs subdirectory of verify_metrics. See the validate_drp directory for an example.

Here is a typical specification written in YAML; in this case for the demo1.ZeropointRMS metric:

name: "minimum"
metric: "ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"
  value: 20.0
tags:
  - "minimum"

The fully-qualified name for this specification is demo1.ZeropointRMS.minimum, following a {package}.{metric}.{spec_name} format. Specification names should be unique, but otherwise can be anything. The Verification Framework does not place special meaning on “minimum,” “design,” and “stretch” specifications. Instead, we recommend that you use tags to designate specifications with operational meaning.

The core of a specification is its test. The demo1.ZeropointRMS.minimum specification defines its test in the threshold YAML field. Here, a measurement passes the specification if \(\mathrm{measurement} \leq 20.0~\mathrm{mmag}\).

We envision other types of specifications beyond thresholds (binary comparisions). Possibilities include ranges and tolerances.

Metadata queries: making specifications only act upon certain measurements

Often you’ll make measurements of a metric in many contexts: with different datasets, from different cameras, in different filters, and so on. A specification we define for one measurement context might not be relevant for other contexts. LPM-17, for example, does this frequently by defining different specifications for \(gri\) datasets than \(uzy\). To prevent false alerts, the Verification Framework allows you to define criteria for when a specification applies to a measurement.

Originally we indended to leverage the provenance of a pipeline execution. Provenance, in general, fully describes the environment of the pipeline run, the datasets that were processed and produced, and the pipeline configuration. We envisioned that specifications might query the provenance of a metric measurement to determine if the specification is applicable. While this is our long-term design intent, a comprehensive pipeline provenance framework does not exist.

To shim the provenance system’s functionality, the Verification Framework introduces a complementary concept called job metadata. Whereas provenance is passively gathered during pipeline execution, metadata is explicitly added by pipeline developers and operators. Metadata could be a task configuration, filter name, dataset name, or any state known during a Task’s execution.

For example, suppose that a specification only applies to CFHT/MegaCam datasets in the \(r\)-band. This requirement is written into the specification’s definition with a metadata_query field:

name: "minimum_megacam_r"
metric: "ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"
  value: 20.0
tags:
  - "minimum"
metadata_query:
  camera: "megacam"
  filter_name: "r"

If a job has metadata with matching camera and filter_name fields, the specification applies:

{
  'camera': 'megacam',
  'filter_name': 'r'
  'dataset_repo': 'https://github.com/lsst/ci_cfht.git'
}

On the other hand, if a job has metadata that is either missing fields, or has conflicting values, the specification does not apply:

{
  'filter_name': 'i'
  'dataset_repo': 'https://github.com/lsst/ci_cfht.git'
}

Specification inheritance

Metadata queries help us write specifications that monitor precisely the pipeline runs we are interested in, with test criteria that make sense. But this also means that we are potentially writing many more specifications for each metric. Most specifications for a given metric share common characteristics, such as units, threshold operators, the metric name, and even some base metadata query terms. To write specifications without repeating outself, we can take advantage of specification inheritance.

As an example, let’s write a basic specification in YAML for the demo1.ZeropointRMS metric, and write another specification that is customized for CFHT/MegaCam \(r\)-band data:

In [6]:
zeropointrms_specs_yaml = """
---
name: "minimum"
metric: "ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"
  value: 20.0
tags:
  - "minimum"

---
name: "minimum_megacam_r"
base: ["ZeropointRMS.minimum"]
threshold:
  value: 15.0
metadata_query:
  camera: "megacam"
  filter_name: "r"
"""

with TemporaryDirectory() as temp_dir:
    # Write YAML to disk, emulating the verify_metrics package for this demo
    specs_dirname = os.path.join(temp_dir, 'demo1')
    os.makedirs(specs_dirname)
    demo1_specs_path = os.path.join(specs_dirname, 'zeropointRMS.yaml')
    with open(demo1_specs_path, mode='w') as f:
        f.write(zeropointrms_specs_yaml)

    # Parse the YAML into a set of Specification objects
    demo1_specs = lsst.verify.SpecificationSet.load_single_package(
        specs_dirname)

demo1_specs
Out[6]:
<Table length=2>
NameTestTags
str36str27str7
demo1.ZeropointRMS.minimum$x$ <= 20.0 $\mathrm{mmag}$minimum
demo1.ZeropointRMS.minimum_megacam_r$x$ <= 15.0 $\mathrm{mmag}$minimum

The demo1.ZeropointRMS.minimum_megacam_r specification indicates that it inherits from demo1.ZeropointRMS.minium by referencing it in the base field.

With inheritance, demo1.ZeropointRMS.minimum_megacam_r includes all fields defined in its base, adds new fields, and overrides values. Notice how the threshold has changed from 20.0 mmag, to 15.0 mmag.

Specification partials for even more composable specifications

Suppose we want to create specifications for many metrics that apply to the megacam camera. Specification inheritance doesn’t help because we need to repeat the metadata query for each metric:

---
# Base specification: demo1.ZeropointRMS.minimum
name: "minimum"
metric: "ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"
  value: 20.0
tags:
  - "minimum"

---
# Base specification: demo1.Completeness.minimum
name: "minimum"
metric: "Completeness"
threshold:
  operator: ">="
  unit: "mag"
  value: 20.0
tags:
  - "minimum"

---
# A demo1.ZeropointRMS specification targetting MegaCam r-band
name: "minimum_megacam_r"
base: ["ZeropointRMS.minimum"]
threshold:
  value: 15.0
metadata_query:
  camera: "megacam"
  filter_name: "r"

---
# A demo1.CompletenessRMS specification targetting MegaCam r-band
name: "minimum_megacam_r"
base: ["Completeness.minimum"]
threshold:
  value: 24.0
metadata_query:
  camera: "megacam"
  filter_name: "r"

To avoid duplicating metadata_query information for all MegaCam \(r\)-band specifications across many metrics, we can extract that information into a partial. Partials are formatted like specifications, but are never parsed as stand-alone specifiations. That means a partial can, as the name implies, define common partial information that can be mixed into many specifications.

Here’s the same example as before, but written with a #megacam_r partial:

---
# Partial for MegaCam r-band specifications
id: "megacam-r"
metadata_query:
  camera: "megacam"
  filter_name: "r"

---
# Base specification: demo1.ZeropointRMS.minimum
name: "minimum"
metric: "ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"
  value: 20.0
tags:
  - "minimum"

---
# Base specification: demo1.Completeness.minimum
name: "minimum"
metric: "Completeness"
threshold:
  operator: ">="
  unit: "mag"
  value: 20.0
tags:
  - "minimum"

---
# A demo1.ZeropointRMS specification targetting MegaCam r-band
name: "minimum_megacam_r"
base: ["ZeropointRMS.minimum", "#megacam-r"]
threshold:
  value: 15.0


---
# A demo1.Completeness specification targetting MegaCam r-band
name: "minimum_megacam_r"
base: ["Completeness.minimum", "#megacam-r]
threshold:
  value: 24.0

As you can see, we’ve added the megacam-r partial to the inheritance chain defined in the base fields. The demo1.ZeropointRMS.minimum_megacam_r and demo1.Completeness.minimum_megacam_r specifications inherit from both specifications and the #megacam-r partial. The # prefix implies a partial, not a specification. It’s also possible to reference partials in other YAML files, see the validate_drp specifications for an example.

Inheritance is evaluated left to right. For example, demo1.Completeness.minimum_megacam_r is built up in this order:

  1. Use the demo1.Completeness.minimum specification.
  2. Override with information from #megacam-r.
  3. Override with information from the demo1.Completeness.minimum_megacam_r specification’s own YAML fields.

Specifications: putting it all together

We’ve seen how to write specification metrics in YAML, and how to write them more efficiently with inheritance and partials. Now let’s write out a full specification set, like we might in verify_metrics:

In [7]:
demo1_specs_yaml = """
# Partials that define metadata queries
# for pipeline execution contexts with
# MegaCam r and u-band data, or HSC r-band.

---
id: "megacam-r"
metadata_query:
  camera: "megacam"
  filter_name: "r"

---
id: "megacam-u"
metadata_query:
  camera: "megacam"
  filter_name: "u"

---
id: "hsc-r"
metadata_query:
  camera: "hsc"
  filter_name: "r"

# We'll also write partials for each metric,
# that set up the basic test. Alternatively
# we could create full specifications to
# inherit from for each camera.

---
id: "ZeropointRMS"
metric: "demo1.ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"

---
id: "Completeness"
metric: "demo1.Completeness"
threshold:
  operator: ">="
  unit: "mag"

# Partials to tag specifications as
# "minimum" requirements or "stretch
# goals"
---
id: "tag-minimum"
tags:
  - "minimum"

---
id: "tag-stretch"
tags:
  - "stretch"

# ZeropointRMS specifications
# tailored for each camera, in
# minimum and stretch goal variants.

---
name: "minimum_megacam_r"
base: ["#ZeropointRMS", "#megacam-r", "#tag-minimum"]
threshold:
  value: 15.0

---
name: "stretch_megacam_r"
base: ["#ZeropointRMS", "#megacam-r", "#tag-stretch"]
threshold:
  value: 10.0

---
name: "minimum_megacam_u"
base: ["#ZeropointRMS", "#megacam-u", "#tag-minimum"]
threshold:
  value: 30.0

---
name: "stretch_megacam_u"
base: ["#ZeropointRMS", "#megacam-u", "#tag-stretch"]
threshold:
  value: 20.0

---
name: "minimum_hsc_r"
base: ["#ZeropointRMS", "#hsc-r", "#tag-minimum"]
threshold:
  value: 12.0

---
name: "stretch_hsc_r"
base: ["#ZeropointRMS", "#hsc-r", "#tag-stretch"]
threshold:
  value: 6.0

# Competeness specifications,
# tailored for each camera in
# minimum and stretch goal variants

---
name: "minimum_megacam_r"
base: ["#Completeness", "#megacam-r", "#tag-minimum"]
threshold:
  value: 24.0

---
name: "stretch_megacam_r"
base: ["#Completeness", "#megacam-r", "#tag-stretch"]
threshold:
  value: 26.0

---
name: "minimum_megacam_u"
base: ["#Completeness", "#megacam-u", "#tag-minimum"]
threshold:
  value: 20.0

---
name: "stretch_megacam_u"
base: ["#Completeness", "#megacam-u", "#tag-stretch"]
threshold:
  value: 24.0

---
name: "minimum_hsc_r"
base: ["#Completeness", "#hsc-r", "#tag-minimum"]
threshold:
  value: 20.0

---
name: "stretch_hsc_r"
base: ["#Completeness", "#hsc-r", "#tag-stretch"]
threshold:
  value: 28.0
"""

with TemporaryDirectory() as temp_dir:
    # Write YAML to disk, emulating the verify_metrics package for this demo
    specs_dirname = os.path.join(temp_dir, 'demo1')
    os.makedirs(specs_dirname)
    demo1_specs_path = os.path.join(specs_dirname, 'demo1.yaml')
    with open(demo1_specs_path, mode='w') as f:
        f.write(demo1_specs_yaml)

    # Parse the YAML into a set of Specification objects
    demo_specs = lsst.verify.SpecificationSet.load_single_package(
        specs_dirname)

demo_specs
Out[7]:
<Table length=12>
NameTestTags
str36str27str7
demo1.Completeness.minimum_hsc_r$x$ >= 20.0 $\mathrm{mag}$minimum
demo1.Completeness.minimum_megacam_r$x$ >= 24.0 $\mathrm{mag}$minimum
demo1.Completeness.minimum_megacam_u$x$ >= 20.0 $\mathrm{mag}$minimum
demo1.Completeness.stretch_hsc_r$x$ >= 28.0 $\mathrm{mag}$stretch
demo1.Completeness.stretch_megacam_r$x$ >= 26.0 $\mathrm{mag}$stretch
demo1.Completeness.stretch_megacam_u$x$ >= 24.0 $\mathrm{mag}$stretch
demo1.ZeropointRMS.minimum_hsc_r$x$ <= 12.0 $\mathrm{mmag}$minimum
demo1.ZeropointRMS.minimum_megacam_r$x$ <= 15.0 $\mathrm{mmag}$minimum
demo1.ZeropointRMS.minimum_megacam_u$x$ <= 30.0 $\mathrm{mmag}$minimum
demo1.ZeropointRMS.stretch_hsc_r$x$ <= 6.0 $\mathrm{mmag}$stretch
demo1.ZeropointRMS.stretch_megacam_r$x$ <= 10.0 $\mathrm{mmag}$stretch
demo1.ZeropointRMS.stretch_megacam_u$x$ <= 20.0 $\mathrm{mmag}$stretch

More metrics and specifications for the demo2 package

All the metrics we’ve created have been associated with the hypothetical “demo1” pipeline package. Let’s quickly create another set of metrics and specifications for a “demo2” pipeline package, which we’ll use later. This is an opportunity to show that metrics and specifications can be created dynamically in Python too.

In [8]:
sourcecount_metric = lsst.verify.Metric(
    'demo2.SourceCount',
    "Number of matched sources.",
    unit=u.dimensionless_unscaled,
    tags=['demo'])
demo_metrics.insert(sourcecount_metric)

print(demo_metrics['demo2.SourceCount'])
demo2.SourceCount (dimensionless_unscaled): Number of matched sources.

Notice that demo2.SourceCount is just a count; it doesn’t have physical units. We designated this type of unit with Astropy’s astropy.units.dimensionless_unscaled unit. Its string form is an empty string:

In [9]:
u.dimensionless_unscaled == u.Unit('')
Out[9]:
True

Next, we’ll create complementary specifications:

In [10]:
sourcecount_minimum_spec = lsst.verify.ThresholdSpecification(
    'demo2.SourceCount.minimum_cfht_r',
    250 * u.dimensionless_unscaled,
    '>=',
    tags=['minimum'],
    metadata_query={
        'camera': 'megacam',
        'filter_name': 'r'
    })
demo_specs.insert(sourcecount_minimum_spec)

sourcecount_stretch_spec = lsst.verify.ThresholdSpecification(
    'demo2.SourceCount.stretch_cfht_r',
    500 * u.dimensionless_unscaled,
    '>=',
    tags=['stretch'],
    metadata_query={
        'camera': 'megacam',
        'filter_name': 'r'
    })
demo_specs.insert(sourcecount_stretch_spec)

That’s it. We now have a set of metrics and specifications defined for two packages, demo1 and demo2. Here are the metrics in full:

In [11]:
demo_metrics
Out[11]:
<Table length=3>
NameDescriptionUnitsReferenceTags
str18str50str15str28str16
demo1.CompletenessMagnitude of the catalog's 50% completeness limit.$\mathrm{mag}$https://example.com/Completedemo, photometry
demo1.ZeropointRMSPhotometric calibration RMS.$\mathrm{mmag}$https://example.com/PhotRMSdemo, photometry
demo2.SourceCountNumber of matched sources.$\mathrm{}$demo

And the specifications in full:

In [12]:
demo_specs
Out[12]:
<Table length=14>
NameTestTags
str36str27str7
demo1.Completeness.minimum_hsc_r$x$ >= 20.0 $\mathrm{mag}$minimum
demo1.Completeness.minimum_megacam_r$x$ >= 24.0 $\mathrm{mag}$minimum
demo1.Completeness.minimum_megacam_u$x$ >= 20.0 $\mathrm{mag}$minimum
demo1.Completeness.stretch_hsc_r$x$ >= 28.0 $\mathrm{mag}$stretch
demo1.Completeness.stretch_megacam_r$x$ >= 26.0 $\mathrm{mag}$stretch
demo1.Completeness.stretch_megacam_u$x$ >= 24.0 $\mathrm{mag}$stretch
demo1.ZeropointRMS.minimum_hsc_r$x$ <= 12.0 $\mathrm{mmag}$minimum
demo1.ZeropointRMS.minimum_megacam_r$x$ <= 15.0 $\mathrm{mmag}$minimum
demo1.ZeropointRMS.minimum_megacam_u$x$ <= 30.0 $\mathrm{mmag}$minimum
demo1.ZeropointRMS.stretch_hsc_r$x$ <= 6.0 $\mathrm{mmag}$stretch
demo1.ZeropointRMS.stretch_megacam_r$x$ <= 10.0 $\mathrm{mmag}$stretch
demo1.ZeropointRMS.stretch_megacam_u$x$ <= 20.0 $\mathrm{mmag}$stretch
demo2.SourceCount.minimum_cfht_r$x$ >= 250.0 $\mathrm{}$minimum
demo2.SourceCount.stretch_cfht_r$x$ >= 500.0 $\mathrm{}$stretch

Of course, these examples are contrived for this tutorial. Normally metrics and specifications aren’t defined in notebooks or code, but with a pull request to the verify_metrics GitHub repository.

Making measurements

Now that we’ve defined metrics, we can measure them. Measurements happen in Pipelines code, either within regular Tasks, or in dedicated afterburner Tasks.

The Verification Framework provides two patterns for making measurements: either using the full measurement API, or a more lightweight capture of measurement quantities. For the demo1 package we’ll use the more comprehensive approach, and then make lightweight measurements for the demo2 package.

Measuring ZeropointRMS

In our Task, we might have arrays of matched photometry and catalogs stars with known photometry:

In [13]:
catalog_mags = np.random.uniform(18, 26, size=100)*u.mag

obs_mags = catalog_mags - 25*u.mag + np.random.normal(scale=12.0, size=100)*u.mmag

From these the task might estimate a zeropoint:

In [14]:
zp = np.median(catalog_mags - obs_mags)

And a scatter:

In [15]:
zp_rms = np.std(catalog_mags - obs_mags)

zp_rms is a measurement of the demo1.ZeropointRMS metric that we’d like to capture. Let’s create a lsst.verify.Measurement object to do that:

In [16]:
zp_meas = lsst.verify.Measurement('demo1.ZeropointRMS', zp_rms)

We’ve captured the measurement, but there’s more information that will be useful for later understanding the measurement. These additional data are called measurement extras:

In [17]:
zp_meas.extras['zp'] = lsst.verify.Datum(
    zp, label="m_0", description="Estimated zeropoint.")
zp_meas.extras['catalog_mags'] = lsst.verify.Datum(
    catalog_mags, label="m_cat", description="Catalog magnitudes.")
zp_meas.extras['obs_mags'] = lsst.verify.Datum(
    obs_mags, label="m_obs", description="Instrument magnitudes.")

The Datum objects act as wrappers for information, like Astropy quantities, that adds plotting labels and descriptions to help document our datasets.

In a Task, we might want to add annotations about the Task’s configuration. These annotations will be added to the metadata of the pipeline execution. For example, this is an annotation of the function used to estimate the RMS:

In [18]:
zp_meas.notes['estimator'] = 'numpy.std'

Measuring Completeness

Our task also measures photometric completeness. Let’s making another Measurement to record this metric measurement, along with extras:

In [19]:
# Here's a mock dataset
mag_grid = np.linspace(22, 28, num=50, endpoint=True)
c_percent = 1. / np.cosh((mag_grid - mag_grid.min()) / 2.) * 100.

# Make the measurement
completeness_mag = np.interp(50.0, c_percent[::-1], mag_grid[::-1]) * u.mag

# Package the measurement
completeness_meas = lsst.verify.Measurement(
    'demo1.Completeness',
    completeness_mag,
)
completeness_meas.extras['mag_grid'] = lsst.verify.Datum(
    mag_grid * u.mag, label="m", description="Magnitude")
completeness_meas.extras['c_frac'] = lsst.verify.Datum(
    c_percent * u.percent,
    label="C",
    description="Photometric catalog completeness.")

Packaging measurements in a Verification Job

In the Verification Framework, a “job” is a pipeline run that produces metric measurements. The lsst.verify.Job class allows us to package several measurements from the pipeline run. With a Job object, we can then analyze the measurements, save verification datasets to disk, and dispatch datasets to the SQUASH database.

Normally when we create a Job object from scratch we seed it with the metrics and specifications defined in the verify_metrics repo:

In [20]:
job = lsst.verify.Job.load_metrics_package()

Of course, we created ad hoc metrics and specifications outside of verify_metrics. We can add those to the job:

In [21]:
job.metrics.update(demo_metrics)
job.specs.update(demo_specs)

Now add the measurements:

In [22]:
job.measurements.insert(zp_meas)
job.measurements.insert(completeness_meas)

The pipeline Tasks that is making this Job knows about the camera and filter of the dataset. The Task code can record this metadata:

In [23]:
job.meta.update({'camera': 'megacam', 'filter_name': 'r'})

Job metadata is a dict-like mapping. Here’s the full set of metadata recorded for the job:

In [24]:
print(job.meta)
{
    "camera": "megacam",
    "demo1.ZeropointRMS.estimator": "numpy.std",
    "filter_name": "r"
}

As expected, the camera and filter_name is present, but so is the estimator annotation that we attached to the demo1.ZeropointRMS measurement. Measurement annotations are automatically included in a Job’s metadata, but keys are prefixed with the measurement’s metric name. Specification metadata_query definitions can act on both job and measurement-level metadata.

Before a Task exits, it should write the verification Job dataset to disk. Serialization to disk is a temporary shim until Job datasets can be persisted through the Butler.

The native serialization format of the Verification Framework is JSON:

In [25]:
job.write('demo1.verify.json')

Making lightweight quantity-only measurements with output_quantities()

lsst.verify.Measurement and lsst.verify.Job classes are necessary for producing rich job datasets (for example, associating extras with measurements). Many Tasks, though, won’t need this functionality. A Task might record a measurement as an Astropy quantity and persist that measurement with as little overhead as possible. The lsst.verify.output_quantities function enables this use case.

First, a Task will create a dictionary to collect measurements throughout the lifetime of the Task’s execution:

In [26]:
demo2_measurements = {}

Then the task measures the demo2.SourceCount metric:

In [27]:
demo2_measurements['demo2.SourceCount'] = 350*u.dimensionless_unscaled

Measurements are always Astropy quantities.

Finally, before the Task returns, it can output measurements to disk. The default filename format for the Verification job dataset file is {package}.verify.json.

In [28]:
lsst.verify.output_quantities('demo2', demo2_measurements)
Out[28]:
'demo2.verify.json'

Post processing verification jobs

Our hypothetical pipeline has produced measurements for two packages: demo1 and demo2. These measurements are persisted to demo1.verify.json and demo2.verify.json files on disk. Now we’d like to gather these measurements and either submit them to the SQUASH dashboard, or collate the measurements for local analysis.

The dispatch_verify.py tool lets us do this. Before uploading measurements to SQUASH, let us see how to combine the mesurements into a single JSON file.

In [29]:
%%bash
export DYLD_LIBRARY_PATH=$LSST_LIBRARY_PATH
dispatch_verify.py --test --ignore-lsstsw --write demo.verify.json demo1.verify.json demo2.verify.json
verify.bin.dispatchverify.main INFO: Loading demo1.verify.json
verify.bin.dispatchverify.main INFO: Loading demo2.verify.json
verify.bin.dispatchverify.main INFO: Merging verification Job JSON.
verify.bin.dispatchverify.main INFO: Refreshing metric definitions from verify_metrics
verify.bin.dispatchverify.main INFO: Writing Job JSON to demo.verify.json.

The flags used here are:

  • --test: prevents dispatch_verify.py from attempting to upload to the SQUASH service.
  • --ignore-lsstsw: since this notebook may not be run from an lsstsw-based installation, we’ll avoid scraping it for information (such as Git commits and branches of packages included in the Pipeline stack).
  • --write demo.verify.json: Write the merged job dataset to demo.verify.json.
  • demo1.verify.json and demo2.verify.json are inputs, as positional arguments, pointing to the job JSON files that we created earlier with metric measurements.

See dispatch_verify.py --help for more information.

Dispatching verification jobs to SQUASH

We’ll use a sandbox instance of SQUASH specially deployed for this tutorial. We’ll show how to register a new user and update the SQUASH database with the metrics and specifications defined earlier. Finally, we’ll upload a verification job to SQUASH with dispatch_verify.py.

The SQUASH RESTful API is used for managing the SQUASH metrics dashboard. In the sandbox instance it can be reached by the following URL:

In [30]:
squash_api_url = "https://squash-restful-api-sandbox.lsst.codes"

Here we create a new user in SQUASH. An authenticated user is required to make POST requests to the SQUASH RESTful API.

In [31]:
import getpass
username = getpass.getuser()
password = getpass.getpass(prompt='Password for user `{}`: '.format(username))
Password for user `afausti`: ········
In [32]:
import requests
credentials = {'username': username, 'password': password}
r = requests.post('{}/register'.format(squash_api_url), json=credentials)
r.json()
Out[32]:
{'message': 'User created successfully.'}

Uploading metrics definition and specifications

In practice, a change in the verify_metrics package would automatically trigger an update to SQUASH. However, the metrics and specifications defined in this tutorial for the demo1 and demo2 packages must be manually loaded to SQUASH. This can be done throught the SQUASH RESTful API.

In [33]:
r = requests.post('{}/auth'.format(squash_api_url), json=credentials)
r.json()
Out[33]:
{'access_token': 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE2MDI1MzU0OTAsImlhdCI6MTUyNDc3NTQ5MCwibmJmIjoxNTI0Nzc1NDkwLCJpZGVudGl0eSI6Mn0.cyS6ozuO8eCxjSHaCZplKqv2YAn52YtHca0PB9dAuI8'}
In [34]:
headers = {'Authorization': 'JWT {}'.format(r.json()['access_token'])}
In [35]:
r = requests.post(
    '{}/metrics'.format(squash_api_url),
    json={'metrics': demo_metrics.json},
    headers=headers)
r.json()
Out[35]:
{'message': 'List of metrics successfully created.'}
In [36]:
r = requests.post(
    '{}/specs'.format(squash_api_url),
    json={'specs': demo_specs.json},
    headers=headers)
r.json()
Out[36]:
{'message': 'List of metric specificationss successfully created.'}

Uploading verification jobs

Finally, let us upload the demo.verify.json file to SQUASH so that we can visualize the results in the metrics dashboard.

In [37]:
%%bash -s "$squash_api_url" "$username" "$password"
export DYLD_LIBRARY_PATH=$LSST_LIBRARY_PATH
dispatch_verify.py --ignore-lsstsw --url $1 --user $2 --password $3 demo.verify.json
verify.bin.dispatchverify.main INFO: Loading demo.verify.json
verify.bin.dispatchverify.main INFO: Refreshing metric definitions from verify_metrics
verify.bin.dispatchverify.main INFO: Uploading Job JSON to https://squash-restful-api-sandbox.lsst.codes.
verify.squash.get INFO: GET https://squash-restful-api-sandbox.lsst.codes status: 200
verify.squash.post INFO: POST https://squash-restful-api-sandbox.lsst.codes/auth status: 200
verify.squash.post INFO: POST https://squash-restful-api-sandbox.lsst.codes/job status: 202

The flags used here are:

  • --ignore-lsstsw: since this notebook may not be run from an lsstsw-based installation, we’ll avoid scraping it for information (such as Git commits and branches of packages included in the Pipeline stack).
  • --url: points to the SQUASH RESTful API URL
  • --user and --password: credentials for the SQUASH user
  • demo.verify.json the job JSON file that we created earlier.

See dispatch_verify.py --help for more information.

Analyze verification results locally

For code development, it’s convenient to look at the results of verification measurements locally, rather than in SQUASH. The Verification Framework is designed for this workflow, with special affordances for Jupyter Notebook users.

The collated measurement dataset produced by dispatch_verify.py earlier is in the file demo.verify.json. Let’s open this dataset using the Job.deserialize class method:

In [38]:
with open('demo.verify.json') as f:
    job = lsst.verify.Job.deserialize(**json.load(f))

Making reports

With a job dataset, we can make a report that summarizes the pass/fail status of specifications that have a corresponding measurement. Reports, lsst.verify.Report instances, are thin wrappers around Astropy Tables, and look great in Jupyter Notebooks:

In [39]:
job.report().show()
Out[39]:
<Table length=6>
StatusSpecificationMeasurementTestMetric TagsSpec. Tags
demo1.Completeness.minimum_megacam_r24.6 $\mathrm{mag}$$x$ >= 24.0 $\mathrm{mag}$demo, photometryminimum
demo1.Completeness.stretch_megacam_r24.6 $\mathrm{mag}$$x$ >= 26.0 $\mathrm{mag}$demo, photometrystretch
demo1.ZeropointRMS.minimum_megacam_r12.2 $\mathrm{mmag}$$x$ <= 15.0 $\mathrm{mmag}$demo, photometryminimum
demo1.ZeropointRMS.stretch_megacam_r12.2 $\mathrm{mmag}$$x$ <= 10.0 $\mathrm{mmag}$demo, photometrystretch
demo2.SourceCount.minimum_cfht_r350.0 $\mathrm{}$$x$ >= 250.0 $\mathrm{}$demominimum
demo2.SourceCount.stretch_cfht_r350.0 $\mathrm{}$$x$ >= 500.0 $\mathrm{}$demostretch

Notice that the report only shows specification tests that are relevant to the measurements. Recall that the job metadata indicates these measurements are with CFHT/MegaCam in the \(r\)-band:

In [40]:
print(job.meta)
{
    "camera": "megacam",
    "demo1.ZeropointRMS.estimator": "numpy.std",
    "filter_name": "r",
    "packages": {}
}

Thus all the specifications having to do with HSC or the \(u\)-band aren’t tested because those tests are meaningless with the current measurements.

When there are many measurements and specifications, you might be more interested in producing reports around specific topics. Such tailored reports can be made by passing arguments to the Job.report method. For example, this is a report listing only demo1 package metrics:

In [41]:
job.report(name='demo1').show()
Out[41]:
<Table length=4>
StatusSpecificationMeasurementTestMetric TagsSpec. Tags
demo1.Completeness.minimum_megacam_r24.6 $\mathrm{mag}$$x$ >= 24.0 $\mathrm{mag}$demo, photometryminimum
demo1.Completeness.stretch_megacam_r24.6 $\mathrm{mag}$$x$ >= 26.0 $\mathrm{mag}$demo, photometrystretch
demo1.ZeropointRMS.minimum_megacam_r12.2 $\mathrm{mmag}$$x$ <= 15.0 $\mathrm{mmag}$demo, photometryminimum
demo1.ZeropointRMS.stretch_megacam_r12.2 $\mathrm{mmag}$$x$ <= 10.0 $\mathrm{mmag}$demo, photometrystretch

And this report shows results for the demo1.ZeropointRMS metrics:

In [42]:
job.report(name='demo1.ZeropointRMS').show()
Out[42]:
<Table length=2>
StatusSpecificationMeasurementTestMetric TagsSpec. Tags
demo1.ZeropointRMS.minimum_megacam_r12.2 $\mathrm{mmag}$$x$ <= 15.0 $\mathrm{mmag}$demo, photometryminimum
demo1.ZeropointRMS.stretch_megacam_r12.2 $\mathrm{mmag}$$x$ <= 10.0 $\mathrm{mmag}$demo, photometrystretch

Recall that we added tags to the specifications to designate minimum and stretch goals, as in seen in the demo1.ZeropointRMS.minimum_megacam_r specification:

In [43]:
job.specs['demo1.ZeropointRMS.minimum_megacam_r'].tags
Out[43]:
{'minimum'}

We can tailor the report to show tests only against these minimum specifications:

In [44]:
job.report(spec_tags=['minimum']).show()
Out[44]:
<Table length=3>
StatusSpecificationMeasurementTestMetric TagsSpec. Tags
demo1.Completeness.minimum_megacam_r24.6 $\mathrm{mag}$$x$ >= 24.0 $\mathrm{mag}$demo, photometryminimum
demo1.ZeropointRMS.minimum_megacam_r12.2 $\mathrm{mmag}$$x$ <= 15.0 $\mathrm{mmag}$demo, photometryminimum
demo2.SourceCount.minimum_cfht_r350.0 $\mathrm{}$$x$ >= 250.0 $\mathrm{}$demominimum

Notice that the spec_tags argument takes a sequence of tags. Each tag is treated as an AND filter with the others. For example, there are no specifications that are both minimum and stretch, so the report is empty:

In [45]:
job.report(spec_tags=['minimum', 'stretch']).show()
Out[45]:
<Table length=0>
StatusSpecificationMeasurementTestMetric TagsSpec. Tags

In addition to specification tags, you can filter by metric tags by setting the metric_tags argument.

Finally, these filters can be combined. For example, this report summarizes specification tests for metrics from the demo1 package against minimum goals:

In [46]:
job.report(name='demo1', spec_tags=['minimum']).show()
Out[46]:
<Table length=2>
StatusSpecificationMeasurementTestMetric TagsSpec. Tags
demo1.Completeness.minimum_megacam_r24.6 $\mathrm{mag}$$x$ >= 24.0 $\mathrm{mag}$demo, photometryminimum
demo1.ZeropointRMS.minimum_megacam_r12.2 $\mathrm{mmag}$$x$ <= 15.0 $\mathrm{mmag}$demo, photometryminimum

Data behind the measurements

Besides reports of specifications that were met or failed during a job, we’re also interested in the context of the measurements. What was the distribution of points? Where were sources on the detector? These questions cannot be answered by metrics, which are scalars by definition. But they might be answered by the blob datasets that accompany measurements.

Recall that during the demo1 measurements we added “extras,” consisting of raw arrays of magnitudes, as well as the fitted zeropoint. We can access these blob datasets and make plots for deeper investigation.

First, we access the demo1.ZeropointRMS metric measurement in the job:

In [47]:
m = job.measurements['demo1.ZeropointRMS']

The extra data associated with the measurement are stored as key-value items in the measurement’s extras attribute:

In [48]:
list(m.extras.keys())
Out[48]:
['zp', 'catalog_mags', 'obs_mags']

For this tutorial we’ll use Bokeh to make interactive plots with this data. Often it’s easiest to pack a Pandas DataFrame for plotting with Bokeh. We’ll make the DataFrame from the Astropy Quantity array, accessed from the quantity attributes of each item:

In [49]:
df = pandas.DataFrame({
    "obs_mags":
    m.extras['obs_mags'].quantity,
    "catalog_mags":
    m.extras['catalog_mags'].quantity,
    "delta_mags":
    m.extras['catalog_mags'].quantity - m.extras['obs_mags'].quantity
})

These items, obs_mags and catalog_mags, are lsst.verify.Datum instances. Datum objects allow us to pack information with data, such as plot labels. Here we’ll use that metadata to build plot labels:

In [50]:
# Scatter plot of observed vs. catalog stellar photometry
p = figure(
    title="Zeropoint stellar sample",
    x_axis_label="{0.label} [{0.unit}]".format(m.extras['obs_mags']),
    y_axis_label="{0.label} [{0.unit}]".format(m.extras['catalog_mags']),
    plot_width=350,
    plot_height=350)
p.circle(df['obs_mags'], df['catalog_mags'], size=5)

# Histogram of zeropoint estimates from individual matched stars.
# We're not using the Histogram Bokeh chart for some extra control.
hist_counts, hist_edges = np.histogram(df['delta_mags'], bins=10)
h = figure(
    tools="xpan, xwheel_zoom, reset",
    active_scroll="xwheel_zoom",
    y_range=(0, hist_counts.max() + 2),
    y_axis_label="Count",
    x_axis_label="{0.label} - {1.label} [{0.unit}]".format(
        m.extras['obs_mags'], m.extras['catalog_mags']),
    plot_width=350,
    plot_height=350)

# Draw histogram edges on the figure
h.quad(
    bottom=0,
    left=hist_edges[:-1],
    right=hist_edges[1:],
    top=hist_counts,
    color="lightblue",
    line_color="#3A5785")
# Line at zeropoint estimate
span = Span(
    location=m.extras['zp'].quantity.value,
    dimension='height',
    line_color="black",
    line_dash='dashed',
    line_width=3)
h.add_layout(span)
In [51]:
# Plot side-by-side
show(row(p, h), notebook_handle=True)
Out[51]:

<Bokeh Notebook handle for In[51]>

The key to building useful plots is packing the right blob data with measurements to begin with. As you write your code, imagine what plots might usefully augment metric measurements.

Summary and outlook

This technical note has demonstrated the full usage cycle of lsst.verify:

  • Defining metrics.
  • Defining specifications of metrics.
  • Measuring metrics.
  • Associating extra datasets with measurements.
  • Integration with the SQUASH dashboard application.
  • Analyzing verification pipeline jobs, including building pass/fail reports and making plots.

We encourage Data Management engineers and scientists to consider how you might instrument your own code, particularly pipeline Tasks, with verification measurements. By systematically monitoring performance metrics in your code, you will gain a clearer picture of how code development is affecting your systems.

With SQUASH, your metric measurements are centrally available to the whole organization. We believe that lsst.verify and SQUASH will become an everday service for DM developers to ensure that code contributions do not introduce adverse performance side-effects across the Stack.

References

Astropy Collaboration et al (2013). Astropy: A community Python package for astronomy. A&A, 558, A33, 10.1051/0004-6361/201322068.

Fausti, Angelo (2016). SQUASH dashboard prototype. SQuaRE Technical Note SQR-009. https://sqr-009.lsst.io.

Ivezić, Željko, and The LSST Science Collaboration (2011). LSST Science Requirements Document. LPM-17. https://ls.st/LPM-17.

Parejko, John and Sick, Jonathan (2017). Validation Metrics Framework. SQuaRE Technical Note SQR-017. https://sqr-017.lsst.io.

Wood-Vasey, Michael (2016). Introducing validate_drp: Calculate SRD Key Performance Metrics for an output repository. Data Management Technical Note DMTN-008. https://dmtn-008.lsst.io.