SQR-019: LSST Verification Framework API Demonstration

  • Jonathan Sick

Latest Revision: 2017-05-01

Introduction

lsst.verify is the new framework for making verification measurements in the LSST Science Pipelines. Verification is an activity where well-defined quantities, called metrics, are measured to ensure that LSST’s data and pipelines meet requirements, which we call specifications.

You might be familar with validate_drp. That package that currently measures metrics of ProcessCcdTask outputs and posts results to SQUASH. By tracking metric measurements we are able to understand trends in the algorithmic performance of the LSST Science Pipelines, and ultimately verify that we will meet our requirements.

With lsst.verify we sought to generalize the process of defining metrics, measuring those metrics, and tracking those measurements in SQUASH. Rather than supporting only specially-design verification afterburner tasks, our goal is to empower developers to track performance metrics of their own specific pipeline Tasks. By defining metrics relevant to specific Tasks, verification becomes a highly relevant integration testing activity for day-to-day pipelines development.

This tutorial demonstrates key features and patterns in the lsst.verify framework, from defining metrics and specifications, to making measurements, to analyzing and summarizing performance.

In [1]:
import astropy.units as u
import numpy as np
import yaml

import lsst.verify

Defining metrics

Metrics are definitions of measurable things that you want to track. A measureable thing could be anything: the \(\chi^2\) of a fit, the number of sources identified and measured, or even the latency or memory usage of a function.

In the verification framework, all metrics are centrally defined in the verify_metrics package. To define a new metric, simply add or modify a YAML file in the /metrics directory of verify_metrics. Each Stack package that measures metrics has its own YAML definitions file (jointcal.yaml, validate_drp.yaml, and so on).

SQUASH watches verify_metrics so that when a metric is committed to the GitHub repo is it also known to the SQUASH dashboard.

For this tutorial, we will create metrics for hypothetical demo1 and demo2 packages. Here are some metric definitions.

First, content for a hypothetical /metrics/demo1.yaml file:

In [2]:
demo1_metrics_yaml = """
ZeropointRMS:
  unit: mmag
  description: >
    Photometric calibration RMS.
  reference:
    url: https://example.com/PhotRMS
  tags:
    - photometry

Completeness:
  unit: mag
  description: >
    Magnitude of the catalog's 50% completeness limit.
  reference:
    url: https://example.com/Complete
  tags:
    - photometry

SourceCount:
  unit: ''
  description: >
    Number of objects in the output catalog.
  tags:
    - photometry
"""

This YAML defines three metrics: demo1.ZeropointRMS, demo1.Complete and demo1.SourceCount. A metric consists of:

  • A name. Names are prefixed by name of the package that defines them.
  • A description. This helps to document metrics, even if they are more thoroughly defined in other documentation (see the reference field).
  • A unit. Units are astropy.units-compatibble strings. Metrics of unitless quantities should use the [dimensionless_unscaled}(http://docs.astropy.org/en/stable/units/standard_units.html#the-dimensionless-unit) unit, an empty string.
  • References. References can be made to URLs, or even to document handles and page numbers. We can expand the reference field’s schema to accomodate formalize reference identifiers in the future.
  • Tags. These help us group metrics together in reports.

For the purposes of this demo, we’ll parse this YAML object into a lsst.verify.MetricSet collection. Normally this doesn’t need to be done since metrics should be pre-defined in verify_metrics, which are automatically loaded as we’ll see later.

In [3]:
from tempfile import TemporaryDirectory
import os

with TemporaryDirectory() as temp_dir:
    demo1_metrics_path = os.path.join(temp_dir, 'demo1.yaml')
    with open(demo1_metrics_path, mode='w') as f:
        f.write(demo1_metrics_yaml)
    demo_metrics = lsst.verify.MetricSet.load_single_package(demo1_metrics_path)
print(demo_metrics)
<MetricSet: 3 Metrics>

Defining metric specifications

Specifications are tests of metric measurements. A specification can be thought of as a milestone; if a measurement passes a specification then data and code are working as expected.

Like metrics, specifications can be centrally defined in the verify_metrics repository. Specifications for each package are defined in one or more YAML files in the specs subdirectory of verify_metrics. See the validate_drp directory for an example.

Here is a typical specification, in this case for the demo1.ZeropointRMS metric:

name: "minimum"
metric: "ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"
  value: 20.0
tags:
  - "demo-minimum"

The fully-qualified name for this specification is demo1.ZeropointRMS.minimum, following a {package}.{metric}.{spec_name} format. Specifications names should be unique, but otherwise can be anything. The Verification Framework does not place special meaning on “minimum,” “design,” and “stretch” specifications. Instead, we recommend that you use tags to designate specifiations with operational meaning.

The core of a specification is its test, and the demo1.ZeropointRMS.minimum specification defines its test in the threshold YAML field. Here, a measurement passes the specification if \(\mathrm{measurement} \leq 20.0~\mathrm{mmag}\).

We envision other types of specifications beyond thresholds (binary comparisions). Possible specification types include ranges and tolerances.

Metadata queries: making specifications only act upon certain measurements

Often you’ll make measurements of a metric in many contexts. With different datasets, from different cameras, in different filters. A specification we define for one measurement context might not be relevant for other contexts. LPM-17, for example, does this frequently by defining different specifications for \(gri\) datasets than \(uzy\). To prevent false alerts, the Verification Framework allows you to define criteria for when a specification applies to a measurement.

Originally we indended to leverage the provenance of a pipeline execution. Provenance, in general, fully describes the environment of the pipeline run, the datasets that were processed and produced, and the pipeline configuration. We envisioned that specifications might query the provenance of a metric measurement to determine if the specification’s test is applicable. While this is our long term design intent, a comprehensive pipeline provenance framework does not exist.

To shim the provenance system’s functionality, the Verification Framework introduces a complementary concept called job metadata. Whereas provenance is passively gathered during pipeline execution, metadata is explicitly added by pipeline developers and operators. Metadata could be a task configuration, filter name, dataset name, or any state known during a Task’s execution.

For example, suppose that a specification only applies to CFHT/MegaCam datasets in the \(r\)-band. This requirement is written into the specification’s definition with a metadata_query field:

name: "minimum_megacam_r"
metric: "ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"
  value: 20.0
tags:
  - "demo-minimum"
metadata_query:
  camera: "megacam"
  filter_name: "r"

If a job has metadata with matching camera and filter_name fields, the specification applies:

{
  'camera': 'megacam',
  'filter_name': 'r'
  'dataset_repo': 'https://github.com/lsst/ci_cfht.git'
}

On the other hand, if a job has metadata that is either missing fields, or has conflicting values, the specification does not apply:

{
  'filter_name': 'i'
  'dataset_repo': 'https://github.com/lsst/ci_cfht.git'
}

Specification inheritance

Metadata queries help us write specifications that monitor precisely the pipeline runs we are interested in, with test criteria that make sense. But this also means that we are potentially writing many more specifications for each metric. Most specifications for a given metric share common characteristics, such as units, threshold operators, the metric name, and even some base metadata query terms. To write specifications without repeating outself, we can take advantage of specification inheritance.

As an example, let’s write a basic specification in YAML for the demo1.ZeropointRMS metric, and write another specification that is customized for CFHT/MegaCam \(r\)-band data:

In [4]:
zeropointrms_specs_yaml = """
---
name: "minimum"
metric: "ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"
  value: 20.0
tags:
  - "demo-minimum"

---
name: "minimum_megacam_r"
base: ["ZeropointRMS.minimum"]
threshold:
  value: 15.0
metadata_query:
  camera: "megacam"
  filter_name: "r"
"""

with TemporaryDirectory() as temp_dir:
    # Write YAML to disk, emulating the verify_metrics package for this demo
    specs_dirname = os.path.join(temp_dir, 'demo1')
    os.makedirs(specs_dirname)
    demo1_specs_path = os.path.join(specs_dirname, 'zeropointRMS.yaml')
    with open(demo1_specs_path, mode='w') as f:
        f.write(zeropointrms_specs_yaml)

    # Parse the YAML into a set of Specification objects
    demo1_specs = lsst.verify.SpecificationSet.load_single_package(specs_dirname)

print(demo1_specs)
<SpecificationSet: 2 Specifications>

The demo1.ZeropointRMS.minimum_megacam_r specification indicates that it inherits from demo1.ZeropointRMS.minium by referencing it in the base field.

With inheritance, demo1.ZeropointRMS.minimum_megacam_r includes all fields defined in its base, adds new fields, and overrides values. Notice how the threshold has changed from 20.0 mmag, to 10.0 mmag:

In [5]:
demo1_specs['demo1.ZeropointRMS.minimum_megacam_r'].json
Out[5]:
{'metadata_query': {'camera': 'megacam', 'filter_name': 'r'},
 'name': 'demo1.ZeropointRMS.minimum_megacam_r',
 'tags': ['demo-minimum'],
 'threshold': {'operator': '<=', 'unit': 'mmag', 'value': 15.0},
 'type': 'threshold'}

Specification partials for even more composable specifications

Suppose we want to create specifications for many metrics that apply to the megacam camera. Specification inheritance doesn’t help because we need to repeat the metadata query for each metric:

---
# Base specification: demo1.ZeropointRMS.minimum
name: "minimum"
metric: "ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"
  value: 20.0
tags:
  - "demo-minimum"

---
# Base specification: demo1.Completeness.minimum
name: "minimum"
metric: "Completeness"
threshold:
  operator: ">="
  unit: "mag"
  value: 20.0
tags:
  - "demo-minimum"

---
# A demo1.ZeropointRMS specification targetting MegaCam r-band
name: "minimum_megacam_r"
base: ["ZeropointRMS.minimum"]
threshold:
  value: 15.0
metadata_query:
  camera: "megacam"
  filter_name: "r"

---
# A demo1.CompletenessRMS specification targetting MegaCam r-band
name: "minimum_megacam_r"
base: ["Completeness.minimum"]
threshold:
  value: 24.0
metadata_query:
  camera: "megacam"
  filter_name: "r"

To avoid duplicating metadata_query information for all MegaCam \(r\)-band specifications across many metrics, we can extract that information into a partial. Partials are formatted like specifications, but are never parsed as stand-alone specifiations. That means a partial can, as the name implies, define common partial information that can be mixed into many specifications.

Here’s the same example as before, but written with a #megacam_r partial:

---
# Partial for MegaCam r-band specifications
id: "megacam-r"
metadata_query:
  camera: "megacam"
  filter_name: "r"

---
# Base specification: demo1.ZeropointRMS.minimum
name: "minimum"
metric: "ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"
  value: 20.0
tags:
  - "demo-minimum"

---
# Base specification: demo1.Completeness.minimum
name: "minimum"
metric: "Completeness"
threshold:
  operator: ">="
  unit: "mag"
  value: 20.0
tags:
  - "demo-minimum"

---
# A demo1.ZeropointRMS specification targetting MegaCam r-band
name: "minimum_megacam_r"
base: ["ZeropointRMS.minimum", "#megacam-r"]
threshold:
  value: 15.0


---
# A demo1.Completeness specification targetting MegaCam r-band
name: "minimum_megacam_r"
base: ["Completeness.minimum", "#megacam-r]
threshold:
  value: 24.0

As you can see, we’ve added the megacam-r partial to the inheritance chain defined in the base fields. The demo1.ZeropointRMS.minimum_megacam_r and demo1.Completeness.minimum_megacam_r specifications inherit form both specifications and the #megacam-r partial. The # prefix implies an partial, not a specification. It’s also possible to reference partials in other YAML files, see the validate_drp specifications for an example.

Inheritance is evaluated left to right. For example, demo1.Completeness.minimum_megacam_r is built up in this order:

  1. Use the demo1.Completeness.minimum specification.
  2. Override with information from #megacam-r.
  3. Override with information from the demo1.Completeness.minimum specification’s own YAML fields.

Specifications: putting it all together

We’ve seen how to write specification metrics in YAML, and how to write them more efficiently with inheritance and partials. Now let’s write out a full specification set, like we might in verify_metrics:

In [6]:
demo1_specs_yaml = """
# Partials that define metadata queries
# for pipeline execution contexts with
# MegaCam r and u-band data, or HSC r-band.

---
id: "megacam-r"
metadata_query:
  camera: "megacam"
  filter_name: "r"

---
id: "megacam-u"
metadata_query:
  camera: "megacam"
  filter_name: "u"

---
id: "hsc-r"
metadata_query:
  camera: "hsc"
  filter_name: "r"

# We'll also write partials for each metric,
# that set up the basic test. Alternatively
# we could create full specifications to
# inherit from for each camera.

---
id: "ZeropointRMS"
metric: "demo1.ZeropointRMS"
threshold:
  operator: "<="
  unit: "mmag"

---
id: "Completeness"
metric: "demo1.Completeness"
threshold:
  operator: ">="
  unit: "mag"

# Partials to tag specifications as
# "minimum" requirements or "stretch
# goals"
---
id: "tag-minimum"
tags:
  - "minimum"

---
id: "tag-stretch"
tags:
  - "stretch"

# ZeropointRMS specifications
# tailored for each camera, in
# minimum and stretch goal variants.

---
name: "minimum_megacam_r"
base: ["#ZeropointRMS", "#megacam-r", "#tag-minimum"]
threshold:
  value: 15.0

---
name: "stretch_megacam_r"
base: ["#ZeropointRMS", "#megacam-r", "#tag-stretch"]
threshold:
  value: 10.0
tags:
  - "stretch"

---
name: "minimum_megacam_u"
base: ["#ZeropointRMS", "#megacam-u", "#tag-minimum"]
threshold:
  value: 30.0
tags:
  - "minimum"

---
name: "stretch_megacam_u"
base: ["#ZeropointRMS", "#megacam-u", "#tag-stretch"]
threshold:
  value: 20.0
tags:
  - "stretch"

---
name: "minimum_hsc_r"
base: ["#ZeropointRMS", "#hsc-r", "#tag-minimum"]
threshold:
  value: 12.0
tags:
  - "minimum"

---
name: "stretch_hsc_r"
base: ["#ZeropointRMS", "#hsc-r", "#tag-stretch"]
threshold:
  value: 6.0
tags:
  - "stretch"

# Competeness specifications,
# tailored for each camera in
# minimum and stretch goal variants

---
name: "minimum_megacam_r"
base: ["#Completeness", "#megacam-r", "#tag-minimum"]
threshold:
  value: 24.0
tags:
  - "minimum"

---
name: "stretch_megacam_r"
base: ["#Completeness", "#megacam-r", "#tag-stretch"]
threshold:
  value: 26.0
tags:
  - "stretch"

---
name: "minimum_megacam_u"
base: ["#Completeness", "#megacam-u", "#tag-minimum"]
threshold:
  value: 20.0
tags:
  - "minimum"

---
name: "stretch_megacam_u"
base: ["#Completeness", "#megacam-u", "#tag-stretch"]
threshold:
  value: 24.0
tags:
  - "stretch"

---
name: "minimum_hsc_r"
base: ["#Completeness", "#hsc-r", "#tag-minimum"]
threshold:
  value: 20.0
tags:
  - "minimum"

---
name: "stretch_hsc_r"
base: ["#Completeness", "#hsc-r", "#tag-stretch"]
threshold:
  value: 28.0
tags:
  - "stretch"
"""

with TemporaryDirectory() as temp_dir:
    # Write YAML to disk, emulating the verify_metrics package for this demo
    specs_dirname = os.path.join(temp_dir, 'demo1')
    os.makedirs(specs_dirname)
    demo1_specs_path = os.path.join(specs_dirname, 'demo1.yaml')
    with open(demo1_specs_path, mode='w') as f:
        f.write(demo1_specs_yaml)

    # Parse the YAML into a set of Specification objects
    demo_specs = lsst.verify.SpecificationSet.load_single_package(specs_dirname)

print(demo1_specs)
<SpecificationSet: 2 Specifications>

More metrics and specifications for the demo1 package

All the metrics we’ve created have been associated with the hypothetical “demo1” pipeline package. Let’s quickly create another set of metrics and specifications for a “demo1” pipeline package, which we’ll use later. This is an excuse to show that metrics and specifications can be created dynamically in Python too.

In [7]:
sourcecount_metric = lsst.verify.Metric(
    'demo2.SourceCount',
    "Number of matched sources.",
    unit=u.dimensionless_unscaled,
    tags=['demo'])
demo_metrics.insert(sourcecount_metric)

demo_metrics['demo2.SourceCount']
Out[7]:
<lsst.verify.metric.Metric at 0x10e936dd8>

Notice that demo1.SourceCount is just a count; it doesn’t have physical units. We designated this type of unit with Astropy’s astropy.units.dimensionless_unscaled unit. Its string form is an empty string:

In [8]:
u.dimensionless_unscaled == u.Unit('')
Out[8]:
True

Next, we’ll create complementary specifications:

In [9]:
sourcecount_minimum_spec = lsst.verify.ThresholdSpecification(
    'demo2.SourceCount.minimum_cfht_r',
    250 * u.dimensionless_unscaled,
    '>=',
    tags=['minimum'],
    metadata_query={'camera': 'megacam', 'filter_name': 'r'}
)
demo_specs.insert(sourcecount_minimum_spec)

sourcecount_stretch_spec = lsst.verify.ThresholdSpecification(
    'demo2.SourceCount.stretch_cfht_r',
    500 * u.dimensionless_unscaled,
    '>=',
    tags=['stretch'],
    metadata_query={'camera': 'megacam', 'filter_name': 'r'}
)
demo_specs.insert(sourcecount_stretch_spec)

That’s it. We now have a set of metrics and specifications defined for two packages, demo1 and demo2.

Of course, these examples are contrived for this tutorial. Normally metrics and specifications aren’t defined in notebooks or code, but with a pull request to the verify_metrics GitHub repository.

Making measurements

Now that we’ve defined metrics, we can measure them. Measurements happen in Pipelines code, either within regular Tasks, or in dedicated afterburner Tasks.

The Verification Framework provides two patterns for making measurements: either using the full measurement API, or a more lightweight capture of measurement quantities. For the demo1 package we’ll use the more comprehensive approach, and make lightweight measurements for the demo2 package.

Measuring ZeropointRMS

In our Task, we might have arrays of matched photometry and catalogs stars with known photometry:

In [10]:
catalog_mags = np.random.uniform(18, 26, size=100)*u.mag

obs_mags = catalog_mags - 25*u.mag + np.random.normal(scale=12.0, size=100)*u.mmag

From these the task might estimate a zeropoint:

In [11]:
zp = np.median(catalog_mags - obs_mags)

And a scatter:

In [12]:
zp_rms = np.std(catalog_mags - obs_mags)

zp_rms is a measurement of the demo1.ZeropointRMS metric that we’d like to capture. Let’s create a lsst.verify.Measurement object to do that:

In [13]:
zp_meas = lsst.verify.Measurement('demo1.ZeropointRMS', zp_rms)

We’ve captured the measurement, but there’s more information that we be useful for later understanding the measurement. These additional data are called measurement extras:

In [14]:
zp_meas.extras['zp'] = lsst.verify.Datum(zp, label="$m_0$",
                                         description="Estimated zeropoint.")
zp_meas.extras['catalog_mags'] = lsst.verify.Datum(catalog_mags, label="$m_\mathrm{cat}$",
                                                   description="Catalog magnitudes.")
zp_meas.extras['obs_mags'] = lsst.verify.Datum(obs_mags, label="$m_\mathrm{obs}",
                                               description="Instrument magnitudes.")

Datum acts as a wrapper as information, like Astropy quantities, that adds plotting labels and descriptions to help document our datasets.

In a Task, we might want to add annotations about the Task’s configuration. These annotations will be added to the metadata of the pipeline execution. For example, this is an annotation of the function used to estimate the RMS:

In [15]:
zp_meas.notes['estimator'] = 'numpy.std'

Measuring Completeness

Our task also measures photometric completeness. Let’s making another Measurement to record this metric measurement, along with extras:

In [16]:
# Here's a mock dataset
mag_grid = np.linspace(22, 28, num=50, endpoint=True)
c_percent = 1. / np.cosh((mag_grid - mag_grid.min()) / 2.) * 100.

# Make the measurement
completeness_mag = np.interp(50.0, c_percent[::-1], mag_grid[::-1]) * u.mag

# Package the measurement
completeness_meas = lsst.verify.Measurement(
    'demo1.Completeness',
    completeness_mag,
)
completeness_meas.extras['mag_grid'] = lsst.verify.Datum(
    mag_grid*u.mag,
    label="$m$",
    description="Magnitude")
completeness_meas.extras['c_frac'] = lsst.verify.Datum(
    c_percent*u.percent,
    label="$C$",
    description="Photometric catalog completeness.")

Packaging measurements in a Verification Job

In the Verification Framework, a “job” is a run of pipeline that produces metric measurements. The lsst.verify.Job class allows us to package several measurements from the pipeline run. With a Job object, we can then analyze the measurements, save verification datasets to disk, and dispatch datasets to the SQUASH database.

Normally when we create a Job object from scratch we seed it with the metrics and specifications defined in the verify_metrics repo:

In [17]:
job = lsst.verify.Job.load_metrics_package()

Of course, we created ad hoc metrics and specifications outside of verify_metrics. We can add those to the job:

In [18]:
job.metrics.update(demo_metrics)
job.specs.update(demo_specs)

Now add the measurements:

In [19]:
job.measurements.insert(zp_meas)
job.measurements.insert(completeness_meas)

The pipeline Tasks that is making this Job knows about the camera and filter of the dataset. The Task code can record this metadata:

In [20]:
job.meta.update({'camera': 'megacam', 'filter_name': 'r'})

Job metadata is a dict-like mapping. Here’s the full set of metadata recorded for the job:

In [21]:
job.meta
Out[21]:
ChainMap({'filter_name': 'r', 'camera': 'megacam'})

As expected, the camera and filter_name is present, but so is the estimator annotation that we attached to the demo1.ZeropointRMS measurement. Measurement annotations are automatically included in a Job’s metadata, but keys are prefixed with the measurement’s metric name. Specification metadata_query definitions can act on both job and measurement-level metadata.

Before a Task exits, it should write the verification Job dataset to disk. Serialization to disk is a temporary shim until Job datasets can be persisted through the Butler.

The native serialization format of the Verification Framework is JSON:

In [22]:
job.write('demo1.verify.json')

Making lightweight quantity-only measurements with output_quantities()

lsst.verify.Measurement and lsst.verify.Job classes are necessary for producing rich job datasets (for example, associating extras with measurements. Many Tasks, though, won’t need this functionality. A Task might record a measurement as an astropy quantity and persist that measurement with as little overhead as possible. The lsst.verify.output_quantities function enables this usecase.

First, a Task will create a dictionary to collect measurements throughout the lifetime of the Task’s execution:

In [23]:
demo2_measurements = {}

Then the task measures the demo2.SourceCount metric:

In [24]:
demo2_measurements['demo2.SourceCount'] = 350*u.dimensionless_unscaled

Measurements are always Astropy quantities.

Finally, before the Task returns, it can output measurements to disk. The default filename format for the Verification job dataset file is {package}.verify.json.

In [25]:
lsst.verify.output_quantities('demo2', demo2_measurements)
Out[25]:
'demo2.verify.json'

Post processing verification jobs

Our hypothetical pipeline has produced measurements for two packages: demo1 and demo2. These measurements are persisted to the demo1.verify.json and demo2.verify.json files on disk. Now we’d like to gather these measurements and either submit them to the SQUASH dashboard, or otherwise collate the measurements for convenient local analysis.

The dispatch_verify.py tool lets us do this. For this demo we won’t upload measurements to squash, but instead we will combine the mesurements into one JSON file, and also add metadata about the versions of the pipeline packages that produced the measurements.

In [26]:
%%bash
echo $LSSTSW
export DYLD_LIBRARY_PATH=$LSST_LIBRARY_PATH
dispatch_verify.py --test --ignore-lsstsw --write demo.verify.json demo1.verify.json demo2.verify.json

verify.bin.dispatchverify.main INFO: Loading demo1.verify.json
verify.bin.dispatchverify.main INFO: Loading demo2.verify.json
verify.bin.dispatchverify.main INFO: Merging verification Job JSON.
verify.bin.dispatchverify.main INFO: Refreshing metric definitions from verify_metrics
verify.bin.dispatchverify.main INFO: Writing Job JSON to demo.verify.json.

The flags used here are:

  • --test: prevents dispatch_verify.py from attempting to upload to the SQUASH service.
  • --ignore-lsstsw: since the $LSSTSW environment variable may not be available in this notebook context, we’ll avoid scraping it for information (such as Git commits and branches of packages included in the Pipeline stack).
  • --write demo.verify.json: Write the merged job dataset to demo.verify.json.
  • demo1.verify.json and demo2.verify.json are inputs, as position arguments, pointing to the job JSON files that we created earlier with metric measurements.

See dispatch_verify.py for help.

In [ ]: