Intro to Unit Metrics

To turn complex evidence into actionable insights, you can run a unit metric as a single-value measure to quantify and monitor risks throughout a model's lifecycle.

In this interactive notebook, we introduce the concept of unit metric and provide a step-by-step guide on how to define, execute and extract results from these measures. As an example, we use data from a customer churn use case to fit a binary classification model. To illustrate the application of these measures, we show you how to run sklearn classification metrics as unit metrics, demonstrating their utility in quantifying model performance and risk.

In Model Risk Management (MRM), the primary objective is to identify, assess, and mitigate the risks associated with the development, implementation, and ongoing use of quantitative models. The process of measuring risk involves the understanding and assessment of evidence generated throw multiple tests across all the model development lifecycle stages, from data collection and data quality to model performance and explainability.

Evidence versus risk

The distinction between evidence and quantifiable risk measures is a critical aspect of MRM. Evidence, in this context, refers to the outputs from various tests conducted throughout the model lifecycle. For instance, a table displaying the number of missing values per feature in a dataset is a form of evidence. It shows where data might be incomplete, which can affect the model's performance and reliability. Similarly, a Receiver Operating Characteristic (ROC) curve is a graphical plot that illustrates the diagnostic ability of a binary classifier system as its discrimination threshold is varied. The curve is evidence of the model's classification performance.

However, these pieces of evidence do not offer a direct measure of risk. To quantify risk, one must derive metrics from this evidence that reflect the potential impact on the model's performance and the decisions it informs. For example, the missing data rate, calculated as the percentage of missing values in the dataset, is a quantifiable risk measure that indicates the risk associated with data quality. Similarly, the accuracy score, which measures the proportion of correctly classified labels, acts as an indicator of performance risk in a classification model.

Unit metric

A Unit metric is a single value measure that is used to identify and monitor risks arising from the development of Machine Learning or AI models. This metric simplifies evidence into a single actionable number, that can be monitored and compared over time or across different models or datasets.

Properties:

• They are the fundamental computation unit that returns a single value.
• They quantify risk and can be used to monitor and assess risks associated with a model's entire lifecycle.
• Measurable, relevant, and linked to risk areas and critical business processes - e.g., regulatory requirements, risk appetite, model performance, data quality.
• Standalone in nature, meaning they do not rely on other metrics for their calculation or interpretation.

Incorporating unit metrics into your ML workflow streamlines risk assessment, turning complex analyses into clear, actionable insights.

Contents

• About ValidMind
  • Before you begin
    
  • New to ValidMind?
    
• Install the ValidMind Library
  
• Initialize the ValidMind Library
  • Get your code snippet
    
• Load the demo dataset
  
• Train a model for testing
  
• Initialize ValidMind objects
  
• Assign predictions
  
• Running Unit Metrics
  • List Available Metrics
    
  • Learn More About a Metric
    
  • Run ROC AUC Metric
    
  • Pass parameters
    
  • Other Model Performance Unit Metrics
    
• Running the ROC AUC Metric across Multiple Datasets
  • Run multiple unit metrics as a single test
    
• Comparing Composite Metrics for Multiple Input Combinations
  
• Next steps
  • Work with your model documentation
    
  • Discover more learning resources
    
• Upgrade ValidMind

About ValidMind

ValidMind is a suite of tools for managing model risk, including risk associated with AI and statistical models.

You use the ValidMind Library to automate documentation and validation tests, and then use the ValidMind Platform to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators.

Before you begin

This notebook assumes you have basic familiarity with Python, including an understanding of how functions work. If you are new to Python, you can still run the notebook but we recommend further familiarizing yourself with the language.

If you encounter errors due to missing modules in your Python environment, install the modules with pip install, and then re-run the notebook. For more help, refer to Installing Python Modules.

New to ValidMind?

If you haven't already seen our documentation on the ValidMind Library, we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.

For access to all features available in this notebook, create a free ValidMind account.

Signing up is FREE — Register with ValidMind

Install the ValidMind Library

To install the library:

%pip install -q validmind

Initialize the ValidMind Library

ValidMind generates a unique code snippet for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.

Get your code snippet

1. In a browser, log in to ValidMind.

2. In the left sidebar, navigate to Model Inventory and click + Register Model.

3. Enter the model details and click Continue. (Need more help?)

   For example, to register a model for use with this notebook, select:

   • Documentation template: Binary classification
   • Use case: Marketing/Sales - Analytics

   You can fill in other options according to your preference.

4. Go to Getting Started and click Copy snippet to clipboard.

Next, load your model identifier credentials from an .env file or replace the placeholder with your own code snippet:

# Load your model identifier credentials from an `.env` file

%load_ext dotenv
%dotenv .env

# Or replace with your code snippet

import validmind as vm

vm.init(
    # api_host="...",
    # api_key="...",
    # api_secret="...",
    # model="...",
)

Load the demo dataset

In this example, we load a demo dataset to fit a customer churn model.

from validmind.datasets.classification import customer_churn as demo_dataset

print(
    f"Loaded demo dataset with: \n\n\t• Target column: '{demo_dataset.target_column}' \n\t• Class labels: {demo_dataset.class_labels}"
)

raw_df = demo_dataset.load_data()
raw_df.head()

Train a model for testing

We train a simple customer churn model for our test.

import xgboost as xgb

train_df, validation_df, test_df = demo_dataset.preprocess(raw_df)

x_train = train_df.drop(demo_dataset.target_column, axis=1)
y_train = train_df[demo_dataset.target_column]
x_val = validation_df.drop(demo_dataset.target_column, axis=1)
y_val = validation_df[demo_dataset.target_column]

model = xgb.XGBClassifier(early_stopping_rounds=10)
model.set_params(
    eval_metric=["error", "logloss", "auc"],
)
model.fit(
    x_train,
    y_train,
    eval_set=[(x_val, y_val)],
    verbose=False,
)

Initialize ValidMind objects

Once the datasets and model are prepared for validation, we initialize ValidMind dataset and model, specifying features and targets columns. The property input_id allows users to uniquely identify each dataset and model. This allows for the creation of multiple versions of datasets and models, enabling us to compute metrics by specifying which versions we want to use as inputs.

import validmind as vm

vm_train_ds = vm.init_dataset(
    input_id="train_dataset",
    dataset=train_df,
    target_column=demo_dataset.target_column,
)
vm_test_ds = vm.init_dataset(
    input_id="test_dataset",
    dataset=test_df,
    target_column=demo_dataset.target_column,
)

vm_model = vm.init_model(model=model, input_id="xbg_model")

Assign predictions

We can now use the assign_predictions() method from the Dataset object to link existing predictions to any model. If no prediction values are passed, the method will compute predictions automatically:

vm_train_ds.assign_predictions(model=vm_model)
vm_test_ds.assign_predictions(model=vm_model)

Running Unit Metrics

List Available Metrics

We can list all available unit metrics by using the list_metrics() function from the unit_metrics module. This function returns a list of all available metric functions.

from validmind.unit_metrics import list_metrics

metrics = list_metrics()
metrics

Learn More About a Metric

We can use the describe_metric() function to learn more about a metric, specifically its required inputs and optional parameters. Let's check out the details for the ROC AUC metric.

from validmind.unit_metrics import describe_metric

describe_metric("validmind.unit_metrics.classification.ROC_AUC")

Run ROC AUC Metric

The following snippet shows how to set up and execute a unit metric implementation of the ROC AUC score. We are going to use the run_metric() function from the validmind.unit_metrics module. This function has a signature very similar to run_test() from the validmind.tests module, but it is specifically designed to compute unit metrics. It takes the following arguments:

• metric_id: The unique identifier for the metric, in this case validmind.unit_metrics.classification.ROC_AUC
• inputs: A dictionary containing the input dataset and model or their respective input IDs
• params: A dictionary containing the keyword arguments for the unit metric (in this case, none are required, but we can pass any kwargs that the underlying sklearn implementation accepts)

The function returns and displays a result object like a normal ValidMind test, accept that it simply shows the unit metrics value. This result object has a .log() method that will log the result to the ValidMind Platform where it can be used for monitoring and model documentation.

from validmind.unit_metrics import run_metric

result = run_metric(
    "validmind.unit_metrics.classification.ROC_AUC",
    inputs={
        "model": vm_model,
        "dataset": vm_test_ds,
    },
)

# Log the result to the ValidMind Platform
result.log()

Pass parameters

Unit Metrics can accept parameters as mentioned above. In the case of the ROC AUC and many of the other unit metrics that use an underlying sklearn implementation, these metrics support all parameters of the associated sklearn metric functions. Let's see how we can do this in the case of the above ROC AUC metric.

Available parameters for the ROC AUC metric include:

• average: This parameter is used to compute the average of the ROC AUC score across all classes. The options include micro, macro, weighted, and samples.
• sample_weight: This parameter is used to pass sample weights to the metric.
• max_fpr: This parameter is used to compute the partial AUC ROC over the range [0, max_fpr].

See the sklearn documentation for more details on these parameters.

result = run_metric(
    "validmind.unit_metrics.classification.ROC_AUC",
    inputs={
        "model": vm_model,
        "dataset": vm_test_ds,
    },
    params={
        "average": "weighted",
        "max_fpr": 1.0,
    },
)

# Log the result to the ValidMind Platform
result.log()

Other Model Performance Unit Metrics

In addition to the ROC AUC score, you can compute other classification metrics, such as accuracy score, precision, recall etc. Let's list out all metrics and filter down to classification metrics. Then we can run and log them in a loop.

metrics = [metric for metric in list_metrics() if "classification" in metric]

for metric_id in metrics:
    describe_metric(metric_id)

for metric_id in metrics:
    result = run_metric(
        metric_id,
        inputs={
            "model": vm_model,
            "dataset": vm_test_ds,
        },
        show=False, # set show to False to avoid showing an ipywidget output
    )
    result.log()

    # instead of showing the widget, we can print the result value
    print(f"{metric_id}: {result.metric}")

Comparing Unit Metrics for Different Inputs

Running the ROC AUC Metric across Multiple Datasets

We can use the run_test() function from the validmind.tests module and pass in a unit metric ID to run the metric as a test. In practice this is useful for composing and comparing one or more metrics across one or more inptu combinations. If you have followed along some of the other instructional notebooks, you'll remember that run_test() accepts an input_grid parameter that will allow you to pass multiple values for each input. This will result in the metric being computed for each combination of input values and then combined into a single test result object. For instance, we can compute the ROC AUC score for the train and test datasets of our model. Let's do that now.

from validmind.tests import run_test

result = run_test(
    "validmind.unit_metrics.classification.ROC_AUC",
    input_grid={
        "model": [vm_model],
        "dataset": [vm_train_ds, vm_test_ds],
    },
)

result.log()

Composing Test Results from Individual Unit Metrics

Run multiple unit metrics as a single test

Another higher-level way to use unit metrics is to compose multiple metrics into a single test result. This is useful, for instance, if you want to create a single test result for your documentation that contains a table of model performance metrics. In the case of our customer churn model, we can compute the F1, Precision, Recall, Accuracy and ROC AUC scores for the test dataset and combine them into a single test result. Let's do that now.

from validmind.tests import run_test

result = run_test(
    name="Model Performance",
    unit_metrics=[
        "validmind.unit_metrics.classification.F1",
        "validmind.unit_metrics.classification.Precision",
        "validmind.unit_metrics.classification.Recall",
        "validmind.unit_metrics.classification.Accuracy",
        "validmind.unit_metrics.classification.ROC_AUC",
    ],
    inputs={
        "model": vm_model,
        "dataset": vm_test_ds,
    },
)

If we take a look at the result_id for the result, we'll see that it is a unique identifier that starts with validmind.composite_metric.<user-supplied-metric-name>. This will be used to identify this result as coming from a composite metric and is used to rebuild the composite metric as we will see in the next section.

result.result_id

Let's go ahead and log the result to save it to the ValidMind Platform.

result.log()

Bringing It All Together

Comparing Composite Metrics for Multiple Input Combinations

We can bring all of the above concepts together to create a very sophisticated test result that compares multiple metrics across multiple inputs. This is a very modular and flexible way to create complex test results for your model documentation. Let's go ahead and compare the training and test performance of our model across Accuracy, Precision, Recall, F1 and ROC AUC scores.

result = run_test(
    name="Model Performance Comparison Train vs Test",
    unit_metrics=[
        "validmind.unit_metrics.classification.F1",
        "validmind.unit_metrics.classification.Precision",
        "validmind.unit_metrics.classification.Recall",
        "validmind.unit_metrics.classification.Accuracy",
        "validmind.unit_metrics.classification.ROC_AUC",
    ],
    input_grid={
        "model": [vm_model],
        "dataset": [vm_train_ds, vm_test_ds],
    },
)

result.log()

Next steps

You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way — use the ValidMind Platform to work with your model documentation.

Work with your model documentation

1. From the Model Inventory in the ValidMind Platform, go to the model you registered earlier. (Need more help?)

2. Click and expand the Model Development section.

What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. Learn more ...

Discover more learning resources

We offer many interactive notebooks to help you document models:

• Run tests & test suites
• Code samples

Or, visit our documentation to learn more about ValidMind.

Upgrade ValidMind

After installing ValidMind, you’ll want to periodically make sure you are on the latest version to access any new features and other enhancements.

Retrieve the information for the currently installed version of ValidMind:

%pip show validmind

If the version returned is lower than the version indicated in our production open-source code, restart your notebook and run:

%pip install --upgrade validmind

You may need to restart your kernel after running the upgrade package for changes to be applied.