ValidMind for model development 2 — Start the model development process

Learn how to use ValidMind for your end-to-end model documentation process with our series of four introductory notebooks. In this second notebook, you'll run tests and investigate results, then add the results or evidence to your documentation.

You'll become familiar with the individual tests available in ValidMind, as well as how to run them and change parameters as necessary. Using ValidMind's repository of individual tests as building blocks helps you ensure that a model is being built appropriately.

For a full list of out-of-the-box tests, refer to our Test descriptions or try the interactive Test sandbox.

Learn by doing

Our course tailor-made for developers new to ValidMind combines this series of notebooks with more a more in-depth introduction to the ValidMind Platform — Developer Fundamentals

Prerequisites

In order to log test results or evidence to your model documentation with this notebook, you'll need to first have:

• Registered a model within the ValidMind Platform with a predefined documentation template
• Installed the ValidMind Library in your local environment, allowing you to access all its features

Need help with the above steps?

Refer to the first notebook in this series: 1 — Set up the ValidMind Library

Setting up

Initialize the ValidMind Library

First, let's connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:

1. In a browser, log in to ValidMind.

2. In the left sidebar, navigate to Inventory and select the model you registered for this "ValidMind for model development" series of notebooks.

3. Go to Getting Started and click Copy snippet to clipboard.

Next, load your model identifier credentials from an .env file or replace the placeholder with your own code snippet:

# Make sure the ValidMind Library is installed

%pip install -q validmind

# Load your model identifier credentials from an `.env` file

%load_ext dotenv
%dotenv .env

# Or replace with your code snippet

import validmind as vm

vm.init(
    # api_host="...",
    # api_key="...",
    # api_secret="...",
    # model="...",
)

Note: you may need to restart the kernel to use updated packages.

2025-12-31 01:01:03,425 - INFO(validmind.api_client): 🎉 Connected to ValidMind!
📊 Model: [ValidMind Academy] Model development (ID: cmalgf3qi02ce199qm3rdkl46)
📁 Document Type: model_documentation

Import sample dataset

Then, let's import the public Bank Customer Churn Prediction dataset from Kaggle.

In our below example, note that:

• The target column, Exited has a value of 1 when a customer has churned and 0 otherwise.
• The ValidMind Library provides a wrapper to automatically load the dataset as a Pandas DataFrame object. A Pandas Dataframe is a two-dimensional tabular data structure that makes use of rows and columns.

from validmind.datasets.classification import customer_churn as demo_dataset

print(
    f"Loaded demo dataset with: \n\n\t• Target column: '{demo_dataset.target_column}' \n\t• Class labels: {demo_dataset.class_labels}"
)

raw_df = demo_dataset.load_data()
raw_df.head()

Loaded demo dataset with: 

    • Target column: 'Exited' 
    • Class labels: {'0': 'Did not exit', '1': 'Exited'}

	CreditScore	Geography	Gender	Age	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
0	619	France	Female	42	2	0.00	1	1	1	101348.88	1
1	608	Spain	Female	41	1	83807.86	1	0	1	112542.58	0
2	502	France	Female	42	8	159660.80	3	1	0	113931.57	1
3	699	France	Female	39	1	0.00	2	0	0	93826.63	0
4	850	Spain	Female	43	2	125510.82	1	1	1	79084.10	0

Identify qualitative tests

Next, let's say we want to do some data quality assessments by running a few individual tests.

Use the vm.tests.list_tests() function introduced by the first notebook in this series in combination with vm.tests.list_tags() and vm.tests.list_tasks() to find which prebuilt tests are relevant for data quality assessment:

• tasks represent the kind of modeling task associated with a test. Here we'll focus on classification tasks.
• tags are free-form descriptions providing more details about the test, for example, what category the test falls into. Here we'll focus on the data_quality tag.

# Get the list of available task types
sorted(vm.tests.list_tasks())

['classification',
 'clustering',
 'data_validation',
 'feature_extraction',
 'monitoring',
 'nlp',
 'regression',
 'residual_analysis',
 'text_classification',
 'text_generation',
 'text_qa',
 'text_summarization',
 'time_series_forecasting',
 'visualization']

# Get the list of available tags
sorted(vm.tests.list_tags())

['AUC',
 'analysis',
 'anomaly_detection',
 'bias_and_fairness',
 'binary_classification',
 'calibration',
 'categorical_data',
 'classification',
 'classification_metrics',
 'clustering',
 'correlation',
 'credit_risk',
 'data_analysis',
 'data_distribution',
 'data_quality',
 'data_validation',
 'descriptive_statistics',
 'dimensionality_reduction',
 'distribution',
 'embeddings',
 'feature_importance',
 'feature_selection',
 'few_shot',
 'forecasting',
 'frequency_analysis',
 'kmeans',
 'linear_regression',
 'llm',
 'logistic_regression',
 'metadata',
 'model_comparison',
 'model_diagnosis',
 'model_explainability',
 'model_interpretation',
 'model_performance',
 'model_predictions',
 'model_selection',
 'model_training',
 'model_validation',
 'multiclass_classification',
 'nlp',
 'normality',
 'numerical_data',
 'outliers',
 'qualitative',
 'rag_performance',
 'ragas',
 'regression',
 'retrieval_performance',
 'scorecard',
 'seasonality',
 'senstivity_analysis',
 'sklearn',
 'stationarity',
 'statistical_test',
 'statistics',
 'statsmodels',
 'tabular_data',
 'text_data',
 'threshold_optimization',
 'time_series_data',
 'unit_root_test',
 'visualization',
 'zero_shot']

You can pass tags and tasks as parameters to the vm.tests.list_tests() function to filter the tests based on the tags and task types.

For example, to find tests related to tabular data quality for classification models, you can call list_tests() like this:

vm.tests.list_tests(task="classification", tags=["tabular_data", "data_quality"])

ID	Name	Description	Has Figure	Has Table	Required Inputs	Params	Tags	Tasks
validmind.data_validation.ClassImbalance	Class Imbalance	Evaluates and quantifies class distribution imbalance in a dataset used by a machine learning model....	True	True	['dataset']	{'min_percent_threshold': {'type': 'int', 'default': 10}}	['tabular_data', 'binary_classification', 'multiclass_classification', 'data_quality']	['classification']
validmind.data_validation.DescriptiveStatistics	Descriptive Statistics	Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's...	False	True	['dataset']	{}	['tabular_data', 'time_series_data', 'data_quality']	['classification', 'regression']
validmind.data_validation.Duplicates	Duplicates	Tests dataset for duplicate entries, ensuring model reliability via data quality verification....	False	True	['dataset']	{'min_threshold': {'type': '_empty', 'default': 1}}	['tabular_data', 'data_quality', 'text_data']	['classification', 'regression']
validmind.data_validation.HighCardinality	High Cardinality	Assesses the number of unique values in categorical columns to detect high cardinality and potential overfitting....	False	True	['dataset']	{'num_threshold': {'type': 'int', 'default': 100}, 'percent_threshold': {'type': 'float', 'default': 0.1}, 'threshold_type': {'type': 'str', 'default': 'percent'}}	['tabular_data', 'data_quality', 'categorical_data']	['classification', 'regression']
validmind.data_validation.HighPearsonCorrelation	High Pearson Correlation	Identifies highly correlated feature pairs in a dataset suggesting feature redundancy or multicollinearity....	False	True	['dataset']	{'max_threshold': {'type': 'float', 'default': 0.3}, 'top_n_correlations': {'type': 'int', 'default': 10}, 'feature_columns': {'type': 'list', 'default': None}}	['tabular_data', 'data_quality', 'correlation']	['classification', 'regression']
validmind.data_validation.MissingValues	Missing Values	Evaluates dataset quality by ensuring missing value ratio across all features does not exceed a set threshold....	False	True	['dataset']	{'min_threshold': {'type': 'int', 'default': 1}}	['tabular_data', 'data_quality']	['classification', 'regression']
validmind.data_validation.MissingValuesBarPlot	Missing Values Bar Plot	Assesses the percentage and distribution of missing values in the dataset via a bar plot, with emphasis on...	True	False	['dataset']	{'threshold': {'type': 'int', 'default': 80}, 'fig_height': {'type': 'int', 'default': 600}}	['tabular_data', 'data_quality', 'visualization']	['classification', 'regression']
validmind.data_validation.Skewness	Skewness	Evaluates the skewness of numerical data in a dataset to check against a defined threshold, aiming to ensure data...	False	True	['dataset']	{'max_threshold': {'type': '_empty', 'default': 1}}	['data_quality', 'tabular_data']	['classification', 'regression']
validmind.plots.BoxPlot	Box Plot	Generates customizable box plots for numerical features in a dataset with optional grouping using Plotly....	True	False	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'group_by': {'type': 'Optional', 'default': None}, 'width': {'type': 'int', 'default': 1800}, 'height': {'type': 'int', 'default': 1200}, 'colors': {'type': 'Optional', 'default': None}, 'show_outliers': {'type': 'bool', 'default': True}, 'title_prefix': {'type': 'str', 'default': 'Box Plot of'}}	['tabular_data', 'visualization', 'data_quality']	['classification', 'regression', 'clustering']
validmind.plots.HistogramPlot	Histogram Plot	Generates customizable histogram plots for numerical features in a dataset using Plotly....	True	False	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'bins': {'type': 'Union', 'default': 30}, 'color': {'type': 'str', 'default': 'steelblue'}, 'opacity': {'type': 'float', 'default': 0.7}, 'show_kde': {'type': 'bool', 'default': True}, 'normalize': {'type': 'bool', 'default': False}, 'log_scale': {'type': 'bool', 'default': False}, 'title_prefix': {'type': 'str', 'default': 'Histogram of'}, 'width': {'type': 'int', 'default': 1200}, 'height': {'type': 'int', 'default': 800}, 'n_cols': {'type': 'int', 'default': 2}, 'vertical_spacing': {'type': 'float', 'default': 0.15}, 'horizontal_spacing': {'type': 'float', 'default': 0.1}}	['tabular_data', 'visualization', 'data_quality']	['classification', 'regression', 'clustering']
validmind.stats.DescriptiveStats	Descriptive Stats	Provides comprehensive descriptive statistics for numerical features in a dataset....	False	True	['dataset']	{'columns': {'type': 'Optional', 'default': None}, 'include_advanced': {'type': 'bool', 'default': True}, 'confidence_level': {'type': 'float', 'default': 0.95}}	['tabular_data', 'statistics', 'data_quality']	['classification', 'regression', 'clustering']

Want to learn more about navigating ValidMind tests?

Refer to our notebook outlining the utilities available for viewing and understanding available ValidMind tests: Explore tests

Initialize the ValidMind datasets

With the individual tests we want to run identified, the next step is to connect your data with a ValidMind Dataset object. This step is always necessary every time you want to connect a dataset to documentation and produce test results through ValidMind, but you only need to do it once per dataset.

Initialize a ValidMind dataset object using the init_dataset function from the ValidMind (vm) module. For this example, we'll pass in the following arguments:

• dataset — The raw dataset that you want to provide as input to tests.
• input_id — A unique identifier that allows tracking what inputs are used when running each individual test.
• target_column — A required argument if tests require access to true values. This is the name of the target column in the dataset.

# vm_raw_dataset is now a VMDataset object that you can pass to any ValidMind test
vm_raw_dataset = vm.init_dataset(
    dataset=raw_df,
    input_id="raw_dataset",
    target_column="Exited",
)

Running tests

Now that we know how to initialize a ValidMind dataset object, we're ready to run some tests!

You run individual tests by calling the run_test function provided by the validmind.tests module. For the examples below, we'll pass in the following arguments:

• test_id — The ID of the test to run, as seen in the ID column when you run list_tests.
• params — A dictionary of parameters for the test. These will override any default_params set in the test definition.

Run tabular data tests

The inputs expected by a test can also be found in the test definition — let's take validmind.data_validation.DescriptiveStatistics as an example.

Note that the output of the describe_test() function below shows that this test expects a dataset as input:

vm.tests.describe_test("validmind.data_validation.DescriptiveStatistics")

▶ Test: Descriptive Statistics ('validmind.data_validation.DescriptiveStatistics')

Descriptive Statistics

Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's dataset.

Purpose

The purpose of the Descriptive Statistics metric is to provide a comprehensive summary of both numerical and categorical data within a dataset. This involves statistics such as count, mean, standard deviation, minimum and maximum values for numerical data. For categorical data, it calculates the count, number of unique values, most common value and its frequency, and the proportion of the most frequent value relative to the total. The goal is to visualize the overall distribution of the variables in the dataset, aiding in understanding the model's behavior and predicting its performance.

Test Mechanism

The testing mechanism utilizes two in-built functions of pandas dataframes: describe() for numerical fields and value_counts() for categorical fields. The describe() function pulls out several summary statistics, while value_counts() accounts for unique values. The resulting data is formatted into two distinct tables, one for numerical and another for categorical variable summaries. These tables provide a clear summary of the main characteristics of the variables, which can be instrumental in assessing the model's performance.

Signs of High Risk

• Skewed data or significant outliers can represent high risk. For numerical data, this may be reflected via a significant difference between the mean and median (50% percentile).
• For categorical data, a lack of diversity (low count of unique values), or overdominance of a single category (high frequency of the top value) can indicate high risk.

Strengths

• Provides a comprehensive summary of the dataset, shedding light on the distribution and characteristics of the variables under consideration.
• It is a versatile and robust method, applicable to both numerical and categorical data.
• Helps highlight crucial anomalies such as outliers, extreme skewness, or lack of diversity, which are vital in understanding model behavior during testing and validation.

Limitations

• While this metric offers a high-level overview of the data, it may fail to detect subtle correlations or complex patterns.
• Does not offer any insights on the relationship between variables.
• Alone, descriptive statistics cannot be used to infer properties about future unseen data.
• Should be used in conjunction with other statistical tests to provide a comprehensive understanding of the model's data.

Required Inputs: dataset

Parameters:

Parameter	Default Value

How to Run:

Code:

        
import validmind as vm

# inputs dictionary maps your inputs to the expected input names
# keys are the expected input names and values are the actual inputs
# values may be string input_ids or the actual VMDataset or VMModel objects
inputs = {
    "dataset": "my_vm_dataset"
}
params = {}

# to run and view the result of this test, run the following code:
result = vm.tests.run_test(
  "validmind.data_validation.DescriptiveStatistics", inputs=inputs, params=params
)

# To see the result of the test, ensure that you have called `vm.init()` and then run:
result.log()
    

Now, let's run a few tests to assess the quality of the dataset:

result = vm.tests.run_test(
    test_id="validmind.data_validation.DescriptiveStatistics",
    inputs={"dataset": vm_raw_dataset},
)

Descriptive Statistics

Descriptive Statistics is designed to provide a comprehensive summary of both numerical and categorical data within a dataset. The primary purpose of this test is to visualize the overall distribution of the variables, which aids in understanding the model's behavior and predicting its performance. By summarizing key statistics such as count, mean, standard deviation, and frequency, the test offers insights into the data's characteristics and potential anomalies.

The test operates by utilizing two in-built functions of pandas dataframes: describe() for numerical fields and value_counts() for categorical fields. The describe() function extracts summary statistics for numerical data, including count, mean, standard deviation, minimum, and maximum values, as well as percentiles. These metrics provide a snapshot of the data's central tendency, dispersion, and range. For categorical data, value_counts() calculates the count of each category, the number of unique values, the most common value, and its frequency. This helps identify the distribution and dominance of categories within the dataset. The results are formatted into two tables, one for numerical and another for categorical variables, offering a clear summary of the dataset's main characteristics.

The primary advantages of this test include its ability to provide a detailed overview of the dataset, highlighting the distribution and characteristics of the variables. It is versatile and robust, applicable to both numerical and categorical data, and helps identify crucial anomalies such as outliers, extreme skewness, or lack of diversity. These insights are vital for understanding model behavior during testing and validation. By offering a high-level summary, the test aids in identifying potential data quality issues and informs further analysis.

It should be noted that while this test offers a high-level overview of the data, it may not detect subtle correlations or complex patterns. It does not provide insights into the relationships between variables and should be used in conjunction with other statistical tests for a comprehensive understanding of the model's data. Additionally, skewed data or significant outliers can represent high risk, as indicated by a significant difference between the mean and median for numerical data. For categorical data, a lack of diversity or overdominance of a single category can also indicate high risk.

This test shows the results in two tables: one for numerical variables and another for categorical variables. The numerical variables table includes columns for count, mean, standard deviation, minimum, and maximum values, as well as percentiles (25%, 50%, 75%, 90%, and 95%). These metrics provide a detailed view of the data's distribution, central tendency, and variability. The categorical variables table includes columns for count, number of unique values, the most common value, its frequency, and the proportion of the most frequent value relative to the total. This helps identify the distribution and dominance of categories within the dataset. Notable observations include the range and scale of values, with specific attention to any significant outliers or skewness in the data.

The test results reveal the following key insights:

• Credit Score Distribution: The credit score data shows a mean of 650.16 with a standard deviation of 96.85, indicating a moderate spread around the mean. The scores range from 350 to 850, with the median at 652, suggesting a relatively normal distribution.
• Age Variability: The age data has a mean of 38.95 and a standard deviation of 10.46, with ages ranging from 18 to 92. The median age is 37, indicating a slightly younger population skew.
• Balance Characteristics: The balance data shows a mean of 76,434.10 with a high standard deviation of 62,612.25, indicating significant variability. The median balance is 97,264, with a notable number of zero balances, suggesting a skewed distribution.
• Geography Dominance: In the categorical data, France is the most common geography, representing 50.12% of the dataset, indicating a potential bias towards this region.
• Gender Distribution: The gender data shows a slight male dominance, with males representing 54.95% of the dataset, suggesting a relatively balanced gender distribution.

Based on these results, the dataset exhibits a diverse range of numerical values with some skewness and variability, particularly in balance and credit score data. The categorical data shows a dominance of certain categories, such as France in geography, which could influence model behavior. These insights suggest that while the dataset provides a broad representation of variables, attention should be given to the skewness and dominance observed, as they may impact the model's performance and generalizability. Understanding these characteristics is crucial for interpreting the model's predictions and ensuring robust performance across different scenarios.

Tables

Numerical Variables

Name	Count	Mean	Std	Min	25%	50%	75%	90%	95%	Max
CreditScore	8000.0	650.1596	96.8462	350.0	583.0	652.0	717.0	778.0	813.0	850.0
Age	8000.0	38.9489	10.4590	18.0	32.0	37.0	44.0	53.0	60.0	92.0
Tenure	8000.0	5.0339	2.8853	0.0	3.0	5.0	8.0	9.0	9.0	10.0
Balance	8000.0	76434.0965	62612.2513	0.0	0.0	97264.0	128045.0	149545.0	162488.0	250898.0
NumOfProducts	8000.0	1.5325	0.5805	1.0	1.0	1.0	2.0	2.0	2.0	4.0
HasCrCard	8000.0	0.7026	0.4571	0.0	0.0	1.0	1.0	1.0	1.0	1.0
IsActiveMember	8000.0	0.5199	0.4996	0.0	0.0	1.0	1.0	1.0	1.0	1.0
EstimatedSalary	8000.0	99790.1880	57520.5089	12.0	50857.0	99505.0	149216.0	179486.0	189997.0	199992.0

Categorical Variables

Name	Count	Number of Unique Values	Top Value	Top Value Frequency	Top Value Frequency %
Geography	8000.0	3.0	France	4010.0	50.12
Gender	8000.0	2.0	Male	4396.0	54.95

result2 = vm.tests.run_test(
    test_id="validmind.data_validation.ClassImbalance",
    inputs={"dataset": vm_raw_dataset},
    params={"min_percent_threshold": 30},
)

❌ Class Imbalance

Class Imbalance is designed to evaluate the distribution of target classes in a dataset used by a machine learning model. Its primary purpose is to ensure that the classes are not overly skewed, which could lead to bias in the model's predictions. A balanced training dataset is crucial to avoid creating a model that performs well for the majority class but poorly for the minority class.

The test operates by calculating the frequency of each class in the target column of the dataset, expressed as a percentage. It checks whether each class appears in at least a set minimum percentage of the total records, with the default threshold set at 10%. This involves counting the occurrences of each class and dividing by the total number of records to obtain the percentage. The test then compares these percentages against the threshold to determine if any class is under-represented. A class is marked as high risk if its percentage falls below the threshold, indicating potential imbalance. The typical range for these percentages is from 0% to 100%, with values below the threshold considered poor.

The primary advantages of this test include its ability to quickly identify under-represented classes that could affect model efficiency. The straightforward calculation makes it swift and easy to implement. It is highly informative, not only spotting imbalance but also quantifying the degree of imbalance. The adjustable threshold allows for flexibility, adapting to different use-cases or domain-specific needs. Additionally, the test provides a visual plot showing class proportions, enhancing interpretability and comprehension of the data.

It should be noted that the test might struggle with datasets containing a high number of classes, where imbalance could be inherent. Sensitivity to the threshold value might lead to incorrect imbalance detection if set too high. The test does not account for varying costs or impacts of misclassifying different classes, which can vary by application. While it identifies imbalances, it does not offer methods to address them. The test is applicable only for classification tasks and unsuitable for regression or clustering.

This test shows the results in both tabular and graphical formats. The table presents the percentage of rows for each class and whether they pass or fail the threshold test. The plot visually represents the class distribution, with the y-axis showing the percentage and the x-axis representing the classes. The table indicates that class '0' comprises 79.80% of the dataset and passes the threshold, while class '1' comprises 20.20% and fails. The plot corroborates this, showing a significant disparity between the two classes, with class '0' dominating the dataset. The threshold for passing is set at 30%, highlighting the imbalance as class '1' falls below this mark.

The test results reveal the following key insights:

• Dominance of Class '0': Class '0' constitutes 79.80% of the dataset, significantly exceeding the 30% threshold, indicating a strong presence.
• Under-representation of Class '1': Class '1' makes up only 20.20% of the dataset, failing the threshold and highlighting a potential imbalance issue.

Based on these results, the dataset exhibits a notable class imbalance, with class '0' being predominant and class '1' under-represented. This imbalance could lead to biased model predictions, favoring the majority class. The visual and tabular data both emphasize the need for addressing this imbalance to ensure fair and accurate model performance across all classes. The insights suggest that without intervention, the model may struggle to accurately predict the minority class, impacting its overall effectiveness.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	79.80%	Pass
1	20.20%	Fail

Figures

The output above shows that the class imbalance test did not pass according to the value we set for min_percent_threshold.

To address this issue, we'll re-run the test on some processed data. In this case let's apply a very simple rebalancing technique to the dataset:

import pandas as pd

raw_copy_df = raw_df.sample(frac=1)  # Create a copy of the raw dataset

# Create a balanced dataset with the same number of exited and not exited customers
exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 1]
not_exited_df = raw_copy_df.loc[raw_copy_df["Exited"] == 0].sample(n=exited_df.shape[0])

balanced_raw_df = pd.concat([exited_df, not_exited_df])
balanced_raw_df = balanced_raw_df.sample(frac=1, random_state=42)

With this new balanced dataset, you can re-run the individual test to see if it now passes the class imbalance test requirement.

As this is technically a different dataset, remember to first initialize a new ValidMind Dataset object to pass in as input as required by run_test():

# Register new data and now 'balanced_raw_dataset' is the new dataset object of interest
vm_balanced_raw_dataset = vm.init_dataset(
    dataset=balanced_raw_df,
    input_id="balanced_raw_dataset",
    target_column="Exited",
)

# Pass the initialized `balanced_raw_dataset` as input into the test run
result = vm.tests.run_test(
    test_id="validmind.data_validation.ClassImbalance",
    inputs={"dataset": vm_balanced_raw_dataset},
    params={"min_percent_threshold": 30},
)

✅ Class Imbalance

Class Imbalance is designed to evaluate the distribution of target classes in a dataset used by a machine learning model. Its primary purpose is to ensure that the classes are not overly skewed, which could lead to bias in the model's predictions. A balanced training dataset is crucial to avoid creating a model that is biased with high accuracy for the majority class and low accuracy for the minority class.

The test operates by calculating the frequency of each class in the target column of the dataset, expressed as a percentage. It checks whether each class appears in at least a set minimum percentage of the total records, with the default threshold set to 10%. This involves counting the occurrences of each class and dividing by the total number of records to obtain the percentage. The test then compares these percentages against the threshold to determine if any class is under-represented. A class is marked as high risk if it falls below the threshold, indicating potential imbalance. The typical range for these percentages is from 0% to 100%, with values below the threshold considered poor.

The primary advantages of this test include its ability to spot under-represented classes that could affect the efficiency of a machine learning model. The calculation is straightforward and swift, making it easy to implement. The test is highly informative as it not only spots imbalance but also quantifies the degree of imbalance. The adjustable threshold allows flexibility and adaptation to different use-cases or domain-specific needs. Additionally, the test provides a visually insightful plot showing the classes and their corresponding proportions, enhancing interpretability and comprehension of the data.

It should be noted that the test might struggle to perform well or provide vital insights for datasets with a high number of classes, where imbalance could be inevitable due to inherent class distribution. Sensitivity to the threshold value might result in faulty detection of imbalance if the threshold is set excessively high. Regardless of the percentage threshold, it doesn't account for varying costs or impacts of misclassifying different classes, which might fluctuate based on specific applications or domains. While it can identify imbalances in class distribution, it doesn't provide direct methods to address or correct these imbalances. The test is only applicable for classification operations and unsuitable for regression or clustering tasks.

This test shows a table and a plot representing the class distribution in the dataset. The table titled "Exited Class Imbalance" displays two classes, 0 and 1, each with a percentage of rows at 50.00%, and both classes pass the test based on the set threshold of 30%. The plot visually confirms this balance, with two bars of equal height representing the classes, each reaching the 0.5 mark on the y-axis, indicating 50% representation. The x-axis represents the class labels, while the y-axis shows the percentage. The equal distribution of the bars suggests a balanced dataset with no class falling below the threshold, indicating no immediate risk of class imbalance.

The test results reveal the following key insights:

• Balanced Class Distribution: Both classes, 0 and 1, have an equal representation of 50% in the dataset, indicating a perfectly balanced class distribution.
• Threshold Compliance: Each class exceeds the minimum percentage threshold of 30%, passing the test and suggesting no risk of class imbalance.

Based on these results, the dataset demonstrates a balanced class distribution, with both classes equally represented at 50%. This balance suggests that the model is unlikely to be biased towards any particular class, as each class meets and exceeds the minimum threshold requirement. The equal distribution ensures that the model can potentially perform well across different classes without favoring one over the other. This balance is crucial for maintaining the model's predictive accuracy and fairness, particularly in applications where class representation is critical.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	50.00%	Pass
1	50.00%	Pass

Figures

Utilize test output

You can utilize the output from a ValidMind test for further use, for example, if you want to remove highly correlated features. Removing highly correlated features helps make the model simpler, more stable, and easier to understand.

Below we demonstrate how to retrieve the list of features with the highest correlation coefficients and use them to reduce the final list of features for modeling.

First, we'll run validmind.data_validation.HighPearsonCorrelation with the balanced_raw_dataset we initialized previously as input as is for comparison with later runs:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)

❌ High Pearson Correlation

High Pearson Correlation is designed to identify highly correlated feature pairs in a dataset, suggesting feature redundancy or multicollinearity. The primary purpose of this test is to measure the linear relationship between features, which can indicate potential issues such as feature redundancy or multicollinearity that may affect the performance and interpretability of machine learning models.

The test operates by calculating pairwise Pearson correlations for all features in the dataset. It measures the strength and direction of the linear relationship between two variables, with the correlation coefficient ranging from -1 to 1. A value close to 1 indicates a strong positive linear relationship, while a value close to -1 indicates a strong negative linear relationship. A value around 0 suggests no linear relationship. The test sorts these correlations, removes duplicates and self-correlations, and evaluates them against a pre-set threshold, which is 0.3 by default. If the absolute value of a correlation exceeds this threshold, it is flagged as a potential issue. The test also returns the top n strongest correlations, providing a clear view of the most significant relationships in the dataset.

The primary advantages of this test include its ability to quickly and effectively identify linear relationships between feature pairs, which is crucial for understanding potential multicollinearity issues. This transparency allows developers and risk management teams to address these issues early in the model development process. The test's output is straightforward, displaying pairs of correlated variables along with their Pearson correlation coefficients and a Pass or Fail status. This makes it particularly useful for scenarios where model interpretability and performance are critical, as it helps ensure that the model is not unduly influenced by redundant features.

It should be noted that the test is limited to identifying linear relationships and does not account for nonlinear dependencies, which may also be present in the data. Additionally, the Pearson correlation coefficient is sensitive to outliers, which can skew the results and lead to misleading conclusions. The test only identifies redundancy within feature pairs, potentially missing more complex relationships involving three or more variables. High correlation coefficients exceeding the threshold indicate a risk of multicollinearity, which can lead to model overfitting and reduced interpretability.

This test shows the results in a tabular format, where each row represents a pair of features along with their correlation coefficient and a Pass or Fail status. The table includes columns for the feature pair, the calculated Pearson correlation coefficient, and whether the correlation passes the threshold test. The coefficients range from -0.1972 to 0.3489, with the threshold set at 0.3. The table highlights the feature pair (Age, Exited) with a coefficient of 0.3489, which fails the test, indicating a potential issue with multicollinearity. Other feature pairs, such as (IsActiveMember, Exited) and (Balance, NumOfProducts), have lower coefficients and pass the test, suggesting weaker linear relationships.

The test results reveal the following key insights:

• Age and Exited Correlation: The feature pair (Age, Exited) has a correlation coefficient of 0.3489, which exceeds the threshold and fails the test, indicating a strong linear relationship that may suggest multicollinearity.
• IsActiveMember and Exited Correlation: The feature pair (IsActiveMember, Exited) has a correlation coefficient of -0.1972, which passes the test, indicating a moderate negative linear relationship that does not pose a significant risk of multicollinearity.
• Balance and NumOfProducts Correlation: The feature pair (Balance, NumOfProducts) has a correlation coefficient of -0.1758, passing the test and suggesting a weak negative linear relationship.
• Overall Correlation Patterns: Most feature pairs exhibit weak correlations, with coefficients close to zero, indicating limited linear relationships and reduced risk of multicollinearity.

Based on these results, the test highlights a significant linear relationship between Age and Exited, which may require further investigation to address potential multicollinearity issues. The other feature pairs show weaker correlations, suggesting that they are less likely to contribute to redundancy or multicollinearity in the model. These insights provide a clearer understanding of the dataset's structure, allowing for more informed decisions regarding feature selection and model development. The results emphasize the importance of addressing the identified high correlation to ensure the model's robustness and interpretability.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3489	Fail
(IsActiveMember, Exited)	-0.1972	Pass
(Balance, NumOfProducts)	-0.1758	Pass
(Balance, Exited)	0.1637	Pass
(Age, Balance)	0.0552	Pass
(NumOfProducts, Exited)	-0.0513	Pass
(HasCrCard, IsActiveMember)	-0.0445	Pass
(Age, NumOfProducts)	-0.0405	Pass
(Tenure, EstimatedSalary)	0.0386	Pass
(NumOfProducts, IsActiveMember)	0.0381	Pass

The output above shows that the test did not pass according to the value we set for max_threshold.

corr_result is an object of type TestResult. We can inspect the result object to see what the test has produced:

print(type(corr_result))
print("Result ID: ", corr_result.result_id)
print("Params: ", corr_result.params)
print("Passed: ", corr_result.passed)
print("Tables: ", corr_result.tables)

<class 'validmind.vm_models.result.result.TestResult'>
Result ID:  validmind.data_validation.HighPearsonCorrelation
Params:  {'max_threshold': 0.3}
Passed:  False
Tables:  [ResultTable]

Let's remove the highly correlated features and create a new VM dataset object.

We'll begin by checking out the table in the result and extracting a list of features that failed the test:

# Extract table from `corr_result.tables`
features_df = corr_result.tables[0].data
features_df

	Columns	Coefficient	Pass/Fail
0	(Age, Exited)	0.3489	Fail
1	(IsActiveMember, Exited)	-0.1972	Pass
2	(Balance, NumOfProducts)	-0.1758	Pass
3	(Balance, Exited)	0.1637	Pass
4	(Age, Balance)	0.0552	Pass
5	(NumOfProducts, Exited)	-0.0513	Pass
6	(HasCrCard, IsActiveMember)	-0.0445	Pass
7	(Age, NumOfProducts)	-0.0405	Pass
8	(Tenure, EstimatedSalary)	0.0386	Pass
9	(NumOfProducts, IsActiveMember)	0.0381	Pass

# Extract list of features that failed the test
high_correlation_features = features_df[features_df["Pass/Fail"] == "Fail"]["Columns"].tolist()
high_correlation_features

['(Age, Exited)']

Next, extract the feature names from the list of strings (example: (Age, Exited) > Age):

high_correlation_features = [feature.split(",")[0].strip("()") for feature in high_correlation_features]
high_correlation_features

['Age']

Now, it's time to re-initialize the dataset with the highly correlated features removed.

Note the use of a different input_id. This allows tracking the inputs used when running each individual test.

# Remove the highly correlated features from the dataset
balanced_raw_no_age_df = balanced_raw_df.drop(columns=high_correlation_features)

# Re-initialize the dataset object
vm_raw_dataset_preprocessed = vm.init_dataset(
    dataset=balanced_raw_no_age_df,
    input_id="raw_dataset_preprocessed",
    target_column="Exited",
)

Re-running the test with the reduced feature set should pass the test:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

✅ High Pearson Correlation

High Pearson Correlation is designed to identify highly correlated feature pairs in a dataset, suggesting feature redundancy or multicollinearity. The primary purpose of this test is to measure the linear relationship between features, which can indicate potential issues such as feature redundancy or multicollinearity that may affect the performance and interpretability of machine learning models.

The test operates by calculating pairwise Pearson correlations for all features in the dataset. It measures the strength and direction of the linear relationship between two variables, with the correlation coefficient ranging from -1 to 1. A value of 1 indicates a perfect positive linear relationship, -1 indicates a perfect negative linear relationship, and 0 indicates no linear relationship. The test sorts these correlations, removing duplicates and self-correlations, and evaluates them against a pre-set threshold (defaulted at 0.3). If the absolute value of a correlation exceeds this threshold, it suggests a significant linear relationship. The test then returns the top n strongest correlations, providing a Pass or Fail status based on the threshold.

The primary advantages of this test include its ability to quickly and simply identify relationships between feature pairs, which is crucial for early detection of multicollinearity issues that could disrupt model training. The test's transparent output displays pairs of correlated variables along with their Pearson correlation coefficients and Pass or Fail status, making it easy to interpret. This feature is particularly useful in scenarios where model interpretability is critical, as it helps in understanding which features may be redundant and could potentially be removed or adjusted to improve model performance.

It should be noted that the test is limited to identifying linear relationships and does not account for nonlinear dependencies, which could also affect model performance. Additionally, the Pearson correlation coefficient is sensitive to outliers, meaning that a few extreme values can significantly impact the results. The test is also limited to identifying redundancy within feature pairs and may not detect more complex relationships involving three or more variables. High correlation coefficients exceeding the threshold indicate a risk of multicollinearity, which can lead to model overfitting and reduced interpretability.

This test shows the results in a tabular format, listing feature pairs, their correlation coefficients, and a Pass or Fail status. Each row represents a pair of features, with the "Columns" field indicating the feature pair, the "Coefficient" field showing the Pearson correlation coefficient, and the "Pass/Fail" field indicating whether the correlation exceeds the threshold. The coefficients range from -0.1972 to 0.0386, all of which are below the threshold of 0.3, resulting in a Pass status for all pairs. Notable observations include the highest correlation of -0.1972 between "IsActiveMember" and "Exited" and the lowest of -0.0328 between "Tenure" and "IsActiveMember," indicating weak linear relationships across the dataset.

The test results reveal the following key insights:

• Weak Correlations Across Features: All feature pairs exhibit weak correlations, with coefficients ranging from -0.1972 to 0.0386, indicating no significant linear relationships.
• No Multicollinearity Detected: Since all correlation coefficients are below the threshold of 0.3, there is no evidence of multicollinearity among the features.
• Balanced Feature Relationships: The distribution of correlation coefficients suggests balanced relationships among features, with no pair showing a dominant linear relationship.

Based on these results, the dataset does not exhibit significant linear relationships between feature pairs, suggesting that multicollinearity is not a concern. The weak correlations indicate that the features are relatively independent, which is beneficial for model interpretability and performance. This analysis provides confidence that the features can be used in modeling without the risk of redundancy or overfitting due to multicollinearity. The results support the robustness of the dataset for further analysis and model development.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(IsActiveMember, Exited)	-0.1972	Pass
(Balance, NumOfProducts)	-0.1758	Pass
(Balance, Exited)	0.1637	Pass
(NumOfProducts, Exited)	-0.0513	Pass
(HasCrCard, IsActiveMember)	-0.0445	Pass
(Tenure, EstimatedSalary)	0.0386	Pass
(NumOfProducts, IsActiveMember)	0.0381	Pass
(Tenure, HasCrCard)	0.0367	Pass
(CreditScore, EstimatedSalary)	-0.0332	Pass
(Tenure, IsActiveMember)	-0.0328	Pass

You can also plot the correlation matrix to visualize the new correlation between features:

corr_result = vm.tests.run_test(
    test_id="validmind.data_validation.PearsonCorrelationMatrix",
    inputs={"dataset": vm_raw_dataset_preprocessed},
)

Pearson Correlation Matrix

Pearson Correlation Matrix is designed to evaluate the extent of linear dependency between numerical variables in a dataset. The primary purpose is to identify potential redundancy by revealing high correlations, which can help in reducing the dimensionality of the dataset without significantly impacting the model's performance.

The test operates by generating a correlation matrix for all numerical variables using the Pearson correlation formula. This formula measures the linear relationship between two variables, producing a coefficient that ranges from -1 to 1. A value of 1 indicates a perfect positive linear relationship, -1 indicates a perfect negative linear relationship, and 0 indicates no linear relationship. The test visualizes this matrix as a heat map, where the color intensity represents the magnitude and direction of the correlation. High correlations, typically above 0.7 in absolute terms, are highlighted to indicate potential redundancy.

The primary advantages of this test include its ability to detect and quantify the linearity of relationships between variables, which aids in identifying redundant variables. This can simplify models and potentially improve performance by reducing complexity. The heatmap visualization provides an intuitive overview of correlations, making it accessible even to users who may not be comfortable with numerical matrices. This visual representation helps in quickly identifying areas of concern or interest within the dataset.

It should be noted that this test is limited to detecting linear relationships, potentially missing non-linear dependencies that could also be important for dimensionality reduction. It measures only the degree of linear relationship, not the strength of one variable's effect on another. The chosen threshold of 0.7 for high correlation is somewhat arbitrary and might exclude valid dependencies with lower coefficients. Additionally, a large number of highly correlated variables can indicate redundancy, posing a risk of overfitting if not addressed.

This test shows a heat map representing the Pearson correlation coefficients between pairs of numerical variables in the dataset. The axes of the heat map list the variables, and each cell's color indicates the correlation coefficient between the corresponding pair of variables. The scale ranges from -1 to 1, with darker colors representing stronger correlations. Notably, the heat map highlights correlations above 0.7 in white, indicating significant linear relationships. In this particular dataset, most correlations are low, with values close to zero, suggesting minimal linear dependency between the variables. However, some variables, such as "CreditScore" and "Tenure," show perfect correlation with themselves, as expected.

The test results reveal the following key insights:

• Low Overall Correlation: Most variable pairs exhibit low correlation coefficients, indicating minimal linear dependency.
• Notable Correlation: The "Balance" and "Exited" variables show a moderate positive correlation of 0.16, suggesting a potential relationship worth further exploration.
• Self-Correlation: Variables like "CreditScore" and "Tenure" show perfect self-correlation, as expected, confirming the accuracy of the matrix.

Based on these results, the dataset exhibits generally low linear correlations between variables, suggesting minimal redundancy. The moderate correlation between "Balance" and "Exited" may warrant further investigation to understand its implications for the model. Overall, the low correlation levels indicate that the dataset's dimensionality is unlikely to be significantly reduced through linear dependency analysis alone. This suggests that other methods, such as exploring non-linear relationships, may be necessary to achieve further dimensionality reduction or model simplification.

Figures

Documenting test results

Now that we've done some analysis on two different datasets, we can use ValidMind to easily document why certain things were done to our raw data with testing to support it.

Every test result returned by the run_test() function has a .log() method that can be used to send the test results to the ValidMind Platform:

• When using run_documentation_tests(), documentation sections will be automatically populated with the results of all tests registered in the documentation template.
• When logging individual test results to the platform, you'll need to manually add those results to the desired section of the model documentation.

To demonstrate how to add test results to your model documentation, we'll populate the entire Data Preparation section of the documentation using the clean vm_raw_dataset_preprocessed dataset as input, and then document an additional individual result for the highly correlated dataset vm_balanced_raw_dataset.

Run and log multiple tests

run_documentation_tests() allows you to run multiple tests at once and automatically log the results to your documentation. Below, we'll run the tests using the previously initialized vm_raw_dataset_preprocessed as input — this will populate the entire Data Preparation section for every test that is part of the documentation template.

For this example, we'll pass in the following arguments:

• inputs: Any inputs to be passed to the tests.
• config: A dictionary <test_id>:<test_config> that allows configuring each test individually. Each test config requires the following:
  • params: Individual test parameters.
  • inputs: Individual test inputs. This overrides any inputs passed from the run_documentation_tests() function.

When including explicit configuration for individual tests, you'll need to specify the inputs even if they mirror what is included in your global configuration.

# Individual test config with inputs specified
test_config = {
    "validmind.data_validation.ClassImbalance": {
        "params": {"min_percent_threshold": 30},
        "inputs": {"dataset": vm_raw_dataset_preprocessed},
    },
    "validmind.data_validation.HighPearsonCorrelation": {
        "params": {"max_threshold": 0.3},
        "inputs": {"dataset": vm_raw_dataset_preprocessed},
    },
}

# Global test config
tests_suite = vm.run_documentation_tests(
    inputs={
        "dataset": vm_raw_dataset_preprocessed,
    },
    config=test_config,
    section=["data_preparation"],
)

Test suite complete!

26/26 (100.0%)

Test Suite Results: Binary Classification V2

---

Check out the updated documentation on ValidMind.

Template for binary classification models.

▶ Data Preparation

Data Preparation

▶ Test Result: Dataset Description (validmind.data_validation.DatasetDescription)

Dataset Description

Dataset Description is designed to provide a comprehensive analysis and statistical summary of each column in a machine learning model's dataset. The primary purpose of this test is to offer a detailed overview of the dataset's characteristics, which includes vital statistics such as count, distinct values, missing values, and histograms for various data types. This analysis aids in understanding the data's structure and distribution, which is crucial for model training and evaluation.

The test operates by first inferring the data type of each column in the dataset, categorizing them as numeric, categorical, boolean, or text. For each column, the test collects statistical information, including the total count of entries, the number of missing values and their proportion, and the count of unique values along with their proportion. For numeric columns, histograms are generated to depict the distribution of values, while categorical, boolean, and text columns have histograms showing the frequency of each unique value. The test uses methods like "get_numerical_histograms" for numeric data and similar approaches for other data types. The typical range of values for these metrics varies, with missing and distinct percentages ranging from 0% to 100%. A high percentage of missing values or distinct values can indicate potential data quality issues or complexity in pattern recognition for the model.

The primary advantages of this test include its ability to provide a detailed analysis of the dataset, offering versatile summaries such as counts, unique values, and histograms. This flexibility allows it to handle different data types effectively, making it a valuable tool for detecting potential issues like missing values, unsupported data types, and irregular data distributions. By providing a comprehensive understanding of dataset features, developers can make informed decisions about data preprocessing and model training strategies. The test's ability to highlight data characteristics ensures that any anomalies or patterns are identified early in the model development process.

It should be noted that the test has some limitations and potential risks. The computational cost can be high, especially for large datasets with numerous columns, which may require significant resources. The use of an arbitrary number of bins for histograms might not always capture the optimal data distribution, potentially leading to misinterpretations. Unsupported data types in columns will raise errors, limiting the test's applicability. Additionally, columns with all null or missing values are excluded from histogram computations, which might overlook certain data characteristics. While the test provides insights into dataset quality, it does not directly address the model's performance, necessitating further analysis for comprehensive model evaluation.

This test shows a table format output that provides a detailed summary of each column in the dataset. The table includes columns for the name, type, count, missing values, missing percentage, distinct values, and distinct percentage for each dataset feature. The "Count" column indicates the total number of entries for each feature, while the "Missing" and "Missing %" columns show the number and proportion of missing values, respectively. The "Distinct" and "Distinct %" columns provide the count and proportion of unique values. For instance, the "CreditScore" column is numeric with 3232 entries, no missing values, and 438 distinct values, representing 13.55% of the total. The "EstimatedSalary" column, also numeric, has 3232 distinct values, indicating each entry is unique. The categorical columns like "Geography" and "Gender" have fewer distinct values, reflecting limited categories. This detailed breakdown allows for a clear understanding of the dataset's structure and potential areas of concern, such as high distinct percentages or unique value distributions.

The test results reveal the following key insights:

• Uniform Data Completeness: All columns in the dataset have complete data with no missing values, ensuring data integrity and reliability for model training.
• High Distinct Values in EstimatedSalary: The "EstimatedSalary" column has a distinct value for each entry, indicating a wide range of salary data, which could impact model predictions related to financial metrics.
• Limited Categories in Categorical Features: Categorical columns like "Geography" and "Gender" have a small number of distinct values, suggesting a limited set of categories that may simplify model interpretation but could also limit diversity.
• Varied Numeric Distributions: Numeric columns such as "CreditScore" and "Balance" show a significant number of distinct values, indicating diverse data distributions that may require normalization or scaling for effective model training.

Based on these results, the dataset exhibits a high level of completeness with no missing values, which is advantageous for model training as it reduces the need for imputation or data cleaning. The presence of high distinct values in certain numeric columns, particularly "EstimatedSalary," suggests a diverse range of data that could enhance the model's ability to generalize across different financial scenarios. However, the limited categories in categorical features may restrict the model's ability to capture nuanced patterns within those variables. The varied distributions in numeric columns highlight the need for careful preprocessing to ensure that the model can effectively learn from the data. Overall, the dataset's characteristics provide a solid foundation for model development, with specific areas identified for potential preprocessing to optimize model performance.

Tables

Dataset Description

Name	Type	Count	Missing	Missing %	Distinct	Distinct %
CreditScore	Numeric	3232.0	0	0.0	438	0.1355
Geography	Categorical	3232.0	0	0.0	3	0.0009
Gender	Categorical	3232.0	0	0.0	2	0.0006
Tenure	Numeric	3232.0	0	0.0	11	0.0034
Balance	Numeric	3232.0	0	0.0	2184	0.6757
NumOfProducts	Numeric	3232.0	0	0.0	4	0.0012
HasCrCard	Categorical	3232.0	0	0.0	2	0.0006
IsActiveMember	Categorical	3232.0	0	0.0	2	0.0006
EstimatedSalary	Numeric	3232.0	0	0.0	3232	1.0000
Exited	Categorical	3232.0	0	0.0	2	0.0006

▶ Test Result: Class Imbalance (validmind.data_validation.ClassImbalance)

✅ Class Imbalance

Class Imbalance is designed to evaluate the distribution of target classes in a dataset used by a machine learning model. Its primary purpose is to ensure that the classes are not overly skewed, which could lead to bias in the model's predictions. A balanced training dataset is crucial to avoid creating a model that is biased with high accuracy for the majority class and low accuracy for the minority class.

The test operates by calculating the frequency of each class in the target column of the dataset, expressed as a percentage. It checks whether each class appears in at least a set minimum percentage of the total records, with the default threshold set to 10%. This involves counting the occurrences of each class and dividing by the total number of records to obtain the percentage. The test then compares these percentages against the threshold to determine if any class is under-represented. A class is marked as high risk if it falls below the threshold, indicating potential imbalance. The typical range for these percentages is from 0% to 100%, with values below the threshold considered poor.

The primary advantages of this test include its ability to spot under-represented classes that could affect the efficiency of a machine learning model. The calculation is straightforward and swift, making it easy to implement. The test is highly informative as it not only spots imbalance but also quantifies the degree of imbalance. The adjustable threshold allows for flexibility and adaptation to different use-cases or domain-specific needs. Additionally, the test provides a visually insightful plot showing the classes and their corresponding proportions, enhancing interpretability and comprehension of the data.

It should be noted that the test might struggle to perform well or provide vital insights for datasets with a high number of classes, where imbalance could be inevitable due to inherent class distribution. Sensitivity to the threshold value might result in faulty detection of imbalance if the threshold is set excessively high. Regardless of the percentage threshold, it doesn't account for varying costs or impacts of misclassifying different classes, which might fluctuate based on specific applications or domains. While it can identify imbalances in class distribution, it doesn't provide direct methods to address or correct these imbalances. The test is only applicable for classification operations and unsuitable for regression or clustering tasks.

This test shows a table and a plot representing the class distribution in the dataset. The table titled "Exited Class Imbalance" displays two classes, 0 and 1, each with a "Percentage of Rows (%)" of 50.00%, and both classes have a "Pass" status under the "Pass/Fail" column. This indicates that both classes meet the minimum percentage threshold of 30%, as specified in the test parameters. The accompanying plot visually confirms this balance, with two bars of equal height representing the two classes, each reaching the 0.5 mark on the y-axis, which corresponds to 50%. The x-axis represents the class labels, and the y-axis represents the percentage of the total dataset. The plot provides a clear visual confirmation of the balanced distribution between the classes.

The test results reveal the following key insights:

• Balanced Class Distribution: Both classes, 0 and 1, have an equal distribution of 50% each, indicating a perfectly balanced dataset.
• Threshold Compliance: Each class exceeds the minimum percentage threshold of 30%, resulting in a "Pass" status for both classes.

Based on these results, the dataset used for the model demonstrates a balanced class distribution, with each class meeting and exceeding the specified minimum percentage threshold. This balance suggests that the model is unlikely to be biased towards any particular class, as both classes are equally represented. The equal distribution ensures that the model can potentially achieve consistent accuracy across different classes, reducing the risk of bias in predictions. The visual and tabular representations confirm the dataset's compliance with the class balance requirements, supporting the model's robustness in handling class predictions.

Parameters:

{
  "min_percent_threshold": 30
}
            

Tables

Exited Class Imbalance

Exited	Percentage of Rows (%)	Pass/Fail
0	50.00%	Pass
1	50.00%	Pass

Figures

▶ Test Result: Duplicates (validmind.data_validation.Duplicates)

✅ Duplicates

Duplicates is designed to ensure the reliability of a model by verifying the quality of the dataset through the detection of duplicate entries. The primary purpose of this test is to identify and quantify duplicate rows within the dataset, which can adversely affect model performance by introducing bias or redundancy. By identifying duplicates, the test helps maintain the integrity of the dataset, ensuring that the model is trained on unique and diverse data points.

The test operates by examining each row in the dataset to identify duplicates. If a specific text column is designated, the test focuses on that column; otherwise, it evaluates all feature columns. The test calculates both the number and percentage of duplicate rows, presenting these metrics in a DataFrame. The methodology involves comparing each row against others to detect exact matches, which are then counted and expressed as a percentage of the total dataset. The test is considered successful if the number of duplicates is below a user-defined threshold, indicating a dataset free from significant redundancy. The typical range for the percentage of duplicates is from 0% to 100%, with lower percentages indicating better data quality.

The primary advantages of this test include its ability to enhance the reliability of the model's training process by ensuring that the dataset is not contaminated with duplicate entries. This is crucial for maintaining the statistical validity of analyses and preventing overfitting, where the model might perform well on training data but poorly on unseen data. The test's provision of both absolute numbers and percentage values of duplicates offers a comprehensive overview of data quality. Additionally, its customizable threshold allows users to define acceptable levels of duplication based on specific project requirements, making it adaptable to various scenarios.

It should be noted that the test has limitations, such as its inability to differentiate between benign duplicates, which may occur naturally, and problematic duplicates resulting from data collection or processing errors. As the dataset size increases, the test becomes more computationally intensive, potentially limiting its applicability to very large datasets. Furthermore, the test only identifies exact duplicates, potentially overlooking semantically similar entries that are not identical in form but may still impact model performance.

This test shows the results in a tabular format, specifically focusing on the number of duplicate rows and their percentage relative to the total dataset. The table titled "Duplicate Rows Results for Dataset" presents two key metrics: the "Number of Duplicates" and the "Percentage of Rows (%)". In this case, the table indicates that there are zero duplicate entries, with a corresponding percentage of 0.0%. This suggests that the dataset is free from exact duplicate rows, which is a positive indicator of data quality. The table is straightforward to interpret, with the absence of duplicates suggesting that the dataset is well-prepared for model training without the risk of redundancy affecting the model's learning process.

The test results reveal the following key insights:

• No Duplicate Entries Detected: The dataset contains zero duplicate rows, indicating a high level of data quality and integrity.
• Zero Percentage of Duplicates: The percentage of duplicate rows is 0.0%, confirming that the dataset is free from redundancy and potential bias due to repeated entries.

Based on these results, the dataset demonstrates excellent data quality with no detected duplicates, which is beneficial for the model's training process. The absence of duplicate entries suggests that the dataset is diverse and unique, reducing the risk of overfitting and ensuring that the model can generalize well to new, unseen data. This outcome supports the reliability of the model's predictions and enhances its performance by providing a robust foundation for learning. The results underscore the importance of thorough data preprocessing in achieving accurate and reliable model outcomes.

Tables

Duplicate Rows Results for Dataset

Number of Duplicates	Percentage of Rows (%)
0	0.0

▶ Test Result: High Cardinality (validmind.data_validation.HighCardinality)

✅ High Cardinality

High Cardinality is designed to evaluate the number of unique values present in the categorical columns of a dataset to detect high cardinality and potential overfitting. The primary purpose of this test is to identify columns with a large number of unique, non-repetitive values, which can indicate a risk of overfitting and introduce unwanted noise into the model.

The test operates by first identifying the categorical columns within the dataset. It then calculates the number of distinct values (n_distinct) and the percentage of distinct values (p_distinct) for each categorical column. The test uses a predefined numeric threshold to determine whether a column passes or fails. If the number of distinct values in a column is less than this threshold, the column passes the test. The threshold is static and is determined based on the test parameters. The results are compiled into a table that includes the column name, the number of distinct values, the percentage of distinct values, and the pass/fail status. This approach helps in assessing the potential risk of overfitting by highlighting columns with high cardinality.

The primary advantages of this test include its ability to detect potential overfitting early in the model development process. By identifying columns with high cardinality, the test helps in reducing noise and improving data quality. It is versatile and can be applied to both classification and regression tasks, making it a valuable tool across different types of models. Additionally, the test aids in identifying potential outliers and inconsistencies within the dataset, which can further enhance the model's performance by ensuring that the data is clean and reliable.

It should be noted that the test is limited to categorical data types and does not apply to numerical or continuous features. This restriction limits its scope, as it cannot assess high cardinality in non-categorical data. Furthermore, the test does not consider the relevance or importance of unique values within categorical features, which means it might overlook critical data points that are significant for the model's performance. The static threshold used in the test may not be optimal for all datasets, and there is a need for mechanisms to adjust and refine this threshold to improve the test's effectiveness across diverse datasets and applications.

This test shows the results in a tabular format, presenting the analysis of categorical columns in terms of their distinct values. The table includes columns for the name of the categorical feature, the number of distinct values, the percentage of distinct values relative to the total number of entries, and the pass/fail status based on the predefined threshold. The "Number of Distinct Values" column indicates how many unique entries exist within each categorical feature, while the "Percentage of Distinct Values" provides a relative measure of these unique entries as a percentage of the total dataset size. The "Pass/Fail" column indicates whether the number of distinct values exceeds the threshold, with "Pass" suggesting that the column does not exhibit high cardinality. Notable observations include the fact that both the "Geography" and "Gender" columns have passed the test, indicating that they do not pose a high risk of overfitting due to high cardinality.

The test results reveal the following key insights:

• Low Cardinality in Geography: The "Geography" column has only 3 distinct values, representing 0.0928% of the dataset, which is well below the threshold, indicating low cardinality and a pass status.
• Minimal Distinct Values in Gender: The "Gender" column contains 2 distinct values, accounting for 0.0619% of the dataset, also below the threshold, resulting in a pass status.

Based on these results, the categorical columns "Geography" and "Gender" do not exhibit high cardinality, suggesting that they are unlikely to contribute to overfitting in the model. The low number of distinct values in these columns indicates that they are stable and consistent, which is beneficial for the model's performance. The pass status for both columns confirms that they meet the criteria for acceptable cardinality levels, reducing the risk of introducing noise or overfitting. This analysis provides confidence in the dataset's categorical features, ensuring that they are suitable for use in the model without the need for further modification or reduction in cardinality.

Tables

Column	Number of Distinct Values	Percentage of Distinct Values (%)	Pass/Fail
Geography	3	0.0928	Pass
Gender	2	0.0619	Pass

▶ Test Result: Missing Values (validmind.data_validation.MissingValues)

✅ Missing Values

Missing Values is designed to evaluate the quality of a dataset by measuring the number of missing values across all features. The primary purpose of this test is to ensure that the ratio of missing data to total data is less than a predefined threshold, which is crucial for maintaining the data quality necessary for reliable predictive strength in a machine learning model.

The test operates by iterating through each column of the dataset, counting missing values represented as NaNs, and calculating the percentage they represent against the total number of rows. This percentage is then compared to a predefined min_threshold to determine if the column passes or fails the test. The methodology involves a straightforward calculation where the number of missing entries in a column is divided by the total number of entries, resulting in a percentage that indicates the proportion of missing data. This percentage typically ranges from 0% to 100%, where a lower percentage is desirable. A column passes the test if its missing value percentage is below the threshold, indicating acceptable data quality.

The primary advantages of this test include its ability to quickly and granularly identify missing data across each feature in the dataset. This capability is particularly useful for data preprocessing in machine learning, where maintaining high data quality is essential for model accuracy and performance. By providing a clear and concise summary of missing data, the test allows data scientists to make informed decisions about data cleaning and imputation strategies. Additionally, the test's simplicity and efficiency make it a valuable tool for large datasets, where manual inspection of missing values would be impractical.

It should be noted that the test has limitations, such as not suggesting the root causes of the missing values or recommending ways to impute or handle them. It may also overlook features with significant missing data that still fall below the min_threshold, potentially impacting the model's performance. Furthermore, the test does not account for data encoded as values like "-999" or "None," which might not technically classify as missing but could bear similar implications. These limitations highlight the importance of complementing the test with additional data quality assessments and domain knowledge to fully understand and address missing data issues.

This test shows a table format output that summarizes the missing value analysis for each column in the dataset. The table includes columns for the feature name, the number of missing values, the percentage of missing values, and a Pass/Fail status based on the threshold comparison. Each column in the dataset is evaluated, and the results are presented in a clear and structured manner. The key measurements include the absolute count of missing values and their percentage relative to the total number of entries. Notable observations from the table indicate that all columns have zero missing values, resulting in a 0.0% missing value percentage for each feature. Consequently, every column passes the test, suggesting that the dataset is complete with respect to missing values.

The test results reveal the following key insights:

• Complete Data Across All Features: Every column in the dataset has zero missing values, indicating a complete dataset with no missing entries.
• Uniform Pass Status: All features pass the missing values test, as the percentage of missing values is 0.0% for each column, well below any reasonable threshold.
• High Data Quality: The absence of missing values across all features suggests a high level of data quality, which is beneficial for subsequent data analysis and modeling efforts.

Based on these results, the dataset demonstrates excellent data quality with respect to missing values, as all features have complete data with no missing entries. This uniformity across the dataset ensures that the data is well-prepared for further analysis and modeling without the need for additional imputation or data cleaning related to missing values. The absence of missing data across all features provides a strong foundation for building reliable and accurate machine learning models, as it eliminates potential biases and inaccuracies that could arise from incomplete data.

Tables

Column	Number of Missing Values	Percentage of Missing Values (%)	Pass/Fail
CreditScore	0	0.0	Pass
Geography	0	0.0	Pass
Gender	0	0.0	Pass
Tenure	0	0.0	Pass
Balance	0	0.0	Pass
NumOfProducts	0	0.0	Pass
HasCrCard	0	0.0	Pass
IsActiveMember	0	0.0	Pass
EstimatedSalary	0	0.0	Pass
Exited	0	0.0	Pass

▶ Test Result: Skewness (validmind.data_validation.Skewness)

❌ Skewness

Skewness is designed to evaluate the asymmetry in the distribution of numerical data within a dataset, ensuring data quality and optimizing model performance. The primary purpose of this test is to measure how much the distribution of data deviates from a normal distribution, which is crucial for maintaining the foundational assumptions of many machine learning models. By identifying skewness, the test helps in detecting potential data quality issues that could affect model predictions and inferences.

The test operates by calculating the skewness of each numerical column in the dataset. Skewness is a statistical measure that quantifies the degree of asymmetry of a distribution around its mean. A skewness value of zero indicates a perfectly symmetrical distribution, while positive or negative values indicate right or left skew, respectively. The test compares these calculated skewness values against a predefined threshold, typically set at 1. If the skewness of a column exceeds this threshold, it suggests a significant deviation from normality, and the test marks it as a fail. This mechanism allows for quick identification of columns that may require transformation or other preprocessing steps to improve model performance.

The primary advantages of this test include its ability to quickly and efficiently identify unequal data distributions that could impact model performance. By providing a quantitative measure of skewness, the test allows for easy comparison across different datasets or columns. The adjustable threshold parameter offers flexibility, enabling users to tailor the test to specific requirements or domain standards. This adaptability makes it particularly useful in scenarios where data normalization is critical for model accuracy and reliability, such as in financial modeling or risk assessment.

It should be noted that the test has limitations, including its focus solely on numeric columns, which may overlook skewness or bias in categorical or text data. Additionally, the assumption that data should follow a normal distribution may not always be applicable, especially in domains where non-normal distributions are the norm. The subjective nature of the threshold for skewness also requires expert input to ensure it aligns with the specific context and objectives of the analysis. High skewness levels that exceed the threshold can indicate potential issues with the model's assumptions, leading to suboptimal performance or biased predictions.

This test shows the skewness results for various columns in the dataset, presented in a tabular format. Each row of the table corresponds to a specific column, with columns for the skewness value and a pass/fail indicator based on the predefined threshold. The skewness values range from negative to positive, indicating the direction and degree of asymmetry. For instance, the "NumOfProducts" column has a skewness of 1.2404, which exceeds the threshold and results in a fail, suggesting a right-skewed distribution. In contrast, columns like "CreditScore" and "Balance" have skewness values close to zero, indicating near-symmetrical distributions and passing the test. The table provides a clear overview of which columns may require further attention to address potential skewness-related issues.

The test results reveal the following key insights:

• Overall Skewness Distribution: Most columns exhibit skewness values close to zero, indicating symmetrical distributions and passing the test.
• NumOfProducts Skewness: The "NumOfProducts" column shows a skewness of 1.2404, failing the test due to significant right skewness, which may require transformation.
• Symmetrical Columns: Columns such as "CreditScore" and "Balance" have skewness values of -0.0921 and -0.2476, respectively, indicating near-normal distributions.
• Stable Columns: Columns like "EstimatedSalary" and "Exited" show minimal skewness, with values of 0.0054 and 0.0, respectively, suggesting stable distributions.

Based on these results, the dataset predominantly consists of columns with symmetrical distributions, which is favorable for model performance. However, the "NumOfProducts" column's significant skewness suggests a need for data transformation to mitigate potential impacts on the model's assumptions and predictions. The overall low skewness in most columns indicates that the dataset is well-suited for models that assume normality, reducing the risk of biased inferences. This analysis provides a clear understanding of the data's distribution characteristics, guiding further preprocessing steps to enhance model accuracy and reliability.

Tables

Skewness Results for Dataset

Column	Skewness	Pass/Fail
CreditScore	-0.0921	Pass
Tenure	0.0593	Pass
Balance	-0.2476	Pass
NumOfProducts	1.2404	Fail
HasCrCard	-0.8614	Pass
IsActiveMember	0.1290	Pass
EstimatedSalary	0.0054	Pass
Exited	0.0000	Pass

▶ Test Result: Unique Rows (validmind.data_validation.UniqueRows)

❌ Unique Rows

Unique Rows is designed to verify the diversity of the dataset by ensuring that the count of unique rows exceeds a prescribed threshold. This test is crucial for assessing the quality of data used in training machine learning models, as it helps ensure that the dataset is varied enough to support robust and unbiased model development.

The test operates by first calculating the total number of rows in the dataset. It then determines the count of unique rows for each column, calculating the percentage of unique rows as the ratio of unique rows to the total row count. This percentage is compared against a predefined minimum threshold. If the percentage of unique rows in a column is less than this threshold, the column fails the test. The test results are compiled to provide a pass or fail verdict for each column, indicating whether the dataset meets the diversity requirements necessary for effective model training. The typical range for the percentage of unique values is from 0% to 100%, where higher percentages indicate greater diversity. A result below the threshold suggests insufficient diversity, which could lead to model overfitting and poor generalization.

The primary advantages of this test include its efficiency in evaluating data diversity across each column in the dataset. By systematically assessing the uniqueness of data, the test provides a quick and reliable method to gauge data quality, which is pivotal in developing effective machine learning models. This test is particularly useful in scenarios where data variety is crucial for model performance, as it helps identify columns that may contribute to biased or overfitted models due to lack of diversity.

It should be noted that the Unique Rows test assumes that data quality is directly proportional to its uniqueness, which may not always be the case. In some contexts, non-unique rows might be essential and should not be disregarded. Additionally, the test does not account for the relative importance of each column in predicting the output, treating all columns equally. This limitation can be significant in datasets with categorical variables, where the number of unique categories is inherently limited, potentially leading to misleading results about data quality.

This test shows the results in a tabular format, where each row represents a column from the dataset, and the columns of the table include the column name, the number of unique values, the percentage of unique values, and a pass/fail status. The table provides a clear overview of the diversity of each column, with the percentage of unique values indicating the level of diversity. Key measurements include the number of unique values and their percentage relative to the total dataset size. Notable observations include columns like "EstimatedSalary" with 100% unique values, indicating high diversity, while columns like "Geography" and "Gender" have very low percentages, failing the diversity threshold.

The test results reveal the following key insights:

• High Diversity in Estimated Salary: The "EstimatedSalary" column shows 100% unique values, indicating excellent diversity and passing the test.
• Low Diversity in Categorical Columns: Columns such as "Geography," "Gender," and "HasCrCard" have very low percentages of unique values, all below 1%, resulting in a fail status.
• Moderate Diversity in Credit Score and Balance: The "CreditScore" and "Balance" columns have moderate diversity with 13.552% and 67.5743% unique values, respectively, both passing the test.
• Uniformity in Binary Columns: Columns like "IsActiveMember" and "Exited" show minimal diversity with only 2 unique values each, failing the test.

Based on these results, the dataset exhibits a mix of high and low diversity across different columns. The high diversity in "EstimatedSalary" suggests that this column is well-suited for model training, while the low diversity in categorical and binary columns indicates potential risks of bias and overfitting. The moderate diversity in "CreditScore" and "Balance" suggests these columns may contribute positively to model robustness. Overall, the results highlight the need for careful consideration of column diversity in model development to ensure balanced and effective learning.

Tables

Column	Number of Unique Values	Percentage of Unique Values (%)	Pass/Fail
CreditScore	438	13.5520	Pass
Geography	3	0.0928	Fail
Gender	2	0.0619	Fail
Tenure	11	0.3403	Fail
Balance	2184	67.5743	Pass
NumOfProducts	4	0.1238	Fail
HasCrCard	2	0.0619	Fail
IsActiveMember	2	0.0619	Fail
EstimatedSalary	3232	100.0000	Pass
Exited	2	0.0619	Fail

▶ Test Result: Too Many Zero Values (validmind.data_validation.TooManyZeroValues)

❌ Too Many Zero Values

Too Many Zero Values is designed to identify numerical columns in a dataset that contain an excessive number of zero values, which may indicate data sparsity or a lack of variation. The primary purpose of this test is to highlight columns where the proportion of zero values exceeds a predefined threshold, set by default at 0.03%, potentially impacting the effectiveness of these columns in a machine learning model.

The test operates by iterating through each numerical column in the dataset, calculating the total number of zero values, and determining their ratio to the total number of rows. This ratio is then compared against a threshold percentage, which is set at 0.03% by default. If the percentage of zero values in a column surpasses this threshold, the column is flagged as having too many zero values. The test results are presented in a summary format, indicating the count and percentage of zero values for each numerical column, along with a pass or fail status. This mechanism allows for the identification of columns that may require further investigation due to potential data quality issues.

The primary advantages of this test include its ability to efficiently highlight columns with an excessive number of zero values, which might otherwise be overlooked in large datasets. By providing both counts and percentages of zero values, the test offers a detailed view of the distribution and proportion of zeros within each column. This can be particularly useful for identifying data sparsity issues that could affect model performance. Additionally, the test's flexibility in allowing the threshold to be adjusted makes it adaptable to the specific needs of different analyses or models, ensuring that it can be tailored to various contexts and requirements.

It should be noted that this test is limited to detecting zero values and does not account for other potential data issues, such as extreme values, missing data, or outliers. It also does not consider the context of the dataset, where a high number of zero values might be normal and not indicative of poor data quality. Furthermore, the test does not evaluate non-numerical columns, which may present different types of data quality concerns. In some cases, zero values may be meaningful, and labeling them as excessive could lead to misinterpretation of the data. Additionally, the test does not identify patterns of zeros that could be significant in time-series or longitudinal data.

This test shows the results in a tabular format, where each row represents a numerical column from the dataset. The table includes columns for the variable name, total row count, number of zero values, percentage of zero values, and a pass or fail status. The "Variable" column lists the names of the numerical columns being evaluated. The "Row Count" column indicates the total number of observations in each column. The "Number of Zero Values" column provides the count of zero values found in each column, while the "Percentage of Zero Values (%)" column shows the proportion of zero values relative to the total row count, expressed as a percentage. The "Pass/Fail" column indicates whether the column has passed or failed the test based on the predefined threshold. Notable observations include columns with a high percentage of zero values, such as "IsActiveMember" with 53.2178% zero values, which significantly exceeds the threshold and suggests potential data sparsity issues.

The test results reveal the following key insights:

• High Zero Value Proportion in 'IsActiveMember': The 'IsActiveMember' column has a notably high percentage of zero values at 53.2178%, indicating a significant lack of variation and potential data sparsity.
• Significant Zero Values in 'Balance': The 'Balance' column shows 32.4567% zero values, suggesting that a substantial portion of the dataset may not have balance information, which could impact model predictions.
• Zero Values in 'HasCrCard': The 'HasCrCard' column contains 30.229% zero values, indicating a considerable number of entries without credit card information, which may affect the model's ability to accurately assess credit-related features.
• Moderate Zero Values in 'Tenure': The 'Tenure' column has 3.7129% zero values, which, while lower than other columns, still exceeds the threshold and may warrant further investigation.

Based on these results, the test highlights several columns with a high proportion of zero values, which could indicate data sparsity and potentially impact the model's performance. The 'IsActiveMember' column, in particular, shows a significant lack of variation, which may affect its utility in predictive modeling. Similarly, the 'Balance' and 'HasCrCard' columns exhibit substantial zero values, suggesting that a large portion of the dataset lacks critical financial information. These insights underscore the importance of addressing data quality issues related to zero values, as they can influence the model's ability to make accurate predictions. The test results provide a clear indication of which columns may require further examination or data preprocessing to ensure the robustness and reliability of the model.

Tables

Variable	Row Count	Number of Zero Values	Percentage of Zero Values (%)	Pass/Fail
Tenure	3232	120	3.7129	Fail
Balance	3232	1049	32.4567	Fail
HasCrCard	3232	977	30.2290	Fail
IsActiveMember	3232	1720	53.2178	Fail

▶ Test Result: IQR Outliers Table (validmind.data_validation.IQROutliersTable)

IQR Outliers Table

IQR Outliers Table is designed to identify and summarize outliers within numerical features of a dataset using the Interquartile Range (IQR) method. This test is crucial for data preprocessing as outliers can significantly distort statistical analysis and impact the performance of machine learning models.

The test operates by calculating the IQR for each numerical feature in the dataset, which is the range between the first quartile (25th percentile) and the third quartile (75th percentile). An outlier is defined as any data point that falls below "Q1 - 1.5 * IQR" or above "Q3 + 1.5 * IQR". This method is robust because it relies on quartile calculations, which are less sensitive to extreme values than other measures like the mean. The test outputs the number of outliers and their summary statistics, including minimum, 25th percentile, median, 75th percentile, and maximum values for each feature. The default threshold for identifying outliers is set at 1.5 but can be adjusted based on user requirements. This approach is particularly useful for datasets where numerical features are prone to extreme values that could skew analysis.

The primary advantages of this test include its ability to provide a comprehensive summary of outliers for each numerical feature, which helps in pinpointing features with potential quality issues. The IQR method is particularly robust to extremely high or low outlier values because it is based on quartile calculations rather than mean or standard deviation, which can be heavily influenced by outliers. Additionally, the test can be customized to focus on selected features and allows users to set specific thresholds for outlier detection, making it adaptable to various datasets and analytical needs. This flexibility is beneficial in scenarios where certain features are known to have different distributions or when the dataset has undergone significant preprocessing.

It should be noted that the test might produce false positives if the variable deviates from a normal or near-normal distribution, especially in the case of skewed distributions. The IQR method does not provide interpretation or recommendations for addressing outliers, leaving further analysis to users or data scientists. It is also only applicable to numerical features, excluding categorical data from analysis. The default thresholds may not be optimal for datasets with heavy preprocessing, manipulation, or inherently high kurtosis, which can lead to an overestimation of outliers. Additionally, the presence of a large number of outliers in multiple features or outliers significantly distanced from the mean value of variables can indicate data entry errors or other data quality issues.

This test shows the results in a tabular format, summarizing the outliers detected by the IQR method for each numerical feature. The table includes columns for the variable name, total count of outliers, mean value of the variable, and summary statistics of the outliers such as minimum, 25th percentile, median, 75th percentile, and maximum values. The "CreditScore" variable has 8 outliers, with a mean value of 647.8555, and outlier values ranging from 350 to 367. The "NumOfProducts" variable has 44 outliers, with a mean value of 1.5065, and all outlier values are consistently at 4. This indicates that the outliers for "NumOfProducts" are concentrated at a single value, whereas the "CreditScore" outliers are more varied. The table provides a clear overview of the outlier distribution and allows for easy identification of features with significant outlier presence.

The test results reveal the following key insights:

• Concentration of Outliers in NumOfProducts: The "NumOfProducts" variable shows a high concentration of outliers, with all 44 outliers having a value of 4, indicating a potential data entry issue or a specific condition leading to this value.
• Varied Outlier Range in CreditScore: The "CreditScore" variable has 8 outliers with values ranging from 350 to 367, suggesting a more varied distribution of outliers compared to "NumOfProducts".
• Mean Value Context: The mean value of "CreditScore" is 647.8555, which is significantly higher than the outlier values, indicating that these outliers are substantially lower than the average credit score in the dataset.

Based on these results, the analysis highlights that the "NumOfProducts" variable has a significant number of outliers concentrated at a single value, which may require further investigation to determine if this is due to data entry errors or a specific business rule. In contrast, the "CreditScore" variable exhibits a more varied range of outlier values, which could indicate different underlying factors affecting these scores. The presence of outliers in these features suggests potential areas for data quality improvement or further exploration to understand the causes of these anomalies. Understanding these patterns is crucial for ensuring the accuracy and reliability of subsequent analyses and model predictions.

Tables

Summary of Outliers Detected by IQR Method

Variable	Total Count of Outliers	Mean Value of Variable	Minimum Outlier Value	Outlier Value at 25th Percentile	Outlier Value at 50th Percentile	Outlier Value at 75th Percentile	Maximum Outlier Value
CreditScore	8	647.8555	350	350.0	350.0	360.5	367
NumOfProducts	44	1.5065	4	4.0	4.0	4.0	4

▶ Test Result: IQR Outliers Bar Plot (validmind.data_validation.IQROutliersBarPlot)

IQR Outliers Bar Plot

IQR Outliers Bar Plot is designed to visually analyze and evaluate the extent of outliers in numeric variables based on percentiles. Its primary purpose is to clarify the dataset's distribution, flag possible abnormalities, and gauge potential risks associated with processing potentially skewed data, which can affect the machine learning model's predictive prowess.

The test operates by calculating the 25th percentile (Q1) and 75th percentile (Q3) for each numeric feature in the dataset to derive the Interquartile Range (IQR), which is the difference between Q1 and Q3. It then determines the lower and upper thresholds by subtracting Q1 from the threshold times IQR and adding Q3 to the threshold times IQR, with the default threshold set at 1.5. Values falling below the lower threshold or exceeding the upper threshold are labeled as outliers. The number of outliers is tallied for different percentiles, such as [0-25], [25-50], [50-75], and [75-100], and these counts are used to construct a bar plot for each feature, showcasing the distribution of outliers across different percentiles.

The primary advantages of this test include its ability to effectively identify outliers in the data through visual means, facilitating easier comprehension and offering insights into the outliers' possible impact on the model. It provides flexibility by accommodating all numeric features or a chosen subset and is task-agnostic, making it viable for both classification and regression tasks. Additionally, it can handle large datasets as its operation does not rely on computationally heavy processes, making it efficient and scalable.

It should be noted that the test's application is limited to numerical variables and does not extend to categorical ones. It only reveals the presence and distribution of outliers and does not provide insights into how these outliers might affect the model's predictive performance. The assumption that data is unimodal and symmetric may not always hold true, and in cases with non-normal distributions, the results can be misleading. Signs of high risk include a prevalence of outliers potentially skewing the distribution, outliers dominating higher percentiles (75-100), and certain features harboring most of their values as outliers, which may not contribute positively to the model's forecasting ability.

This test shows the distribution of outliers across different percentiles for numeric features using bar plots. Each plot represents a feature, with the x-axis indicating the percentile ranges [0-25], [25-50], [50-75], and [75-100], and the y-axis showing the outlier count. The plots provide a visual representation of where outliers are concentrated within the data distribution. For instance, the "CreditScore" plot shows outliers primarily in the 50-75 percentile range, with a smaller number in the 75-100 range, while the "NumOfProducts" plot indicates a significant concentration of outliers in the 75-100 percentile range. These visualizations help identify features with potential skewness or extreme values that could impact model performance.

The test results reveal the following key insights:

• Concentration of Outliers in CreditScore: The "CreditScore" feature shows a concentration of outliers in the 50-75 percentile range, indicating potential skewness in this segment of the data.
• High Outlier Count in NumOfProducts: The "NumOfProducts" feature has a significant number of outliers in the 75-100 percentile range, suggesting the presence of extreme values that could influence model predictions.

Based on these results, the distribution of outliers across different percentiles highlights potential areas of concern for model performance. The concentration of outliers in specific percentile ranges suggests that certain features may have skewed distributions or extreme values that could affect the model's predictive accuracy. Understanding these patterns allows for better data preprocessing and feature engineering to mitigate the impact of outliers on the model's behavior. The insights gained from this analysis can guide further investigation into the underlying causes of outliers and inform strategies for handling them in the modeling process.

Figures

▶ Test Result: Descriptive Statistics (validmind.data_validation.DescriptiveStatistics)

Descriptive Statistics

Descriptive Statistics is designed to provide a comprehensive summary of both numerical and categorical data within a dataset. The primary purpose of this test is to visualize the overall distribution of the variables, which aids in understanding the model's behavior and predicting its performance. By summarizing key statistics such as count, mean, standard deviation, and frequency, this test offers insights into the data's characteristics and potential anomalies.

The test operates by utilizing two in-built functions of pandas dataframes: describe() for numerical fields and value_counts() for categorical fields. The describe() function extracts summary statistics like count, mean, standard deviation, and percentiles, which help in understanding the central tendency and dispersion of numerical data. For categorical data, value_counts() calculates the count of each category, the number of unique values, and the frequency of the most common category. These metrics are crucial for identifying patterns such as skewness or dominance of a particular category. The typical range for numerical metrics like mean and standard deviation varies depending on the data, while categorical metrics focus on the proportion of the top category, which can indicate a lack of diversity if excessively high.

The primary advantages of this test include its ability to provide a detailed overview of the dataset, highlighting the distribution and characteristics of the variables. This versatility makes it applicable to both numerical and categorical data, allowing for a broad analysis of the dataset. By identifying anomalies such as outliers or extreme skewness, the test helps in understanding potential risks and behaviors of the model during testing and validation. This comprehensive summary is instrumental in assessing the model's performance and ensuring data quality.

It should be noted that while this test offers a high-level overview of the data, it may not detect subtle correlations or complex patterns. It does not provide insights into the relationships between variables, which are crucial for understanding interactions within the dataset. Additionally, descriptive statistics alone cannot infer properties about future unseen data, making it necessary to use this test in conjunction with other statistical analyses. High-risk signs include skewed data or significant outliers, which can be indicated by a large difference between the mean and median. For categorical data, a low count of unique values or a high frequency of a single category can suggest a lack of diversity.

This test shows the results in two tables: one for numerical variables and another for categorical variables. The numerical table includes columns for count, mean, standard deviation, minimum, maximum, and various percentiles, providing a detailed view of the data's distribution. For instance, the "CreditScore" variable has a mean of 647.86 and a standard deviation of 98.80, with values ranging from 350 to 850. The categorical table presents the count, number of unique values, and the frequency of the top category. For example, the "Geography" variable shows "France" as the most common category, representing 46.57% of the data. These tables allow for easy identification of key metrics and notable observations, such as the dominance of certain categories or the presence of outliers in numerical data.

The test results reveal the following key insights:

• Credit Score Distribution: The "CreditScore" variable has a mean of 647.86, with a standard deviation of 98.80, indicating a moderate spread around the mean. The scores range from 350 to 850, with the median at 649, suggesting a relatively normal distribution with slight skewness.
• Balance Variability: The "Balance" variable shows significant variability, with a mean of 80,986.99 and a standard deviation of 61,455.80. The median balance is 102,211, indicating a right-skewed distribution, as the mean is lower than the median.
• Geographical Dominance: In the categorical data, "France" is the most frequent category in the "Geography" variable, accounting for 46.57% of the entries, which may suggest a geographical bias in the dataset.
• Gender Proportion: The "Gender" variable is almost evenly split, with "Male" being the top category at 50.12%, indicating a balanced representation of genders in the dataset.

Based on these results, the dataset exhibits a mix of balanced and skewed distributions across its variables. The numerical data, such as "CreditScore" and "Balance," show varying degrees of spread and skewness, which could impact the model's predictions if not addressed. The categorical data highlights potential biases, particularly in geographical representation, which may influence model outcomes. These insights suggest that while the dataset provides a broad overview of the variables, further analysis is needed to explore relationships and interactions that could affect the model's performance.

Tables

Numerical Variables

Name	Count	Mean	Std	Min	25%	50%	75%	90%	95%	Max
CreditScore	3232.0	647.8555	98.7962	350.0	579.0	649.0	719.0	776.0	813.0	850.0
Tenure	3232.0	4.9876	2.8964	0.0	3.0	5.0	8.0	9.0	10.0	10.0
Balance	3232.0	80986.9904	61455.7996	0.0	0.0	102211.0	128274.0	150114.0	164815.0	250898.0
NumOfProducts	3232.0	1.5065	0.6699	1.0	1.0	1.0	2.0	2.0	3.0	4.0
HasCrCard	3232.0	0.6977	0.4593	0.0	0.0	1.0	1.0	1.0	1.0	1.0
IsActiveMember	3232.0	0.4678	0.4990	0.0	0.0	0.0	1.0	1.0	1.0	1.0
EstimatedSalary	3232.0	99985.0091	57955.2865	12.0	50159.0	99749.0	150232.0	179704.0	190134.0	199929.0

Categorical Variables

Name	Count	Number of Unique Values	Top Value	Top Value Frequency	Top Value Frequency %
Geography	3232.0	3.0	France	1505.0	46.57
Gender	3232.0	2.0	Male	1620.0	50.12

▶ Test Result: Pearson Correlation Matrix (validmind.data_validation.PearsonCorrelationMatrix)

Pearson Correlation Matrix

Pearson Correlation Matrix is designed to evaluate the extent of linear dependency between numerical variables in a dataset. The primary purpose is to identify potential redundancy by revealing high correlations, which can help in reducing the dimensionality of the dataset without significantly impacting the model's performance.

The test operates by generating a correlation matrix for all numerical variables using the Pearson correlation formula. This formula measures the linear relationship between two variables, providing a coefficient that ranges from -1 to 1. A value of 1 indicates a perfect positive correlation, -1 a perfect negative correlation, and 0 no correlation. The test visualizes these relationships in a heat map, where the color intensity represents the magnitude and direction of the correlation. High correlations, typically above 0.7 in absolute terms, are highlighted to indicate potential redundancy.

The primary advantages of this test include its ability to detect and quantify linear relationships between variables, which aids in identifying redundant variables. This can simplify models and potentially improve performance by reducing complexity. The heatmap visualization offers an intuitive overview of correlations, making it accessible even to users who may not be comfortable with numerical matrices. This visual representation helps in quickly identifying areas of concern or interest within the dataset.

It should be noted that this test is limited to detecting linear relationships, potentially missing non-linear dependencies that could also be significant. It measures only the degree of linear relationship, not the strength of one variable's effect on another. The threshold of 0.7 for high correlation is somewhat arbitrary and might exclude valid dependencies with lower coefficients. Additionally, a large number of highly correlated variables can indicate redundancy, posing a risk of overfitting if not addressed.

This test shows a heat map representing the Pearson correlation coefficients between various numerical variables in the dataset. Each axis of the heat map corresponds to a different variable, and the color of each cell indicates the strength and direction of the correlation between the variables at that intersection. The scale ranges from -1 to 1, with darker colors representing stronger correlations. Notably, the heat map highlights correlations above 0.7 in white, although in this particular dataset, no such high correlations are present. The matrix reveals that most variables have low correlation coefficients, suggesting minimal linear dependency. The highest observed correlation is between "Balance" and "Exited" at 0.16, which is still relatively low.

The test results reveal the following key insights:

• Low Overall Correlation: Most variables exhibit low correlation coefficients, indicating minimal linear dependency across the dataset.
• Balance and Exited Correlation: The highest correlation observed is between "Balance" and "Exited" at 0.16, suggesting a weak relationship.
• Minimal Redundancy: The lack of high correlations suggests that there is minimal redundancy among the variables, reducing the risk of overfitting.

Based on these results, the dataset shows a generally low level of linear dependency among its numerical variables, indicating that each variable may contribute unique information to the model. The weak correlation between "Balance" and "Exited" suggests a slight relationship, but not strong enough to warrant immediate concern for redundancy. The absence of high correlations implies that the dataset is well-suited for modeling without significant risk of overfitting due to redundant variables. This analysis provides a clear understanding of the linear relationships present, guiding further exploration or model refinement.

Figures

▶ Test Result: High Pearson Correlation (validmind.data_validation.HighPearsonCorrelation)

✅ High Pearson Correlation

High Pearson Correlation is designed to identify highly correlated feature pairs in a dataset, suggesting potential feature redundancy or multicollinearity. The primary purpose of this test is to measure the linear relationship between features, which can impact the performance and interpretability of machine learning models. By detecting high correlations, developers and risk management teams can address these issues to ensure robust model development.

The test operates by calculating pairwise Pearson correlations for all features in the dataset. It then sorts these correlations, removing duplicates and self-correlations. The Pearson correlation coefficient, which ranges from -1 to 1, quantifies the linear relationship between two variables. A value close to 1 or -1 indicates a strong linear relationship, while a value near 0 suggests no linear correlation. The test assigns a Pass or Fail status based on whether the absolute value of the correlation coefficient exceeds a pre-set threshold, which is 0.3 by default. The test also returns the top n strongest correlations, providing a clear view of the most significant relationships in the dataset.

The primary advantages of this test include its ability to quickly and simply identify relationships between feature pairs, which is crucial for early detection of multicollinearity issues that could disrupt model training. The test's transparent output displays pairs of correlated variables, the Pearson correlation coefficient, and a Pass or Fail status, making it easy to interpret. This feature is particularly useful in scenarios where model interpretability and performance are critical, as it allows for the identification and mitigation of potential issues before they affect the model's outcomes.

It should be noted that the test has limitations, such as its focus on linear relationships, which means it cannot detect nonlinear dependencies. Additionally, the Pearson correlation coefficient is sensitive to outliers, which can significantly affect the results. The test is also limited to identifying redundancy within feature pairs, potentially missing more complex relationships involving three or more variables. High correlation coefficients exceeding the threshold indicate a risk of multicollinearity and model overfitting, which can undermine the model's interpretability and predictive power.

This test shows the results in a tabular format, listing feature pairs, their Pearson correlation coefficients, and a Pass or Fail status. Each row represents a pair of features, with the correlation coefficient indicating the strength and direction of their linear relationship. The table includes columns for the feature pair, the calculated coefficient, and the Pass/Fail status based on the threshold of 0.3. The coefficients range from -0.1972 to 0.1637, all of which are below the threshold, resulting in a Pass status for each pair. Notable observations include the highest correlation between "IsActiveMember" and "Exited" with a coefficient of -0.1972, and the lowest between "Tenure" and "IsActiveMember" with a coefficient of -0.0328.

The test results reveal the following key insights:

• Low Correlation Across Features: All feature pairs exhibit low correlation coefficients, with none exceeding the threshold of 0.3, indicating minimal risk of multicollinearity.
• Negative Correlations Predominate: The majority of the correlations are negative, with the strongest being between "IsActiveMember" and "Exited" at -0.1972, suggesting an inverse relationship.
• Minimal Linear Relationships: The low coefficients across all pairs suggest that the features do not share strong linear relationships, reducing concerns about feature redundancy.

Based on these results, the dataset appears to have minimal issues with feature redundancy or multicollinearity, as indicated by the low Pearson correlation coefficients across all feature pairs. The absence of high correlations suggests that the features are relatively independent, which is beneficial for model interpretability and performance. The predominance of negative correlations, particularly between "IsActiveMember" and "Exited," may warrant further investigation to understand the underlying relationships. Overall, the test results provide confidence in the dataset's suitability for modeling, with no immediate concerns about linear dependencies affecting the model's behavior.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(IsActiveMember, Exited)	-0.1972	Pass
(Balance, NumOfProducts)	-0.1758	Pass
(Balance, Exited)	0.1637	Pass
(NumOfProducts, Exited)	-0.0513	Pass
(HasCrCard, IsActiveMember)	-0.0445	Pass
(Tenure, EstimatedSalary)	0.0386	Pass
(NumOfProducts, IsActiveMember)	0.0381	Pass
(Tenure, HasCrCard)	0.0367	Pass
(CreditScore, EstimatedSalary)	-0.0332	Pass
(Tenure, IsActiveMember)	-0.0328	Pass

Run and log an individual test

Next, we'll use the previously initialized vm_balanced_raw_dataset (that still has a highly correlated Age column) as input to run an individual test, then log the result to the ValidMind Platform.

When running individual tests, you can use a custom result_id to tag the individual result with a unique identifier:

• This result_id can be appended to test_id with a : separator.
• The balanced_raw_dataset result identifier will correspond to the balanced_raw_dataset input, the dataset that still has the Age column.

result = vm.tests.run_test(
    test_id="validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset",
    params={"max_threshold": 0.3},
    inputs={"dataset": vm_balanced_raw_dataset},
)
result.log()

❌ High Pearson Correlation Balanced Raw Dataset

High Pearson Correlation: Balanced Raw Dataset is designed to identify highly correlated feature pairs in a dataset, which may suggest feature redundancy or multicollinearity. The primary purpose of this test is to detect linear relationships between features that could impact the performance and interpretability of machine learning models. By identifying these correlations, developers and risk management teams can address potential issues that may arise from multicollinearity, such as model overfitting and reduced interpretability.

The test operates by calculating pairwise Pearson correlation coefficients for all features in the dataset. The Pearson correlation coefficient is a statistical measure that quantifies the linear relationship between two variables, ranging from -1 to 1. A value of 1 indicates a perfect positive linear relationship, -1 indicates a perfect negative linear relationship, and 0 indicates no linear relationship. The test sorts these coefficients, removes duplicate and self-correlations, and evaluates each pair against a pre-set threshold (0.3 in this case). If the absolute value of the coefficient exceeds this threshold, the pair is flagged as a potential issue. The test outputs the top n strongest correlations, providing a Pass or Fail status based on the threshold.

The primary advantages of this test include its ability to quickly and effectively identify linear relationships between feature pairs, which is crucial for early detection of multicollinearity issues. This transparency allows for straightforward interpretation of the results, as the test outputs a clear list of correlated variable pairs along with their correlation coefficients and Pass/Fail status. This makes it particularly useful in scenarios where model interpretability is critical, as it helps ensure that each feature's predictive power is authentic and not overshadowed by redundant variables.

It should be noted that the test is limited to detecting linear relationships and may not capture nonlinear dependencies between features. Additionally, the Pearson correlation coefficient is sensitive to outliers, which can skew results and lead to misleading interpretations. The test also focuses on pairwise correlations, potentially missing more complex multicollinearity involving three or more variables. High correlation coefficients exceeding the threshold indicate a risk of multicollinearity, which can lead to model overfitting and reduced interpretability.

This test shows the results in a tabular format, listing feature pairs, their Pearson correlation coefficients, and a Pass or Fail status based on the threshold of 0.3. The table provides a clear view of which feature pairs exhibit significant linear relationships. The columns represent the feature pairs, the calculated correlation coefficient, and the Pass/Fail status. The coefficients range from -0.1972 to 0.3489, with the highest correlation observed between "Age" and "Exited" at 0.3489, which fails the test due to exceeding the threshold. The table allows for easy identification of potential multicollinearity issues, with the Pass/Fail status indicating whether each pair's correlation is within acceptable limits.

The test results reveal the following key insights:

• High Correlation Between Age and Exited: The feature pair "Age" and "Exited" shows a correlation coefficient of 0.3489, which exceeds the threshold and is marked as Fail, indicating a strong linear relationship that could affect model performance.
• Low Correlation Among Other Features: Other feature pairs, such as "IsActiveMember" and "Exited" or "Balance" and "NumOfProducts," have correlation coefficients well below the threshold, indicating weak linear relationships and passing the test.
• Overall Low Multicollinearity Risk: Most feature pairs exhibit low correlation coefficients, suggesting minimal risk of multicollinearity affecting the model's interpretability and performance.

Based on these results, the dataset shows a generally low risk of multicollinearity, with most feature pairs exhibiting weak linear relationships. The exception is the "Age" and "Exited" pair, which demonstrates a significant correlation that may warrant further investigation to ensure it does not adversely impact the model. The overall low correlation among other features suggests that the dataset is well-suited for modeling, with minimal redundancy and a clear distinction between the predictive power of individual variables. This analysis provides a comprehensive understanding of the dataset's feature relationships, guiding further model development and refinement.

Parameters:

{
  "max_threshold": 0.3
}
            

Tables

Columns	Coefficient	Pass/Fail
(Age, Exited)	0.3489	Fail
(IsActiveMember, Exited)	-0.1972	Pass
(Balance, NumOfProducts)	-0.1758	Pass
(Balance, Exited)	0.1637	Pass
(Age, Balance)	0.0552	Pass
(NumOfProducts, Exited)	-0.0513	Pass
(HasCrCard, IsActiveMember)	-0.0445	Pass
(Age, NumOfProducts)	-0.0405	Pass
(Tenure, EstimatedSalary)	0.0386	Pass
(NumOfProducts, IsActiveMember)	0.0381	Pass

2025-12-31 01:03:26,426 - INFO(validmind.vm_models.result.result): Test driven block with result_id validmind.data_validation.HighPearsonCorrelation:balanced_raw_dataset does not exist in model's document

Note the output returned indicating that a test-driven block doesn't currently exist in your model's documentation for this particular test ID.

That's expected, as when we run individual tests the results logged need to be manually added to your documentation within the ValidMind Platform.

Add individual test results to model documentation

With the test results logged, let's head to the model we connected to at the beginning of this notebook and insert our test results into the documentation (Need more help?):

1. From the Inventory in the ValidMind Platform, go to the model you connected to earlier.

2. In the left sidebar that appears for your model, click Documentation under Documents.

3. Locate the Data Preparation section and click on 2.3. Correlations and Interactions to expand that section.

4. Hover under the Pearson Correlation Matrix content block until a horizontal dashed line with a + button appears, indicating that you can insert a new block.

   
   
   

5. Click + and then select Test-Driven Block under FROM LIBRARY:

   • Click on VM Library under TEST-DRIVEN in the left sidebar.
   • In the search bar, type in HighPearsonCorrelation.
   • Select HighPearsonCorrelation:balanced_raw_dataset as the test.

   A preview of the test gets shown:

   
   
   

6. Finally, click Insert 1 Test Result to Document to add the test result to the documentation.

   Confirm that the individual results for the high correlation test has been correctly inserted into section 2.3. Correlations and Interactions of the documentation.

7. Finalize the documentation by editing the test result's description block to explain the changes you made to the raw data and the reasons behind them as shown in the screenshot below:

   

Model testing

So far, we've focused on the data assessment and pre-processing that usually occurs prior to any models being built. Now, let's instead assume we have already built a model and we want to incorporate some model results into our documentation.

Train simple logistic regression model

Using ValidMind tests, we'll train a simple logistic regression model on our dataset and evaluate its performance by using the LogisticRegression class from the sklearn.linear_model.

To start, let's grab the first few rows from the balanced_raw_no_age_df dataset with the highly correlated features removed we initialized earlier:

balanced_raw_no_age_df.head()

	CreditScore	Geography	Gender	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited
4721	850	Spain	Male	7	115378.94	1	0	1	162087.82	0
6987	574	Spain	Male	5	120355.00	1	1	0	137793.35	0
2183	620	France	Male	8	0.00	2	1	1	145937.99	0
395	531	France	Female	6	0.00	1	0	0	194998.34	1
7245	702	Germany	Female	5	138597.54	2	1	1	41536.59	1

Before training the model, we need to encode the categorical features in the dataset:

• Use the OneHotEncoder class from the sklearn.preprocessing module to encode the categorical features.
• The categorical features in the dataset are Geography and Gender.

balanced_raw_no_age_df = pd.get_dummies(
    balanced_raw_no_age_df, columns=["Geography", "Gender"], drop_first=True
)
balanced_raw_no_age_df.head()

	CreditScore	Tenure	Balance	NumOfProducts	HasCrCard	IsActiveMember	EstimatedSalary	Exited	Geography_Germany	Geography_Spain	Gender_Male
4721	850	7	115378.94	1	0	1	162087.82	0	False	True	True
6987	574	5	120355.00	1	1	0	137793.35	0	False	True	True
2183	620	8	0.00	2	1	1	145937.99	0	False	False	True
395	531	6	0.00	1	0	0	194998.34	1	False	False	False
7245	702	5	138597.54	2	1	1	41536.59	1	True	False	False

We'll split our preprocessed dataset into training and testing, to help assess how well the model generalizes to unseen data:

• We start by dividing our balanced_raw_no_age_df dataset into training and test subsets using train_test_split, with 80% of the data allocated to training (train_df) and 20% to testing (test_df).
• From each subset, we separate the features (all columns except "Exited") into X_train and X_test, and the target column ("Exited") into y_train and y_test.

from sklearn.model_selection import train_test_split

train_df, test_df = train_test_split(balanced_raw_no_age_df, test_size=0.20)

X_train = train_df.drop("Exited", axis=1)
y_train = train_df["Exited"]
X_test = test_df.drop("Exited", axis=1)
y_test = test_df["Exited"]

Then using GridSearchCV, we'll find the best-performing hyperparameters or settings and save them:

from sklearn.linear_model import LogisticRegression

# Logistic Regression grid params
log_reg_params = {
    "penalty": ["l1", "l2"],
    "C": [0.001, 0.01, 0.1, 1, 10, 100, 1000],
    "solver": ["liblinear"],
}

# Grid search for Logistic Regression
from sklearn.model_selection import GridSearchCV

grid_log_reg = GridSearchCV(LogisticRegression(), log_reg_params)
grid_log_reg.fit(X_train, y_train)

# Logistic Regression best estimator
log_reg = grid_log_reg.best_estimator_

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1135: FutureWarning:

'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', and C=np.inf instead of penalty=None.

/opt/hostedtoolcache/Python/3.11.14/x64/lib/python3.11/site-packages/sklearn/linear_model/_logistic.py:1160: UserWarning:

Inconsistent values: penalty=l1 with l1_ratio=0.0. penalty is deprecated. Please use l1_ratio only.

Initialize model evaluation objects

The last step for evaluating the model's performance is to initialize the ValidMind Dataset and Model objects in preparation for assigning model predictions to each dataset.

# Initialize the datasets into their own dataset objects
vm_train_ds = vm.init_dataset(
    input_id="train_dataset_final",
    dataset=train_df,
    target_column="Exited",
)

vm_test_ds = vm.init_dataset(
    input_id="test_dataset_final",
    dataset=test_df,
    target_column="Exited",
)

You'll also need to initialize a ValidMind model object (vm_model) that can be passed to other functions for analysis and tests on the data for each of our three models.

You simply initialize this model object with vm.init_model():

# Register the model
vm_model = vm.init_model(log_reg, input_id="log_reg_model_v1")

Assign predictions

Once the model has been registered you can assign model predictions to the training and testing datasets.

• The assign_predictions() method from the Dataset object can link existing predictions to any number of models.
• This method links the model's class prediction values and probabilities to our vm_train_ds and vm_test_ds datasets.

If no prediction values are passed, the method will compute predictions automatically:

vm_train_ds.assign_predictions(model=vm_model)
vm_test_ds.assign_predictions(model=vm_model)

2025-12-31 01:03:27,435 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2025-12-31 01:03:27,437 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2025-12-31 01:03:27,438 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2025-12-31 01:03:27,440 - INFO(validmind.vm_models.dataset.utils): Done running predict()
2025-12-31 01:03:27,443 - INFO(validmind.vm_models.dataset.utils): Running predict_proba()... This may take a while
2025-12-31 01:03:27,443 - INFO(validmind.vm_models.dataset.utils): Done running predict_proba()
2025-12-31 01:03:27,445 - INFO(validmind.vm_models.dataset.utils): Running predict()... This may take a while
2025-12-31 01:03:27,446 - INFO(validmind.vm_models.dataset.utils): Done running predict()

Run the model evaluation tests

In this next example, we'll focus on running the tests within the Model Development section of the model documentation. Only tests associated with this section will be executed, and the corresponding results will be updated in the model documentation.

• Note the additional config that is passed to run_documentation_tests() — this allows you to override inputs or params in certain tests.
• In our case, we want to explicitly use the vm_train_ds for the validmind.model_validation.sklearn.ClassifierPerformance:in_sample test, since it's supposed to run on the training dataset and not the test dataset.

test_config = {
    "validmind.model_validation.sklearn.ClassifierPerformance:in_sample": {
        "inputs": {
            "dataset": vm_train_ds,
            "model": vm_model,
        },
    }
}
results = vm.run_documentation_tests(
    section=["model_development"],
    inputs={
        "dataset": vm_test_ds,  # Any test that requires a single dataset will use vm_test_ds
        "model": vm_model,
        "datasets": (
            vm_train_ds,
            vm_test_ds,
        ),  # Any test that requires multiple datasets will use vm_train_ds and vm_test_ds
    },
    config=test_config,
)

Test suite complete!

34/34 (100.0%)

Test Suite Results: Binary Classification V2

---

Check out the updated documentation on ValidMind.

Template for binary classification models.

▶ Model Development

Model Development

▶ Test Result: Model Metadata (validmind.model_validation.ModelMetadata)

Model Metadata

Model Metadata is designed to compare the metadata of different models, focusing on their architecture, framework, framework version, and programming language. This test aims to provide a comprehensive overview of the models' technical specifications, facilitating easier comparison and identification of potential integration or deployment challenges.

The test operates by utilizing the get_model_info function to extract metadata from each model. This metadata includes key attributes such as the modeling technique, framework, framework version, and programming language. The function then standardizes these attributes by renaming columns according to a predefined set of labels, ensuring consistency across different models. The compiled information is presented in a summary table, which allows for straightforward comparison. This process is crucial for identifying discrepancies or inconsistencies in model documentation, which could indicate potential risks in model management. The test does not delve into detailed parameter information but focuses on high-level metadata, which is essential for understanding the broader technical landscape of the models.

The primary advantages of this test include its ability to provide a clear and standardized comparison of essential model metadata. By focusing on high-level attributes such as framework and programming language, the test helps identify potential compatibility issues that could arise during model integration or deployment. This is particularly useful in environments where multiple models are used, and consistency is crucial for seamless operation. The test's ability to standardize metadata labels also enhances interpretability, making it easier for stakeholders to understand and compare different models' technical specifications. This can be especially beneficial in collaborative settings where multiple teams may be involved in model development and deployment.

It should be noted that the test has certain limitations and potential risks. It assumes that the get_model_info function accurately retrieves all necessary metadata fields, which may not always be the case if the metadata is incomplete or incorrectly documented. Additionally, the test focuses on high-level metadata and does not include detailed parameter information, which could be a limitation in scenarios where such details are critical for decision-making. Inconsistent or missing metadata across models can indicate potential issues in model documentation or management, posing a risk to model reliability. Furthermore, significant differences in framework versions or programming languages might pose challenges in model integration and deployment, especially if backward compatibility is not maintained.

This test shows a summary table output format, which presents the metadata of the models in a structured manner. The table includes columns for the modeling technique, modeling framework, framework version, and programming language. Each row represents a different model, allowing for easy comparison across these key attributes. The table format is straightforward to read, with each column clearly labeled to indicate the type of metadata being compared. Key measurements include the specific framework version and programming language used, which are critical for assessing compatibility and integration potential. Notable observations from the table can include any discrepancies in framework versions or programming languages, which could highlight potential integration challenges. The range and scale of values are determined by the specific versions and languages used, with the expectation that consistent use of frameworks and languages across models is generally preferable for ease of integration.

The test results reveal the following key insights:

• Consistent Use of Python: All models utilize Python as the programming language, indicating a uniform approach to model development that can facilitate easier integration and maintenance.
• Framework Version Uniformity: The use of sklearn version 1.8.0 across models suggests a standardized framework version, which is beneficial for ensuring compatibility and reducing the risk of integration issues.
• Single Modeling Technique: The presence of a single modeling technique, SKlearnModel, across the models indicates a consistent approach to model architecture, which can simplify the deployment process.

Based on these results, the metadata comparison highlights a consistent use of Python and sklearn version 1.8.0 across the models, suggesting a standardized approach to model development. This uniformity in programming language and framework version is advantageous for integration and deployment, as it reduces the likelihood of compatibility issues. The use of a single modeling technique further supports this consistency, indicating a streamlined approach to model architecture. These insights suggest that the models are well-aligned in terms of their technical specifications, which can facilitate smoother operational processes and reduce the complexity of managing multiple models within the same environment.

Tables

Modeling Technique	Modeling Framework	Framework Version	Programming Language
SKlearnModel	sklearn	1.8.0	Python

▶ Test Result: Dataset Split (validmind.data_validation.DatasetSplit)

Dataset Split

Dataset Split is designed to evaluate and visualize the distribution of data among training, testing, and validation datasets within a machine learning model. The primary purpose of this test is to ensure that the datasets are split appropriately, as an imbalanced distribution can negatively impact the model's ability to learn effectively and generalize to new, unseen data.

The test operates by first calculating the total size of all datasets involved in the model. It then determines the size of each individual dataset and calculates its proportion relative to the total dataset size. This involves summing the sizes of the training, testing, and validation datasets to obtain a total size. Each dataset's size is then divided by this total to compute its proportion, expressed as a percentage. The results are summarized in a table that displays the dataset names, their sizes, and their respective proportions of the total dataset size. This approach provides a clear quantitative measure of how the data is distributed across different datasets, with proportions typically ranging from 0% to 100%. A balanced distribution is generally considered beneficial, with a larger proportion allocated to the training dataset, while ensuring sufficient data is reserved for testing and validation to assess model performance.

The primary advantages of this test include its ability to provide a clear and understandable visualization of dataset split proportions, which can quickly highlight any potential imbalances. This is particularly useful for identifying issues that may arise from an inappropriate data split, such as overfitting or underfitting. The test is versatile and applicable to a wide range of task types, including classification, regression, and text-related tasks. Additionally, it is not tied to any specific data type, making it suitable for tabular data, time series data, or text data. This flexibility allows it to be used across various domains and applications, providing valuable insights into the data distribution for different machine learning models.

It should be noted that the Dataset Split test has certain limitations. It does not provide insights into the quality or diversity of the data within each split, focusing solely on size and proportion. This means that while it can identify imbalances in dataset sizes, it does not address potential issues related to data quality or representativeness. Additionally, the test does not offer recommendations or adjustments for imbalanced datasets, leaving it up to the user to interpret the results and make necessary changes. Furthermore, the test may have limited applicability for more complex data splitting methods, such as stratified or time-based splits, which could affect its usefulness in certain scenarios. High-risk signs include a very small training dataset, which may hinder the model's learning, or a large training dataset with a small test dataset, potentially leading to overfitting.

This test shows the distribution of data across the training and testing datasets through a table format. The table presents three columns: Dataset, Size, and Proportion. The 'Dataset' column lists the names of the datasets, including 'train_dataset_final', 'test_dataset_final', and 'Total'. The 'Size' column indicates the number of data points in each dataset, with 'train_dataset_final' containing 2,585 data points and 'test_dataset_final' containing 647 data points, summing up to a total of 3,232 data points. The 'Proportion' column shows the percentage of the total dataset size that each dataset represents, with the training dataset accounting for 79.98% and the testing dataset for 20.02%. The table provides a clear view of how the data is split, allowing for easy identification of any imbalances. The values are presented as whole numbers for sizes and percentages for proportions, facilitating straightforward interpretation. Notable observations include the substantial allocation of data to the training dataset, which is typical for ensuring robust model training, while the testing dataset is sufficiently sized to evaluate model performance.

The test results reveal the following key insights:

• Training Dataset Dominance: The training dataset, 'train_dataset_final', comprises 79.98% of the total data, indicating a strong focus on model training.
• Adequate Testing Proportion: The testing dataset, 'test_dataset_final', represents 20.02% of the total data, providing a reasonable amount of data for evaluating model performance.
• Balanced Total Distribution: The total dataset size of 3,232 data points is distributed in a manner that supports both training and testing needs, with a clear emphasis on training.

Based on these results, the dataset split demonstrates a typical distribution pattern where the majority of data is allocated to the training dataset, ensuring the model has ample data to learn from. The testing dataset is sufficiently sized to allow for effective evaluation of the model's performance on unseen data. This distribution is generally favorable for machine learning tasks, as it balances the need for comprehensive training with the necessity of robust testing. The results suggest that the current split is appropriate for the model's objectives, providing a solid foundation for both learning and performance assessment.

Tables

Dataset	Size	Proportion
train_dataset_final	2585	79.98%
test_dataset_final	647	20.02%
Total	3232	100%

▶ Test Result: Population Stability Index (validmind.model_validation.sklearn.PopulationStabilityIndex)

Population Stability Index

Population Stability Index is designed to assess the stability of a machine learning model's predictions by comparing the output distributions across different datasets. This test is crucial for identifying any significant shifts in model performance over time or changes in the population characteristics that the model predicts.

The test operates by calculating the Population Stability Index (PSI) for each feature between two datasets, typically a training and a test dataset. The data is sorted into bins based on the distribution of the training data. The proportion of data in each bin is calculated for both datasets. The PSI is then derived by applying a logarithmic transformation to the ratio of these proportions. This metric quantifies the stability of the model's predictions, with values typically ranging from 0 to infinity. A PSI value below 0.1 indicates no significant change, between 0.1 and 0.2 suggests moderate change, and above 0.2 indicates a significant shift.

The primary advantages of this test include its ability to provide a quantitative measure of model stability over time or across different samples. It facilitates direct comparisons across features based on PSI values, making it a valuable tool for model risk management. The straightforward calculation and interpretation of PSI, along with visual aids like tables and charts, enhance its utility in understanding model performance changes.

It should be noted that the PSI test does not account for interdependencies between features, which may result in similar shifts in distributions and PSI values. It does not provide insights into the reasons behind distribution differences or PSI changes. The test may struggle with features containing significant outliers and focuses on model predictions rather than underlying data distributions. Changes in PSI could result from model drift, concept drift, or both, but distinguishing these causes is challenging.

This test shows the results in a table and a plot, illustrating the PSI values across different bins. The table presents the count and percentage of data in each bin for both initial and new datasets, along with the PSI for each bin. The plot visually represents the population ratio for each bin and the corresponding PSI values. Key metrics include the percentage of data in each bin and the PSI, which helps identify shifts in data distribution. Notable observations include the PSI values across bins, with the total PSI indicating overall stability.

The test results reveal the following key insights:

• Overall Stability: The total PSI value of 0.0151 suggests no significant change between the datasets.
• Bin-Specific Variations: Bin 6 shows the highest PSI value of 0.0063, indicating a noticeable shift in this segment.
• Consistent Distribution: Most bins exhibit low PSI values, reflecting stable distributions across datasets.

Based on these results, the model demonstrates overall stability with no significant shifts in data distribution between the initial and new datasets. The low total PSI value indicates that the model's predictions remain consistent over time. However, the higher PSI in Bin 6 suggests a specific area where the distribution has changed more noticeably, which may warrant further investigation to understand the underlying causes. Overall, the model appears reliable for future predictions, with only minor variations observed in specific segments.

Tables

Population Stability Index for train_dataset_final and test_dataset_final Datasets

Bin	Count Initial	Percent Initial (%)	Count New	Percent New (%)	PSI
0	138	5.3385	30	4.6368	0.0010
1	216	8.3559	60	9.2736	0.0010
2	307	11.8762	82	12.6739	0.0005
3	340	13.1528	84	12.9830	0.0000
4	321	12.4178	70	10.8192	0.0022
5	411	15.8994	92	14.2195	0.0019
6	287	11.1025	90	13.9104	0.0063
7	249	9.6325	58	8.9645	0.0005
8	131	5.0677	29	4.4822	0.0007
9	185	7.1567	52	8.0371	0.0010
Total	2585	100.0000	647	100.0000	0.0151

Figures

▶ Test Result: Confusion Matrix (validmind.model_validation.sklearn.ConfusionMatrix)

Confusion Matrix

Confusion Matrix is designed to evaluate and visually represent the classification ML model's predictive performance using a Confusion Matrix heatmap. The primary purpose is to assess how well the model can correctly classify True Positives, True Negatives, False Positives, and False Negatives, which are fundamental aspects of model accuracy.

The test operates by comparing the predicted results (y_test_predict) from the classification model against the actual values (y_test_true). A confusion matrix is constructed using the unique labels from y_test_true, utilizing scikit-learn's metrics. This matrix is then visually rendered with Plotly's create_annotated_heatmap function, providing a two-dimensional graphical representation of the model's performance. The matrix highlights the distribution of True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN). The values in the matrix indicate the count of each classification type, offering insights into the model's accuracy and error types. A high number of TPs and TNs is generally desirable, indicating effective classification, while high FPs and FNs suggest areas for improvement.

The primary advantages of this test include its ability to provide a simplified yet comprehensive visual snapshot of the classification model's predictive performance. It distinctly highlights True Positives, True Negatives, False Positives, and False Negatives, making it easier to identify potential areas for improvement. The matrix is particularly useful in multi-class classification problems, offering a straightforward view of complex model performances. Additionally, it aids in understanding the different types of errors the model could make, providing insights into Type-I and Type-II errors, which are crucial for refining model accuracy and reliability.

It should be noted that the test has limitations, particularly in cases of unbalanced classes, where the confusion matrix might misrepresent the model's accuracy by focusing on the majority class. It does not provide a single unified statistic to evaluate overall performance, requiring separate calculations for metrics like precision, recall, and F1-score. The matrix serves primarily as a descriptive tool and lacks the capability for statistical hypothesis testing. There is also a risk of misinterpretation, as the matrix does not directly provide precision, recall, or F1-score data, necessitating additional computations for a complete performance evaluation.

This test shows a confusion matrix heatmap that visually represents the classification results. The matrix is divided into four quadrants, each representing a different classification outcome: True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN). The x-axis represents the predicted labels, while the y-axis represents the true labels. The values within each quadrant indicate the count of instances for each classification type. In this matrix, there are 208 True Positives, 196 True Negatives, 126 False Positives, and 117 False Negatives. The heatmap uses color intensity to reflect the magnitude of these values, with darker shades indicating higher counts. This visual representation allows for quick assessment of the model's performance, highlighting areas where the model excels or needs improvement.

The test results reveal the following key insights:

• High True Positive Rate: The model correctly identifies 208 instances as True Positives, indicating effective classification for this category.
• Significant False Positives: There are 126 False Positives, suggesting the model occasionally misclassifies negative instances as positive.
• Moderate True Negatives: With 196 True Negatives, the model shows reasonable accuracy in identifying negative instances.
• Notable False Negatives: The presence of 117 False Negatives indicates some positive instances are incorrectly classified as negative.

Based on these results, the model demonstrates a strong ability to identify True Positives, which is a positive aspect of its performance. However, the presence of significant False Positives and False Negatives suggests areas for improvement, particularly in reducing misclassification errors. The balance between True Negatives and False Positives indicates a need for further refinement to enhance overall accuracy. These insights provide a clear understanding of the model's current behavior, highlighting strengths in positive identification while pointing out specific areas where adjustments could lead to improved performance.

Figures

▶ Test Result: Classifier Performance In Sample (validmind.model_validation.sklearn.ClassifierPerformance:in_sample)

Classifier Performance In Sample

Classifier Performance: In Sample is designed to evaluate the performance of classification models by calculating key metrics such as precision, recall, F1-Score, accuracy, and ROC AUC. These metrics provide a comprehensive view of how well a model distinguishes between classes, making it suitable for both binary and multiclass classification tasks.

The test operates by utilizing scikit-learn's classification_report to compute precision, recall, F1-Score, and accuracy. Precision measures the proportion of true positive predictions among all positive predictions, indicating the model's ability to avoid false positives. Recall, or sensitivity, assesses the proportion of true positive predictions among all actual positives, reflecting the model's capacity to capture all relevant instances. The F1-Score, a harmonic mean of precision and recall, balances these two metrics, providing a single score that accounts for both false positives and false negatives. Accuracy represents the proportion of correct predictions among all predictions, offering a straightforward measure of overall performance. The ROC AUC score, calculated using the multiclass_roc_auc_score function, evaluates the model's ability to distinguish between classes, with values ranging from 0 to 1. A score closer to 1 indicates excellent discriminative ability, while a score near 0.5 suggests random guessing.

The primary advantages of this test include its versatility in handling both binary and multiclass models, making it applicable across various classification scenarios. By employing a range of performance metrics, the test provides a detailed analysis of model behavior, highlighting strengths and weaknesses in classification tasks. The inclusion of ROC AUC is particularly beneficial for datasets with class imbalance, as it offers insights into the model's discriminative power beyond simple accuracy. This comprehensive approach ensures that the test can identify subtle performance nuances that might be overlooked by single-metric evaluations.

It should be noted that the test assumes correctly identified labels for binary classification models, which may not always be the case in real-world applications. Additionally, it is specifically designed for classification models and is not applicable to regression tasks. The test's effectiveness can be limited if the dataset used does not accurately represent real-world conditions, potentially leading to misleading performance assessments. Signs of high risk include low precision, recall, F1-Score, accuracy, and ROC AUC values, which indicate poor model performance. An imbalance between precision and recall scores can also signal issues with the model's ability to generalize across different classes.

This test shows the results in two tables: one for precision, recall, and F1-Score, and another for accuracy and ROC AUC. The first table presents class-specific metrics for two classes, labeled "0" and "1", along with weighted and macro averages. Precision, recall, and F1-Score for Class 0 are 0.6461, 0.6406, and 0.6434, respectively, while for Class 1, they are 0.6429, 0.6483, and 0.6456. The weighted and macro averages for these metrics are all 0.6445, indicating a balanced performance across classes. The second table shows an accuracy of 0.6445 and a ROC AUC of 0.6915. The accuracy reflects the overall proportion of correct predictions, while the ROC AUC score suggests moderate discriminative ability, as it is above 0.5 but not close to 1. These tables provide a clear view of the model's performance, with precision, recall, and F1-Score offering insights into class-specific behavior, and accuracy and ROC AUC providing an overall performance assessment.

The test results reveal the following key insights:

• Balanced Class Performance: The precision, recall, and F1-Score for both classes are closely aligned, with values around 0.64, indicating consistent performance across classes.
• Moderate Discriminative Ability: The ROC AUC score of 0.6915 suggests that the model has a moderate ability to distinguish between classes, performing better than random guessing but not achieving high discriminative power.
• Overall Accuracy: The accuracy of 0.6445 reflects a moderate level of correct predictions, consistent with the precision and recall metrics, indicating that the model performs similarly across different evaluation criteria.

Based on these results, the model demonstrates a balanced performance across classes, with precision, recall, and F1-Score values indicating consistent classification ability. The moderate ROC AUC score suggests that while the model can distinguish between classes better than random guessing, there is room for improvement in its discriminative power. The overall accuracy aligns with the class-specific metrics, reinforcing the observation of a moderate performance level. These insights provide a comprehensive understanding of the model's behavior, highlighting areas of strength and potential improvement in its classification capabilities.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6461	0.6406	0.6434
1	0.6429	0.6483	0.6456
Weighted Average	0.6445	0.6445	0.6445
Macro Average	0.6445	0.6445	0.6445

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6445
ROC AUC	0.6915

▶ Test Result: Classifier Performance Out Of Sample (validmind.model_validation.sklearn.ClassifierPerformance:out_of_sample)

Classifier Performance Out Of Sample

Classifier Performance: Out of Sample is designed to evaluate the performance of classification models by calculating key metrics such as precision, recall, F1-Score, accuracy, and ROC AUC scores. This test provides a comprehensive analysis of a model's ability to correctly classify instances in both binary and multiclass scenarios, offering insights into its predictive power and reliability.

The test operates by utilizing scikit-learn's classification_report to compute precision, recall, F1-Score, and accuracy. Precision measures the proportion of true positive predictions among all positive predictions, indicating the model's ability to avoid false positives. Recall, or sensitivity, assesses the proportion of true positive predictions among all actual positives, reflecting the model's capacity to capture all relevant instances. The F1-Score, a harmonic mean of precision and recall, balances these two metrics, providing a single measure of a model's accuracy. Accuracy represents the proportion of correctly classified instances over the total instances. For multiclass models, macro and weighted averages of these scores are calculated to account for class imbalances. The ROC AUC score, ranging from 0 to 1, evaluates the model's ability to distinguish between classes, with values closer to 1 indicating better performance. This score is particularly useful for assessing models on unbalanced datasets, as it considers the trade-off between true positive and false positive rates.

The primary advantages of this test include its versatility in handling both binary and multiclass classification models, making it applicable across a wide range of scenarios. By employing a variety of performance metrics, the test offers a detailed view of a model's strengths and weaknesses, allowing for a nuanced understanding of its behavior. The inclusion of ROC AUC as a metric is particularly beneficial for evaluating models on unbalanced datasets, as it provides a robust measure of discriminatory power. This comprehensive approach ensures that the test can effectively highlight areas where a model excels or requires improvement, aiding in the development of more accurate and reliable predictive models.

It should be noted that the test has certain limitations, including its assumption of correctly identified labels for binary classification models, which may not always hold true in real-world applications. Additionally, the test is specifically designed for classification models and is not suitable for regression models, limiting its applicability in scenarios where continuous outcomes are of interest. The test's effectiveness is also contingent on the representativeness of the test dataset; if the dataset does not accurately reflect real-world conditions, the insights gained may be limited. Furthermore, signs of high risk, such as low precision, recall, F1-Score, accuracy, and ROC AUC scores, can indicate poor model performance, necessitating further investigation and potential model refinement.

This test shows the results in tabular format, with two tables presented. The first table, titled "Precision, Recall, and F1," displays these metrics for each class, as well as their weighted and macro averages. Precision, recall, and F1-Score are shown for Class 0 and Class 1, with values indicating the model's performance in distinguishing between these classes. The second table, titled "Accuracy and ROC AUC," presents the overall accuracy and ROC AUC score of the model. Accuracy is a straightforward measure of the proportion of correctly classified instances, while the ROC AUC score provides insight into the model's ability to differentiate between classes. The tables are easy to read, with each metric clearly labeled and values presented to four decimal places, allowing for precise interpretation. Notable observations include the relatively balanced precision and recall scores across classes, as well as a moderate ROC AUC score, suggesting a reasonable level of discriminatory power.

The test results reveal the following key insights:

• Balanced Class Performance: The precision and recall scores for Class 0 and Class 1 are relatively balanced, with Class 0 having a precision of 0.6262 and recall of 0.6087, while Class 1 has a precision of 0.6228 and recall of 0.64. This indicates that the model performs consistently across both classes.
• Moderate Overall Performance: The weighted and macro averages for precision, recall, and F1-Score are all approximately 0.624, suggesting a moderate level of overall performance. The accuracy of 0.6244 further supports this observation, indicating that the model correctly classifies approximately 62.44% of instances.
• Discriminatory Power: The ROC AUC score of 0.6658 suggests that the model has a moderate ability to distinguish between classes. While this score is above the 0.5 threshold, indicating better-than-random performance, there is room for improvement to achieve higher discriminatory power.

Based on these results, the model demonstrates a moderate level of performance, with balanced precision and recall scores across classes and a reasonable overall accuracy. The ROC AUC score indicates that the model has some ability to distinguish between classes, though there is potential for enhancement. These insights suggest that while the model is functional, further refinement may be necessary to improve its predictive accuracy and discriminatory power, particularly in scenarios where higher precision and recall are critical. The balanced performance across classes is a positive aspect, indicating that the model does not disproportionately favor one class over another, which is crucial for applications where class balance is important.

Tables

Precision, Recall, and F1

Class	Precision	Recall	F1
0	0.6262	0.6087	0.6173
1	0.6228	0.6400	0.6313
Weighted Average	0.6245	0.6244	0.6243
Macro Average	0.6245	0.6243	0.6243

Accuracy and ROC AUC

Metric	Value
Accuracy	0.6244
ROC AUC	0.6658

▶ Test Result: Precision Recall Curve (validmind.model_validation.sklearn.PrecisionRecallCurve)

Precision Recall Curve

Precision Recall Curve is designed to evaluate the trade-off between precision and recall in binary classification models. Its primary purpose is to assess the model's ability to produce accurate results while capturing a majority of positive instances. This is particularly useful in scenarios where both precision and recall are critical, such as in medical diagnosis or fraud detection.

The test operates by extracting ground truth labels and prediction probabilities from the model's test dataset. It utilizes the precision_recall_curve method from the sklearn metrics module, which computes precision-recall pairs for each possible threshold. This involves calculating precision, the ratio of true positive predictions to the total positive predictions, and recall, the ratio of true positive predictions to the total actual positives. The resulting precision and recall scores are plotted against each other to form the Precision-Recall Curve. The curve is visualized using Plotly's scatter plot, providing a graphical representation of the model's performance across different threshold levels. The values range from 0 to 1, where a higher area under the curve indicates better performance.

The primary advantages of this test include its ability to effectively represent the balance between precision and recall, which is crucial in applications where both false positives and false negatives have significant consequences. The visual representation of the Precision-Recall Curve allows for an intuitive understanding of how the model performs at various thresholds, making it easier to identify the optimal balance between precision and recall. This is particularly beneficial in domains where the cost of errors is high, enabling stakeholders to make informed decisions based on the model's performance.

It should be noted that this test is limited to binary classification models and is not applicable to multiclass or foundation models. Additionally, it may not fully capture the overall accuracy of the model if the costs of false positives and false negatives differ significantly, or if the dataset is heavily imbalanced. A lower area under the Precision-Recall Curve signifies a higher risk, indicating a model with a high number of false positives or false negatives. If the curve is closer to the bottom left of the plot, it suggests a higher risk, as opposed to being closer to the top right corner.

This test shows a Precision-Recall Curve plotted with precision on the y-axis and recall on the x-axis. The curve provides a visual representation of the trade-off between precision and recall at various thresholds. The plot indicates how precision decreases as recall increases, which is typical in classification models. The curve starts at a low recall with high precision and gradually moves towards higher recall with decreasing precision. The area under the curve is a key metric, with values closer to 1 indicating better performance. The plot reveals that the model maintains a relatively high precision across a range of recall values, but there is a noticeable decline in precision as recall approaches 1. This suggests that while the model is effective at identifying positive instances, there is a trade-off in terms of precision as more positives are captured.

The test results reveal the following key insights:

• Overall Precision-Recall Trade-off: The model demonstrates a balanced trade-off between precision and recall, maintaining a reasonable level of precision across various recall levels.
• High Precision at Low Recall: At lower recall values, the model achieves high precision, indicating effective identification of positive instances with minimal false positives.
• Decline in Precision with Increased Recall: As recall increases, precision declines, highlighting the trade-off and the model's tendency to produce more false positives as it captures more positives.

Based on these results, the model exhibits a balanced performance in terms of precision and recall, with a notable ability to maintain high precision at lower recall levels. However, as recall increases, the precision decreases, indicating a trade-off that is typical in classification models. The insights suggest that while the model is effective in identifying positive instances, there is a need to consider the implications of increased false positives as recall improves. This understanding of the model's behavior across different thresholds can guide stakeholders in making informed decisions about threshold selection and model deployment in scenarios where precision and recall are critical.

Figures

▶ Test Result: ROC Curve (validmind.model_validation.sklearn.ROCCurve)

ROC Curve

ROC Curve is designed to evaluate the performance of binary classification models by plotting the Receiver Operating Characteristic (ROC) curve and calculating the Area Under Curve (AUC) score. The ROC curve provides a visual representation of a model's ability to distinguish between two classes by plotting the True Positive Rate (TPR) against the False Positive Rate (FPR) at various threshold levels. The AUC score quantifies this performance, with higher values indicating better discrimination between the positive and negative classes.

The test operates by selecting the target model and datasets for binary classification, calculating predicted probabilities for the test set, and using these along with true outcomes to generate the ROC curve. The ROC curve plots TPR against FPR across different thresholds, providing a comprehensive view of the model's performance. The AUC score, ranging from 0 to 1, is calculated to summarize the ROC curve's overall performance. A score of 0.5 suggests no discrimination (random performance), while a score closer to 1 indicates excellent discrimination. The test also includes a line representing random performance (AUC of 0.5) for comparison. Infinite values in the ROC threshold are removed to ensure accuracy, and the results are saved for future reference.

The primary advantages of this test include its ability to provide a complete visual depiction of a model's discriminative power across all possible classification thresholds, unlike metrics that only evaluate performance at a single threshold. The AUC score remains consistent regardless of dataset proportions, making it an ideal choice for evaluating models in various scenarios. This consistency allows for a reliable comparison of model performance, even when class distributions are imbalanced, as it focuses on ranking predictions rather than absolute values.

It should be noted that this test is limited to binary classification tasks, restricting its applicability to other model types. Additionally, models that produce probabilities skewed towards 0 or 1 may not perform well under this evaluation. The ROC curve can sometimes indicate high performance even when most classifications are incorrect, as long as the model maintains a correct ranking order. This is known as the "Class Imbalance Problem," where the ROC curve may not fully reflect the model's practical utility in imbalanced datasets.

This test shows a plot of the ROC curve for the model log_reg_model_v1 on the test_dataset_final, with the ROC curve depicted in magenta and a dashed line representing random performance. The x-axis represents the False Positive Rate, while the y-axis represents the True Positive Rate. The AUC score of 0.67 is displayed, indicating the model's ability to distinguish between classes. The curve's proximity to the diagonal line suggests moderate discriminative power. The plot provides a visual comparison of the model's performance against random chance, with the AUC score offering a numerical summary of this performance. The range of the ROC curve spans from 0 to 1 on both axes, with the curve's shape and position providing insights into the model's classification capabilities.

The test results reveal the following key insights:

• Moderate Discriminative Power: The AUC score of 0.67 indicates that the model has moderate ability to distinguish between the positive and negative classes, performing better than random chance but not reaching high levels of accuracy.
• Curve Proximity to Randomness: The ROC curve's proximity to the diagonal line suggests that while the model is better than random, there is room for improvement in its discriminative capabilities.
• Threshold Sensitivity: The shape of the ROC curve indicates how the model's performance varies across different threshold levels, highlighting areas where the model may perform better or worse.

Based on these results, the model demonstrates moderate discriminative power, as evidenced by the AUC score of 0.67. The ROC curve's position relative to the line of randomness suggests that while the model is capable of distinguishing between classes, there is potential for enhancement. The insights into threshold sensitivity provide valuable information for understanding how the model's performance changes with different decision thresholds, offering opportunities for optimization. The results underscore the importance of considering both the ROC curve and AUC score when evaluating model performance, particularly in scenarios with class imbalance or varying threshold requirements.

Figures

▶ Test Result: Training Test Degradation (validmind.model_validation.sklearn.TrainingTestDegradation)

✅ Training Test Degradation

Training Test Degradation is designed to assess the extent of performance degradation between a model's training and test datasets, ensuring that the model's ability to generalize to unseen data remains within acceptable limits. The primary purpose of this test is to evaluate the model's robustness and reliability by comparing key classification metrics such as precision, recall, and f1 score across both datasets. By doing so, it helps in identifying any significant drop in performance that could indicate overfitting or other generalization issues.

The test operates by calculating several classification metrics, including precision, recall, and f1 score, for both the training and test datasets. Precision measures the accuracy of positive predictions, recall assesses the ability to identify all relevant instances, and the f1 score provides a balance between precision and recall. These metrics are computed by comparing the model's predictions against the actual labels in both datasets. The degradation is quantified as the percentage difference between the training and test scores, relative to the training score. A degradation threshold, typically set at 10%, is used to determine if the model's performance drop is acceptable. The results are presented in a table format, showing the train and test scores, degradation percentage, and a pass/fail status for each metric.

The primary advantages of this test include its ability to provide a quantitative measure of the model's generalization capability, which is crucial for predicting real-world performance. By evaluating multiple metrics, the test offers a comprehensive view of the model's performance across different aspects, such as precision and recall, which are critical for understanding the model's behavior in various scenarios. Additionally, the flexibility to adjust the degradation threshold allows for customization based on specific requirements or risk tolerances, making it adaptable to different contexts and applications.

It should be noted that while this test provides valuable insights into performance degradation, it does not account for the nature or distribution of the data. This means that areas with poor representation in the training set might still lead to suboptimal performance on the test data, even if the degradation is within acceptable limits. Furthermore, the test assumes that both datasets are well-balanced and representative, which may not always be the case. The test is also limited to classification tasks, and its applicability to other types of models or tasks is restricted. High degradation percentages, especially those exceeding the 10% threshold, are indicative of potential overfitting or other generalization issues.

This test shows the results in a tabular format, where each row corresponds to a specific class and metric combination. The table includes columns for the metric name, train dataset score, test dataset score, degradation percentage, and pass/fail status. The train and test scores are presented as decimal values, typically ranging from 0 to 1, where higher values indicate better performance. The degradation percentage reflects the relative drop in performance from the training to the test dataset, with lower percentages being preferable. The pass/fail status indicates whether the degradation is within the acceptable threshold. Notable observations include the fact that all metrics for both classes have degradation percentages well below the 10% threshold, resulting in a 'Pass' status across the board.

The test results reveal the following key insights:

• Class 1 Precision Stability: The precision for Class 1 shows a degradation of 3.13%, with scores of 0.6429 for training and 0.6228 for testing, indicating stable performance.
• Class 1 Recall Consistency: Recall for Class 1 has a minimal degradation of 1.29%, with training and test scores of 0.6483 and 0.64, respectively, demonstrating consistent recall capability.
• Class 1 F1-Score Robustness: The f1 score for Class 1 degrades by 2.22%, with scores of 0.6456 for training and 0.6313 for testing, reflecting robust performance.
• Class 0 Precision Reliability: Precision for Class 0 shows a degradation of 3.09%, with scores of 0.6461 for training and 0.6262 for testing, indicating reliable precision.
• Class 0 Recall Variation: Recall for Class 0 experiences a degradation of 4.99%, with training and test scores of 0.6406 and 0.6087, respectively, showing some variation but still within acceptable limits.
• Class 0 F1-Score Dependability: The f1 score for Class 0 degrades by 4.05%, with scores of 0.6434 for training and 0.6173 for testing, indicating dependable performance.

Based on these results, the model demonstrates a strong ability to generalize from the training to the test dataset, with all metrics for both classes showing degradation well within the acceptable threshold of 10%. This suggests that the model is not overfitting and maintains consistent performance across different data splits. The slight variations in degradation percentages across metrics and classes highlight areas where the model's performance could be further scrutinized, but overall, the results indicate a robust and reliable model suitable for deployment in real-world scenarios.

Tables

Class	Metric	train_dataset_final Score	test_dataset_final Score	Degradation (%)	Pass/Fail
1	Precision	0.6429	0.6228	3.1271	Pass
1	Recall	0.6483	0.6400	1.2855	Pass
1	F1-Score	0.6456	0.6313	2.2189	Pass
0	Precision	0.6461	0.6262	3.0866	Pass
0	Recall	0.6406	0.6087	4.9877	Pass
0	F1-Score	0.6434	0.6173	4.0506	Pass

▶ Test Result: Minimum Accuracy (validmind.model_validation.sklearn.MinimumAccuracy)

❌ Minimum Accuracy

Minimum Accuracy is designed to ensure that a model's prediction accuracy meets or exceeds a specified threshold, which is crucial for validating the model's performance in both binary and multiclass classification tasks. The primary purpose of this test is to confirm that the model can reliably predict outcomes with a level of accuracy that is deemed acceptable for its intended application, thereby providing confidence in its deployment.

The test operates by calculating the model's accuracy score using the accuracy_score method from sklearn, which compares the true labels (y_true) with the predicted labels (class_pred). This score represents the proportion of correct predictions out of the total number of predictions made by the model. The accuracy score is then compared to a predefined threshold, typically set at 0.7, to determine if the model passes the test. An accuracy score above this threshold indicates satisfactory model performance, while a score below suggests potential issues with prediction reliability. The accuracy score ranges from 0 to 1, where a score closer to 1 indicates better performance. This test is particularly useful for providing a quick assessment of the model's overall predictive capability.

The primary advantages of this test include its simplicity and effectiveness in providing a straightforward measure of a model's performance across all classes. It is particularly beneficial when the dataset is balanced, as it offers a clear indication of the model's ability to correctly classify instances. The test's versatility allows it to be applied to both binary and multiclass classification problems, making it a widely applicable tool in model evaluation. By focusing on the overall accuracy, it provides a holistic view of the model's performance, which is essential for initial assessments and comparisons between different models or configurations.

It should be noted that the Minimum Accuracy test has limitations, particularly in scenarios where the dataset is imbalanced. In such cases, the accuracy score may be misleading, as it can disproportionately favor the majority class, giving an inaccurate perception of the model's true performance. This test does not account for other important metrics such as precision, recall, or the model's ability to handle false positives and false negatives, which are crucial for a comprehensive evaluation. Additionally, persistent scores below the threshold indicate a high risk of inaccurate predictions, which could have significant implications depending on the application.

This test shows a table format output that presents the model's accuracy score, the threshold used for comparison, and the pass/fail status of the test. The table includes columns for the "Score," which is the calculated accuracy of the model, the "Threshold," which is the minimum acceptable accuracy level set for the test, and the "Pass/Fail" status, indicating whether the model's performance meets the required standard. In this specific test result, the model achieved an accuracy score of 0.6244, which is below the threshold of 0.7, resulting in a "Fail" status. This indicates that the model's predictions are not sufficiently accurate according to the predefined criteria, highlighting a potential area for improvement in model performance.

The test results reveal the following key insights:

• Model Accuracy Below Threshold: The model's accuracy score of 0.6244 falls short of the 0.7 threshold, indicating that the model does not meet the minimum accuracy requirement for reliable predictions.
• Fail Status Indicates Performance Concerns: The "Fail" status in the test results suggests that the model's current configuration or data handling may need to be revisited to enhance its predictive accuracy.

Based on these results, the model's current accuracy level is insufficient to meet the established performance criteria, as evidenced by the score of 0.6244, which is below the threshold of 0.7. This shortfall suggests that the model may not be adequately capturing the underlying patterns in the data, leading to a higher rate of incorrect predictions. The "Fail" status highlights the need for further investigation into the model's design, data preprocessing, or feature selection processes to identify potential areas for improvement. This analysis underscores the importance of achieving a balance between model complexity and accuracy, particularly in applications where prediction reliability is critical.

Tables

Score	Threshold	Pass/Fail
0.6244	0.7	Fail

▶ Test Result: Minimum F1 Score (validmind.model_validation.sklearn.MinimumF1Score)

✅ Minimum F1 Score

Minimum F1 Score is designed to ensure that the model's F1 score on the validation set meets a predefined minimum threshold, thereby confirming balanced performance between precision and recall. This test is particularly crucial in classification tasks where the distribution of positive and negative classes is skewed, as it provides a more comprehensive measure of a model's effectiveness than accuracy alone.

The test operates by calculating the F1 score using scikit-learn's metrics in Python. For binary classification problems, the F1 score is computed directly, while for multi-class problems, macro averaging is employed. The F1 score is a harmonic mean of precision and recall, which are derived from the true positive, false positive, and false negative rates. Precision measures the accuracy of positive predictions, while recall assesses the ability to identify all positive instances. The F1 score ranges from 0 to 1, where 1 indicates perfect precision and recall, and 0 indicates the worst performance. A score above the predefined threshold is considered satisfactory, indicating a balanced trade-off between precision and recall.

The primary advantages of this test include its ability to provide a balanced measure of a model's performance by accounting for both false positives and false negatives. This is particularly useful in scenarios with imbalanced class distributions, where accuracy can be misleading. The flexibility in setting the threshold value allows practitioners to tailor the minimum acceptable performance standards to specific requirements, ensuring that the model meets the desired level of effectiveness in identifying positive cases while minimizing false positives.

It should be noted that the F1 score assumes an equal cost for false positives and false negatives, which may not be true in some real-world scenarios. Additionally, while the F1 score is a useful metric, it may not be suitable for all types of models and machine learning tasks. Practitioners might need to rely on other metrics such as precision, recall, or the ROC-AUC score that align more closely with specific requirements. Furthermore, if a model returns an F1 score that is less than the established threshold, it is regarded as high risk, suggesting that the model is not finding an optimal balance between precision and recall.

This test shows a table format output that presents the F1 score, the predefined threshold, and the pass/fail status of the test. The table includes columns for the Score, Threshold, and Pass/Fail status. The Score column represents the calculated F1 score of the model on the validation dataset, which in this case is 0.6313. The Threshold column indicates the minimum acceptable F1 score, set at 0.5 for this test. The Pass/Fail column shows whether the model's performance meets the threshold, with a "Pass" indicating that the model's F1 score is above the threshold. The table is straightforward to read, with each row representing a single test result. The key measurement here is the F1 score, which is compared against the threshold to determine the model's pass/fail status. Notably, the model's F1 score of 0.6313 exceeds the threshold of 0.5, resulting in a "Pass" status.

The test results reveal the following key insights:

• Model Meets Performance Threshold: The model achieves an F1 score of 0.6313, which surpasses the predefined threshold of 0.5, indicating satisfactory performance in balancing precision and recall.
• Balanced Precision and Recall: The F1 score suggests that the model effectively identifies positive instances while maintaining a reasonable rate of false positives, reflecting a balanced trade-off between precision and recall.

Based on these results, the model demonstrates a satisfactory level of performance by achieving an F1 score that exceeds the minimum threshold, indicating a balanced approach to handling precision and recall. This suggests that the model is effective in identifying positive cases while minimizing false positives, which is crucial in scenarios with imbalanced class distributions. The results provide confidence in the model's ability to perform well in the validation dataset, meeting the predefined performance standards. The insights gained from this test highlight the model's capability to maintain a balanced performance, which is essential for its deployment in real-world applications where both precision and recall are critical.

Tables

Score	Threshold	Pass/Fail
0.6313	0.5	Pass

▶ Test Result: Minimum ROCAUC Score (validmind.model_validation.sklearn.MinimumROCAUCScore)

✅ Minimum ROCAUC Score

Minimum ROC AUC Score is designed to validate the model's performance by ensuring that the Receiver Operating Characteristic Area Under the Curve (ROC AUC) score on the validation dataset meets or exceeds a predefined threshold. This test is crucial for assessing the model's ability to distinguish between different classes, which is a fundamental requirement in both binary and multiclass classification tasks.

The test operates by calculating the multiclass ROC AUC score using the true target values and the model's predictions. It first converts the multi-class target variables into a binary format using a technique called LabelBinarizer. This transformation is essential for computing the ROC AUC score, which measures the model's ability to differentiate between classes. The ROC AUC score ranges from 0 to 1, where a score closer to 1 indicates excellent class separation, while a score near 0.5 suggests no better performance than random guessing. The test passes if the ROC AUC score exceeds the predefined threshold, typically set at 0.5, indicating that the model performs better than random chance.

The primary advantages of this test include its comprehensive assessment of the model's performance by considering both the true positive rate and false positive rate. This makes the ROC AUC score a robust measure as it evaluates the model's quality across various classification thresholds, independent of any specific threshold. Additionally, the test's applicability to both binary and multiclass classification problems enhances its utility in diverse scenarios, providing a reliable metric for model evaluation across different datasets and classification challenges.

It should be noted that the test has limitations, particularly in scenarios with highly imbalanced class distributions. In such cases, the ROC AUC score might still appear favorable even if the model fails to predict the minority class effectively. Furthermore, the test does not offer insights into the specific aspects of the model that may be causing poor performance if the ROC AUC score is unsatisfactory. The use of a macro average for multiclass ROC AUC scoring implies equal weightage to each class, which might not be appropriate if the classes are imbalanced, potentially skewing the interpretation of the model's performance.

This test shows the results in a tabular format, presenting key metrics such as the ROC AUC score, the threshold, and the pass/fail status. The table is straightforward to read, with columns labeled for each metric. The ROC AUC score is a numerical value representing the model's ability to distinguish between classes, with a typical range from 0 to 1. The threshold column indicates the minimum score required for the model to pass the test, set at 0.5 in this case. The pass/fail column provides a binary indication of whether the model's performance meets the required standard. In this test, the ROC AUC score is 0.6658, which surpasses the threshold of 0.5, resulting in a "Pass" status. This indicates that the model performs better than random chance in distinguishing between classes.

The test results reveal the following key insights:

• Model Performance Exceeds Threshold: The ROC AUC score of 0.6658 is significantly above the threshold of 0.5, indicating that the model has a good ability to distinguish between different classes.
• Pass Status Achieved: The model successfully passes the test, demonstrating that it meets the minimum performance criteria set by the ROC AUC threshold.

Based on these results, the model demonstrates a satisfactory level of performance in terms of class separation, as evidenced by the ROC AUC score exceeding the predefined threshold. This suggests that the model is capable of distinguishing between classes more effectively than random guessing, which is a positive indicator of its classification capabilities. The pass status further confirms that the model meets the necessary performance standards, providing confidence in its ability to handle the classification task at hand. These insights collectively highlight the model's robustness in distinguishing between classes, making it a reliable tool for classification purposes within the tested context.

Tables

Score	Threshold	Pass/Fail
0.6658	0.5	Pass

▶ Test Result: Permutation Feature Importance (validmind.model_validation.sklearn.PermutationFeatureImportance)

Permutation Feature Importance

Permutation Feature Importance is designed to evaluate the significance of each feature in a machine learning model by measuring the impact on model performance when feature values are randomly rearranged. This test helps identify which features are most influential in the model's predictions, providing insights into the model's reliance on specific data inputs.

The test operates by utilizing the permutation_importance method from the sklearn.inspection module. This method involves shuffling the values of each feature in the dataset and observing the resulting change in the model's performance. The primary metric used is the decrease in performance, typically measured by accuracy or another relevant metric, when a feature's values are permuted. A significant drop in performance indicates that the feature is important, while little to no change suggests the feature may not be crucial. The method is model-agnostic, meaning it can be applied to any model that provides a measure of prediction accuracy. The typical range of values for permutation importance is from 0 to 1, where higher values indicate greater importance.

The primary advantages of this test include its ability to provide clear insights into the importance of different features, which can reveal underlying data structures and potential overfitting if certain features overly influence predictions. It is particularly useful in scenarios where understanding feature importance is critical, such as in model interpretability and feature selection. The test's model-agnostic nature allows it to be applied across various types of classifiers, enhancing its versatility. By identifying key features, it aids in refining models and improving their generalization capabilities.

It should be noted that this test does not imply causality; it only indicates the amount of information a feature provides for the prediction task. Additionally, it does not account for interactions between features, which can lead to misleading importance scores if features are correlated. The test's applicability is limited with certain libraries like statsmodels, pytorch, and catboost, which may restrict its use in some contexts. High-risk signs include reliance on features with highly variable values, indicating potential instability, and discrepancies between model-determined importance and domain knowledge expectations.

This test shows a bar plot illustrating the permutation importance of each feature. The x-axis represents the importance score, while the y-axis lists the features. The plot is read by observing the length of each bar, which indicates the relative importance of the feature. Key measurements include the importance scores, which range from 0 to approximately 0.06. Notable observations include the high importance of features like IsActiveMember and Geography_Germany, which have the longest bars, indicating they significantly impact model performance. Conversely, features like HasCrCard and Tenure have minimal impact, as shown by their shorter bars.

The test results reveal the following key insights:

• High Importance of IsActiveMember: The feature IsActiveMember shows the highest importance score, indicating it plays a crucial role in the model's predictions.
• Significant Role of Geography_Germany: Geography_Germany also has a high importance score, suggesting geographical location is a significant factor in the model.
• Moderate Influence of Gender_Male: The feature Gender_Male has a moderate impact, highlighting gender as a relevant but less critical factor.
• Minimal Impact of HasCrCard: Features like HasCrCard and Tenure show minimal importance, suggesting they do not significantly influence the model's outcomes.

Based on these results, the model heavily relies on features such as IsActiveMember and Geography_Germany, which are crucial for its predictions. This reliance indicates that these features provide substantial information for the model's decision-making process. The moderate importance of Gender_Male suggests that while gender contributes to predictions, it is not as pivotal as other features. The minimal impact of features like HasCrCard and Tenure implies they do not add significant predictive value, which could inform future feature selection and model refinement efforts. These insights collectively enhance the understanding of the model's behavior and the relative importance of different features in its predictive framework.

Figures

▶ Test Result: SHAP Global Importance (validmind.model_validation.sklearn.SHAPGlobalImportance)

SHAP Global Importance

SHAP Global Importance is designed to evaluate and visualize global feature importance using SHAP values for model explanation and risk identification. This test aims to elucidate model outcomes by attributing them to the contributing features, assigning a quantifiable global importance to each feature via their respective absolute Shapley values. It is particularly suitable for tasks like classification, both binary and multiclass, and forms an essential part of model risk management.

The test operates by selecting a suitable explainer that aligns with the model's type. For tree-based models like XGBClassifier, RandomForestClassifier, and CatBoostClassifier, the TreeExplainer is used, whereas for linear models like LogisticRegression and LinearRegression, the LinearExplainer is employed. The explainer calculates the Shapley values, which are then visualized using two specific graphical representations: the Mean Importance Plot and the Summary Plot. The Mean Importance Plot portrays the significance of individual features based on their absolute Shapley values, calculating the average of these values across all instances to highlight global importance. The Summary Plot combines feature importance with their effects, where each dot represents a Shapley value for a certain feature in a specific case, with a color gradient indicating the feature's value.

The primary advantages of this test include its ability to provide a detailed perspective on how different features shape the model's decision-making logic for each instance. SHAP not only illustrates global feature significance but also offers clear insights into model behavior, making it particularly useful for understanding complex models. By visualizing the contribution of each feature, stakeholders can gain a deeper understanding of the model's inner workings, which is crucial for tasks involving model risk management and explanation.

It should be noted that high-dimensional data can convolute interpretations, and associating importance with tangible real-world impact still involves a certain degree of subjectivity. Signs of high risk include overemphasis on certain features in SHAP importance plots, which might hint at model overfitting, and anomalies such as unexpected or illogical features showing high importance, suggesting incorrect or undesirable reasoning. A SHAP summary plot filled with high variability or scattered data points may also indicate potential issues.

This test shows two key visualizations: the Mean Importance Plot and the Summary Plot. The Mean Importance Plot displays normalized feature importance, with features like "IsActiveMember" and "Geography_Germany" showing the highest importance, while "EstimatedSalary" shows the least. The horizontal axis represents the normalized SHAP value as a percentage, indicating the contribution of each feature to the model's predictions. The Summary Plot provides a more detailed view, showing the impact of each feature on the model output. Each dot represents a Shapley value for a feature in a specific instance, with the color gradient indicating the feature's value from low to high. The horizontal axis shows the SHAP value's impact on the model output, with features like "IsActiveMember" having a significant positive impact.

The test results reveal the following key insights:

• IsActiveMember Dominance: "IsActiveMember" is the most influential feature, with a normalized SHAP value close to 100%, indicating its strong impact on model predictions.
• Geography and Gender Influence: "Geography_Germany" and "Gender_Male" also show high importance, suggesting these features significantly affect the model's decisions.
• Low Impact of EstimatedSalary: "EstimatedSalary" has minimal influence, with a near-zero normalized SHAP value, indicating it contributes little to the model's output.
• Feature Impact Variability: The Summary Plot shows variability in feature impacts, with "IsActiveMember" and "Geography_Germany" having a wide range of SHAP values, reflecting diverse effects on predictions.

Based on these results, the model's behavior is heavily influenced by membership status and geographical location, with these features playing a crucial role in decision-making. The low impact of "EstimatedSalary" suggests it is not a significant factor in the model's predictions. The variability in feature impacts, particularly for "IsActiveMember" and "Geography_Germany," indicates that these features have diverse effects across different instances. This understanding of feature importance and variability is essential for assessing model reliability and identifying potential areas of overfitting or bias.

Figures

▶ Test Result: Weakspots Diagnosis (validmind.model_validation.sklearn.WeakspotsDiagnosis)

❌ Weakspots Diagnosis

Weakspots Diagnosis is designed to identify and visualize weak spots in a machine learning model's performance across various sections of the feature space. The primary purpose is to evaluate the model's performance within specific regions, highlighting areas where it falls below set thresholds for metrics like accuracy, precision, recall, and F1 scores.

The test operates by dividing the feature space into multiple bins, each representing a specific range of feature values. For each bin, the model's performance metrics are calculated on both the training and test datasets. Metrics such as accuracy, precision, recall, and F1 score are used to assess the model's ability to correctly predict outcomes. Accuracy measures the proportion of correct predictions, precision indicates the proportion of true positive results in all positive predictions, recall reflects the proportion of true positive results in all actual positives, and F1 score is the harmonic mean of precision and recall. These metrics typically range from 0 to 1, where higher values indicate better performance. A weak spot is identified if any metric falls below a predetermined threshold in a bin on the test dataset.

The primary advantages of this test include its ability to pinpoint specific regions in the feature space where the model's performance is suboptimal, allowing for targeted improvements. The graphical representation of performance metrics provides an intuitive understanding of how the model performs across different feature areas. This test is flexible, allowing users to set different thresholds for various metrics based on application-specific requirements, making it adaptable to diverse scenarios.

It should be noted that the test's limitations include the potential oversimplification of model behavior due to the binning system, which depends on the chosen 'bins' parameter. The effectiveness of the test is also contingent on the selection of performance metric thresholds, which may not be universally applicable. Additionally, the test is limited to numerical or categorical data types, excluding datasets with text columns. While the test highlights problematic regions, it does not provide direct suggestions for model improvement, requiring further analysis to address identified issues.

This test shows the results in both tabular and graphical formats. The tables present detailed metrics for each feature slice, including the number of records, accuracy, precision, recall, and F1 score for both training and test datasets. The bar charts visually represent these metrics, with separate plots for each feature and metric. The x-axis represents the feature slices, while the y-axis shows the metric values. A red dashed line indicates the threshold for each metric, helping to identify weak spots where the model's performance falls below the threshold. Notable observations include regions with zero accuracy, precision, recall, and F1 scores, indicating significant performance issues in those slices.

The test results reveal the following key insights:

• Significant Performance Variability: The model exhibits considerable variability in performance across different feature slices, with some regions showing zero accuracy and others achieving perfect scores.
• Disparity Between Datasets: There is a noticeable disparity in performance metrics between the training and test datasets, particularly in certain feature slices, suggesting potential overfitting.
• Consistent Weak Spots: Certain feature slices consistently fall below the threshold across multiple metrics, indicating areas where the model struggles to make accurate predictions.

Based on these results, the model demonstrates varied performance across the feature space, with specific regions identified as weak spots due to low metric values. The disparity between training and test datasets in some slices suggests overfitting, where the model performs well on training data but poorly on unseen data. Consistent weak spots across multiple metrics highlight areas where the model's predictive capability is limited, necessitating further investigation and potential model adjustments. These insights provide a comprehensive view of the model's behavior, guiding targeted improvements to enhance overall performance.

Tables

Slice	Number of Records	Feature	Accuracy	Precision	Recall	F1	Dataset
(-250.898, 25089.809]	195	Balance	0.6308	0.5116	0.3014	0.3793	test_dataset_final
(25089.809, 50179.618]	3	Balance	0.0000	0.0000	0.0000	0.0000	test_dataset_final
(50179.618, 75269.427]	26	Balance	0.5385	0.6364	0.4667	0.5385	test_dataset_final
(75269.427, 100359.236]	83	Balance	0.5783	0.5000	0.6571	0.5679	test_dataset_final
(100359.236, 125449.045]	167	Balance	0.6766	0.7018	0.8000	0.7477	test_dataset_final
(125449.045, 150538.854]	114	Balance	0.6667	0.6988	0.8169	0.7532	test_dataset_final
(150538.854, 175628.663]	43	Balance	0.4651	0.4400	0.5500	0.4889	test_dataset_final
(175628.663, 200718.472]	14	Balance	0.5714	0.6250	0.6250	0.6250	test_dataset_final
(200718.472, 225808.281]	1	Balance	1.0000	1.0000	1.0000	1.0000	test_dataset_final
(225808.281, 250898.09]	1	Balance	1.0000	1.0000	1.0000	1.0000	test_dataset_final
(-250.898, 25089.809]	857	Balance	0.6453	0.5610	0.3495	0.4307	train_dataset_final
(25089.809, 50179.618]	21	Balance	0.5238	0.7143	0.3846	0.5000	train_dataset_final
(50179.618, 75269.427]	93	Balance	0.5484	0.6061	0.4082	0.4878	train_dataset_final
(75269.427, 100359.236]	298	Balance	0.6074	0.5890	0.6575	0.6214	train_dataset_final
(100359.236, 125449.045]	582	Balance	0.6959	0.7092	0.8152	0.7585	train_dataset_final
(125449.045, 150538.854]	475	Balance	0.6653	0.6718	0.8037	0.7319	train_dataset_final
(150538.854, 175628.663]	192	Balance	0.5885	0.5870	0.7864	0.6722	train_dataset_final
(175628.663, 200718.472]	49	Balance	0.5510	0.5333	0.6667	0.5926	train_dataset_final
(200718.472, 225808.281]	17	Balance	0.5294	0.8182	0.6000	0.6923	train_dataset_final
(225808.281, 250898.09]	1	Balance	0.0000	0.0000	0.0000	0.0000	train_dataset_final
(349.5, 400.0]	3	CreditScore	0.6667	1.0000	0.6667	0.8000	test_dataset_final
(400.0, 450.0]	14	CreditScore	0.4286	0.4286	0.4286	0.4286	test_dataset_final
(450.0, 500.0]	36	CreditScore	0.7500	0.7895	0.7500	0.7692	test_dataset_final
(500.0, 550.0]	51	CreditScore	0.5686	0.6400	0.5517	0.5926	test_dataset_final
(550.0, 600.0]	106	CreditScore	0.6792	0.6094	0.8125	0.6964	test_dataset_final
(600.0, 650.0]	113	CreditScore	0.5664	0.5926	0.5424	0.5664	test_dataset_final
(650.0, 700.0]	127	CreditScore	0.6693	0.6471	0.7097	0.6769	test_dataset_final
(700.0, 750.0]	94	CreditScore	0.6064	0.6250	0.5319	0.5747	test_dataset_final
(750.0, 800.0]	61	CreditScore	0.6066	0.5789	0.7333	0.6471	test_dataset_final
(800.0, 850.0]	42	CreditScore	0.5952	0.5882	0.5000	0.5405	test_dataset_final
(349.5, 400.0]	11	CreditScore	0.5455	1.0000	0.5455	0.7059	train_dataset_final
(400.0, 450.0]	47	CreditScore	0.6596	0.7200	0.6667	0.6923	train_dataset_final
(450.0, 500.0]	116	CreditScore	0.6293	0.5821	0.7222	0.6446	train_dataset_final
(500.0, 550.0]	287	CreditScore	0.6341	0.6265	0.7075	0.6645	train_dataset_final
(550.0, 600.0]	368	CreditScore	0.6359	0.6452	0.6383	0.6417	train_dataset_final
(600.0, 650.0]	476	CreditScore	0.6282	0.6230	0.6414	0.6320	train_dataset_final
(650.0, 700.0]	479	CreditScore	0.6388	0.6325	0.6298	0.6311	train_dataset_final
(700.0, 750.0]	393	CreditScore	0.6565	0.6631	0.6327	0.6475	train_dataset_final
(750.0, 800.0]	244	CreditScore	0.6967	0.6870	0.6752	0.6810	train_dataset_final
(800.0, 850.0]	164	CreditScore	0.6524	0.6528	0.5949	0.6225	train_dataset_final
(-108.087, 20075.492]	65	EstimatedSalary	0.6462	0.6400	0.5333	0.5818	test_dataset_final
(20075.492, 40059.234]	69	EstimatedSalary	0.6232	0.5946	0.6667	0.6286	test_dataset_final
(40059.234, 60042.976]	74	EstimatedSalary	0.5811	0.5789	0.5946	0.5867	test_dataset_final
(60042.976, 80026.718]	56	EstimatedSalary	0.7321	0.7241	0.7500	0.7368	test_dataset_final
(80026.718, 100010.46]	57	EstimatedSalary	0.5789	0.5625	0.6429	0.6000	test_dataset_final
(100010.46, 119994.202]	75	EstimatedSalary	0.6000	0.5676	0.6000	0.5833	test_dataset_final
(119994.202, 139977.944]	59	EstimatedSalary	0.6949	0.7297	0.7714	0.7500	test_dataset_final
(139977.944, 159961.686]	59	EstimatedSalary	0.5932	0.6774	0.6000	0.6364	test_dataset_final
(159961.686, 179945.428]	67	EstimatedSalary	0.5672	0.5429	0.5938	0.5672	test_dataset_final
(179945.428, 199929.17]	66	EstimatedSalary	0.6515	0.6364	0.6562	0.6462	test_dataset_final
(-108.087, 20075.492]	260	EstimatedSalary	0.7154	0.7319	0.7319	0.7319	train_dataset_final
(20075.492, 40059.234]	242	EstimatedSalary	0.6281	0.6241	0.6748	0.6484	train_dataset_final
(40059.234, 60042.976]	258	EstimatedSalary	0.6473	0.6031	0.6695	0.6345	train_dataset_final
(60042.976, 80026.718]	284	EstimatedSalary	0.6479	0.6593	0.6224	0.6403	train_dataset_final
(80026.718, 100010.46]	256	EstimatedSalary	0.6133	0.5956	0.6480	0.6207	train_dataset_final
(100010.46, 119994.202]	245	EstimatedSalary	0.6041	0.6271	0.5827	0.6041	train_dataset_final
(119994.202, 139977.944]	259	EstimatedSalary	0.6216	0.6190	0.6094	0.6142	train_dataset_final
(139977.944, 159961.686]	251	EstimatedSalary	0.6733	0.6522	0.6410	0.6466	train_dataset_final
(159961.686, 179945.428]	279	EstimatedSalary	0.6452	0.6739	0.6327	0.6526	train_dataset_final
(179945.428, 199929.17]	251	EstimatedSalary	0.6454	0.6364	0.6720	0.6537	train_dataset_final
(-0.001, 0.1]	327	Gender_Male	0.6177	0.6104	0.8011	0.6929	test_dataset_final
(0.9, 1.0]	320	Gender_Male	0.6312	0.6505	0.4497	0.5317	test_dataset_final
(-0.001, 0.1]	1285	Gender_Male	0.6428	0.6537	0.7708	0.7075	train_dataset_final
(0.9, 1.0]	1300	Gender_Male	0.6462	0.6225	0.4939	0.5508	train_dataset_final
(-0.001, 0.1]	443	Geography_Germany	0.6140	0.5743	0.4404	0.4985	test_dataset_final
(0.9, 1.0]	204	Geography_Germany	0.6471	0.6613	0.9318	0.7736	test_dataset_final
(-0.001, 0.1]	1804	Geography_Germany	0.6297	0.5852	0.4512	0.5095	train_dataset_final
(0.9, 1.0]	781	Geography_Germany	0.6786	0.6911	0.9387	0.7961	train_dataset_final
(-0.001, 0.1]	491	Geography_Spain	0.6191	0.6129	0.6840	0.6465	test_dataset_final
(0.9, 1.0]	156	Geography_Spain	0.6410	0.6727	0.4933	0.5692	test_dataset_final
(-0.001, 0.1]	1999	Geography_Spain	0.6483	0.6507	0.6940	0.6716	train_dataset_final
(0.9, 1.0]	586	Geography_Spain	0.6314	0.5990	0.4627	0.5221	train_dataset_final
(-0.001, 0.1]	208	HasCrCard	0.6202	0.6316	0.6606	0.6457	test_dataset_final
(0.9, 1.0]	439	HasCrCard	0.6264	0.6182	0.6296	0.6239	test_dataset_final
(-0.001, 0.1]	769	HasCrCard	0.6164	0.6148	0.6260	0.6203	train_dataset_final
(0.9, 1.0]	1816	HasCrCard	0.6564	0.6549	0.6578	0.6564	train_dataset_final
(-0.001, 0.1]	343	IsActiveMember	0.6152	0.6346	0.8168	0.7143	test_dataset_final
(0.9, 1.0]	304	IsActiveMember	0.6349	0.5811	0.3496	0.4365	test_dataset_final
(-0.001, 0.1]	1377	IsActiveMember	0.6311	0.6507	0.8164	0.7242	train_dataset_final
(0.9, 1.0]	1208	IsActiveMember	0.6598	0.6137	0.3586	0.4527	train_dataset_final
(0.997, 1.3]	364	NumOfProducts	0.6429	0.7019	0.6822	0.6919	test_dataset_final
(1.9, 2.2]	231	NumOfProducts	0.6190	0.3700	0.5968	0.4568	test_dataset_final
(2.8, 3.1]	46	NumOfProducts	0.5435	0.9583	0.5349	0.6866	test_dataset_final
(3.7, 4.0]	6	NumOfProducts	0.3333	1.0000	0.3333	0.5000	test_dataset_final
(0.997, 1.3]	1508	NumOfProducts	0.6492	0.7148	0.6935	0.7040	train_dataset_final
(1.9, 2.2]	896	NumOfProducts	0.6641	0.3631	0.5837	0.4477	train_dataset_final
(2.8, 3.1]	143	NumOfProducts	0.5315	1.0000	0.5109	0.6763	train_dataset_final
(3.7, 4.0]	38	NumOfProducts	0.4211	1.0000	0.4211	0.5926	train_dataset_final
(-0.01, 1.0]	89	Tenure	0.5955	0.5926	0.6957	0.6400	test_dataset_final
(1.0, 2.0]	67	Tenure	0.6269	0.5312	0.6296	0.5763	test_dataset_final
(2.0, 3.0]	67	Tenure	0.5224	0.5000	0.5312	0.5152	test_dataset_final
(3.0, 4.0]	60	Tenure	0.6333	0.6207	0.6207	0.6207	test_dataset_final
(4.0, 5.0]	59	Tenure	0.5932	0.6061	0.6452	0.6250	test_dataset_final
(5.0, 6.0]	68	Tenure	0.6176	0.6053	0.6765	0.6389	test_dataset_final
(6.0, 7.0]	52	Tenure	0.5962	0.6500	0.4815	0.5532	test_dataset_final
(7.0, 8.0]	84	Tenure	0.6429	0.6957	0.6667	0.6809	test_dataset_final
(8.0, 9.0]	63	Tenure	0.7460	0.7667	0.7188	0.7419	test_dataset_final
(9.0, 10.0]	38	Tenure	0.7105	0.7222	0.6842	0.7027	test_dataset_final
(-0.01, 1.0]	375	Tenure	0.6773	0.7136	0.6893	0.7012	train_dataset_final
(1.0, 2.0]	274	Tenure	0.5839	0.5556	0.6489	0.5986	train_dataset_final
(2.0, 3.0]	290	Tenure	0.6276	0.6090	0.6690	0.6376	train_dataset_final
(3.0, 4.0]	274	Tenure	0.6022	0.6122	0.6338	0.6228	train_dataset_final
(4.0, 5.0]	251	Tenure	0.6773	0.6791	0.7054	0.6920	train_dataset_final
(5.0, 6.0]	250	Tenure	0.6360	0.6239	0.6083	0.6160	train_dataset_final
(6.0, 7.0]	247	Tenure	0.6356	0.5847	0.6273	0.6053	train_dataset_final
(7.0, 8.0]	225	Tenure	0.6578	0.6699	0.6161	0.6419	train_dataset_final
(8.0, 9.0]	273	Tenure	0.6923	0.6967	0.6439	0.6693	train_dataset_final
(9.0, 10.0]	126	Tenure	0.6508	0.7170	0.5672	0.6333	train_dataset_final

Figures

▶ Test Result: Overfit Diagnosis (validmind.model_validation.sklearn.OverfitDiagnosis)

Overfit Diagnosis

Overfit Diagnosis is designed to assess potential overfitting in a model's predictions by identifying regions where performance between training and testing sets deviates significantly. The primary purpose of this test is to pinpoint specific regions or feature segments where the model may be overfitting, thereby providing insights into areas that may require further model tuning or data augmentation.

The test operates by comparing the model's performance on training versus test data, grouped by feature columns. It calculates the difference between the training and test performance for each group and identifies regions where this difference exceeds a specified threshold. For classification models, the test uses the AUC metric, which measures the model's ability to distinguish between classes, with values ranging from 0 to 1. A higher AUC indicates better performance. For regression models, the MSE metric is used, which quantifies the average squared difference between predicted and actual values. The test identifies overfitting regions where the performance gap exceeds a default threshold of 0.04, highlighting areas where the model's generalization may be compromised.

The primary advantages of this test include its ability to identify specific areas where overfitting occurs, providing targeted insights for model improvement. It supports multiple performance metrics, offering flexibility in application across different model types, including both classification and regression models. The visualization of overfitting segments aids in better understanding and debugging, allowing practitioners to visually assess where the model's performance diverges between training and testing datasets. This targeted approach helps in efficiently addressing overfitting issues without overhauling the entire model.

It should be noted that the test has limitations, including the default threshold, which may not be suitable for all use cases and requires tuning based on specific model characteristics and data distributions. The test may not capture more subtle forms of overfitting that do not exceed the threshold, potentially overlooking areas where the model's performance is suboptimal. Additionally, the test assumes that the binning of features adequately represents the data segments, which may not always be the case, leading to potential misinterpretation of results if the feature segments are not well-defined.

This test shows the results in both tabular and graphical formats, providing a comprehensive view of the model's performance across different feature segments. The tables present key metrics such as the number of training and test records, training and test AUC values, and the performance gap for each feature slice. The plots visually depict the AUC gap for each feature segment, with a red line indicating the cut-off threshold of 0.04. The x-axis represents the feature slices, while the y-axis shows the AUC gap. Notable observations include significant gaps in certain feature segments, such as a gap of 0.2565 for the CreditScore slice (400.0, 450.0], indicating potential overfitting in this region.

The test results reveal the following key insights:

• Significant Overfitting in Low Credit Scores: The CreditScore slice (400.0, 450.0] shows a substantial AUC gap of 0.2565, indicating severe overfitting in this region.
• Moderate Overfitting in Estimated Salary: The EstimatedSalary slice (139977.944, 159961.686] exhibits a gap of 0.1775, suggesting moderate overfitting.
• High Overfitting in Low Balance: The Balance slice (25089.809, 50179.618] has a gap of 0.5577, highlighting a critical overfitting issue.
• Stable Performance in Tenure: The Tenure slice (1.0, 2.0] shows a minimal gap of 0.0409, indicating stable model performance.

Based on these results, the model exhibits varying degrees of overfitting across different feature segments, with significant issues in low credit scores and balance ranges. These insights suggest that the model's generalization ability is compromised in specific areas, necessitating targeted interventions to improve performance. The stable performance in certain tenure segments indicates areas where the model is well-calibrated, providing a benchmark for further tuning. Overall, the test highlights critical regions for improvement, guiding efforts to enhance the model's robustness and predictive accuracy.

Tables

Overfit Diagnosis

Feature	Slice	Number of Training Records	Number of Test Records	Training AUC	Test AUC	Gap
CreditScore	(400.0, 450.0]	47	14	0.7463	0.4898	0.2565
CreditScore	(500.0, 550.0]	287	51	0.6713	0.5784	0.0930
CreditScore	(600.0, 650.0]	476	113	0.6877	0.5785	0.1092
CreditScore	(750.0, 800.0]	244	61	0.7184	0.6194	0.0990
Tenure	(-0.01, 1.0]	375	89	0.7190	0.6542	0.0649
Tenure	(1.0, 2.0]	274	67	0.6483	0.6074	0.0409
Tenure	(2.0, 3.0]	290	67	0.6640	0.5875	0.0765
Tenure	(4.0, 5.0]	251	59	0.7242	0.6313	0.0928
Tenure	(7.0, 8.0]	225	84	0.7076	0.6458	0.0618
Balance	(25089.809, 50179.618]	21	3	0.5577	0.0000	0.5577
Balance	(50179.618, 75269.427]	93	26	0.5557	0.5030	0.0526
Balance	(100359.236, 125449.045]	582	167	0.7450	0.6939	0.0511
Balance	(150538.854, 175628.663]	192	43	0.5898	0.4674	0.1224
EstimatedSalary	(-108.087, 20075.492]	260	65	0.7634	0.6667	0.0967
EstimatedSalary	(40059.234, 60042.976]	258	74	0.7240	0.5931	0.1309
EstimatedSalary	(139977.944, 159961.686]	251	59	0.7430	0.5655	0.1775
EstimatedSalary	(159961.686, 179945.428]	279	67	0.6820	0.5839	0.0980

Figures

▶ Test Result: Robustness Diagnosis (validmind.model_validation.sklearn.RobustnessDiagnosis)

✅ Robustness Diagnosis

Robustness Diagnosis is designed to evaluate the resilience of a machine learning model when subjected to perturbations or noise in its input data. This test is crucial for understanding the model's ability to handle real-world scenarios where data may be imperfect or corrupted, ensuring that the model maintains its performance under such conditions.

The test operates by introducing Gaussian noise to the numeric input features of the datasets at varying scales of standard deviation. The model's performance is then assessed using metrics such as the Area Under the Curve (AUC) for classification tasks. AUC measures the model's ability to distinguish between classes, with values ranging from 0 to 1, where 1 indicates perfect classification and 0.5 suggests no discriminative power. The test involves adding noise to the input features, evaluating the model's performance on the perturbed data, and aggregating the results to visualize performance decay relative to the perturbation size. This approach helps in identifying how noise affects the model's predictive capabilities.

The primary advantages of this test include providing insights into the model's robustness against noisy or corrupted data, which is essential for applications in dynamic environments. It utilizes a variety of performance metrics suitable for both classification and regression tasks, allowing for a comprehensive assessment of the model's behavior. The visualization of results aids in understanding the extent of performance degradation, making it easier to identify specific perturbation levels that significantly impact the model's performance. This test is particularly useful for models deployed in real-world settings where data quality cannot always be guaranteed.

It should be noted that Gaussian noise might not adequately represent all types of real-world data perturbations, which can limit the test's applicability in certain scenarios. The performance thresholds used to evaluate the model's robustness are somewhat arbitrary and may require tuning to align with specific application requirements. Additionally, the test may not account for more complex or unstructured noise patterns that could affect model robustness, potentially leading to an incomplete assessment of the model's resilience. Signs of high risk include a significant drop in performance metrics with minimal noise and consistent failure to meet performance standards across multiple perturbation scales.

This test shows the results in both tabular and graphical formats. The tables present data on perturbation size, dataset, row count, AUC, performance decay, and pass status. The AUC values indicate the model's classification performance, with higher values representing better performance. Performance decay measures the change in AUC relative to the baseline, with positive values indicating a decline. The plot visualizes AUC against perturbation size, showing how performance changes as noise increases. The x-axis represents perturbation size, while the y-axis shows AUC values. Notable observations include the stability of AUC at lower perturbation levels and a gradual decline as noise increases, particularly in the test dataset.

The test results reveal the following key insights:

• Stable Performance at Low Perturbation: At a perturbation size of 0.1, the model maintains stable AUC values, with minimal performance decay observed in both training and test datasets.
• Gradual Decline with Increased Noise: As perturbation size increases to 0.3 and 0.4, a noticeable decline in AUC is observed, particularly in the test dataset, indicating reduced robustness.
• Test Dataset Sensitivity: The test dataset shows greater sensitivity to noise, with a more pronounced performance decay compared to the training dataset, highlighting potential vulnerabilities in real-world applications.

Based on these results, the model demonstrates reasonable robustness at lower levels of noise, maintaining performance with minimal decay. However, as noise levels increase, particularly beyond a perturbation size of 0.3, the model's performance begins to degrade, especially in the test dataset. This suggests that while the model can handle minor perturbations effectively, its resilience diminishes with more significant noise, which could impact its reliability in unpredictable environments. The insights gained from this test provide a clear understanding of the model's behavior under noisy conditions, emphasizing the need for further evaluation and potential adjustments to enhance robustness in real-world applications.

Tables

Perturbation Size	Dataset	Row Count	AUC	Performance Decay	Passed
Baseline (0.0)	train_dataset_final	2585	0.6915	0.0000	True
Baseline (0.0)	test_dataset_final	647	0.6658	0.0000	True
0.1	train_dataset_final	2585	0.6915	-0.0000	True
0.1	test_dataset_final	647	0.6649	0.0010	True
0.2	train_dataset_final	2585	0.6873	0.0041	True
0.2	test_dataset_final	647	0.6609	0.0049	True
0.3	train_dataset_final	2585	0.6859	0.0055	True
0.3	test_dataset_final	647	0.6504	0.0155	True
0.4	train_dataset_final	2585	0.6789	0.0126	True
0.4	test_dataset_final	647	0.6639	0.0020	True
0.5	train_dataset_final	2585	0.6735	0.0179	True
0.5	test_dataset_final	647	0.6615	0.0043	True

Figures

In summary

In this second notebook, you learned how to:

• Import a sample dataset
• Identify which tests you might want to run with ValidMind
• Initialize ValidMind datasets and model objects
• Run individual tests
• Utilize the output from tests you've run
• Log test results from sets of or individual tests as evidence to the ValidMind Platform
• Add supplementary individual test results to your documentation
• Assign model predictions to your ValidMind model objects

Next steps

Integrate custom tests

Now that you're familiar with the basics of using the ValidMind Library to run and log tests to provide evidence for your model documentation, let's learn how to incorporate your own custom tests into ValidMind: 3 — Integrate custom tests