Inference meta-monitoring for Amazon SageMaker AI endpoints with Amazon Quick

<p>Without the ability to track machine learning (ML) model prediction quality, organizations only realize they have issues when their customers complain or when they conduct spot checks, which jeopardizes customer trust. This post introduces inference meta-monitoring for Amazon SageMaker AI endpoints. It provides a governance layer that sits above production ML inference pipelines to continuously track prediction and data quality metrics and visualize trends. You will learn how to design and implement an inference meta-monitoring system specifically for predictive models, using Amazon Quick and optionally with Amazon SageMaker AI MLflow App. The system includes drift detection, delayed ground truth data integration, and automated performance dashboards.</p> <h2 id="the-inference-monitoring-gap">The inference monitoring gap</h2> <p>Developing predictive ML models is often a resource-intensive process. You invest months building training pipelines to drive strong validation accuracy for use cases such as fraud detection, credit scoring or demand forecasting. The performance of the models you deploy might silently degrade, and your teams might notice it only weeks later. Fraud case handlers start to see false positives spiking, and loan officers start to see more applications that should have been previously flagged. Corporate resource planners find themselves with excess inventory because of overestimated demand forecasts.</p> <p>ML teams therefore need a system that provides them with continuous feedback on the performance of their models in production. The system should immediately alert them whenever model quality or data drift is detected. Teams can then take action early so that model performance remains consistent over time, and customer trust remains high.</p> <h2 id="solution-overview">Solution overview</h2> <p>This inference meta-monitoring solution combines AWS managed services (<a href="https://aws.amazon.com/sagemaker/ai/" target="_blank" rel="noopener">Amazon SageMaker AI</a>, <a href="https://aws.amazon.com/athena/" target="_blank" rel="noopener">Amazon Athena</a>, <a href="https://aws.amazon.com/pm/lambda/" target="_blank" rel="noopener">AWS Lambda</a>, <a href="https://aws.amazon.com/eventbridge/" target="_blank" rel="noopener">Amazon EventBridge</a>, <a href="https://docs.aws.amazon.com/quick/latest/userguide/what-is.html" target="_blank" rel="noopener">Amazon Quick</a>) with open source ML tools (SageMaker AI MLflow Apps, Evidently AI).</p> <h2 id="prerequisites">Prerequisites</h2> <ul> <li><a href="https://signin.aws.amazon.com/signup?request_type=register" target="_blank" rel="noopener">AWS account</a>.</li> <li>Use the <a href="https://github.com/aws-samples/sample-mlops-bestpractices/blob/main/sagemaker-automated-drift-and-trend-monitoring/cloudformation/sagemaker-mlflow-setup.yaml" target="_blank" rel="noopener">CloudFormation template</a> provided in the repo to create the virtual private cloud (VPC), subnets, SageMaker AI domain, user profile, and JupyterLab space. The template also clones the git repo and updates the <code>.env</code> with the actual values.</li> </ul> <h2 id="setup">Setup</h2> <p>The CloudFormation template automates the entire setup. If you intend to use your existing domains, you can clone the repository, update the env values and run the notebooks in sequence.</p> <h3 id="environment-setup">Environment setup</h3> <p>Running the CloudFormation setup creates and initializes the variables.</p> <p><a href="https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=fraud-detection-monitoring&amp;templateURL=https://frpaolo-public-blogs.s3.amazonaws.com/sagemaker-mlflow-setup.yaml" target="_blank" rel="noopener"><img loading="lazy" class="alignnone wp-image-56268 size-full" src="https://d2908q01vomqb2.cloudfront.net/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59/2023/05/21/ML-14328-cloudformation-launch-stack.png" alt="Launch CloudFormation stack