Evaluating candidate models with MLOps
After you add a candidate model to a prediction, Prediction Studio configures and validates the new model, and provides comparison data to help you evaluate the new model. Decide whether you want to approve the new model for deployment to production or reject it.
Model evaluation is part of the Machine Learning Operations (MLOps) feature of Pega Platform.
- In the navigation pane of Prediction Studio, click Predictions.
- From the list of predictions, open a prediction that contains a candidate model, and then click the Models tab.
- Display the candidate model by expanding the twist arrow to the left of the current model.
- Ensure that the status of the candidate model is READY FOR REVIEW. If the configuration or validation of the model is still in progress, wait several seconds, and then refresh the page by clicking .
- Open the candidate model by clicking its name.
Result: In the Model comparison window, the Validation tab displays a basic comparison of the two models. Prediction Studio marks the better model as Recommended
based on the receiver operating characteristic (ROC) and transparency. If a data set was added for use in the comparison, Prediction Studio generates charts that show how the two models compare in terms of score distribution, ROC, lift, and gains. For more information, see Active and candidate model comparison charts.Model comparison 
- Optional: If you want to add a data set for the comparison or use a different data set,
click Configure data set.
- In the Validation data set window, select the data
set and the outcome column to compare the two models.The outcome column contains the response labels for the outcome.
- Review the properties in the data set and their values.In the Records field, you can select the number of records for the data set preview. The values are recalculated after each change.
- Click Submit.
- In the top-right corner of the Model comparison window, click Save.
- In the Confirm save dialog box, click
Yes.
Result: You return to the Prediction window. - Wait several seconds for the validation to complete, and then refresh the page by clicking .
- When the status of the model changes to READY FOR REVIEW, open the model by clicking its name, and review the new comparison charts.
- In the Validation data set window, select the data
set and the outcome column to compare the two models.
- Review the comparison charts to decide which model is better for your use case.
- Optional: Export the analysis data to a CSV file by clicking Download analysis data.
- In the top-right corner, click Evaluate.
- In the evaluation window, approve or reject the model:
- To let the new model shadow the current model, click Approve new candidate model and start shadowing (recommended).
- To replace the current model, click Approve candidate model and replace current active model.
- To reject the new model, select Reject candidate model.
Note: Shadowing means that the new model receives production data and the system tracks the outcomes of the model, but does not use the outcomes to make business decisions. This option is only available for predictive models. For more information, see Active and candidate model type combinations. Approving a candidate model deletes other candidate models that are associated with the active model.
A rejected model and its related assets are archived. You can view them in the model history.
- In the Reason field, provide a comment on the approval or rejection.
- Click Save.
In Pega Customer Decision Hub environments, if you approve a model update, the system automatically creates and resolves a Change prediction request in Pega 1:1 Operations Manager. A revision manager can then deploy the model to production.
When the deployment is complete, log in to the production instance and verify that the model is updated.
If you approved a predictive model in shadow mode, review its performance after a period of time. For more information, see Monitoring predictions.
Based on your analysis and your business needs, you can decide to promote the model as the active model or reject it. For more information, see:
Previous topic Replacing models in predictions with MLOps Next topic Promoting shadow models with MLOps