Prediction Studio notification types
Prediction Studio provides various notifications that describe the status of adaptive and predictive models, as well as predictions. Use the details that are gathered in these actionable insights to monitor your models, update their configuration as necessary, and improve the predictions that support your customer-oriented strategies.
Some notifications have customizable thresholds that you can modify to define when you would like to receive a specific notification type.
Adaptive model notifications
| Notification | Description | Customizable threshold | Category | Impact | Available from Pega Platform version |
| No monitoring information for model in the last 2 days. | The model instance has not had any monitoring data for the last two days. | No | Responses | High | 8.3 |
| No updates to model in the last 2 days. | The model instance has not been updated for the last two days.
The number of responses gathered between two days ago and the time
when the monitoring data was last saved has not changed. This notification occurs only when historical monitoring data is saved. | No | Responses | High | 8.3 |
| Model performance is at its minimum of 50 AUC. | The model instance area under the curve (AUC) performance equals 50. | No | Performance | High | 8.3 |
| Zero responses received for adaptive model. | The model instance has not received any responses yet. | No | Responses | High | 8.3 |
| Response count dropped by 20% or more compared to the last week. | This week’s monitoring data for the model instance contains at least 20% fewer responses compared to the previous week. | Yes | Responses | Medium | 8.3 |
| No responses have been received for one of the outcomes of a binary outcome model. | No responses have been collected for a positive or negative outcome, while the alternative outcome has gathered evidence of more than 1000 responses. | No | Responses | High | 8.4 |
| Performance has dropped by 20% compared to the previous day. | The model performance has decreased by more than 20% in the current monitoring data, compared to the previous day. | Yes | Performance | Medium | 8.3 |
| Success rate has dropped by more than 50% compared to the previous day. | The model success rate has decreased by more than 50% in the current monitoring data, compared to the previous day. | Yes | Performance | High | 8.3 |
| Model performance is low for adaptive model. | The model instance AUC performance is between 50 and 52. | No | Performance | Medium | 8.3 |
| Model has no active predictors. | All predictors have a predictive performance score below the
performance threshold and, as a result, are not active. The
performance threshold for predictors is set to 52 AUC. For more information, see Troubleshooting an adaptive model with no active predictors. | No | Performance | High | 8.4 |
| Model only has 1 active predictor. | Only one predictor has a predictive performance score above the
performance threshold. The predictors with a predictive performance
score below the threshold are not active. The performance threshold
for predictors is set to 52 AUC. For more information, see Troubleshooting an adaptive model with only one active predictor. | No | Performance | Medium | 8.4 |
| Model performance is very high. | The performance score for the adaptive model is above 95
AUC. For more information, see Troubleshooting an adaptive model with exceedingly high performance. | No | Performance | Low | 8.4 |
| The Median value of propensity has decreased by 10 percent in current week when compared to last week. | This notification applies to predictions with predictive or adaptive models. | Yes | Output | Low | 8.7 |
| The number of errors (number_of_errors) during model execution is more than the threshold (threshold_value). | The number of errors (number_of_errors) during model execution is more than the threshold (threshold_value). | Yes | Output | High | 8.7 |
Predictive model notifications
| Notification | Description | Customizable threshold | Category | Impact | Available from Pega Platform version |
| Response count dropped by 20% or more compared to the last week. | This week’s monitoring data for the model instance contains at least 20% fewer responses compared to the previous week. | Yes | Responses | Medium | 8.3 |
| Performance has dropped by more than 20% compared to the previous day. | The model performance has decreased by more than 20% in the
current monitoring data, compared to the previous day. For continuous outcome models, this notification means that the root-mean-square error (RMSE) value has increased by more than 20%, which reflects the lower performance. | Yes | Performance | Medium | 8.3 |
| Success rate has dropped by more than 50% compared to the previous day. | The success rate for the binary outcome model has decreased by more than 50% in the current monitoring data, compared to the previous day. | Yes | Performance | High | 8.3 |
| Model performance is low for the current week. | The weekly performance of the binary outcome model is lower than 52 AUC, based on sufficient evidence of the least frequent outcome with more than 100 responses. | No | Performance | Medium | 8.3 |
| ROC curve is under the diagonal. | The ROC curve for the binary outcome model is under the diagonal,
with the AUC value below 50. This status means that the labels for the outcomes or the output score of the model should be reversed. | No | Performance | High | 8.3 |
| Performance of the model over the last month is less than 20% of the expected performance. | The model monthly performance for each version is below 20% of
the expected performance. The measurement is based on the sufficient evidence of more than 100 positive and 100 negative responses for the binary outcome models, and more than 500 responses in the case of categorical and continuous outcome models. | Yes | Performance | Medium | 8.3 |
| The Median value of propensity has decreased by 10 percent in current week when compared to last week. | This notification applies to predictions with predictive or adaptive models. | Yes | Output | Low | 8.7 |
| There is a change in the output label distribution of more than expected in current week when compared to last week. | This notification applies to predictions with predictive models only if the segment value is available. | No | Output | Low | 8.7 |
| The 90th percentile value of this models numerical output has increased/decreased by 50 percent in current week when compared to last week. | This notification applies to predictions with predictive models. | Yes | Output | Low | 8.7 |
| The number of errors (number_of_errors) during model execution is more than the threshold (threshold_value). | This notification applies to predictions with predictive or adaptive models. | Yes | Output | High | 8.7 |
| The number of errors (number_of_errors) during model execution is more than the threshold (threshold_value). | The number of errors (number_of_errors) during model execution is more than the threshold (threshold_value). | Yes | Output | High | 8.7 |
Prediction notifications
| Notification | Description | Customizable threshold | Category | Impact | Available from Pega Platform version |
| No lift observed over the last week. | The prediction did not generate any lift in the last week. The success rate in the target group was not larger than the success rate in the model control group. The success rate in the model control group should be greater than 0. | No | Performance | High | 8.5 |
| Lift has dropped by more than 10% compared to the previous week. | The weekly lift decreased by more than 10% compared to the previous week. | Yes | Performance | High | 8.5 |
| There is a model name waiting for approval to replace component name. Reference ID is M-number. | A model update awaits approval. View the results of the analysis, and then approve or reject the model update. For more information, see Evaluating candidate models with MLOps. | Not applicable | Model approval | Medium | 8.6 |
| Model approval process is stopped for model name. Issue description. Reference ID is M-number | A model update process failed due to the issue described in the
notification. Here are some of the issues that you might
encounter:
| Not applicable | Model approval | Medium | 8.6 |
| The model name has been rejected with the comment user comment. Reference ID is ID. | A data scientist rejected the request. | Not applicable | Model approval | Low | 8.6 |
| The model name has been approved with the comment user comment. Reference ID is ID. | A data scientist approved the request. | Not applicable | Model approval | Low | 8.6 |
| The Median value of propensity has decreased by 10 percent in current week when compared to last week. | This notification applies to predictions with predictive or adaptive models. | Yes | Output | Low | 8.7 |
| There is a change in the output label distribution of more than expected in current week when compared to last week. | This notification applies to predictions with predictive models only if the segment value is available. | No | Output | Low | 8.7 |
| The 90th percentile value of this models numerical output has increased/decreased by 50 percent in current week when compared to last week. | This notification applies to predictions with predictive models. | Yes | Output | Low | 8.7 |
| The 90th percentile value of this model's numerical output has increased/decreased by 50 percent in current week when compared to last week | The 90th percentile value of this model's numerical output has increased or decreased by 50 percent in current week when compared to last week | Yes | Output | High | 8.7 |
| The number of errors (number_of_errors) during model execution is more than the threshold (threshold_value). | The number of errors (number_of_errors) during model execution is more than the threshold (threshold_value). | Yes | Output | High | 8.7 |
| Deployment of text prediction prediction_name failed due to error | Deployment of the text prediction from a job scheduler failed. If automatic deployment fails, data scientist can deploy the text prediction manually. | Not applicable | Prediction deployment | Medium | 8.7 |
Predictor (input) notifications
| Notification | Description | Customizable threshold | Category | Impact | Available from Pega Platform version |
| The predictor missing values ratio has crossed 1 percent since the last snapshot. | The predictor missing values ratio has crossed 1 percent since the last snapshot. | Yes | Predictors | High | 8.7 |
| There is a change in the input predictor distribution of more than expected in current week when compared to last week | There is a change in the input predictor distribution of more than expected in current week when compared to last week | No | Predictors | Low | 8.7 |
| There is a change in the median value for input predictor distribution in current week when compared to last week. | The Median of predictor output value has increased or decreased by 50 percent in the current week when compared to the last week. | Yes | Predictors | Low | 8.7 |
Previous topic Viewing Prediction Studio notifications Next topic Analyzing natural language