Monitoring predictions

Updated on July 5, 2022

Analyze how successful your predictions are in predicting the outcomes that bring value to your business. Gain insights by reviewing metrics for predictions and the models that drive them.

Note: This procedure applies to customer engagement and case management predictions. For information about analyzing text predictions, see Monitoring text predictions.

Prediction Studio provides the following metric types:

Performance: Performance metrics help you evaluate how effective and accurate the prediction and its models are in predicting outcomes. Performance metrics are based on the feedback, and include the success rate, lift, AUC (Area Under Curve), and total responses.
Output: Output metrics are based on the data that models generate, including propensities, numeric output for continuous models, and symbolic output (labels) for binary and categorical models. Output metrics can help you identify issues and improve your models irrespective of responses.
Predictors (input): Predictor metrics are based on model input, such as age, income, location, or current product subscriptions, that models use to predict customer responses or case outcomes. Predictor metrics can help you identify issues and improve your models irrespective of responses.

Note: You can determine the amount of output and predictor data that you want to monitor, and enable the recording of monitoring data in the Prediction Studio settings. For more information, see Configuring the monitoring of model input and output.

In the navigation pane of Prediction Studio, click Predictions.
In the list of predictions, open a prediction that you want to analyze.
Click the Notifications tab, and then review the messages to identify a performance issue that you want to investigate, for example, a sudden drop in the prediction's lift.
Click the Analysis tab, and then click Prediction.
1. In the Outcomes list, select the outcome to analyze.
  If a prediction predicts a single outcome (single-stage prediction), the outcome is already selected. If a prediction predicts outcomes that occur in a sequence (multistage prediction), you can select each individual outcome or the two outcomes combined.
  For example: For a multistage prediction that predicts whether a customer who is likely to click a web banner is also likely to accept the corresponding offer and convert, you can select from the following outcomes:
  - Clicks + Conversion
  - Clicks
  - Conversion
  Outcome selection on the Prediction tab
2. In the Chart type list, select the type of data to focus on:
  - To view all available charts, click All.
  - To filter for performance charts, click Performance only.
  - To filter for output charts, click Output only.
  Note:
  Performance charts are based on responses and show how effective a prediction is in predicting outcomes. Performance charts cover such metrics as success rate, lift, AUC, and total responses.
  Output charts track the propensities that your models generate, irrespective of responses. You can enable and disable output monitoring in the Prediction Studio settings.
3. In the Time frame list, select the period that you want to analyze.Result: The charts display the metrics that are relevant to the selected outcome. For example, an analysis with regard to churn covers the success rate, lift, AUC, and total responses, as shown in the following figure:
  Prediction analysis
4. On the charts, look for any declines in performance and other anomalies in model behavior.
  Tip: To read a description of a chart, including guidelines on how to use this data to improve your predictions and models, click Learn more in the upper-right corner of the chart. For more information, see Prediction analysis charts.
Review a model that has poor performance or displays unexpected behavior:
1. Click the Notifications tab, and then filter for messages for that model.
  Note: Read the notifications to learn more about recent changes in the model's performance. This information can help you narrow down the scope of your analysis.
2. Click the Analysis tab, and then click Models.
3. If the prediction uses more that one model, in the Models list, select the model that you want to investigate.
4. In the Time frame list, select the period that you want to analyze.
5. Review the model charts that are relevant to the issue that you are investigating.
Review issues related to predictors:
1. Click the Notifications tab, and then filter for messages for that model in the predictors category.
2. Click the Predictors tab.
3. In the Model list, select a model that you want to investigate.
4. In the Predictor list, select a predictor for the associated model.
5. In the Time frame list, select the period that you want to analyze.
  Predictor charts in a prediction
Optional: To refresh the charts with the latest data, in the header of the prediction, click Refresh data.