Skip to main content


         This documentation site is for previous versions. Visit our new documentation site for current releases.      
 

Troubleshooting an adaptive model with exceedingly high performance

Updated on May 17, 2024

Beginning with Pega Platform version 8.4, Prediction Studio informs you when an adaptive model has an unrealistically high predictive performance score, and as a result, might be overoptimistic.

Increase the accuracy of the model's predictions by identifying and addressing the root cause of the issue. The possible root causes include the model continuing to collect a sufficient number of responses, or feeding the historical data to the model instance in a certain order, for example, negative responses first, and then positive responses.

Prediction Studio triggers the following notification for an adaptive model that has a performance score above 95 AUC (Area Under the Curve):

Adaptive Model modelName Model performance is very high

Cause

Identify and address the root cause of the issue by performing the following procedures.

Solution

    Verify the number of registered responses

  1. In Prediction Studio, open the adaptive model by clicking the corresponding notification.
  2. On the Monitor tab, for the affected model instance, click Model report.
  3. Expand the Model details section, and then, in the Number of responses section, sum the number of responses that the model instance has registered for each response type:
    • If each response type has less than 100 responses, wait until the model receives more responses.
    • If each response type has more than 100 responses, reset the model.
  4. Reset the model

    Note: By resetting the model, you cause the model to start learning from the beginning based on new incoming responses. Consider resetting the model after verifying that the model instance received the learning data in a particular order, for example, by viewing the interaction history. For more information, see Monitoring interaction results.

  5. In the header of Dev Studio, click ConfigureDecisioningModel Management.
  6. On the Model Management landing page, click Adaptive.
  7. Select the adaptive model for which you received the notification by selecting the corresponding check box.
  8. Click Clear.
  • Previous topic Timeout exceptions during snapshot generation
  • Next topic Troubleshooting an adaptive model with no active predictors

Have a question? Get answers now.

Visit the Support Center to ask questions, engage in discussions, share ideas, and help others.

Did you find this content helpful?

Want to help us improve this content?

We'd prefer it if you saw us at our best.

Pega.com is not optimized for Internet Explorer. For the optimal experience, please use:

Close Deprecation Notice
Contact us