INC-228430 · Issue 738822
Enhanced diagnostics for NLP
Resolved in Pega Version 8.8
To enhance diagnostics for issues related to natural language processing (NLP) and Apache Rule-based Text Annotation (RUTA), support has been added for runtime metrics that will allow benchmarking and debug test scenarios over a set amount of time. NLP diagnostics have been introduced which will capture the text analyzed for all incomplete executions in Data-NLP-Report-Summary table. Every text analyzer execution will insert a record into summary table with text and text analyzer details and at end of execution the row is deleted. On system crash, texts with incomplete executions will be present in the table to help identify any special patterns that could cause issues. This summary record insert/delete will occur only if DSS(PEGA-NLP!ENABLENLPDIAGNOSTIC) is set to false. Currently the DSS is shipped with true. To enable/disable, change the value in DSS and then re-validate and save for the changed DSS to reflect (Configure > System > Release > Upgrade > Validate > click on re-validate and save - Type:rule-NLP-PredictiveModel dropdown and ruleset: give the application ruleset - ruleset in which the text analyzers are created. Click on list and then Click on run.) Additional support: - The activity "pygetdiagnosticnlpsummaryrecords" has been added to retrieve the summary diagnostic records present in summary table whose NLP analysis has not been completed. - The "pyDumpNLPProcessTimersInRepository" activity has been created to dump process timers to a repository. This activity must be called from a job scheduler configured to run on background processing nodes to dump process timers. This will be done on-demand to analyze the performance(time taken) for RUTA execution etc. Before running this JS, configure the repository in prediction studio settings to dump the CSV in that repository. As soon as 1 CSV is dumped, the JS can be disabled. - Debug logs have been enabled for com.pega.nlp.ner.command.RutaCommand (system>operations>log level settings) which prints the RUTA entity type start and end of execution.
INC-228430 · Issue 744988
RUTA handling improved in Prediction Studio
Resolved in Pega Version 8.8
Out of memory errors were seen when using natural language processing (NLP). Investigation showed that certain Apache Rule-based Text Annotation (RUTA) scripts had disjunctive rules which were not able to handle certain types of texts having base64 characters which were introduced in emails via attachments, images, logos etc, and which caused excessive system loads. This has been resolved by modifying the RUTA handling in the Prediction Studio settings to better manage the scenario.
INC-228935 · Issue 731524
Writes to movie data set made optional in Delayed learning flow
Resolved in Pega Version 8.8
In order to prevent the Cassandra event store from accumulating excessive tombstones, an option has been added on the movie landing page to disable writes to the event store dataset.
INC-229319 · Issue 739145
Service registry queries optimized
Resolved in Pega Version 8.8
Full table scans were being performed by queries running against pr_sys_serviceregistry_kvs and pr_sys_serviceregistry table. This has been resolved by optimizing service registry queries to avoid full table scans.
INC-229371 · Issue 730011
Corrected Schema inconsistency in pxDR after update
Resolved in Pega Version 8.8
After running make decision and capture response data flows using delayed learning and populating the pxDecisionResults data set, updating the system to Pega 8.7 and running capture response data flow with interactions from before the update resulted in the error "DataStoreConnection$InconsistentTableSchemaException: Inconsistency in table [data.pxDecisionResults_Pega_DecisionEngine_ab5de04587] schema between data store and schema repository". This was traced to a difference in DDS Data Sets pre- and post- introduction of DDS SDK and Data-Admin-DDS-Table. To ensure better backwards compatibility, an update has been made to ensure DdsDataSetFactory checks for Data-Admin-DDS-Table record existence in the dataabse, and if found uses tableName from that D-A-D-T. This resolves inconsistencies of the case of the table name.
INC-229682 · Issue 731965
Repository schema updated for DdsDataSetFactory
Resolved in Pega Version 8.8
After update, running the CDH NBA Scheduler resulted in the error "UnsupportedOperationException: This operation is not supported by NonVersionedJDBCJarReader". This has been resolved with an update to use the explicit construction of PRPCSchemaRepository in DdsDataSetFactory rather than loading SchemaRepository via SPI ServiceLoader.
INC-229717 · Issue 730669
Cassandra startup calls reordered to avoid deadlock
Resolved in Pega Version 8.8
Nodes received a service request but it became stuck. This was traced to a deadlock related to CassandraSessionCache.getSession, and has been resolved by reordering the method calls used to initialize the Cassandra session to delay adding the session change listener and avoid a deadlock scenario.
INC-230327 · Issue 738104
Updated DDS table migration handling
Resolved in Pega Version 8.8
In order to prevent data loss during update, Data-Admin-DataSet-DDS activities have been modified so that an existing pyCorrespondingTableName is not overwritten on subsequent save calls for data sets already saved to the database.
INC-230436 · Issue 733943
PropositionFilter logic updated
Resolved in Pega Version 8.8
In order to reduce processing overhead and better handle custom code, new LogicStringParser has been added for evaluating complex logic in PropositionFilter.
INC-231505 · Issue 733974
Error handling improved for NBA run
Resolved in Pega Version 8.8
The data flow partition on a node was sporadically becoming stuck during a Next Best Action Campaign outbound run when it encountered an issue such as a NoClassDefFoundError, requiring the node to be restarted to clear the problem. This has been resolved by explicitly making sure the flow partition is failed when an input thread dies or there is an error in the dataflow.