SR-C6486 · Issue 349824
Check added for Kafka dataflow error handling
Resolved in Pega Version 7.4
When an event dataflow has failed because too many errors were detected, it is possible to continue it. However, it was noticed that the "continue" button had to be used twice: The first time, the dataflow failed again and the "Input record" was increased more than expected. For instance, if the event flow was sent 2 incorrect events, the "Input record" was increased by 3 instead of 1. The second time, the event dataflow was resumed correctly. Investigation showed that when a bad record is inserted into Kafka, a dummy error record is generated that has no information of the partition and position, so the data flow cannot update the partition table correctly. In order to correct this, the partition and position information will be given on the error record. The data flow execution has also been updated such that when there is an onError call, a check will be performed to assess whether the error originated on the primary source and an input record is present. If so, then the partition table will be updated from that record.
SR-C691 · Issue 349064
VBD reworked to retrieve IH DB metadata
Resolved in Pega Version 7.4
Campaign dashboard performance was slow when accessed via the Navigation bar button PRPC user configured for app server has restricted access to schema metadata. This causes VBD to be unable to build SQL query used to synchronize Actuals data set with Interaction History. It was possible to optionally configure vbd/useTableMapping = true in PR Config or DSS (owning RS: Pega-DecisionEngine) in order to force VBD to load IH database column metadata using Pega table info mapping instead of using database connection API, but the issue has now been fixed by reworking VBD to use Pega class/database mapping rules instead of JDBC DatabaseMetadata to retrieve IH fact/dim table columns.
SR-C8023 · Issue 350229
Updated dataflow used after pause/continue
Resolved in Pega Version 7.4
When using an event dataflow that has a Kafka dataset as the source, running the flow, pausing it, and importing updated rules was resulting in the resumed flow still using the old rules. This was a known limitation in the data flow metrics management where when a run was resumed, the previous metrics were "merged" with new metrics. In cases where the structure of a data flow changed between pause/resume, merge failed silently and new metrics were not saved to the database. The metric management has now been updated to merge metrics correctly: in case of a data flow structure update (e.g. shapes added/removed), after the data flow is run, stage metrics will be cleared up as it is not possible to match them with the new structure, and all other metrics will be properly resumed from the "paused" position (e.g. number of processed records, throughput etc.).
INC-126129 · Issue 569666
PropertyToColumnMap made more robust
Resolved in Pega Version 8.1.9
The DF_ProcessEmails dataflow was intermittently failing with a StageException error. This was traced to schema changes being propagated asynchronously by system pulse, which seem to have caused PropertyToColumnMap to cache stale schema. To resolve this, if the property mapping is not found the first time, the system will make a second attempt to get the mapping. Additional logging has also been added for better diagnostics.
INC-128385 · Issue 564521
Behavior made consistent between SSA and legacy engines
Resolved in Pega Version 8.1.9
There was a behavioral disparity between the legacy execution engine and the SSA engine where the latter was not creating a new page when the index was one above the size of the page list. This has now been corrected in order to make the SSA behavior fully backward compatible with the legacy engine, i.e. a new blank page will be added to the list if the index is one above the size of the list.
INC-129222 · Issue 568530
Handling improvements for commit logs
Resolved in Pega Version 8.1.9
The ADM commit log logs the number of unconsumed messages that are going to expire. In certain circumstances, it can include unconsumed messages that are not going to expire in the count. Because they are not expired and removed, the environment was running out of disk space due to the ADM commitlogs table growing larger than expected and performance issues were seen. To resolve this, a new adm_commitlog.adm_responses_commit_log_date_tiered table has been created, with a default_time_to_live of 24 hours. DateTieredCompactionStrategy has been set with max_window_size_seconds as 24hrs and tombstone_compaction_interval as 24hrs.
INC-132976 · Issue 580685
Performance improvements for Test Strategy data flow
Resolved in Pega Version 8.1.9
In the Test Strategy panel under Single case -> "Settings", selecting the "Data flow" option and choosing CustomerData dataflow was taking an excessive amount of time to run on a system with an extremely large database. To improve performance, two areas have been addressed: 1) the default behavior for record key suggestions in the test panel has been modified to collect only the ID as the additional data is not necessary at that time; 2) a DSS has been added that will opt out of reading and collecting the customer IDs in order to minimize data stored on the clipboard.
INC-138037 · Issue 586593
Strategy handling updated for very large systems using IH summary
Resolved in Pega Version 8.1.9
When a Strategy in a Real-time dataflow used IH Summary on a system with more than 5000 groups for one eventKey, the message "Error retrieving aggregates from Cassandra KVS" intermittently appeared. Investigation showed that if the number of result rows was greater than the FETCH_SIZE (set to 5000), it meant another read to Cassandra was required and an exception was generated. To resolve this, updates have been made so that instead of returning maps, the system will return iterators and change them to map on the calling thread.
SR-D92734 · Issue 553412
Simulation can take Data flow type as destination
Resolved in Pega Version 8.1.9
Support has been added for Data flow functionality as simulation target and data transform in simulation input.
INC-136264 · Issue 582469
Chatbot text starts scroll at top of answer
Resolved in Pega Version 8.2.8
In order to present an improved interface for customers interfacing with chatbots, the code has been updated so that the recipient's chat scroll will remain at the top of an answer that would scroll off the screen.