INC-126129 · Issue 569666
PropertyToColumnMap made more robust
Resolved in Pega Version 8.1.9
The DF_ProcessEmails dataflow was intermittently failing with a StageException error. This was traced to schema changes being propagated asynchronously by system pulse, which seem to have caused PropertyToColumnMap to cache stale schema. To resolve this, if the property mapping is not found the first time, the system will make a second attempt to get the mapping. Additional logging has also been added for better diagnostics.
INC-128385 · Issue 564521
Behavior made consistent between SSA and legacy engines
Resolved in Pega Version 8.1.9
There was a behavioral disparity between the legacy execution engine and the SSA engine where the latter was not creating a new page when the index was one above the size of the page list. This has now been corrected in order to make the SSA behavior fully backward compatible with the legacy engine, i.e. a new blank page will be added to the list if the index is one above the size of the list.
INC-129222 · Issue 568530
Handling improvements for commit logs
Resolved in Pega Version 8.1.9
The ADM commit log logs the number of unconsumed messages that are going to expire. In certain circumstances, it can include unconsumed messages that are not going to expire in the count. Because they are not expired and removed, the environment was running out of disk space due to the ADM commitlogs table growing larger than expected and performance issues were seen. To resolve this, a new adm_commitlog.adm_responses_commit_log_date_tiered table has been created, with a default_time_to_live of 24 hours. DateTieredCompactionStrategy has been set with max_window_size_seconds as 24hrs and tombstone_compaction_interval as 24hrs.
INC-132976 · Issue 580685
Performance improvements for Test Strategy data flow
Resolved in Pega Version 8.1.9
In the Test Strategy panel under Single case -> "Settings", selecting the "Data flow" option and choosing CustomerData dataflow was taking an excessive amount of time to run on a system with an extremely large database. To improve performance, two areas have been addressed: 1) the default behavior for record key suggestions in the test panel has been modified to collect only the ID as the additional data is not necessary at that time; 2) a DSS has been added that will opt out of reading and collecting the customer IDs in order to minimize data stored on the clipboard.
INC-138037 · Issue 586593
Strategy handling updated for very large systems using IH summary
Resolved in Pega Version 8.1.9
When a Strategy in a Real-time dataflow used IH Summary on a system with more than 5000 groups for one eventKey, the message "Error retrieving aggregates from Cassandra KVS" intermittently appeared. Investigation showed that if the number of result rows was greater than the FETCH_SIZE (set to 5000), it meant another read to Cassandra was required and an exception was generated. To resolve this, updates have been made so that instead of returning maps, the system will return iterators and change them to map on the calling thread.
SR-D92734 · Issue 553412
Simulation can take Data flow type as destination
Resolved in Pega Version 8.1.9
Support has been added for Data flow functionality as simulation target and data transform in simulation input.
SR-B56256 · Issue 335795
VBD cache handling changed to preserve data integrity
Resolved in Pega Version 7.4
When VBD was configured to use on-heap memory, no data was displayed in VBD planner and logs showed queries failing with ArrayIndexOutOfBoundException. The cause of the error was a corruption of VBD cache data in a partition: VBD compresses its partitions after getting no inserts for 15 minutes. When an insert is received by a compressed partition, the partition is first decompressed before writing. During decompression, the buffer position is set to the end of the uncompressed buffer. However, the buffer may include padded nulls added during buffer expansion. The system writes the new VBD records after the padded nulls, but the nulls would then throw off the read logic during query execution. To fix this, during decompression the buffer position will now be set to the end of the data, not the end of the buffer.
SR-B56256 · Issue 337432
VBD cache handling changed to preserve data integrity
Resolved in Pega Version 7.4
When VBD was configured to use on-heap memory, no data was displayed in VBD planner and logs showed queries failing with ArrayIndexOutOfBoundException. The cause of the error was a corruption of VBD cache data in a partition: VBD compresses its partitions after getting no inserts for 15 minutes. When an insert is received by a compressed partition, the partition is first decompressed before writing. During decompression, the buffer position is set to the end of the uncompressed buffer. However, the buffer may include padded nulls added during buffer expansion. The system writes the new VBD records after the padded nulls, but the nulls would then throw off the read logic during query execution. To fix this, during decompression the buffer position will now be set to the end of the data, not the end of the buffer.
SR-B67003 · Issue 329137
Enhancement added to validate mandatory fields in custom proposition definition forms
Resolved in Pega Version 7.4
Enhancements have been added in order to support extended customization of a proposition definition form. Previously, the custom form containing the custom fields was rendered in the context of a requestor-level, editable data page for creating a new proposition. The custom form section itself did not belong to the same class as the data page but was only rendered in the context of the data page at run-time. Therefore, validations defined on the custom form were not triggered due to the mismatch in the context of execution. The system has now been updated with an extension activity which will have the required-field validation pre-loaded. The same activity can then be extended to include additional validations whenever needed. The approach has some limitations as below. Process: On the custom form for the proposition group, the appropriate fields will need to be marked as required-always from the 'Required' dropdown field configuration form. After this, with the local fix branch included, new propositions should be validated for missing required fields. Limitations: The validation will work only on the manual creation of a new proposition. Conversely, the validation will NOT work if the propositions were uploaded using the Import/Export feature on the Decision Data form. The validation error messages for the custom fields will appear on the top of the page unlike that for the proposition name property. However, the message will list all properties that are marked as required but haven't been filled-in on the form.
SR-B72990 · Issue 326318
DataFlowGenerator class files preserved longer
Resolved in Pega Version 7.4
When creating a campaign, running the generated data flow manually resulted in an illegal argument exception indicating a null property name. This was traced to the use of an overwrite data flow API: class files were generated and removed immediately after the DataFlow object is constructed. In some cases, the java classloader may delay loading of the classes, causing the NoClassDefFoundException as files have been already removed. To resolve this, the DataFlowGenerator will now wait one day to remove the classes.