Resolved Issues

Retry logic added for downloading upgraded rules

A Rules upgrade failed while downloading applications from the maintenance server due to an SFTP server connection failure. This has been resolved by adding logic to retry if the first connection attempt fails.

Resolved buffer overflow for Migration loadDatabase

A Rules upgrade failed in the Migration step at loadDatabase stage, involving the move of all the table records from old schema to new schema. This was traced to the inability of Migration to load blob of sizes more than 100 MB, and has been resolved by updating Migration to use byte[] to read the blob content with the help of metadata that contains blob length.

TableRenameUtil hashing improved

During index name generation, the algorithm that was responsible for index name uniqueness was sometimes insufficient and cerated a loop condition. This has been resolved by using a stronger hash algorithm and refactoring the code that could result in a loop.

Enhancements added to S3 repository implementation

The following changes have been made to avoid redundant/unnecessary S3 SDK calls to look-up metadata for files/folders related to integrated S3 Repository API implementation:

 - Provided storage/AWSSDK/enableSDKMetrics DASS to enable AWS SDK metrics for S3 repositories system-wide to troubleshoot performance issues
 - Added timers for Pega platform repository APIs. To see these timers in Pega application log, set log level to WARN for the log category pxIntegration.Repository.S3 and set log level to DEBUG for the logger com.amazonaws.latency
 - Optimized FileS3Adapter.getMetadata() by avoiding object-metadata lookup for directories and caching rootpath directory metadata for 5 mins
 - Handled exceptions while uploading file to S3 bucket and convey appropriate error message to RepositoryManager.createFile() API consumer
 - Refined DEBUG logging and introduced new loggers to emit stacktraces
 - Enhanced RepositoryManager.initCache() and  FileStorageManager.registerStorageClasses() to maintain non-overridable repositories (that are initialized based on DASS/DNode settings on node start-up) and to prohibit overriding the same.

Added queue process tracer cleanup on node restart

If a node restart occurred while a queue process trace was in progress, some entries in the tracer distributed map were not cleared properly and a subsequent tracer failure was encountered indicating a current trace by a requestor on same operator Id was already running. This has been resolved by clearing the tracer distributed map on node startup for stale entries associated with this node.

Extra logging and more resiliency for cleanup daemons

Null pointer exceptions thrown by daemon threads that clean requestors and handle agent changes can result in a high number inactive requestors. Additional logging has been out into place to help capture information to identify the offending data, and master daemons for requestors and for new agents have been made more resilient to any Null pointer exception thrown by the diagnostics code so they do not stop their work.

Resolved alert from disabled job scheduler

An urgent event alert was seen for scheduler which was disabled in production. This was traced to null pointer exceptions originating in the JobSchedulerLifeCycleManagerImpl call, and has been resolved by fixing a potential null dereference. Additional logging has also been introduced.

Updated SLA table clearance

After cases were resolved, SLA entries were not getting cleared from the SLA table. This caused the SLA table to have a huge number of Overall SLA and PendFlow details present for cases that were resolved or moved to the next state, which impacted case processing via SLA Agent. This was traced to SLA queue-items not being removed by delete-deferred as expected, and has been resolved by invalidating the deferred operations and scheduling item removal.

Added database query optimization and cache invalidation logging

Web nodes were going down during deployment due to failed health checks. This was traced to queries firing even when there was no rule change, and was due to unnecessary cache invalidations. An update has been made which will optimize and analyze declarative database queries and capture diagnostic details on any invalidation of the declarative reference information cache.

Added database query optimization and cache invalidation logging

Web nodes were going down during deployment due to failed health checks. This was traced to queries firing even when there was no rule change, and was due to unnecessary cache invalidations. An update has been made which will optimize and analyze declarative database queries and capture diagnostic details on any invalidation of the declarative reference information cache.