Skip to main content


         This documentation site is for previous versions. Visit our new documentation site for current releases.      
 

Security Advisory: Apache Log4j JNDI Zero Day Vulnerability

Updated on July 8, 2022

+

This content applies to On-premises, Client-managed cloud and Pega Cloud environments

Important: This is the primary authoritative source for information related to the status of Pega's products and services.
Important: Pega is closely following this volatile situation.  We continue to patch our products and services as new vulnerabilities in Log4j are identified, and will continue to post updates in this Advisory as they occur.  Log4j 2.17.1 was recently released, and Pega has now provided hotfixes for that Log4j version.

 

Pega Hotfix Details

The following vulnerabilities have been discovered in Apache Log4j:

CVE

Fixed in Apache Log4j version:

CVE-2021-44228

2.15

CVE-2021-45046

2.16

CVE-2021-45105

2.17

CVE-2021-44832

2.17.1

See below for full details on these vulnerabilities.

Pega hotfixes for our Pega Platform  Infinity  software (8.x versions) which leverage the Apache Log4j version 2.17.1 are available.   See details at Pega Security Advisory – Apache Log4j 2.17.1 Vulnerability Hotfixes.

Important: The most recent Pega Platform patch release for your version is a required prerequisite for the 2.17.1 hotfixes.  Clients who are not on the latest patch release should apply the 2.17 hotfixes, which are available in the Pega Security Advisory for Apache Log4j 2.17 Vulnerability Hotfixes.

This Advisory also includes hotfixes for the Pega 7 versions of Pega Platform (7.3, 7.3.1, and 7.4) for Apache Log4j version 2.17.1.

Pega hotfixes for our Stream Service (8.x versions) which leverage the Apache Log4j version 2.17.1 are available.  See details at Stream Security Advisory – Apache Log4j 2.17.1 Vulnerability Hotfixes.

Important: The most recent Pega Platform patch release for your version is a required prerequisite for the 2.17.1 hotfixes.  Clients who are not on the latest patch release should apply the 2.17 hotfixes, which are available in the Stream Security Advisory for Apache Log4j 2.17 Vulnerability Hotfixes.

Clients can submit a hotfix request for any of these types of hotfixes by using My Support Portal.

Overview

A zero-day vulnerability was identified in the Apache Log4j logging software on December 10 (CVE-2021-44228)A related Log4j vulnerability was identified on December 14 (CVE-2021-45046), a third was identified on December 17 (CVE-2021-45105), and a fourth was identified on December 29 (CVE-2021-44832).  These vulnerabilities could allow malicious actors to take control of organizational networks using Log4j.  The Log4j software is ubiquitously used by most organizations around the world.  

Pega software can use the Log4j component in two places:  the Pega Platform software and Pega's Stream service.

  • Log4j is embedded within our Pega Platform product to allow clients to track and record platform activity. 
  • The Pega Stream service enables the asynchronous flow of data between processes in the Pega Platform. The Stream service is a multi-node component that is based on Apache Kafka.

Click here to scroll down to information about:

Pega Cloud Clients

Pega On-Premises or Client-managed Cloud clients

Additional Services:  WFI, Pega Chat, Co-Browse, Digital Messaging, Voice AI, and others

Testing

How Pega is mitigating these vulnerabilities for Pega Cloud clients

To mitigate these vulnerabilities, Pega has applied the following defense-in-depth approach to our Pega Cloud clients.

For Pega Platform

  • We applied security controls at the network level using AWS Web Application Firewall (WAF) on the afternoon of Friday, Dec. 10. 
  • We have disabled the vulnerable portion of Apache Log4j (JNDI) from Pega Platform, followed by a rolling restart, for Pega Cloud clients as of Sunday, December 12, 2021, at 6 pm ET. Through initial testing, we believe this action will not have an adverse impact on the normal use of Pega Platform. 
  • We have scanned all Pega Cloud sites with the Tenable.io Web Application Scanner for Log4j vulnerabilities, and verified that mitigations are in place.
  • The Hazelcast functionality is being updated to mitigate the Log4j vulnerabilities.

For the Stream service, using Apache Kafka:

  • Most Pega Cloud environments, beginning with Pega Platform version 8.4.0, use AWS-managed Apache Kafka, which does not contain the vulnerable log4j libraries.
  • The Kafka service is not directly accessible from the Internet, and is only available as an internal service to the Pega Platform environments on Pega Cloud.
  • As an additional failsafe, we have disabled the vulnerable JNDI portion of Apache Log4j from the Apache Kafka distribution for Pega Cloud clients.

Pega Cloud Web Application Firewall (WAF) rules

Pega Cloud uses the AWS WAF rules.  AWS does not publish the details of these rules, but more information is available at the AWS Managed Rules changelog.

Trend Micro Intrusion Prevention System (IPS)

Pega has applied Trend Micro Intrusion Prevention System (IPS) rules at the host-based agent level on Saturday, Dec. 11, as soon as it became available, to further block malicious attempts.   For details, see their article Security Alert:  Apache Log4j “Log4Shell” Remote Code Execution 0-Day Vulnerability.

For Pega on-premises and self-managed cloud customers 

This vulnerability can affect Pega clients running on-premises or self-managed cloud clients using Pega Platform version 7.3.x - 8.6.x.

Note: For Pega Platform versions prior to 7.3.x, see Security Advisory:  Apache Log4j 1.2 JMSAppender vulnerability

Versions 8.3.x - 8.6.x of the Pega Platform include the Apache Kafka distribution that contains the vulnerable Log4j JNDI libraries.

In addition, it is possible that vendor platforms which clients are using with their Pega software (such as WebSphere or WebLogic) are also affected by the Log4J JNDI vulnerability.  Pega strongly recommends that clients check with their vendors for any required mitigations, and work with their IT and security staff to confirm that other (non-Pega) products are not inadvertently introducing vulnerabilities.

For Pega Platform

Pega hotfixes are available for our Pega Platform (8.x versions) for the following vulnerabilities:

CVE

Fixed in Apache Log4j version:

CVE-2021-44228

2.15

CVE-2021-45046

2.16

CVE-2021-45105

2.17

CVE-2021-448322.17.1

Clients can request these hotfixes through My Support Portal.  Please see details at  Pega Security Advisory – Apache Log4j 2.17.1 Vulnerability Hotfixes.

Important: The most recent Pega Platform patch release for your version is a required prerequisite for the 2.17.1 hotfixes.  Clients who are not on the latest patch release should apply the 2.17 hotfixes, which are available in the Pega Security Advisory for Apache Log4j 2.17 Vulnerability Hotfixes.

NOTES:

  • CVE-2021-45105 describes a vulnerability that could lead to Distributed Denial of Service (DDoS) attacks.  Applications which are not Internet-facing should be at less risk for this vulnerability.  To limit risk from this vulnerability, Pega strongly recommends that clients avoid exposing unnecessary parts of their systems to the Internet, and protect their Internet-facing features by using security functionality such as WAF or IPS.
  • CVE-2021-44832 can only be exploited if the adversary has already gained access to a client’s system through another means (which would indicate a much larger security issue for the organization).  Therefore, these hotfixes (based on Log4j 2.17.1) are only available for the latest patch release of each Pega Platform version, and clients must evaluate whether they require this fix.  Clients who are on prior patch versions must upgrade to the latest patch version in order to receive and apply this hotfix.

Pega Platform hotfixes for version 7.3, 7.3.1, and 7.4 are available for the above-listed vulnerabilities, and are included in Pega Security Advisory – Apache Log4j 2.17.1 Vulnerability Hotfixes.

NOTE:  If clients apply a hotfix for a particular Pega Platform version, and then later update their systems to a newer Platform version, they must apply all critical hotfixes to this newer version.

The below instructions describe how clients can manually disable the vulnerable JNDI portion of Apache Log4j functionality. We urge all Pega on-premises and self-managed cloud clients to apply the correct version of the latest hotfix to their environments, or take the below manual action immediately on their Pega environments.   In addition, clients should also follow the security recommendations and guidelines from your organization. 

Manual Removal of JNDILookup.class

The following steps remove the JNDILookup.class from the pr_engineclasses table. 

Note: This action will not have an adverse impact on the normal use of Pega Platform unless clients have applied the prior Pega Platform hotfixes based on Log4j 2.16.  Clients who installed those prior hotfixes and then manually deleted the JNDILookup.class can run into an issue where log files may be written to the wrong location.  All clients should apply the Pega Platform hotfixes which are based on Log4j 2.17 (linked above).  The Log4j 2.17 hotfixes will fix this issue, in addition to providing more up-to-date protection from vulnerabilities.

1.  Enter the following SQL statement, to confirm one or more of those classes are present in the DB.  You should see at least one record returned.

select pzjar,pzpackage,pzclass,pzlastmodified,pzmoduleversion,pzcodesetversion,pzpatchdate from <Rules Schema Name>.pr_engineclasses where pzclass = 'JndiLookup.class' and pzpackage = 'org/apache/logging/log4j/core/lookup';

2.  Back up the pr_engineclasses table.

3.  Delete the JNDILookup.class.

delete from <Rules Schema Name>.pr_engineclasses where pzclass = 'JndiLookup.class' and pzpackage = 'org/apache/logging/log4j/core/lookup';

4.  Do a full cluster restart.  NOTE:  a Docker restart is not sufficient.  A rolling restart is fine.

5.  Run the select statement again to confirm that the class is removed.  No results should be returned.

select pzjar,pzpackage,pzclass,pzlastmodified,pzmoduleversion,pzcodesetversion,pzpatchdate from <Rules Schema Name>.pr_engineclasses where pzclass = 'JndiLookup.class' and pzpackage = 'org/apache/logging/log4j/core/lookup';

 

For the Stream service, using Apache Kafka

The Apache Kafka service should not be directly accessible from the Internet, and should only be available as an internal service accessible to the Pega Platform environments via a private network.  Pega strongly recommends ensuring that this measure is in place.

Pega hotfixes are available for our Stream Service (8.x versions) for the following vulnerabilities:

CVE

Fixed in Apache Log4j version:

CVE-2021-44228

2.15

CVE-2021-45046

2.16

CVE-2021-45105

2.17

CVE-2021-44832

2.17.1

Clients can request these hotfixes through My Support Portal.  Please see details at Stream Security Advisory – Apache Log4j 2.17.1 Vulnerability Hotfixes.

Important: The most recent Pega Platform patch release for your version is a required prerequisite for the 2.17.1 hotfixes.  Clients who are not on the latest patch release should apply the 2.17 hotfixes, which are available in the Stream Security Advisory for Apache Log4j 2.17 Vulnerability Hotfixes.

We urge all Pega on-premises and self-managed cloud clients to apply the appropriate latest hotfix to their environments, or take the below manual action immediately on their Pega environments.   

If an immediate remediation is required, follow the below procedure on your Pega Platform Stream nodes to replace the vulnerable JNDI libraries with the upgraded files from Apache Kafka. 

The Stream nodes are configured with the -DNodeType=Stream argument.  Important:  You must follow this process for each Stream node. 

  1. Log in to your application server using the command line.
  2. Stop the application server process.
  3. Navigate to java_ee_server_root/kafka-<kafka-version>/libs, where <kafka-version> is either 1.1.0.5 or 1.1.0.4
  4. Delete the following files:
  • log4j-api-2.11.1.jar
  • log4j-core-2.11.1.jar
  • log4j-slf4j-impl-2.11.1.jar
  1. Download the upgraded log4j files from Apache.  These are the files that Apache has provided to mitigate this vulnerability:
  • log4j-api-2.16.0.jar
  • log4j-core-2.16.0.jar
  • log4j-slf4j-impl-2.16.0.jar
  1. Copy these files to java_ee_server_root/kafka-<kafka_version>/libs
  2. Verify that java_ee_server_root/kafka-<kafka_version>/libs contains only the following jars related to log4j:
  • log4j-api-2.16.0.jar
  • log4j-core-2.16.0.jar
  • log4j-slf4j-impl-2.16.0.jar
  • (possibly) kafka-log4j-appender-1.1.0.jar

NOTE:   The 2.16.0 versions of these files are the latest version that Pega has tested for the above manual mitigation steps.  Since Pega provides a hotfix for the 2.17 versions of Log4j, we have not tested the above manual stepsfor that version. 

The verification can be done by running the following command:

 $ ls | grep 'log4j'
 log4j-api-2.16.0.jar
 log4j-core-2.16.0.jar
 log4j-slf4j-impl-2.16.0.jar
 kafka-log4j-appender-1.1.0.jar

  1. Restart the application server.

 

Additional Pega Services

Pega Workforce Intelligence (WFI)

Servers vulnerable to the Log4J JNDI issue (CVE-2021-44228) were disabled as of Saturday, December 11, and have been hotfixed.  Servers vulnerable to the additional Log4J vulnerability (CVE-2021-45046) were disabled as of Wednesday, December 15, and have been hotfixed. 

A hotfix to address CVE-2021-45105  (based on Apache Log4j 2.17)  has been applied to all WFI client environments.  In addition, AWS WAF has been deployed across the AWS account for all client environments.

A new Apache Log4j vulnerability (CVE-2022-0070)has been reported by Amazon, relating to a hotpatch feature released by Amazon in December 2021.  Pega has determined that this vulnerability could potentially impact Workforce Intelligence.  To address this high vulnerability, Workforce Intelligence instances have been hotfixed (version 8.7.2 HF1). 

A hotfix to address the Log4J JNDI issue (CVE-2021-44228) and the additional Log4J vulnerability (CVE-2021-45046) for version 8.7.3 is being applied to all WFI client environments (version 8.7.3 HF1).  NOTE: During this process, data will be collected on client systems and will be available when the servers are back online.  

Pega Chat and Pega Co-Browse

Pega has completed our analysis of Pega Chat and Co-Browse, and has confirmed that Log4J is not included in these products. 

Pega Digital Messaging

The Log4j JNDI vulnerabilities in this service are addressed through our Pega Platform hotfixes.

Pega Voice AI

External-facing servers vulnerable to the Log4J JNDI issue were disabled as of Monday, December 13; they are being hotfixed before being re-enabled. 

BIX

The Log4j JNDI vulnerabilities in this service are addressed through our Pega Platform hotfixes.

Document Processing Service

Servers vulnerable to the Log4J JNDI issue (CVE-2021-44228) and the additional Log4J vulnerability (CVE-2021-45046) have been hotfixed to mitigate this risk.

PDC

The Log4j JNDI vulnerabilities in this service are addressed through our Pega Platform hotfixes.

Pega Platform Personal Edition

Pega has published an updated version of the Personal Edition which mitigates the following vulnerabilities:

This updated version of the Personal Edition has been released with the Pega Platform 8.7 release.  Pega strongly recommends that any client using Personal Edition shut it down until they can download this mitigated version, and then replace their existing Personal Edition with the mitigated version.

Robotics

Robotic Process Automation (Robotics) is made up of several products. 

  • Robotic Runtime, Robot Studio, and Sync Server do not use Log4j.
  • Robot Manager is a Pega Platform application, so the Log4j JNDI vulnerabilities in this service are addressed through our Pega Platform hotfixes.

Pega Search Functionality

Pega Search functionality uses a product called Elasticsearch.  Clients most commonly access this functionality one of several ways:

1.  Connect to Elasticsearch using the Pega Search and Reporting Service (SRS)  (Pega Platform version 8.6 and later)

The SRS is built on the AWS implementation of Elasticsearch, which uses Elasticsearch version 7.10.  According to the Elasticsearch website:  “Elasticsearch 6 and 7 are not susceptible to remote code execution with this vulnerability due to our use of the Java Security Manager.”  NOTE:  Some vulnerability scanners may continue to flag Elasticsearch in association with this vulnerability based solely on the Log4j version; but as stated by Elasticsearch, the vulnerability is mitigated by Java Security Manager. 

https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476

2.  Embedded in the Pega application

Legacy setups which have Elasticsearch embedded in Pega Platform use the Log4j component that is part of Pega Platform.  Therefore, applying the Pega Platform hotfixes will mitigate the Log4j vulnerabilities in this situation.

3.  Connecting to a node running an Elasticsearch Docker image

For clients running a client-managed cloud environment, Pegasystems provides multiple Docker images, including separate Docker images for Pega Platform, Pega Search, and other services.  Pegasystems Search is based on the Elasticsearch product.  We have updated our Pegasystems Search Docker images to incorporate a JVM argument, as recommended by Elasticsearch:

https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476

To mitigate the risk for Elasticsearch, clients need to install only the updated mitigated Docker image for Search (not the Pega Platform Docker image), available from the Pegasystems Search page. 

For versions 8.2 – 8.6, Pega has updated both the main major/minor version of Pegasystems Search (example:  “8.5”) and the latest patch version of that major/minor version (example “8.5.5”).  Either of these Search Docker images are applicable to all Pega patch versions within the minor version (example:  both “8.5” and “8.5.5” apply to 8.5, 8.5.1, 8.5.2, 8.5.3, 8.5.4, and 8.5.5). 

Pega strongly recommends that clients running Pega software in a client-managed cloud download the appropriate version of the Search image and install it.

NOTE:   These Docker images include Elasticsearch version 5.6.14, as the 7.x versions are not backwards compatible with the older versions.   The JVM argument fix is the solution recommended by Elasticsearch.  A scan of the software will still report the older Log4j file, but its vulnerability is mitigated by the recommended fix.  Pega stands by the Elasticsearch recommendation to use this JVM fix. 

4.  Connecting to Elasticsearch in client-server mode

For much older versions of Pega Platform software (version 8.1), Pega provided directions for clients to administer and manage their own external Elasticsearch cluster

This setup was only supported with Elasticsearch version 5.6.9, which uses the vulnerable Log4j components.  If clients wish to use this setup, it is recommended that they remediate by performing the following steps, as recommended by Elasticsearch in their Main Announcement.

1.  Upgrade to Elasticsearch version 5.6.14

2.  Set the JVM option “-Dlog4j2.formatMsgNoLookups=true”

3.  Restart each node of the Elasticsearch cluster

Pega does not recommend that clients continue to use Elasticsearch version 5.6.9.  If clients wish to stay with that version, however, they can still remediate by removing JndiLookup from the log4j-core JAR file,​ as described by Elasticsearch in their Elasticsearch 5.x and 6.x article.

Deployment Manager

Deployment Manager does not contain a Log4j.jar file.  For this utility, the Log4j vulnerabilities listed in this Advisory are addressed through our Pega Platform hotfixes.

PRPCServiceUtils

Pega has developed a new version of the PRPCServiceUtils component, which includes Apache Log4j 2.17.1.  This will mitigate the following vulnerabilities:

This new version has been published to the Pega MarketPlace here.  Pega strongly recommends that clients download this mitigated version, and replace their existing PRPCServiceUtils with the mitigated version.

prpcUtils

prpcUtils is used to install Pega Platform, Pega applications, and (sometimes) hotfixes.  (Hotfix Manager and Designer Studio do not use this utility.)  It is not constantly running, but only used when doing these installations or updates.   To mitigate risk, clients should use other Pega-provided utilities such as Hotfix Manager or Deployment Manager.

Hazelcast

Hazelcast may be vulnerable if it is deployed in client-server mode.  Pega is providing separate hotfixes to address Hazelcast vulnerabilities.  These hotfixes are still in development, and will be provided when available.

Tracerviewer

The Pega-TracerViewer tool presents and summarizes Tracer output data.  This tool has been updated to use Apache Log4j version 2.17.1.  To get the mitigated version, download version 3.4.0 or later.

For details on downloading and using Pega-Tracerviewer, see Offline debugging by using Pega-TracerViewer. 

Testing for the Log4j vulnerabilities

Pega has scanned all Pega Cloud client sites with the Tenable.io Web Application Scanner for Log4j vulnerabilities, and verified that mitigations are in place. 

To determine that the hotfixes have been installed properly, use the System Scanner.

Clients who have questions about validating that the vulnerability has been addressed by application of the Pega hotfixes should work with their security organization, and have their security teams use their preferred scanning tools to test.

 

 

 

Document Revisions

July 8, 2022 – 11:35 am EDT  |  Added information about additional hotfix to WFI.

April 21, 2022 – 4:15 pm EST  |  Added information about new Amazon vulnerability which may affect WFI.

January 27, 2022 – 7:15 pm EST  |  Moved hotfix information to top, and added information about 2.17.1 hotfixes.  Added information about: the Pega Platform Personal Edition, Pega Search functionality, Deployment Manager, PRPCServiceUtils, and TracerViewer. 

January 13, 2022 – 5:45 pm EST  |  Updated date in note about Pega Platform 2.17.1 hotfixes.

January 10, 2022 – 5:45 pm EST  |  Updated information about Pega Search Functionality.

January 6, 2022 – 5:45 pm EST  |  Added note about Pega Platform 2.17.1 hotfixes.

January 4, 2022 – 7:15 pm EST  |  Added note about not doing the Pega Platform manual steps if you’ve already applied the 2.16 hotfix.

December 29, 2021 – 3:30 pm EST  |  Log4j 2.17 hotfixes are now available for Pega Platform, both the 8.x versions, as well as 7.3, 7.3.1, and 7.4.  Pega is investigating Log4j version 2.17.1. 

December 28, 2021 – 2:00 pm EST  |  The Log4j 2.17 hotfix is being applied to WFI client environments.

December 27, 2021 - 3:00 pm EST  |   The Pega Platform hotfix for 7.3.1 is now available.   Added information about Pegasystems Search Docker Images and prpcUtils.

December 23, 2021 – 2:00 pm EST  |  Added note about CVE-2021-45105 being a DDOS vulnerability, and that clients should avoid unnecessarily exposing their systems to the Internet.

December 22, 2021 – 5:30 pm EST  |  Added information about Hazelcast, and a note about Pega Platform 7.3.x and 7.4.x hotfixes.

December 21, 2021 – 5:45pm EST  |  Added information about Pega Cloud clients being scanned with Tenable.io, and about WAF rules and Trend Micro rules.  Links are provided to the new hotfixes are available for the Stream Service; all clients should apply those.  Testing information has been added.

December 20, 2021 – 5:15 pm EST  | Added link to Apache Log4j JMSAppender advisory, and a statement that we are creating hotfixes based on Log4j 2.17 for both Pega Platform and Stream service, and hotfixing WFI.  Added information about Robotics and PRPCServiceUtils.

December 17, 2021 – 3:30 pm EST  | Added link to Stream Services (Kafka) hotfixes.  Also updated information about PDC, Personal Edition, and Docker images.  Added note about clients who update their systems to a later patch must also apply the hotfix that matches that later version.

December 16, 2021 – 5:45 pm EST  |  Added link to Pega Platform hotfixes.  Also updated information about WFI and Chat/Co-Browse; added information about BIX and Document Processing Service.

December 15, 2021 – 12:30 pm EST | Added information about CVE-2021-45046, and updated the Kafka section to reflect that clients should download version 2.16 of the Apache files

December 14, 2021 – 6:24 pm EST  |  Included information about Additional Pega Services, and added links in the Overview for easier scrolling

December 13, 2021 – 7:55 pm EST  |  Added information about Kafka remediation and Hotfixes.

December 11, 2021 – 5:23 pm EST  |  Published article.

 

 

  • Previous topic Security Advisory: Apache Log4j 1.2 JMSAppender vulnerability
  • Next topic Security Advisory: Spring Framework Vulnerability

Have a question? Get answers now.

Visit the Support Center to ask questions, engage in discussions, share ideas, and help others.

Did you find this content helpful?

Want to help us improve this content?

We'd prefer it if you saw us at our best.

Pega.com is not optimized for Internet Explorer. For the optimal experience, please use:

Close Deprecation Notice
Contact us