Troubleshooting: Locks on multiplexer (Muxer) socket threads causing intermittent stalling of user nodes and non-responsive application server (WebLogic)

Updated on September 10, 2021

Symptom

With a Process Commander deployment running on the IBM AIX operating system and using the Oracle WebLogic application server, certain versions of the Oracle Service Bus and the WebLogic Server do not respond. User nodes managed by the WebLogic Server stall intermittently several times a day. The application server freezes and does not respond to any requests. The WebLogic administrative console is non-responsive.

This problem has been reported for the following Oracle components:

Oracle Service Bus Versions 2.6.1RP to 10gR3
Oracle WebLogic Server Versions 9.2 to 10.3

The problem is caused by locks on multiplexer (Muxer) socket threads. Correct this problem by applying patches that are available for Oracle WebLogic 9.2, 10.0, and 10.3 Maintenance Packs (MPs).

Explanation

The weblogic.socket.Muxer module manages the server’s existing socket connections. Socket Reader Threads accept the incoming request from the Listen Thread Queue and put it on the Execute Thread Queue. First the Muxer determines which sockets have incoming requests waiting to be processed. Then it reads enough data to determine the protocol and dispatches the socket to an appropriate runtime layer based on the protocol. In the runtime layer, the Muxer determines which Execute Thread Queue should be used and delegates the request accordingly.

WebLogic 8.1 provided three socket reader threads by default. WebLogic 9 and later versions allocate 33 percent of server threads to be socket readers. While one socket reader polls, the others process requests.

Because of an application server defect, the socket Muxer threads are reported as stuck; they are all locked on the same object. The locks are transitory because they move from thread to thread: A thread gains an exclusive lock on an object to perform some action, after which it releases the lock to allow the next thread to lock the object. For complete information on WebLogic Muxers, including the two types (Java Muxer, Native Muxer), refer to the WebLogic Server Performance and Tuning guide that is cited as a reference.