Memory management revealed - Questions and answers
During each developer webinar, Pegasystems dedicates additional staff to answer your questions via chat. This article summarizes questions from the Process Commander Memory Management Revealed session of October 3, 2007.
To view the webinar recording, see Webinar: Process Commander Memory Management Revealed (webinar no longer available).
Question | Answer |
|---|---|
| JVM Memory Settings | |
| Q-01: Regarding JVM Memory settings, is there a limit to the max heap? Why 768? | A-01: There are limits to the maximum heap size that can be set for each JVM. These limits vary based on JVM version, and also based on whether the JVM is 32-bit or 64-bit. Generally speaking however, using a heap larger than 2 Gigabytes is tricky from a tuning perspective and should only be used in special cases and with significant understanding of the JVM garbage collection routines. |
| Q-02: Regarding JVM Memory settings, How does the Sun JVM differ from other JVMs? | A-02: While the concepts of concurrent garbage collection and memory sizing are the same across JVM vendors, each vendor has their own JVM arguments and syntax. The basic JVM arguments for the Sun JVM match those of other vendors, but the more advanced arguments around concurrent collection and special tuning are unique to Sun. |
| Q-03: Regarding JVM Memory settings, should we use the same sizes for both 32 and 64 bit JVMs? | A-03: Yes. The same settings as are recommended for 32-bit JVMs should be used with 64-bit JVMs to start. As further tuning information is discovered for 64-bit JVMs this may change, however the initial settings provide a good starting point for tuning for either JVM architecture. |
| Q-04: What should be the JVM size settings difference between Dev and QA? Should it be same, or is there perhaps a parameter that can help us decide when moving from environment to environment? | A-04: This is an interesting question, and there is no clear, single answer. The system usage in development is often times very different than that in a test or QA system. Despite this difference though, I would suggest starting with the settings used in development, and then tuning based on garbage collection performance. |
| Differing JVMs | |
| Q-05: What is the main difference between the Sun JVM and IBM JVM? Why is the minimum Heap so different? | A-05: There are many differences between the two JVM architectures; however the primary difference in terms of garbage collection up until the IBM 1.5 JVM was the difference in the heap organization. The Sun JVM uses a generational model breaking the heap in to young, tenured, and permanent objects. And the IBM JVM used the heap as a single segment allocating objects based on a freelist. IBM has produced it's own version of the generational model in the 1.5 JVM however. Many other factors contribute to the differences, and these can be researched on the vendors respective web sites. |
| Q-06: Is one JVM recommended for Process Commander as best overall? | A-06: You should use the JVM type and version recommended for use with your application server software. The Sun and IBM JVMs are the most prevalent versions in the market place at this time. |
| Q-07: How do the settings vary in the Oracle JVM? Are they a closer match to IBM or Sun JVM configuration settings? | A-07: We have not studied the Oracle JVM settings in detail and cannot yet provide a good comparison between them and the competing JVM vendor options. |
| Q-08: Regarding the IBM JDK V1.5: Which algorithm is preferred for PRPC -- Concurrent Mark OR Gencon Garbage Collection? | A-08: This depends slightly on the type of work being performed by the application, but without specific knowledge the generational collector, which also uses the concurrent mark collector for the tenured area, is preferred. |
| Q-09: Process Commander seems to create many short-lived objects. On the Sun JVM, is it good to define the size of the Young as large as possible? Can the size of the Young exceed the size of Old? | A-09: Yes, PRPC does have a large number of short objects. It is a good idea to set the size of the young generation at a size larger than the JVM defaults. A recommended size would be 1/3 the size of your total heap. So, for example, if you are using a 1 GB heap size, set your young generation to 384 MBs. It is also beneficial to pin the minimum and maximum size of the young generation to the same value. So, given the above example, you might set the JVM options as follows:-XX:NewSize=384m -XX:MaxNewSize=384m -Xms1024m -Xmx1024m. |
| Q-10: Have you seen issues with compaction times on IBM 1.5 JDK and if so what was recommended to solve these? | A-10: Yes, we have seen compaction time issues on the IBM JVMs (1.4 and 1.5). Compaction indicates heavy fragmentation, and it is advantageous to reduce the minimum size of the heap and/or enable generational collection (if available in the IBM 1.5 JVM) or concurrent mark/sweep collection. |
| Q-11: So if we use JRockit which is similar to BEA to we have a concurrent mark sweep gc best practice settings? (Oops. JRockit is similar to IBM. Great, thanks for the response, that is good advice.) | A-11: The BEA JRockit JVM has different JVM arguments, but it has very similar concepts to that of the Sun and IBM JVMs. If you have a "Real Time" license with WebLogic SIP server you can use deterministic garbage collection, which should give you the lowest pause times possible. If you don't have the Real Time license, you can use concurrent garbage collection much like you can with both Sun and IBM to reduce pause times. |
| Q-12: Would Tomcat be the same as Sun in terms of settings? | A-12: The Tomcat application server does not specifically require the Sun JVM, although it is most frequently deployed with the Sun JVM. If you are using Tomcat with Sun the settings discussed in this webinar should provide a good starting point for your application. |
| Q-13: Would there be significant differences between running on WebLogic 8 vs. WebSphere 6.0? (4.2SP6) | A-13: No, they're shouldn't be. WebLogic will either use the Sun or JRockit JVM, and WebSphere usually uses the IBM JVM however. |
| Q-14: How do memory sizes change if we alter our configuration from 32 to 64 bit? | A-14: The values provided in the webinar should provide a good starting point for either JVM architecture. Special tuning information is not yet available (as of September 2007) for 64 bit JVMs. |
| Q-15: Can IBM tools be used for Sun JVM or it only for IBM JVM? | A-15: Yes, the IBM PMAT tool can be used to read Sun JVM GC output as well as HP JVM GC output. |
| Physical Memory | |
| Q-16: How much physical memory is needed? | A-16: You need enough physical memory for the entire JVM heap to fit into RAM, keeping in mind you'll need to have RAM left over for the operating system and other basic services on the server. |
| Q-17: Are there any chances of us getting issues because of a limited amount of physical RAM? | A-17: Yes. If you have limited RAM on your server, and the JVM heap is partially or fully in swap space, this will seriously impact the performance of your system. You may not receive OutOfMemory errors, but your performance will be very poor. |
| Q-18: What should be the RAM size for the recommended settings? | A-18: You need enough physical memory for the entire JVM heap to fit into RAM, keeping in mind you'll need to have RAM left over for the operating system and other basic services on the server. |
| Performance Analysis Tools | |
| Q-19: What is the tool you used to analyze and produce the graph shown in this presentation? | A-19: The IBM Pattern Modeling and Analysis Tool (PMAT) was used to produce the graphs of garbage collection data for the presentation. |
| Q-20: Can you provide some examples that may cause large memory requests? | A-20: Yes, large memory requests typically come from large database list requests or looping database open routines. They might also come from large file attachment functions, or from services that parse large files or messages. |
| Specific Memory Errors | |
Q-21: Sometimes, we get the following error when we have CPU Starvation and Full GC's: ThreadMonitor W WSVR0605W: Thread "WebContainer : 147" (000000e2) has been active for 606033 milliseconds and may be hung. There is/are 27 thread(s) in total in the server that may be hung. At that moment, the server hangs. Do Web Container threads get hung when we have Full GC's? | A-21: Yes, when full GCs take place all JVM threads are paused. Full GCs are also known as "stop-the-world" garbage collections, because no thread in the JVM can process until the full GC is complete. If you are having many full GCs, or successive full GCs, this is a serious performance problem and probably means that you are nearly out of memory in the JVM. Investigate this issue by graphing the GC statistics, using the IBM PMAT tool or another equivalent tool, and determine if you have large memory allocation problems or a potential leak. If you need help with this analysis contact Global Customer Support. |
Q-22: I some times get this error: Unable to synchronize on requestor H2EA7298977CA0DE5DFC3DC900E5B034F within 120 seconds: (thisThread = ExecuteThread: '11' for queue: 'weblogic.kernel.Default') (originally locked by = ExecuteThread: '18' for queue: 'webl. What is the reason for this error, and what is the 120 second limit I see here? | A-22: This message indicates that the requestor in question was locked performing some action (perhaps a large report) for over 2 minutes (120 seconds). This is a warning message, not an error message, and does not necessarily indicate a problem in and of itself. |
Q-23: We have three RuleSets. A (Base), B and C are specific. We have two access groups. Access group P have access to RuleSets B and A. Access group Q have access to RuleSets C and A. We have certain rules defined in Base RuleSet A which can be accessed by requestors of both access groups. Does the JVM maintain a single set of base rule class files for all the access groups or keep a separate copy of those rules for each access group? | A-23: The JVM does not manage RuleSet lists or load rule classes, this is done entirely by the PRPC engine. Your question deals with the areas of First Use Assembly (FUA) and caching in the PRPC engine. While this is a fairly complicated subject, here is a short answer to your question. The PRPC engine creates a hash key (a 32 character alphanumeric string) for each unique RuleSet list in the system. This hash key can be found on the end of each generated, compiled, and loaded class. In your example, there are two unique RuleSet lists. One for access group P, and another for access group Q. The PRPC engine will generate different versions of the rules in RuleSet A for the two different RuleSet lists that correspond to access groups P and Q. This means that there will be two generated classes for the same rule in RuleSet A based on the different RuleSet lists. Both versions will be loaded into the JVM by the PRPC runtime classloader and will be stored in the area of the JVM heap designed for class objects. |
| Heap Analysis | |
| Q-24: How do we perform a heap dump? | A-24: Heap dumps are generally requested by issuing a kill -3 command to the JVM process, however this can vary by operating system platform and JVM type and version. |
| Q-25: RE: Insufficient Space in Heap: Can these 2 options (enable the interaction byte threshold alert, and review the Alert log file) be done in v4.2 or only in v5.x? | A-25: Both the Interaction Byte Threshold alert and the Alert log file are available in the V4.2 series starting in Service Pack 5. |
| Q-26: How do you trigger a heap dump if the JVM is running as a Windows service? | A-26: You cannot trigger a heap dump for a Windows service with a simple command or keystroke. For a Windows service, you must trigger a heap dump either programmatically, for example using the JVMPI, or using a real time monitoring tool such as JConsole or YourKit. |
| Q-27: Can you set up a JVM to have a max Heap setting of more than 1 gig? Is it advised, or do we impact the other memory needs by the JVM process? | A-27: You can setup a JVM to have a heap larger than 1 GB. You wouldn't typically start with a heap larger than 1 GB though. The larger you make the heap, the longer it takes for the garbage collector to mark every live object, and sweep away the garbage. You typically only expand past 1 GB when doing tuning for a specific application and it's specific memory requirements. |
| Byte Threshold Alert | |
| Q-28: Looking at the byte threshold alert log, is the database time was also given by threshold alert? | A-28: The Interaction Byte Threshold alert, which is printed in the regular PegaRULES logfile in most product versions, and in both the PegaRULES logfile and the separate Alert logfile in some versions, does not record the time it took for the query or queries to complete. This alert only records the amount of memory requested past the warning or error level, the requestor that requested it, the username for the requestor (if any), and a stack trace of their activity that caused the alert. However, most Interaction Byte Threshold alerts will also trigger a Database Operation alert, and thus you can find two alerts for the same problematic request. |
| Performance Troubleshooting (JVMs, Garbage Collection, and PRPC Configuration) | |
| Q-29: Is there any way to know if there is a memory leak by monitoring the Garbage Collection? | A-29: Yes. You can identify a memory leak by tracking the garbage collection statistics over time. If the memory usage increases steadily but never decreases even after issuing a full GC to the JVM then a memory leak is occurring. Memory leaks are not a very common occurrence generally though. There can be many causes for memory leaks as well, including the JVM itself, the application configuration, the application code, or the application server code. |
| Q-30: What do we do when we have Out of Memory problems? Break down your queries to return back smaller chunks of data or what? Or do you expect us to call Pega Support to get help in determining what to do first? | A-30: When you have out of memory problems the first thing to do is get a picture of how the JVM heap is being used to help determine if they are being caused by large allocation requests, a memory leak, or very high general heap usage. The out of memory error message itself may also indicate the cause if there is a special clause in it detailing the problematic area. For example, if the Sun JVM runs out of space in the permanent generation the out of memory message will include a statement indicating that the PermGen is out of space. |
| Q-31: While doing Obj-List or Obj-Open, it takes a large amount of memory and gets timed out and produces memory errors. How do we avoid this error? | A-31: You should first review the functional requirements of your process and see if you can reduce the resources being consumed. Does your Obj-List need to return all the data it is returning, or could it just retrieve a few columns? If you are performing multiple Obj-Open call in a loop, evaluate the necessity to return this data. If you’re concerned about the memory allocation necessary for a single Obj-Open, evaluate the information that is being stored in the object and that is making it so large. After you've reviewed the process functionally and streamlined it as much as possible, then look at tuning the JVM parameters. |
| Q-32: We have seen a lot of small object creations in PRPC under load, with compaction times that take multiple seconds to complete. Is this consistent with other folks' experience? | A-32: Yes, the PRPC application creates a number of small objects during normal processing, and heap compaction can take multiple seconds to complete - especially in a 1 GB heap. You can address this issue by setting the minimum and maximum heap size for the IBM JVM as recommended in this article and by enabling either concurrent mark/sweep or generational garbage collection (if available). |
Previous topic JVM properties for IBM WebSphere Application Server Next topic Support Play: Analyzing OutOfMemory Exceptions