Escenic 4.3-x Publication Bootstrapping Configurations

Bootstrapping is controlled with the following to configuration files

• ECE_LOCALCONFIG/neo/io/content/InitialBootstrapper.properties
• features (Publication resource - /escenic/features)

ECE_LOCALCONFIG/neo/io/content/InitialBootstrapper.properties

# The default dpeth to try to probe when going through the section tree. By default, a
# publication's top sections along with its children are probed, i.e. the depth is set to 2.
# Setting this property has effect when the bootstrapOnStratup is set to the keyword true
# (through features for example)
depth=3

# The number of seconds that the InitialBootstrapper should wait before trying to load the
# publications. Default value is 60
secondsToWait=120

# The number of seconds to try retrieving the sections and articles in the bootstrap phase.
# By default, if a publication has not finished bootstraping within 30 seconds, the boostrap
# for the publication in question would fail
timeoutSeconds=120

features (Publication resource)
#######################################################################
# Initial Bootstrapper Configurations
#######################################################################
# Can be set to make the InitialBoottrapper run through the publication
# when the server starts
# Example:
# bootstrapOnStartup=ece_frontpage,main,news,sports,football
# bootstrapOnStartup=true
# when set to true the depth property of the InitialBootstrapper
# component is used
bootstrapOnStartup=true
# The initial bootstrapper will by default try to connect to the
# publication's URL.  On a multi-server installation, this will probably
# be a HTTP server spraying the requests to more application servers.
# To make the initial bootstrapper connect to the bootstrapper,
# add a line which specifies the "localUrl" property.  All connections to
# the template system
# Example:
# local.url=http://localhost:8080//
local.url=http://localhost:8080/myPub/

Notes

Once Bootstrapping is complete, the property "bootstrapped" in http:///escenic-admin/browser/neo/io/content/InitialBootstrapper has value "true". The publication is not accessible until bootstrapping is complete.

You can also force mark complete the bootstrapping process by setting the "bootstrapped" property to "true" here: http:///escenic-admin/browser/neo/io/content/InitialBootstrapper?property=bootstrapped

Changing logging level of the Bootstrapping component
trace.properties (or relevant log4j properties)
log4j.category.neo.xredsys.config.InitialBootstrapper=DEBUG, Appender

Tomcat 5.5.27 | Memory Leak In Escenic CMS Based Web Application

Problem Statement
Memory leak in web application running on Tomcat

System Information

Tomcat 5.5.27

Apache HTTPD 2.2.9

Java HotSpot(TM) Server VM (build 1.5.0_14-b03, mixed mode)

Mod_proxy_http connector

Solaris 10 x86

Tomcat JVM heap settings: min:2GB and max:2GB

Observations and Investigation Done

Under load, the web application slowly runs out of heap space. The heap utilization graph suggests a memory leak type pattern.

Tomcat JVM Heap Utilization Pattern

Heap dumps were gathered to determine the root cause and observations are as below from the dumps:

1. HashMap entries from the below object reference tree seem to consume over 80% of the used tenured generation space.

org/apache/jasper/compiler/JspRuntimeContext

↓

Java/util/Collections$SynchronizedMap

↓

Java/util/HashMap

↓

Java/util/HashMap$Entry 

2. The number of objects referenced by each HashMap entry vary between 2-3
   1. These are either a String and JspServletWrapper object

or

2. A String, JspServletWrapper and another HashMap Entry. This call reference tree of HashMap entries referenced by another HashMap entry can repeat to a depth of approximately 8-10 nodes

The above is indicated in an object reference tree obtained after analyzing the Heap dump as shown below

Snapshot from IBM Heap Analyzer - Object Call Reference Tree

4. The maximum percentage of the memory occupied by the HashMap entry object is by a character array that seems like the JSP servlet response. HTML response (tags with content) can be seen in the character array.
5. Heap dumps gathered during different times and after a Full GC always indicate 100 entries. The number of entries does not grow. However the size of these Hash Map entries grows with time under load. After the first Full GC the size of this HashMap was observed to be around 350MB. The next heap dump obtained after the fourth full GC indicated that the size of this Hash Map doubled to around 730MB. The number of entries under this HashMap however remained constant. I use the words "directly under" this HashMap since I am not sure yet if the number of entries within the first level entries increased. A snapshot can be seen below
   

Comparison of size of HashMap after first and fourth full GC cycles

Questions

1. I am stuck at this point and have run out of ideas on how to get to the root cause of this issue. Do you have any ideas/suggestions to help identify the root cause?
2. How do I  confirm if this is a definite leak. How do I determine from the 2 heap dumps I have that objects are persisting in the heap and not getting collected? The address of objects can change across GC cycles. So what approach should be used to determine if an object that was seen in the first dump also exists in the subsequent dump?
3. I googled and found the following interesting links
   1. http://www.mail-archive.com/dev@tomcat.apache.org/msg03395.html Is there any know issues, configuration that causes a memory leak by caching of servlet responses in the container and not flushing the cached objects?
   2. http://mail-archives.apache.org/mod_mbox/tomcat-users/200303.mbox/%3C000c01c2ee2c$41c079a0$03b696c0@garethdqw0t9if%3E We use a lot of JSTL in our web application. Are there any known issues around memory issues as indicated on the link?

Resolution
The objects holding references to the character arrays that ultimately consume all the memory are of type org.apache.jasper.runtime.BodyContentImpl as indicated in the object reference tree below:

BodyContentImpl holding references to memory consuming character arrays

There was a bug reported in Tomcat 5.5.9 which says 

The problem is that this huge array never gets reset due to the object pooling 
implementation in Jasper (JspFactoryImpl maintains a pool of PageContextImpl 
objects. Each PageContextImpl object maintains an array of BodyContentImpl 
objects), so the memory it consumed is never returned to the heap.

 https://issues.apache.org/bugzilla/show_bug.cgi?id=37793

We do not use the suggest JVM argument in our Tomcat 5.5.27 JVM

-Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true

When the tests were repeated using the JVM argument mentioned above, the problem was resolved. A healthy heap utilization graph was obtained as indicate below

Heap Utilization Post Using the Recommended JVM Argument

There was no residual garbage after Full GC cycles as seen before this new JVM argument.


Additional Configurations Added In The Process

• Development mode was turned "off" for Jasper
• Trim white spaces option what turned "on" for Jasper

These are recommended production settings for tomcat as mentioned here - http://tomcat.apache.org/tomcat-5.5-doc/jasper-howto.html

Memcached Telnet Interface

What is memcached?

memcached is a popular distributed memory object caching system. For details refer this link

Connecting to the telnet interface

To connect to the interface you need the IP address and port on which the memcached server is running.

Describe below is a method to determine this information on a solaris environment on which a memcached service is already running

1. Determine the process ID of the memcached process (/service)

bash# ps -eaf | grep mem

    root  6917 25125   0 16:39:40 pts/31      0:00 grep mem

     sto  6337  1152   0 22:15:10 ?           0:07 /opt/memcached/bin/memcached -d -P /tmp/logs/memcached/memcached.pi

2. Once you know the process ID, use the pargs command to determine the port on which the service is running

bash# pargs 6337

6337:   /opt/memcached/bin/memcached -d -P /tmp/logs/memcached/memcached.pi

argv[0]: /opt/memcached/bin/memcached

argv[1]: -d

argv[2]: -P

argv[3]: /tmp/logs/memcached/memcached.pid

argv[4]: -u

argv[5]: anurag

argv[6]: -c

argv[7]: 1024

argv[8]: -m

argv[9]: 64

argv[10]: -p

argv[11]: 11211

The value of the parameter 'p' indicates the port on which the memcached service is running. In the above example this is port: 11211 (which is the default port for the memcached service)

3. Determine the IP address or hostname of the server using the standard ifconfig -a or hostname commands

4. Once you know the IP(or hostname) and port, simply telnet to it as below:

bash# telnet localhost 11211

Trying 127.0.0.1...

Connected to localhost.

Escape character is '^]'.

Once you get the above prompt, you are connected to the telnet interface!

Obtaining memcached stats from the telnet interface

Once you are connected to the telnet interface, simply use one of the following to obtain the necessary statistics:

Command	Description	Example
stats	Prints general statistics	stats
	Prints memory statistics	stats slabs
	Prints memory statistics	stats malloc
	Print higher level allocation statistics	stats items
		stats detail
		stats sizes

Example:

# telnet localhost 11211

Trying 127.0.0.1...

Connected to localhost.

Escape character is '^]'.

stats

STAT pid 6337

STAT uptime 66956

STAT time 1268844664

STAT version 1.2.6

STAT pointer_size 32

STAT rusage_user 2.496175

STAT rusage_system 4.299976

STAT curr_items 108

STAT total_items 114

STAT bytes 748774

STAT curr_connections 9

STAT total_connections 5607

STAT connection_structures 12

STAT cmd_get 407

STAT cmd_set 114

STAT get_hits 174

STAT get_misses 233

STAT evictions 0

STAT bytes_read 1045885

STAT bytes_written 17161371

STAT limit_maxbytes 67108864

STAT threads 1

END

Other uses of the telnet interface

Command	Description	Example
get	Reads a value	get mykey
set	Set a key unconditionally	set mykey 0 60 5
add	Add a new key	add newkey 0 60 5
replace	Overwrite existing key	replace key 0 60 5
append	Append data to existing key	append key 0 60 15
prepend	Prepend data to existing key	prepend key 0 60 15
incr	Increments numerical key value by given number	incr mykey 2
decr	Decrements numerical key value by given number	decr mykey 5
delete	Deletes an existing key	delete mykey
flush_all	Invalidate specific items immediately	flush_all
	Invalidate all items in n seconds	flush_all 900
version	Prints server version.	version
verbosity	Increases log level	verbosity
quit	Terminate telnet session	quit