Tuesday, October 12, 2010

Escenic 4.3-x Publication Bootstrapping Configurations

Bootstrapping is controlled with the following to configuration files

  • ECE_LOCALCONFIG/neo/io/content/InitialBootstrapper.properties
  • features (Publication resource - /escenic/features)


ECE_LOCALCONFIG/neo/io/content/InitialBootstrapper.properties

# The default dpeth to try to probe when going through the section tree. By default, a
# publication's top sections along with its children are probed, i.e. the depth is set to 2.
# Setting this property has effect when the bootstrapOnStratup is set to the keyword true
# (through features for example)
depth=3

# The number of seconds that the InitialBootstrapper should wait before trying to load the
# publications. Default value is 60
secondsToWait=120

# The number of seconds to try retrieving the sections and articles in the bootstrap phase.
# By default, if a publication has not finished bootstraping within 30 seconds, the boostrap
# for the publication in question would fail
timeoutSeconds=120

features (Publication resource)
#######################################################################
# Initial Bootstrapper Configurations
#######################################################################
# Can be set to make the InitialBoottrapper run through the publication
# when the server starts
# Example:
# bootstrapOnStartup=ece_frontpage,main,news,sports,football
# bootstrapOnStartup=true
# when set to true the depth property of the InitialBootstrapper
# component is used
bootstrapOnStartup=true
# The initial bootstrapper will by default try to connect to the
# publication's URL.  On a multi-server installation, this will probably
# be a HTTP server spraying the requests to more application servers.
# To make the initial bootstrapper connect to the bootstrapper,
# add a line which specifies the "localUrl" property.  All connections to
# the template system
# Example:
# local.url=http://localhost:8080//
local.url=http://localhost:8080/myPub/

Notes

Once Bootstrapping is complete, the property "bootstrapped" in http:///escenic-admin/browser/neo/io/content/InitialBootstrapper has value "true". The publication is not accessible until bootstrapping is complete.

You can also force mark complete the bootstrapping process by setting the "bootstrapped" property to "true" here: http:///escenic-admin/browser/neo/io/content/InitialBootstrapper?property=bootstrapped

Changing logging level of the Bootstrapping component
trace.properties (or relevant log4j properties)
log4j.category.neo.xredsys.config.InitialBootstrapper=DEBUG, Appender


Bookmark and Share

Monday, October 11, 2010

Tomcat 5.5.27 | Memory Leak In Escenic CMS Based Web Application


Problem Statement
Memory leak in web application running on Tomcat

System Information
Tomcat 5.5.27
Apache HTTPD 2.2.9
Java HotSpot(TM) Server VM (build 1.5.0_14-b03, mixed mode)
Mod_proxy_http connector
Solaris 10 x86
Tomcat JVM heap settings: min:2GB and max:2GB

Observations and Investigation Done
Under load, the web application slowly runs out of heap space. The heap utilization graph suggests a memory leak type pattern.

Tomcat JVM Heap Utilization Pattern

Heap dumps were gathered to determine the root cause and observations are as below from the dumps:
  1. HashMap entries from the below object reference tree seem to consume over 80% of the used tenured generation space.
org/apache/jasper/compiler/JspRuntimeContext
Java/util/Collections$SynchronizedMap
Java/util/HashMap
Java/util/HashMap$Entry 
  1. The number of objects referenced by each HashMap entry vary between 2-3
    1. These are either a String and JspServletWrapper object
or
    1. A String, JspServletWrapper and another HashMap Entry. This call reference tree of HashMap entries referenced by another HashMap entry can repeat to a depth of approximately 8-10 nodes
The above is indicated in an object reference tree obtained after analyzing the Heap dump as shown below
Snapshot from IBM Heap Analyzer - Object Call Reference Tree
  1. The maximum percentage of the memory occupied by the HashMap entry object is by a character array that seems like the JSP servlet response. HTML response (tags with content) can be seen in the character array.
  2. Heap dumps gathered during different times and after a Full GC always indicate 100 entries. The number of entries does not grow. However the size of these Hash Map entries grows with time under load. After the first Full GC the size of this HashMap was observed to be around 350MB. The next heap dump obtained after the fourth full GC indicated that the size of this Hash Map doubled to around 730MB. The number of entries under this HashMap however remained constant. I use the words "directly under" this HashMap since I am not sure yet if the number of entries within the first level entries increased. A snapshot can be seen below
Comparison of size of HashMap after first and fourth full GC cycles
Questions
  1. I am stuck at this point and have run out of ideas on how to get to the root cause of this issue. Do you have any ideas/suggestions to help identify the root cause?
  2. How do I  confirm if this is a definite leak. How do I determine from the 2 heap dumps I have that objects are persisting in the heap and not getting collected? The address of objects can change across GC cycles. So what approach should be used to determine if an object that was seen in the first dump also exists in the subsequent dump?
  3. I googled and found the following interesting links
    1. http://www.mail-archive.com/dev@tomcat.apache.org/msg03395.html Is there any know issues, configuration that causes a memory leak by caching of servlet responses in the container and not flushing the cached objects?
    2. http://mail-archives.apache.org/mod_mbox/tomcat-users/200303.mbox/%3C000c01c2ee2c$41c079a0$03b696c0@garethdqw0t9if%3E We use a lot of JSTL in our web application. Are there any known issues around memory issues as indicated on the link?
Resolution
The objects holding references to the character arrays that ultimately consume all the memory are of type org.apache.jasper.runtime.BodyContentImpl as indicated in the object reference tree below:
BodyContentImpl holding references to memory consuming character arrays
There was a bug reported in Tomcat 5.5.9 which says 
The problem is that this huge array never gets reset due to the object pooling 
implementation in Jasper (JspFactoryImpl maintains a pool of PageContextImpl 
objects. Each PageContextImpl object maintains an array of BodyContentImpl 
objects), so the memory it consumed is never returned to the heap.
 https://issues.apache.org/bugzilla/show_bug.cgi?id=37793

We do not use the suggest JVM argument in our Tomcat 5.5.27 JVM
-Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true


When the tests were repeated using the JVM argument mentioned above, the problem was resolved. A healthy heap utilization graph was obtained as indicate below
Heap Utilization Post Using the Recommended JVM Argument


There was no residual garbage after Full GC cycles as seen before this new JVM argument.


Additional Configurations Added In The Process

  • Development mode was turned "off" for Jasper
  • Trim white spaces option what turned "on" for Jasper

These are recommended production settings for tomcat as mentioned here - http://tomcat.apache.org/tomcat-5.5-doc/jasper-howto.html



Bookmark and Share

Wednesday, March 17, 2010

Memcached Telnet Interface

What is memcached?
memcached is a popular distributed memory object caching system. For details refer this link

Connecting to the telnet interface
To connect to the interface you need the IP address and port on which the memcached server is running.
Describe below is a method to determine this information on a solaris environment on which a memcached service is already running

1. Determine the process ID of the memcached process (/service)
bash# ps -eaf | grep mem
    root  6917 25125   0 16:39:40 pts/31      0:00 grep mem
     sto  6337  1152   0 22:15:10 ?           0:07 /opt/memcached/bin/memcached -d -P /tmp/logs/memcached/memcached.pi

2. Once you know the process ID, use the pargs command to determine the port on which the service is running
bash# pargs 6337
6337:   /opt/memcached/bin/memcached -d -P /tmp/logs/memcached/memcached.pi
argv[0]: /opt/memcached/bin/memcached
argv[1]: -d
argv[2]: -P
argv[3]: /tmp/logs/memcached/memcached.pid
argv[4]: -u
argv[5]: anurag
argv[6]: -c
argv[7]: 1024
argv[8]: -m
argv[9]: 64
argv[10]: -p
argv[11]: 11211

The value of the parameter 'p' indicates the port on which the memcached service is running. In the above example this is port: 11211 (which is the default port for the memcached service)

3. Determine the IP address or hostname of the server using the standard ifconfig -a or hostname commands

4. Once you know the IP(or hostname) and port, simply telnet to it as below:
bash# telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

Once you get the above prompt, you are connected to the telnet interface!

Obtaining memcached stats from the telnet interface
Once you are connected to the telnet interface, simply use one of the following to obtain the necessary statistics:

Command Description Example
stats Prints general statistics stats
Prints memory statistics stats slabs
Prints memory statistics stats malloc
Print higher level allocation statistics stats items

stats detail

stats sizes

Example:
# telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
stats
STAT pid 6337
STAT uptime 66956
STAT time 1268844664
STAT version 1.2.6
STAT pointer_size 32
STAT rusage_user 2.496175
STAT rusage_system 4.299976
STAT curr_items 108
STAT total_items 114
STAT bytes 748774
STAT curr_connections 9
STAT total_connections 5607
STAT connection_structures 12
STAT cmd_get 407
STAT cmd_set 114
STAT get_hits 174
STAT get_misses 233
STAT evictions 0
STAT bytes_read 1045885
STAT bytes_written 17161371
STAT limit_maxbytes 67108864
STAT threads 1
END

Other uses of the telnet interface

Command Description Example
get Reads a value get mykey
set Set a key unconditionally set mykey 0 60 5
add Add a new key add newkey 0 60 5
replace Overwrite existing key replace key 0 60 5
append Append data to existing key append key 0 60 15
prepend Prepend data to existing key prepend key 0 60 15
incr Increments numerical key value by given number incr mykey 2
decr Decrements numerical key value by given number decr mykey 5
delete Deletes an existing key delete mykey
flush_all Invalidate specific items immediately flush_all
Invalidate all items in n seconds flush_all 900
version Prints server version. version
verbosity Increases log level verbosity
quit Terminate telnet session quit



Bookmark and Share