--Tech Bytes--

Escenic 4.3-x Publication Bootstrapping Configurations

2010-10-12T23:59:00.001+01:00

Bootstrapping is controlled with the following to configuration files

ECE_LOCALCONFIG/neo/io/content/InitialBootstrapper.properties
features (Publication resource - /escenic/features)

ECE_LOCALCONFIG/neo/io/content/InitialBootstrapper.properties

# The default dpeth to try to probe when going through the section tree. By default, a
# publication's top sections along with its children are probed, i.e. the depth is set to 2.
# Setting this property has effect when the bootstrapOnStratup is set to the keyword true
# (through features for example)
depth=3

# The number of seconds that the InitialBootstrapper should wait before trying to load the
# publications. Default value is 60
secondsToWait=120

# The number of seconds to try retrieving the sections and articles in the bootstrap phase.
# By default, if a publication has not finished bootstraping within 30 seconds, the boostrap
# for the publication in question would fail
timeoutSeconds=120

features (Publication resource)
#######################################################################
# Initial Bootstrapper Configurations
#######################################################################
# Can be set to make the InitialBoottrapper run through the publication
# when the server starts
# Example:
# bootstrapOnStartup=ece_frontpage,main,news,sports,football
# bootstrapOnStartup=true
# when set to true the depth property of the InitialBootstrapper
# component is used
bootstrapOnStartup=true
# The initial bootstrapper will by default try to connect to the
# publication's URL. On a multi-server installation, this will probably
# be a HTTP server spraying the requests to more application servers.
# To make the initial bootstrapper connect to the bootstrapper,
# add a line which specifies the "localUrl" property. All connections to
# the template system
# Example:
# local.url=http://localhost:8080//
local.url=http://localhost:8080/myPub/

Notes

Once Bootstrapping is complete, the property "bootstrapped" in http:///escenic-admin/browser/neo/io/content/InitialBootstrapper has value "true". The publication is not accessible until bootstrapping is complete.

You can also force mark complete the bootstrapping process by setting the "bootstrapped" property to "true" here: http:///escenic-admin/browser/neo/io/content/InitialBootstrapper?property=bootstrapped

Changing logging level of the Bootstrapping component
trace.properties (or relevant log4j properties)
log4j.category.neo.xredsys.config.InitialBootstrapper=DEBUG, Appender

Tomcat 5.5.27 | Memory Leak In Escenic CMS Based Web Application

2010-10-11T13:48:00.011+01:00

Problem Statement
Memory leak in web application running on Tomcat

System Information

Tomcat 5.5.27

Apache HTTPD 2.2.9

Java HotSpot(TM) Server VM (build 1.5.0_14-b03, mixed mode)

Mod_proxy_http connector

Solaris 10 x86

Tomcat JVM heap settings: min:2GB and max:2GB

Observations and Investigation Done

Under load, the web application slowly runs out of heap space. The heap utilization graph suggests a memory leak type pattern.

Tomcat JVM Heap Utilization Pattern

Heap dumps were gathered to determine the root cause and observations are as below from the dumps:

HashMap entries from the below object reference tree seem to consume over 80% of the used tenured generation space.

org/apache/jasper/compiler/JspRuntimeContext

↓

Java/util/Collections$SynchronizedMap

↓

Java/util/HashMap

↓

Java/util/HashMap$Entry

The number of objects referenced by each HashMap entry vary between 2-3
1. These are either a String and JspServletWrapper object

A String, JspServletWrapper and another HashMap Entry. This call reference tree of HashMap entries referenced by another HashMap entry can repeat to a depth of approximately 8-10 nodes

The above is indicated in an object reference tree obtained after analyzing the Heap dump as shown below

Snapshot from IBM Heap Analyzer - Object Call Reference Tree

The maximum percentage of the memory occupied by the HashMap entry object is by a character array that seems like the JSP servlet response. HTML response (tags with content) can be seen in the character array.
Heap dumps gathered during different times and after a Full GC always indicate 100 entries. The number of entries does not grow. However the size of these Hash Map entries grows with time under load. After the first Full GC the size of this HashMap was observed to be around 350MB. The next heap dump obtained after the fourth full GC indicated that the size of this Hash Map doubled to around 730MB. The number of entries under this HashMap however remained constant. I use the words "directly under" this HashMap since I am not sure yet if the number of entries within the first level entries increased. A snapshot can be seen below

Comparison of size of HashMap after first and fourth full GC cycles

Questions

I am stuck at this point and have run out of ideas on how to get to the root cause of this issue. Do you have any ideas/suggestions to help identify the root cause?
How do I confirm if this is a definite leak. How do I determine from the 2 heap dumps I have that objects are persisting in the heap and not getting collected? The address of objects can change across GC cycles. So what approach should be used to determine if an object that was seen in the first dump also exists in the subsequent dump?
I googled and found the following interesting links
1. http://www.mail-archive.com/dev@tomcat.apache.org/msg03395.html Is there any know issues, configuration that causes a memory leak by caching of servlet responses in the container and not flushing the cached objects?
2. http://mail-archives.apache.org/mod_mbox/tomcat-users/200303.mbox/%3C000c01c2ee2c$41c079a0$03b696c0@garethdqw0t9if%3E We use a lot of JSTL in our web application. Are there any known issues around memory issues as indicated on the link?

Resolution
The objects holding references to the character arrays that ultimately consume all the memory are of type org.apache.jasper.runtime.BodyContentImpl as indicated in the object reference tree below:

BodyContentImpl holding references to memory consuming character arrays

There was a bug reported in Tomcat 5.5.9 which says

The problem is that this huge array never gets reset due to the object pooling 
implementation in Jasper (JspFactoryImpl maintains a pool of PageContextImpl 
objects. Each PageContextImpl object maintains an array of BodyContentImpl 
objects), so the memory it consumed is never returned to the heap.

https://issues.apache.org/bugzilla/show_bug.cgi?id=37793

We do not use the suggest JVM argument in our Tomcat 5.5.27 JVM

-Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true

When the tests were repeated using the JVM argument mentioned above, the problem was resolved. A healthy heap utilization graph was obtained as indicate below

Heap Utilization Post Using the Recommended JVM Argument

There was no residual garbage after Full GC cycles as seen before this new JVM argument.

Additional Configurations Added In The Process

Development mode was turned "off" for Jasper
Trim white spaces option what turned "on" for Jasper

These are recommended production settings for tomcat as mentioned here - http://tomcat.apache.org/tomcat-5.5-doc/jasper-howto.html

Memcached Telnet Interface

2010-03-17T16:58:00.001Z

What is memcached?

memcached is a popular distributed memory object caching system. For details refer this link

Connecting to the telnet interface

To connect to the interface you need the IP address and port on which the memcached server is running.

Describe below is a method to determine this information on a solaris environment on which a memcached service is already running

1. Determine the process ID of the memcached process (/service)

bash# ps -eaf | grep mem

    root  6917 25125   0 16:39:40 pts/31      0:00 grep mem

     sto  6337  1152   0 22:15:10 ?           0:07 /opt/memcached/bin/memcached -d -P /tmp/logs/memcached/memcached.pi

2. Once you know the process ID, use the pargs command to determine the port on which the service is running

bash# pargs 6337

6337:   /opt/memcached/bin/memcached -d -P /tmp/logs/memcached/memcached.pi

argv[0]: /opt/memcached/bin/memcached

argv[1]: -d

argv[2]: -P

argv[3]: /tmp/logs/memcached/memcached.pid

argv[4]: -u

argv[5]: anurag

argv[6]: -c

argv[7]: 1024

argv[8]: -m

argv[9]: 64

argv[10]: -p

argv[11]: 11211

The value of the parameter 'p' indicates the port on which the memcached service is running. In the above example this is port: 11211 (which is the default port for the memcached service)

3. Determine the IP address or hostname of the server using the standard ifconfig -a or hostname commands

4. Once you know the IP(or hostname) and port, simply telnet to it as below:

bash# telnet localhost 11211

Trying 127.0.0.1...

Connected to localhost.

Escape character is '^]'.

Once you get the above prompt, you are connected to the telnet interface!

Obtaining memcached stats from the telnet interface

Once you are connected to the telnet interface, simply use one of the following to obtain the necessary statistics:

Command	Description	Example
stats	Prints general statistics	stats
	Prints memory statistics	stats slabs
	Prints memory statistics	stats malloc
	Print higher level allocation statistics	stats items
		stats detail
		stats sizes

Example:

# telnet localhost 11211

Trying 127.0.0.1...

Connected to localhost.

Escape character is '^]'.

stats

STAT pid 6337

STAT uptime 66956

STAT time 1268844664

STAT version 1.2.6

STAT pointer_size 32

STAT rusage_user 2.496175

STAT rusage_system 4.299976

STAT curr_items 108

STAT total_items 114

STAT bytes 748774

STAT curr_connections 9

STAT total_connections 5607

STAT connection_structures 12

STAT cmd_get 407

STAT cmd_set 114

STAT get_hits 174

STAT get_misses 233

STAT evictions 0

STAT bytes_read 1045885

STAT bytes_written 17161371

STAT limit_maxbytes 67108864

STAT threads 1

END

Other uses of the telnet interface

Command	Description	Example
get	Reads a value	get mykey
set	Set a key unconditionally	set mykey 0 60 5
add	Add a new key	add newkey 0 60 5
replace	Overwrite existing key	replace key 0 60 5
append	Append data to existing key	append key 0 60 15
prepend	Prepend data to existing key	prepend key 0 60 15
incr	Increments numerical key value by given number	incr mykey 2
decr	Decrements numerical key value by given number	decr mykey 5
delete	Deletes an existing key	delete mykey
flush_all	Invalidate specific items immediately	flush_all
flush_all	Invalidate all items in n seconds	flush_all 900
version	Prints server version.	version
verbosity	Increases log level	verbosity
quit	Terminate telnet session	quit

SSH To Remote Solaris Server With X11 Forwarding From Windows Desktop

2010-03-17T16:16:00.003Z

What is X11?

X11 (also know as X Windows) is basically a GUI for networked computers. Refer this link for details

Softwares used

PuTTy: A ssh and telnet client. Download

Xming: A leading free unlimited X Window server for Microsoft Windows (XP/2003/Vista). Refer this link for details and downloading

Setup

Install and run Xming server on your windows desktop. Next, create a new ssh connection profile on a remote server with X11 forwarding enabled using your PuTTy client. To do this, navigate to your PuTTy configurations under the menu: Connection > SSH > Tunnels and set it up as shown in the snapshot below:

Note: The PuTTy configuration menu might be slightly different depending on the version you are using. This is an example to demostrate the setup.

If you do not have X11 forwarding enabled on the solaris machine you ssh into, the following might help:

vi /etc/ssh/sshd_config

# make sure the following properties are set:
AllowTcpForwarding yes

# X11 tunneling options
X11Forwarding yes
X11DisplayOffset 10
X11UseLocalhost no # this is required because of the IPV6 Solaris problem

svcadm disable /network/ssh
svcadm enable /network/ssh

Escenic 4.3-x Quick Start Developers Guide

2010-01-26T02:43:00.028Z

What is Escenic?

Escenic is a leading template based Strategic Content Management System. It is a family of several products out of which the following are predominantly used in various media applications:

• Escenic Content Engine
• Escenic Web Studio
• Escenic Content Studio
• Escenic Modules
o Article Comments
o Article Categories and Taxonomy
o Forum
o Menu Editor
o Geotagging
o Poll
• Escenic Ad Manager
• Escenic Analysis Engine

For a complete list of Escenic products refer to: http://www.escenic.com/products/

Escenic Content Engine

The Escenic Content Engine is the heart for the Escenic Content System, storing all text, images, audio, video and other digital assets and provides the back-end functionality for applications such as Escenic Content Studio. Content is also made accessible through Java Server Pages tag libraries, RESTful web services and an open Java API.

Escenic templates need this content engine to run on.

Installation Requirements

Operating System
Application Server or Servlet Container
Database Server
JDBC Driver
LDAP Server
JDK
Web Server (optional)

Example Stack

OS: Solaris 10
Servlet Container: Tomcat 5.5.x
Database Server: Oracle 10g
JDBC Driver: Ojdbc 10.x
LDAP Server: Sun One (iPlanet)

Detailed Installation Instructions: http://technet.escenic.com/multimedia/archive/00001/ece-install-guide_pdf_1231a.pdf

Escenic Template System

An Escenic template system is analogous to a file system with the following mapping:
Folder -> Sections
Files -> Articles

Any Escenic site is a collection of templates where Sections and Articles present as data structures in the database are presented to the templates (web pages) in the form of regular Java Beans.

Thus for starters you can assume every page of an Escenic web site as a Section or an Article page. Each of these section or article pages has a template associated with it.
Section pages have ‘section’, ‘grid’ and ‘element’ templates associated with them whereas Article pages are associated to the ‘article’ template.

Typical Request Flow Mechanism of an Escenic Application

HTTP URL Request (by an end users browser)

↓

Escenic Servlet Filter Chain

↓

Escenic Template System

↓

HTTP Response

When the end user requests a URL, the request is handled by the application server, which forwards the request to the appropriate web application (Escenic web app in our case).

The request on reaching the web app is handled by the Escenic filter chain. The various filters configured as a part of the standard Escenic filter chain have the following two tasks:

1. Retrieve content beans from the database and make them available as java beans in the request scope to be used by a template.
2. Forward the request to an appropriate Escenic template page. Unless overridden by WEB-INF/localconfig/defaults.properties the request is always forwarded to a file called WEB_APP_ROOT/template/common.jsp

Article Page Request

Article pages are requested by URL patterns like below containing the Escenic ID of the article being requested.

http://www.mysite.com/ak/sport/football/article7002354.ece

When a request like above is handled by the Servlet chain, it knows it has to extract the content bean of article with ID: 7002354 from the database, within section the ‘football’ sub-section of the ‘sport’ section.

Based on the information in the request URL, the filters will set the following beans in the request scope before forwarding the request to the appropriate template page (WEB_APP_ROOT/template/common.jsp in most cases):

S No	Bean Name	Type	Description
1	publication	neo.xredsys.api.Publication	Publication object. Which publication object to set in request scope is determined from the context root of the URL. In this example, the Servlet filter would set the publication object corresponding to publication named ‘tol’.
2	section	neo.xredsys.api.Section	Again, which section object to set in scope is determined from the URL. In this example, football section, which is a sub-section of sport section will be set in scope
3	article	neo.xredsys.presentation.PresentationArticle	The article object with ID:7002354 will be set in request scope in this example

After setting the above beans, the filter chain will forward control the template page: common.jsp.

Typical Template System Flow for an Article Request

Default Template (template/common.jsp)

↓

Wireframe (template/wireframe/classic.jsp)

↓

Article Layout (template/art_default.jsp or template/art_standard.jsp For an article request, the template system would look for a file with ‘art_’ prefixed to the default article template configured. Thus art_default.jsp or art_standard.jsp)

↓

template/[version number]/article/articleTypeSwitch.jsp (There is a concept of article types in Escenic. This JSP forwards control to a specific JSP after checking the ‘type’ of article bean set in request scope by the filter chain)

↓

Article Template (template/[version number]/article/standardArticle.jsp)

Section Page Request

Section pages are requested by URL patterns like below:
http://www.mysite.com/ak/news/

Just like the case of an article request the following beans are set in request scope on a section page request:
1. publication – neo.xredsys.api.Publication
2. section – neo.xredsys.api.Section
3. pool – neo.xredsys.presentation.PresentationPool

Unlike an article page, which is composed of only one template – the article template, a section page is composed of the following three templates:

a. Section template
b. Grid template
c. Element template

After setting the above beans in scope, the control would be forwarded to common.jsp (unless overridden).

Typical Template System Flow for a Section Request

Default Template (template/common.jsp)

↓

Wireframe (template/wireframe/classic.jsp)

↓

Section Layout (template/sec_default.jsp or template/sec_standard.jsp For a Section request, the template system would look for a file with ‘sec_’ prefixed to the default section template configured. Thus sec_default.jsp or sec_standard.jsp)

↓

template/[version number]/section/standard.jsp

↓

Grid Template (a file with prefix ‘grid_’. The exact file invoked will depend on the grid template associated with the Section requested. For example, if the name of the grid template associated with the section requested is ‘indexPage’, then the file templat/grid_indexPage.jsp will be invoked)

↓ ↓ … ↓

Element template 1 Element template 2 … Element template n

(What element templates are invoked is controlled by the element templates associated with the particular Grid template. This is usually controlled by publishers using Escenic Content Studio)

Each module on the Section page has its own element template. Thus, element templates are the basic building blocks of a section page.

Example, in the following page from newsoftheworld.co.uk, each of the modules with a red outline is rendered by an element template JSP file.

Publication Resources

A resource is nothing but a binary stream of data and Each Escenic publication (web-site) has the following essential resources:

1. Article-type (/escenic/article-type)
For starters, assume that every page in Escenic that is not a section page, is an article page. Now, there are various article types available in each Escenic publication. The article-type resource defines:
• Number of article-types available in the publication
• Name of each article type available in the publication
• Fields associated with each article type in the publication

For example: A publication may have an article type called: ‘standardArticle’ with the fields like: headline, teaser, body, main-image etc.

2. Layout (/escenic/layout)
This resource defines what grid elements are available in the publication and what element templates can constitute a particular grid template.
For example: A publication may have two grid templates defined, namely, ‘indexpage’ and ‘homepage’.

This resource first defines the article and section layouts available in the publication. The names of article and section layouts defined here are used to determine the names of JSP files with prefixes ‘sec_’ and ‘art_’ the template system forwards control to.

Next, the resource defines what grid templates are available in the publication. The name of the grip template is used to determine what JSP file is invoked in a Section request. For example, if section named ‘sport’ is invoked by a URL request and a grid template named ‘indexpage’ is assigned to the sport section then a file named grid_indexpage.jsp would be called. (For the sequence of invocation refer to the request flow diagram described previously)

This resource then defines what element templates are available in the publication. All element templates assigned to the grid template associated with the invoked Section request are called by the template system. The name of the element template JSP files is prefixed by ‘fp_’.

3. Image Versions (/escenic/image-version)

4. Features (/escenic/feature)

For detailed documentation refer to Escenic Page developers guide: http://technet.escenic.com/multimedia/archive/00001/page-dev_pdf_1228a.pdf

Disclaimer: This should not be treated as official Escenic documentation and should be only used as a beginners reference guide. The author does not take any responsibility for errors in the content. Please refer to technet.escenic.com for official documentation.

Hibernate - Escenic Transaction Filter Integration | Error In Initialization

2009-10-07T16:10:00.004+01:00

Issue

When trying to integrate Hibernate with Escenic Transaction Filter I observed the following exceptions, which prevented the Hibernate Session Factory to initialize correctly. Our application has both custom Event handlers and Transaction filters written for implementing certain functionalities. These event handlers and transaction filters in turn use the Hibernate ORM framework for database transactions.

java.lang.ExceptionInInitializerError
        at org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxyFactory(CGLIBLazyInitializer.java:117)
        at org.hibernate.proxy.pojo.cglib.CGLIBProxyFactory.postInstantiate(CGLIBProxyFactory.java:43)
        at org.hibernate.tuple.entity.PojoEntityTuplizer.buildProxyFactory(PojoEntityTuplizer.java:162)
        at org.hibernate.tuple.entity.AbstractEntityTuplizer.(AbstractEntityTuplizer.java:135)
        at org.hibernate.tuple.entity.PojoEntityTuplizer.(PojoEntityTuplizer.java:55)
        at org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping.(EntityEntityModeToTuplizerMapping.java:56)
        at org.hibernate.tuple.entity.EntityMetamodel.(EntityMetamodel.java:295)
        at org.hibernate.persister.entity.AbstractEntityPersister.(AbstractEntityPersister.java:434)
        at org.hibernate.persister.entity.SingleTableEntityPersister.(SingleTableEntityPersister.java:109)
        at org.hibernate.persister.PersisterFactory.createClassPersister(PersisterFactory.java:55)
        at org.hibernate.impl.SessionFactoryImpl.(SessionFactoryImpl.java:226)
        at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1294)
        at com.db.HibernateUtil.(Unknown Source)
        at com.db.MyTimesDAO.getSession(Unknown Source)
        at com.db.PlusDAO.insertRecord(Unknown Source)
        at com.article.ArticleTransactionFilterImpl.insertRecord(Unknown Source)
        at com.article.ArticleTransactionFilterImpl.handleTransaction(Unknown Source)
        at com.service.EscenicTransactionFilter.doUpdate(Unknown Source)
        at neo.xredsys.api.IOObjectUpdater.applyFiltersTo(IOObjectUpdater.java:3390)
        at neo.xredsys.api.IOObjectUpdater.applyUpdateFiltersTo(IOObjectUpdater.java:3369)
        at neo.xredsys.api.IOObjectUpdater.updateArticle(IOObjectUpdater.java:680)
        at neo.xredsys.api.ArticleTransactionImpl.update(ArticleTransactionImpl.java:62)
        at neo.xredsys.rio.RIOAdapterImpl.update(RIOAdapterImpl.java:625)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at neo.xredsys.rio.RIOSessionImpl$1.invoke(RIOSessionImpl.java:80)
        at $Proxy1.update(Unknown Source)
        at neo.xredsys.rio.RIOSessionImpl$12.run(RIOSessionImpl.java:1518)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)
        at neo.xredsys.rio.RIOSessionImpl.update(RIOSessionImpl.java:1514)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
        at sun.rmi.transport.Transport$1.run(Transport.java:153)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
        at java.lang.Thread.run(Thread.java:595)
Caused by: java.security.AccessControlException: access denied (java.lang.reflect.ReflectPermission suppressAccessChecks)
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:264)
        at java.security.AccessController.checkPermission(AccessController.java:427)
        at java.lang.SecurityManager.checkPermission(SecurityManager.java:532)
        at java.lang.reflect.AccessibleObject.setAccessible(AccessibleObject.java:107)
        at net.sf.cglib.core.ReflectUtils.getConstructor(ReflectUtils.java:245)
        at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:220)
        at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:216)
        at net.sf.cglib.core.KeyFactory$Generator.firstInstance(KeyFactory.java:157)
        at net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:225)
        at net.sf.cglib.core.KeyFactory$Generator.create(KeyFactory.java:145)
        at net.sf.cglib.core.KeyFactory.create(KeyFactory.java:117)
        at net.sf.cglib.core.KeyFactory.create(KeyFactory.java:108)
        at net.sf.cglib.core.KeyFactory.create(KeyFactory.java:104)
        at net.sf.cglib.proxy.Enhancer.(Enhancer.java:69)
        ... 44 more

I observed that when the Escenic event handler was invoked before the transaction filter in the JVMs lifecycle, the singleton instance of HibernateSessionFactory initialized successfully and subsequent use of this session factory by both the event handler and transaction filter (or any other component in the JVM using the same session factory) worked as expected. However, when the transaction filter was invoked before the even handler the above-mentioned initialization exception was thrown. Once the above exception was thrown, even an invocation of the Event handler failed to initialize the Hibernate session factory successfully making Hibernate framework unusable by any component in the JVM.

Resolution

1. Added the following in the JVM security policy file:

java.policy

grant codeBase "file: /opt/tomcat/shared/lib/cglib-2.1.3.jar" {

permission java.security.AllPermission;

};

2. Initialized the HibernateSessionFactory using Escenic Initial Context creator.

The following class actually initializes the HibernateSessionFactory:

import org.apache.log4j.Logger;

import org.hibernate.SessionFactory;

import org.hibernate.cfg.Configuration;

import com.myproject.framework.exception.SystemException;

public class HibernateUtil {

private static final Logger logger = Logger.getLogger(HibernateUtil.class);

private static final SessionFactory sessionFactory;

static {

try {

logger.debug("Building hibernate session factory");

// Create the SessionFactory from hibernate.cfg.xml

sessionFactory = new Configuration().configure().buildSessionFactory();

logger.debug("Hibernate session factory built successfully");

} catch (Throwable e) {

throw new SystemException("Initial Hibernate SessionFactory creation failed", e);

}

/**

* Returns the singleton instance of the SessionFactory

* @return SessionFactory

public static SessionFactory getSessionFactory() {

return sessionFactory;

}

This class was invoked during the Escenic initial context creation to initialize the session factory by adding the following entry in Escenic Initial.properties file as below

service.2.0.0-HibernateInit=/com/myproject/HibernateInitializer

and defining a file called HibernateInitializer.properties in [ECE_localconfig]/com/myproject with the following entry:

$class=com.db.HibernateUtil

Note: In non-escenic applications, if a similar problem is encountered, one can try using a start-up filter defined in the web-application deployment descriptor file (web.xml) instead. The idea is to basically initialize the singleton of the HibernateSessionFactory before invocation by the transaction filter

Tomcat 5 | Useful Configurations | Performance Tuning and Monitoring

2009-08-19T21:44:00.016+01:00

JVM Arguments

argv[4]: -Xmx2560m	Heap Size setting. Min and Max should be set to the same value if you know that your application will mostly operate on the max heap value.
argv[5]: -Xms2560m
argv[13]: -Dsun.rmi.dgc.server.gcInterval=3600000	As there exists a bug in JDK 1.5, this is the recommended setting, Sun Bug Database
argv[14]: -Dsun.rmi.dgc.client.gcInterval=3600000	This is used in combination to the above.
argv[17]: -Dsun.rmi.transport.connectionTimeout=20000	For network failure tuning.
argv[18]: -Dsun.rmi.transport.tcp.handshakeTimeout=20000	For network failure tuning.
argv[19]: -Dsun.rmi.transport.tcp.readTimeout=20000	For network failure tuning.
-XX:+DisableExplicitGC	Do not use. Refer: http://forums.sun.com/thread.jspa?threadID=5369546&tstart=0
argv[20]: -verbose:gc	For debugging
argv[21]: -XX:+UseParNewGC	GC Algorithm
argv[22]: -XX:+PrintGCTimeStamps	For debugging
argv[23]: -XX:+PrintGCDetails	For debugging

Tomcat Settings

Connector Settings

maxThreads="300" minSpareThreads="25" maxSpareThreads="77"

enableLookups="false" redirectPort="8443" acceptCount="256"

connectionTimeout="60000" disableUploadTimeout="true" />

The maxThreads and related settings we used were based on the corresponding settings on Apache considering default values Apache MPM Prefork are being used.
-maxSpareThreads was set to 77 instead of the default value of 75 as tomcat is a bit slower than apache in spinning up threads.
-maxThreads value was increased from 150 to 300 - in-line with the maximum number of connections which Apache can receive (256 by default using pre-fork MPM).
-acceptCount was increased from 100 to 256 – in-line with above.

Tip: Ensure that development mode is set to false in TOMCAT/conf/web.xml [We have not tried this so far. Detailed investigation is in the pipeline]

JDBC Connection Pool Settings (context.xml)

connectionProperties="{ValidateConnection=true,ConnectionWaitTimeout=10}" username="*****" password="*****" validationQuery="select 1 from dual" testOnBorrow="true" poolPreparedStatements="true" maxOpenPreparedStatements="10" initialSize="10" minIdle="10" maxActive="25" maxIdle="20" maxWait="30000" removeAbandonedTimeout="60" removeAbandoned="true" logAbandoned="true"/>

-maxWait value was changed from the default value of -1 (indefinite wait) to 300 seconds. This is to prevent exhaustion of threads due to a problems in the DB layer
-Detailed DBCP Configurations: http://commons.apache.org/dbcp/configuration.html

For Oracle 9i onwards you should use oracle.jdbc.OracleDriver rather than oracle.jdbc.driver.OracleDriver as the driverClassName in the DataSource definiton as Oracle have stated that oracle.jdbc.driver.OracleDriver is deprecated and support for this driver class will be discontinued in the next major release.

It is sometimes desirable to place the JNDI resource definitions in the GlobalNamingResources Section rather than under Context elements to enable resources being shared across all contexts. If resources are defined under Contexts then they are duplicated for each context and thereby can exhaust the max connections a DB is configured to handle

Apache Settings

mod_proxy_http was used instead of mod_proxy_ajp as the connector between Tomcat and Apache as it is more stable. This change sometimes requires ProxyPreserveHost setting to be turned ON to ensure correct configurations.

ProxyPass http://localhost:8080/ retry=0

ProxyPassReverse http://www.mysite.co.uk/

The retry=0 settings was used to resolve an issue on our production environment that caused intermittent 503 errors served by Apache. (Refer: http://httpd.apache.org/docs/2.2/mod/mod_proxy.html )

If IPV4 is used in the environment, modify default httpd.conf settings to change the Listener port configurations as indicated below:

0.0.0.0:80 rather than Listen 80

Useful Tools For Tomcat/Apache Monitoring

JConsole
Lamda Probe
JMX Proxy Servlet
Custom Code Using java.lang.management Package
Example:

import java.io.IOException;
import java.io.PrintWriter;
import java.lang.management.GarbageCollectorMXBean;
import java.lang.management.ManagementFactory;
import java.lang.management.MemoryPoolMXBean;
import java.lang.management.MemoryUsage;
import java.util.List;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

/**

* Servlet that provides JVM - Memory monitoring

* @copyright Anurag Kapur

* @version 1.0

* @author Anurag Kapur

public class JVMMonitor extends HttpServlet {

private static final long serialVersionUID = 1L;

/**

* Invoked on servlet initialization

public void init() throws ServletException {

System.out.println("monitoring.JVMMonitor Initilized");

}

/**

* Method that gets invoked on servlet invocation

public void doGet(HttpServletRequest request, HttpServletResponse response)

throws IOException, ServletException {

PrintWriter out = response.getWriter();

getHeapStatistics(out);

getGenerationStatistics(out);

getGCStatistics(out);

getNonHeapStatistics(out);

out.flush();

}

/**

* Invoked when the servlet is removed from the containers context

public void destroy() {

System.out.println("monitoring.JVMMonitor Destroyed");

}

/**

* Gathers Heap statistics from the {@link ManagementFactory} and prints

* this info to the PrintWriter object

* @param out

private void getHeapStatistics(PrintWriter out) {

MemoryUsage heapUsage = ManagementFactory.getMemoryMXBean()

.getHeapMemoryUsage();

out.println("Heap Memory Statistics (in bytes)");

out.println("Initialized Heap Memory :: " + heapUsage.getInit());

out.println("Committed Heap Memory :: " + heapUsage.getCommitted());

out.println("Max Heap Memory :: " + heapUsage.getMax());

out.println("Used Heap Memory :: " + heapUsage.getUsed());

out.println("");

out.flush();

}

/**

* Gathers Non-Heap statistics from the {@link ManagementFactory} and prints

* this info to the PrintWriter object

* @param out

private void getNonHeapStatistics(PrintWriter out) {

MemoryUsage nonHeapUsage = ManagementFactory.getMemoryMXBean()

.getNonHeapMemoryUsage();

out.println("Non-Heap Memory Statistics (in bytes)");

out.println("Initialized Non-Heap Memory :: " + nonHeapUsage.getInit());

out.println("Committed Non-Heap Memory :: "

+ nonHeapUsage.getCommitted());

out.println("Max Non-Heap Memory :: " + nonHeapUsage.getMax());

out.println("Used Non-Heap Memory :: " + nonHeapUsage.getUsed());

out.println("");

out.flush();

}

/**

* Gathers Heap generation statistics from the {@link ManagementFactory} and prints

* this info to the PrintWriter object

* @param out

private void getGenerationStatistics(PrintWriter out) {

List memoryPool = ManagementFactory

.getMemoryPoolMXBeans();

out.println("Heap Partitions (Generations) Statistics (in bytes)");

for (MemoryPoolMXBean memory : memoryPool) {

out.println("Partition Name :: " + memory.getName());

out.println(memory.getName() + " Max Memory :: " + memory.getUsage().getMax());

out.println(memory.getName() + " Used Memory :: " + memory.getUsage().getUsed());

}

out.println("");

out.flush();

}

/**

* Gathers GC statistics from the {@link ManagementFactory} and prints

* this info to the PrintWriter object

* @param out

private void getGCStatistics(PrintWriter out) {

List collectorBeans = ManagementFactory

.getGarbageCollectorMXBeans();

out.println("Garbage Collection Statistics (times are in milliseconds)");

for (GarbageCollectorMXBean collector : collectorBeans) {

long count = collector.getCollectionCount();

long time = collector.getCollectionTime();

String name = collector.getName();

out.println(name + " Collection Count :: " + count);

out.println(name + " Collection Time :: " + time);

if(count != 0)

out.println(name + " Collection Pause Time :: " + time/count);

}

out.println("");

out.flush();

}

Escenic Content Studio Hang Issue - Version 4.3.3

2009-07-19T21:27:00.000+01:00

Problem Statement: ECS Hang

We have a JNLP Based Java Client Application(Escenic Content Studio) which communicates with a Tomcat Servlet Conatiner using RMI. Software stack with versions used is as below:

JVM Version: 1.5.0_14-b03

Server version: Apache Tomcat/5.5.27

OS/uname: SunOS uat5cma52e 5.10 Generic_125101-06 i86pc i386 i86pc

The server application is a web content management system (produced by a company called Escenic) and uses LDAP for its authentication, and Oracle/Apache DBCP for its database connectivity, it also uses RMI to communicate with other nodes for sending events etc.

The problem we are seeing is that after an amount of time(ranging between 20 minutes to 40 hours), the server application appears to hang for the clients connected via RMI, tomcat continues to serve content via HTTP, but after a while (depending on the number of requests) the HTTP threads begin to block, an example of a hung client connection thread is (The RMI connections have been observed to hang at exactly the same point each time the issue has been reproduced). Once hung the threads remain in hung state for as long as 24 hours and don’t recover:

"RMI TCP Connection(1642)-143.252.80.100" daemon prio=10 tid=0x09a972b0 nid=0x872 waiting on condition [0x60916000..0x60917ab8]

at neo.xredsys.auth.JNDIPolicy14.getPermissions(JNDIPolicy14.java:46)

at java.security.Policy.implies(Policy.java:400)

at java.security.ProtectionDomain.implies(ProtectionDomain.java:195)

at java.security.AccessControlContext.checkPermission(AccessControlContext.java:249)

at java.security.AccessController.checkPermission(AccessController.java:427)

at neo.xredsys.api.IOObjectLoader.getArticle(IOObjectLoader.java:306)

at neo.xredsys.api.IOObjectLoader.getObject(IOObjectLoader.java:40)

at neo.xredsys.rio.RIOLockManagerImpl.lockTransaction(RIOLockManagerImpl.java:57)

at neo.xredsys.rio.RIOLockManagerImpl.lockField(RIOLockManagerImpl.java:133)

at neo.xredsys.rio.RIOAdapterImpl.lock(RIOAdapterImpl.java:400)

at neo.xredsys.rio.RIOSessionImpl$3.run(RIOSessionImpl.java:1006)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAsPrivileged(Subject.java:517)

at neo.xredsys.rio.RIOSessionImpl.lock(RIOSessionImpl.java:1003)

at sun.reflect.GeneratedMethodAccessor384.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:585)

at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)

at sun.rmi.transport.Transport$1.run(Transport.java:153)

at java.security.AccessController.doPrivileged(Native Method)

at sun.rmi.transport.Transport.serviceCall(Transport.java:149)

at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466)

at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)

at java.lang.Thread.run(Thread.java:595)

The line of code where this is hanging is performing an instanceof operation:

for (int i=0; i <>

Principal tmp = pd.getPrincipals()[i];

if (tmp instanceof ECEPrincipal) {

principal = (ECEPrincipal) tmp;

break;

}

The class which is “waiting on condition” has the following inheritance hierarchy:

java.security.Policy

↓

neo.xredsys.auth.JNDIPolicy

↓

neo.xredsys.auth.JNDIPolicy14

The class which is having the instanceof operation performed on it has the following class hierarchy:

java.security.Principal

↓

neo.xredsys.auth.ECEPrincipal

We have an example of the truss output, which shows that the light weight processes are waiting on a condition:

25743/6: lwp_cond_wait(0x08311BA8, 0x08311B90, 0xFAF9BC98, 0) Err#62 ETIME

25743/6: condvar type: USYNC_THREAD

25743/6: mutex type: USYNC_THREAD

25743/6: timeout: 0.010999273 sec

We want to know is “what” the java threads/lwp’s are waiting for, and how to trace through the system; and what exactly is happening when java performs an “instanceof” operation - we believe that it is somehow related to LDAP, as we have had some success in relieving the problem by setting the following 2 properties:

com.sun.jndi.ldap.connect.timeout=5000

com.sun.jndi.ldap.read.timeout=1000

Issue Resolution: ECS Hang

The following exception was being masked in the ECS grey screen and was not visible until we applied the (Grey Screen of Death) GSOD patch:

Caused by: java.lang.LinkageError: neo/xredsys/auth/ECEPrincipal

at neo.xredsys.auth.JNDIPolicy14.getPermissions(JNDIPolicy14.java:46)

at java.security.Policy.implies(Policy.java:400)

at java.security.ProtectionDomain.implies(ProtectionDomain.java:195)

at java.security.AccessControlContext.checkPermission(AccessControlContext.java:249)

at java.security.AccessController.checkPermission(AccessController.java:427)

at neo.xredsys.api.IOObjectLoader.getCatalog(IOObjectLoader.java:1059)

[Stacktrace Truncated]

The error can be simulated ‘at will now’ by accessing the web application named ‘audit’. Since this web application has escenic-io.jar and escenic-common.jar inside its WEB-INF/lib folder, it loads a new instance of class neo.nursery.GlobalBus on access.

Ideally, there should be only on instance of class neo.nursery.GlobalBus in a JVM. The spinning up of a second instance causes a java.lang.LinkageError for the class neo.xredsys.auth.ECEPrincipal when an instanceof operation is invoked by JNDIPolicy14.getPermissions()(Line number 46). The exact same line of code on which we see the RMI TCP Connections hung in the thread dumps throws the Linkage error.