Apache Tomcat 5.5.x configuration for Stellar 2.0

We have switched to Apache tomcat 5.5.x from 4.1.x. Obtain the binary distribution from http://tomcat.apache.org/ Use gnu tar, solaris tar does not work correctly.

The following example is for the 2.2 release, we can use a separate installation for every major release.

cd /var/local
gtar -xvzf apache-tomcat-5.5.23.tar.gz
cp -rp /var/local/apache-tomcat-5.5.23 /var/local/main-tomcat-2.2
cp -rp /var/local/apache-tomcat-5.5.23 /var/local/pilot-tomcat-2.2
cp -rp /var/local/apache-tomcat-5.5.23 /var/local/sakai-tomcat-2.2

Configuration

Changes are needed to conf/context.xml, conf/server.xml, and bin/catalina.sh These are available here. For each of the above tomcat directories $CATALINA_HOME:

  • cp ./context.xml $CATALINA_HOME/conf/context.xml but you must now use the shared-build project for alternative setting, including database connections via JNDI for updated applications which use JNDI.
  • cp ./server.xml $CATALINA_HOME/conf/server.xml
  • cp ./catalina.sh $CATALINA_HOME/bin/catalina.sh
  • cp ./catalina.policy $CATALINA_HOME/conf/catalina.policy

In addition, conf/tomcat-users.xml may be modified to add a user to activate the manager webapp. Add a line like the following, and keep the username and password secret. We haven't done this in production.

  <user name="muser" password="mpass" roles="manager"/>

This $CATALINA_HOME/bin/catalina.sh includes the following lines near the top:

#JAVA_HOME="/var/local/java1.4"
#JAVA_HOME="/var/local/java1.5"
#JAVA_HOME="/var/local/java1.6"

CATALINA_OPTS="-server -Dfile.encoding=UTF-8 $CATALINA_OPTS"

CATALINA_TMPDIR=/usr/tmp/tomcats/temp1

umask 2
ulimit -H -n 4096
NOW=`/usr/bin/date +%Y%m%dT%H%M`

JAVA_OPTS="-d64 -Xms5000m -Xmx6000m"

and later 

CATALINA_OUT=/var/local/stellar-logs/tomcat.`/usr/bin/date +%Y%m%dT%H%M`
CATALINA_OUT="$CATALINA_HOME/logs/catalina.out"

Customize this. JAVA_HOME specifies which JRE to use; for Java 1.6 use /var/local/java1.6/ (see Stellar 2 Server Setup for installation of java 1.6). JAVA_OPTS specifies what options to use; the memory settings may need revision. Farther down, the CATALINA_OUT setting controls where output is sent. logs/catalina.out is the Tomcat 5.5 default, but Stellar uses the stellar-logs directory. CATALINA_TMPDIR defaults to $CATALINA_BASE/temp and grows to fill / so move to /usr/tmp where it will be cleaned out. Give multiple tomcats different directories.

For Tomcat 4 and Java 1.4, we used:

ulimit -H -n 4096
NOW=`/usr/bin/date +%Y%m%dT%H%M`
JAVA_OPTS=" -d64 -Xmx12000m -Xms10000m"
JAVA_OPTS="$JAVA_OPTS -Dsun.rmi.dgc.client.gcInterval=7200000"
JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled"
JAVA_OPTS="$JAVA_OPTS -XX:+UseAdaptiveSizePolicy -XX:NewSize=256m -XX:MaxPermSize=128m"
JAVA_OPTS="$JAVA_OPTS -XX:PermSize=64m -XX:MaxTenuringThreshold=3"
JAVA_OPTS="$JAVA_OPTS -XX:CMSInitiatingOccupancyFraction=40"
JAVA_OPTS="$JAVA_OPTS -XX:MinHeapFreeRatio=20"
JAVA_OPTS="$JAVA_OPTS -verbose:gc -Xloggc:/var/local/stellar-logs/gclog-$NOW"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintAdaptiveSizePolicy -XX:+PrintGCApplicationConcurrentTime"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"

For 1.5 the configuration currently being used is:

JAVA_OPTS="-d64 -Xms8000m -Xmx11000m"
JAVA_OPTS="$JAVA_OPTS -XX:NewSize=512m -XX:MaxNewSize=1024m"
JAVA_OPTS="$JAVA_OPTS  -XX:MaxPermSize=192m -XX:NewRatio=2"

JAVA_OPTS="$JAVA_OPTS -Dsun.rmi.dgc.client.gcInterval=7200000"
JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC"
JAVA_OPTS="$JAVA_OPTS -XX:SurvivorRatio=4  -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15"
JAVA_OPTS="$JAVA_OPTS -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing"
JAVA_OPTS="$JAVA_OPTS -XX:CMSIncrementalDutyCycleMin=2"
JAVA_OPTS="$JAVA_OPTS -XX:CMSIncrementalDutyCycle=10"
JAVA_OPTS="$JAVA_OPTS -XX:CMSIncrementalSafetyFactor=20"
JAVA_OPTS="$JAVA_OPTS -XX:MinHeapFreeRatio=20"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
JAVA_OPTS="$JAVA_OPTS -verbose:gc -Xloggc:/var/local/stellar-logs/gclog-$NOW "
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution -XX:+PrintAdaptiveSizePolicy"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationConcurrentTime"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"

For development, it is recommended that this unpacked, merged copy of tomcat be left as a clean copy. Copy it to a directory on /var/local to be used for your sandbox (etc).

Modify $CATALINA_HOME/conf/server.xml to configure AJP on whatever the apache httpd will use, e.g. to use port 8559 instead of 8009, use the following. maxThreads may also be increased or decreased depending on the capacity of the hardware.

Ports we use: main tomcat 8115/8119, pilot tomcat 8225/8229, sakai tomcat 8335/8339.

For each of main, pilot, and sakai; set the jvmRoute, tcpListenPort, tempDir, deployDir, and watchDir to be different from others in the cluster.

<Connector port="8559"
               enableLookups="false" protocol="AJP/1.3" maxThreads="500" />

The provided ./context.xml has been modified to allow Stellar to access other web application contexts. It may be possible and better to use deployXML="true" in the server.xml Host element and use META-INF/context.xml in the stellar webapp instead.

More

Install webapp for monitoring

The ISDA monitor.war webapp should be installed on each tomcat

cp monitor.war /var/local/main-tomcat-2.2/webapps/main-monitor.war
cp monitor.war /var/local/pilot-tomcat-2.2/webapps/pilot-monitor.war
cp monitor.war /var/local/sakai-tomcat-2.2/webapps/sakai-monitor.war

Security settings

We used to use a ./catalina.policy file; -security was made the default in catalina.sh and we used a custom ./catalina.policy file. However, since we are only installing trusted Stellar and Sakai webapps, and setting the security settings for the Sakai webapps is not normally done in Sakai, we will no longer use this.

Monitoring through JMX

For debugging and test monitoring, set the CATALINA_OPTS variable in your command shell before running tomcat as in the example below. Set the jmxremote.access and jmxremote.password files (access should contain monitorRole readonly and password should contain monitorRole whateverpassword, and set the permissions on the jmxremote.password file to 600. See http://tomcat.apache.org/tomcat-5.5-doc/monitoring.html.

setenv CATALINA_OPTS "-Dcom.sun.management.jmxremote \
    -Dcom.sun.management.jmxremote.port=8123 \
    -Dcom.sun.management.jmxremote.ssl=false \
    -Dcom.sun.management.jmxremote.authenticate=true \
    -Dcom.sun.management.jmxremote.password.file=conf/jmxremote.password \
    -Dcom.sun.management.jmxremote.access.file=conf/jmxremote.access"

Then use jconsole localhost:8123 to monitor the tomcat. SSL can be used to secure the connection, see http://java.sun.com/j2se/1.5.0/docs/guide/management/agent.html

However, much of the same information is available more conveniently through the Tomcat Manager App which is enabled by setting the username and password for the manager role in tomcat-users.xml as described above.

Automatically start on boot and stop on shutdown

ln -s /var/local/main-tomcat-2.2/bin/catalina.sh /etc/init.d/tomcat
ln -s /etc/init.d/tomcat /etc/rc0.d/K02tomcat
ln -s /etc/init.d/tomcat /etc/rc2.d/S98tomcat
ln -s /var/local/pilot-tomcat-2.2/bin/catalina.sh /etc/init.d/pilot-tomcat
ln -s /etc/init.d/pilot-tomcat /etc/rc0.d/K02pilot-tomcat
ln -s /etc/init.d/pilot-tomcat /etc/rc2.d/S98pilot-tomcat
ln -s /var/local/sakai-tomcat-2.2/bin/catalina.sh /etc/init.d/sakai-tomcat
ln -s /etc/init.d/sakai-tomcat /etc/rc0.d/K02sakai-tomcat
ln -s /etc/init.d/sakai-tomcat /etc/rc2.d/S98sakai-tomcat

$Revision: 31385 $ $Date: 2008-11-26 11:39:46 -0500 (Wed, 26 Nov 2008) $