As mentioned in my last post, I have recently spent some time improving the Tomcat package on the Ubuntu and Debian Linux distributions. This post goes into more detail on those changes.
For quite some time I have been studying the Tomcat startup and shutdown procedures, and trying to improve the reliability, security, and user experience on Linux. I noticed that the Ubuntu and Debian init scripts were starting Tomcat via the JSVC service runner, which is known to shut down Tomcat abruptly. JSVC also implements unreliable restarts, such that it is possible for the init script to be unable to restart Tomcat. Instead, the init script should start Tomcat by running Tomcat’s catalina.sh script, and that script should in turn run the Java binary, so that the init script can properly handle any issues with starts, stops, or restarts.
However, JSVC is also used to allow binding the Tomcat JVM to privileged server port numbers (port numbers lower than 1024), such as the standard HTTP and HTTPS ports 80 and 443. Without JSVC, the Ubuntu Tomcat 6 init script would need to use some other mechanism to allow Tomcat to bind to privileged ports. As a replacement for binding via JSVC, I proposed using either Linux Capabilities (the setcap command), or the authbind package.
It was difficult to decide whether to use Linux Capabilities or authbind to allow the Tomcat JVM to bind to privileged ports. The most elegant way appeared to be to use Linux Capabilities, because that’s completely out of the execution path of the Tomcat JVM, and it’s built into Linux. But after trying it, I found that the Hotspot JVM currently does not support Linux Capabilities — I tried it on several different Linux distributions and found that Hotspot just doesn’t support it. I filed JDK bug 6919633 about that, which has been accepted by Sun as an original JDK bug report. This means we can’t use setcap with Sun Java, for now. Next, I turned to authbind. Authbind is written specifically to allow a process that is not running as root to bind to one or more privileged ports. It turned out to work well, and that’s now what the Debian Tomcat 6 package uses.
After the above research and testing, and now that I am a Debian committer, here are the changes that I just finished implementing, or helped to implement:
- JSVC is no longer used by the package. Instead, Tomcat is invoked via the stock catalina.sh script. Any script code in the init script that was used to generate the Java startup arguments has been removed because catalina.sh performs that work.
- Authbind is now the standard method for binding Tomcat to ports lower than 1024. There is a new option (disabled by default) to enable authbind in /etc/default/tomcat6. Enabling authbind means that Tomcat will be allowed to bind to privileged ports, if Tomcat or its webapps are configured to do so, and if authbind is configured to allow it. Enabling authbind also presumes the use of IPv4, since authbind only works with IPv4. The permission for Tomcat to use privileged port numbers is now configured at Tomcat package install time. Tomcat itself can be run entirely as an unprivileged user for this reason, tightening the scope of escalated privileges, improving server security.
- The security manager now defaults to the disabled state. It is commented that way in /etc/default/tomcat6. Having it disabled by default (but still supported in the code) means that fewer Tomcat package users will have difficulty getting Tomcat working in the default configuration.
- Reliable restarts are now implemented in the init script. A stop command sends a SIGTERM to the Tomcat JVM, running Tomcat’s shutdown hook. The init script monitors the process after sending this signal and will exit right away when Tomcat’s JVM exits. If Tomcat’s JVM does not exit, the init script waits for 20 seconds, monitoring the JVM process. At the end of the 20 seconds, the init script sends a SIGKILL to the JVM and waits up to 5 additional seconds for the JVM to quit. Of course, you can easily change these timeouts to suit your use of Apache Tomcat by modifying the number in the init script. If the JVM process will not quit (kernel I/O hang problem, or similar), the init script prints a failure message saying that Tomcat will not shut down. Tomcat can now be reliably and gracefully restarted via the init script. Graceful shutdowns mean that no webapp data is lost due to the Tomcat JVM being forcefully killed. Reliable restarts means that it is sufficient to script a “service tomcat6 restart” command, and you can be confident that the currently running Tomcat will be shut down gracefully and that a new Tomcat process will be started only after the first one is gone. Implementing graceful stops means that webapps have the appropriate amount of time to persist their application data to disk before the Tomcat JVM exits.
- Tomcat now sends STDOUT and STDERR to its usual stock log file, $CATALINA_BASE/logs/catalina.out (for example, /var/log/tomcat6/catalina.out in this package) alongside all of the other Tomcat log files, instead of logging STDOUT and STDERR to syslog.
- The Debian Tomcat 6 package is now up to date with the latest stable release of Tomcat, 6.0.24. This fixes a number of important issues, as I described in a recent blog entry .
These improvements have been implemented in time for the Ubuntu Lucid Lynx (10.04) and Debian Squeeze (6.0) releases.
MuleSoft’s contributions to Tomcat have significantly improved the reliability and usability of both the Ubuntu and Debian Tomcat 6 packages, and we hope to do the same for Tomcat 7, which is expected to be released around summer 2010. We’re happy to have contributed these open source changes!
