Month: October 2010

In my previous post, I talked about what devops is and the need for devops tools around Tomcat. In this post, I want to go in depth around collaboration between devs and ops and how it applies to Tcat Server.

A key concept of the devops movement is that not only are there developers and operations, but there are also lots of people in between. Perhaps there is an ultimate authority on the operations team, but there are still many things you might want to enable developers or devops persons to do outside the operations team:

You wouldn’t necessarily be very excited about reliable, graceful app server restarts — unless you go to restart your server and it doesn’t restart, or unless the restart script corrupted your webapp data. There are times when a reasonably fast, fully reliable restart is a very important feature. Some examples:

• You found that your webapp has a new memory leak, and you just fixed it in development, just finished testing it, and you’re about to deploy the fixed version. But, first, you want to undeploy and restart the server to be completely sure the memory leak code is gone. While you’re doing this, your server is offline, and you want to get it serving again as soon as possible, so you run the restart command.. but it doesn’t stop. It stays running, and while you spend time trying to figure out why, your webapp is undeployed.
• You have more traffic on your site, and now your memory utilization is climbing, and you’ve decided you should increase your Tomcat’s heap memory allocation. You make the configuration change, and you run the restart command, which runs and happily completes, but Tomcat doesn’t budge — it’s still running. You spend the next hour or two trying to figure out why.
• You wrote a shell or batch script that changes your web site in a way that it also has to restart Tomcat to make all the right changes take effect. Your script runs Tomcat’s stop command, and then Tomcat’s start command. But, after using it a few times you find that the script isn’t successfully restarting Tomcat like that either due to an error. You spend lots of time looking for the cause of the problem…

If stock Tomcat restarts could both integrate well with the operating system, and also be fully reliable, it would save you time in cases like these, and it would allow you to automate more. We’ve cases like these with stock Tomcat, and we have improved server restarts as part of Tcat Server.

Let’s also define what we mean by reliable server restarts: When the administrator tells the server to restart, the server gracefully shuts down as quickly as reasonably possible, and immediately starts a new Tomcat JVM only after the original Tomcat JVM process is completely gone. When the administrator tells the server to stop, the restart script stops the JVM reliably. If for any reason the original Tomcat JVM was hung up on something during a stop operation and wasn’t going to exit, the restart script handles this case and (after waiting the appropriate/configurable timeout) kills the JVM process to make sure that it does not stay hanging and broken. When the restart script exits, it returns an exit code that reliably denotes whether the start, stop, or restart operation was successful. This exit code is a 100% reliable indicator of what the Tomcat JVM was doing at the time the restart script exited, so the exit code can be used by another script. The restart script also outputs textual information such that the person running the script can reliably see what is going on with the server.