What prevents Tomcat from shutting down?

Every now and then I come across a web application that won't shut down cleanly.

./catalina.sh start

works fine for starting up a Java process with the web application running in the Tomcat servlet container, but 

./catalina.sh stop

leaves the Java process running.

• Checking the catalina.out log file reveals that the Tomcat system has attempted to shut down.
• Http requests to the port(s) that Tomcat has been listening on indicate that the system is no longer accepting requests.
• Repeat attempts at ./catalina.sh stop indicate that Tomcat is no longer listening on the shutdown port.

In my experience, without fail, the underlying cause of this problem has been a non-daemon thread which has been started by a web application that has not been configured to should be shut down when Tomcat is shut down.

If you are not totally familiar with how the application hangs together, you may need some hints about where to look for the offending code.  Triggering a thread dump of the Tomcat process after the shutdown has failed should reveal a list including some non-daemon threads.  The names of the threads should give an indication of whether it is a JVM thread, a Tomcat thread, or a thread that has been started by the application.

Once you have established where the threads are being created, it is time to consider where and how to stop the threads.  Depending on the nature of the functionality being provided by the code, you may need to wait for the current processing to complete before terminating - otherwise the application would have used a daemon thread in the first place.

Dependency injection frameworks, such as Spring, will typically include shutdown hooks for any resources that they have responsibility for creating.