Hello Carl/Everyone,
Just wanted to let you know, that the issue is not resolved yet and the CF10 was down recently. I noticed the following error in the coldfusion-error.log
------------------------------------------------------------------------------------------ ----------------------
Jun 18, 2014 5:01:31 AM org.apache.catalina.core.StandardServer await
WARNING: StandardServer.await: Invalid command '' received
Jun 18, 2014 5:01:31 AM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Jun 18, 2014 5:01:31 AM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-bio-8500"]
Jun 18, 2014 5:02:01 AM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-bio-8012"]
Jun 18, 2014 5:02:37 AM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Exception in thread "Timer-5" Jun 18, 2014 5:31:35 AM org.apache.catalina.core.ContainerBase stopInternal
SEVERE: A child container failed during stop
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at org.apache.catalina.core.ContainerBase.stopInternal(ContainerBase.java:1156)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1540)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1530)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1057)
at sun.misc.URLClassPath.getResource(URLClassPath.java:199)
at java.net.URLClassLoader$1.run(URLClassLoader.java:358)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:412)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1675)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1533)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at java.sql.DriverManager.isDriverAllowed(DriverManager.java:463)
at java.sql.DriverManager.isDriverAllowed(DriverManager.java:455)
at java.sql.DriverManager.getDrivers(DriverManager.java:356)
at org.apache.catalina.loader.JdbcLeakPrevention.clearJdbcDriverRegistrations(JdbcLeakPreven tion.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.catalina.loader.WebappClassLoader.clearReferencesJdbc(WebappClassLoader.java:2 044)
at org.apache.catalina.loader.WebappClassLoader.clearReferences(WebappClassLoader.java:1964)
at org.apache.catalina.loader.WebappClassLoader.stop(WebappClassLoader.java:1876)
at org.apache.catalina.loader.WebappLoader.stopInternal(WebappLoader.java:659)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5499)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1540)
------------------------------------------------------------------------------------------ ----------------------
It seems that error occurred, when the CF was restarted by a script.
I wrote that script which monitor the memory usage. It sends me a message if there is an issue. I received the following last message this morning:
------------------------------------------------------------------------------------------ ----------------------
Apache ran out of memory at 06-18-2014 08:15:01 and restarted. There were 203 httpd processes at the time, with an average size of 42170 KB, for a total Apache usage of 8560 MB.
------------------------------------------------------------------------------------------ ----------------------
Additional info from apache logs and mod_jk
mod_jk.log:
------------------------------------------------------------------------------------------ ----------------------
[Wed Jun 18 05:18:29 2014] [60628:140485010884576] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1293): (cfusion) can't receive the response header message from tomcat, network problems or tomcat (127.0.0.1:8012) is down (errno=104)
[Wed Jun 18 05:18:29 2014] [60628:140485010884576] [error] ajp_get_reply::jk_ajp_common.c (2140): (cfusion) Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Jun 18 05:18:29 2014] [60628:140485010884576] [info] ajp_service::jk_ajp_common.c (2629): (cfusion) sending request to tomcat failed (recoverable), (attempt=1)
[Wed Jun 18 05:18:30 2014] [60747:140485010884576] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1293): (cfusion) can't receive the response header message from tomcat, network problems or tomcat (127.0.0.1:8012) is down (errno=104)
[Wed Jun 18 05:18:30 2014] [60747:140485010884576] [error] ajp_get_reply::jk_ajp_common.c (2140): (cfusion) Tomcat is down or refused connection. No response has been sent to the client (yet)
------------------------------------------------------------------------------------------ ----------------------
error_log:
------------------------------------------------------------------------------------------ ----------------------
[Wed Jun 18 05:18:29 2014] [error] (43)Identifier removed: apr_global_mutex_lock(jk_log_lock) failed
[Wed Jun 18 05:18:29 2014] [error] (43)Identifier removed: apr_global_mutex_unlock(jk_log_lock) failed
[Wed Jun 18 05:18:29 2014] [error] (43)Identifier removed: apr_global_mutex_lock(jk_log_lock) failed
[Wed Jun 18 05:18:29 2014] [error] (43)Identifier removed: apr_global_mutex_unlock(jk_log_lock) failed
[Wed Jun 18 05:18:29 2014] [error] (43)Identifier removed: apr_global_mutex_lock(jk_log_lock) failed
------------------------------------------------------------------------------------------ ----------------------
I am still investigating the issue and will let you know if I find something helpful.
Thanks!
Feb