free(): corrupted unsorted chunks

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

free(): corrupted unsorted chunks

Oliver Kohll
Hi,

I'm a Java developer getting a crash and error message 'free(): corrupted
unsorted chunks' in the log. this sounds like it could be a low level issue
in the core Java libraries so I'm posting here, but if it would be better
in another list (or even a different community) please let me know.

So I'm requesting help because I don't know how to debug this further. In
summary, the issue is

We're getting a crash nearly once a day on our production system, at a time
of day when it's most busy (mid morning). This crash hasn't been seen on
our development environment, which is the same apart from the server has
less memory and CPU resources.

The environment is:
* A VPS host
* Ubuntu LTS 18 (Bionic)
* The default Java 10 packaged as 11 I believe (
https://askubuntu.com/questions/1037646/why-is-openjdk-10-packaged-as-openjdk-11
)
* However, javac seems to be from JDK8:
root@server:~# file /etc/alternatives/java /etc/alternatives/javac
/etc/alternatives/java:  symbolic link to
/usr/lib/jvm/java-11-openjdk-amd64/bin/java
/etc/alternatives/javac: symbolic link to
/usr/lib/jvm/java-8-openjdk-amd64/bin/javac

We're running our webapp on Tomcat 8 (also the Ubuntu default). When this
happens, the server stops responding to HTTP requests (in fact the java
process has quit) and needs to be restarted.

The crash doesn't seem to correspond to a particular activity on the server
as far as I can see, though it's hard to tell as many people are logged in
at the same time.

Does anyone have any pointers of
a) how to track the bug further so I can report it, ideally without a big
impact on our live server
b) more immediately, any workarounds to try

In the meantime I will see if I can downgrade to JDK8 on the test server,
but I'm not sure if that's possible or whether it would make a difference.

Regards
Oliver
Reply | Threaded
Open this post in threaded view
|

Re: free(): corrupted unsorted chunks

Thomas Stüfe-2
Hi Oliver,

Is there a stack trace on stderr?

Thanks, Thomas





On Thu, Jun 21, 2018 at 1:59 PM, Oliver Kohll <[hidden email]> wrote:

> Hi,
>
> I'm a Java developer getting a crash and error message 'free(): corrupted
> unsorted chunks' in the log. this sounds like it could be a low level issue
> in the core Java libraries so I'm posting here, but if it would be better
> in another list (or even a different community) please let me know.
>
> So I'm requesting help because I don't know how to debug this further. In
> summary, the issue is
>
> We're getting a crash nearly once a day on our production system, at a time
> of day when it's most busy (mid morning). This crash hasn't been seen on
> our development environment, which is the same apart from the server has
> less memory and CPU resources.
>
> The environment is:
> * A VPS host
> * Ubuntu LTS 18 (Bionic)
> * The default Java 10 packaged as 11 I believe (
> https://askubuntu.com/questions/1037646/why-is-openjdk-10-packaged-as-openjdk-11
> )
> * However, javac seems to be from JDK8:
> root@server:~# file /etc/alternatives/java /etc/alternatives/javac
> /etc/alternatives/java:  symbolic link to
> /usr/lib/jvm/java-11-openjdk-amd64/bin/java
> /etc/alternatives/javac: symbolic link to
> /usr/lib/jvm/java-8-openjdk-amd64/bin/javac
>
> We're running our webapp on Tomcat 8 (also the Ubuntu default). When this
> happens, the server stops responding to HTTP requests (in fact the java
> process has quit) and needs to be restarted.
>
> The crash doesn't seem to correspond to a particular activity on the server
> as far as I can see, though it's hard to tell as many people are logged in
> at the same time.
>
> Does anyone have any pointers of
> a) how to track the bug further so I can report it, ideally without a big
> impact on our live server
> b) more immediately, any workarounds to try
>
> In the meantime I will see if I can downgrade to JDK8 on the test server,
> but I'm not sure if that's possible or whether it would make a difference.
>
> Regards
> Oliver
Reply | Threaded
Open this post in threaded view
|

Re: free(): corrupted unsorted chunks

Oliver Kohll
Hi Thomas,


Unfortunately not, that was /var/log/tomcat8/catalina.out, a snippet from
just before to afterwards when I restarted is below. Everything after
'free(): corrupted unsorted chunks', from 'Note: Picked up
JDK_JAVA_OPTIONS' on was only logged on restart. The warnings are normal.


The lines before were from my application. However they're different for
each crash.


Thanks

Oliver


---


Thu 2018/06/21 09:29:19.913|Warn|Scheduler|View CTG - (Fitting Sync) has
already been scheduled again, shouldn't be reset

Thu 2018/06/21 09:29:21.639|Info|ReportDownloader|User [username] exporting
report GMP from table n0) ff - GMP

Thu 2018/06/21 09:29:23.506|Info|Scheduler|Starting action for googlesyncer
view b4) Fitting Dates (Remedial Sync)

Thu 2018/06/21 09:29:23.911|Info|Scheduler|Starting action for googlesyncer
view b5) Survey Dates (Survey Sync)

Thu 2018/06/21 09:29:23.914|Warn|GoogleSyncer|1 other calendar syncs taking
place, delaying for a minute

Thu 2018/06/21 09:29:23.941|Warn|Scheduler|View (Survey Sync) has already
been scheduled again, shouldn't be reset

Thu 2018/06/21 09:29:24.106|Info|Scheduler|Starting action for
workflowrunner view m1.0) Customer Order.orders to lock

Thu 2018/06/21 09:29:24.338|Info|Scheduler|Starting action for googlesyncer
view a) leads.AGS (Lead Sync)

*free(): corrupted unsorted chunks*

NOTE: Picked up JDK_JAVA_OPTIONS:
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED

21-Jun-2018 09:35:56.096 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/var/lib/tomcat8/common/classes], exists: [false], isDirectory:
[false], canRead: [false]

21-Jun-2018 09:35:56.102 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/var/lib/tomcat8/common], exists: [false], isDirectory: [false],
canRead: [false]

21-Jun-2018 09:35:56.102 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/usr/share/tomcat8/common/classes], exists: [false],
isDirectory: [false], canRead: [false]

21-Jun-2018 09:35:56.102 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/usr/share/tomcat8/common], exists: [false], isDirectory:
[false], canRead: [false]

21-Jun-2018 09:35:56.104 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/var/lib/tomcat8/server/classes], exists: [false], isDirectory:
[false], canRead: [false]

21-Jun-2018 09:35:56.104 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/var/lib/tomcat8/server], exists: [false], isDirectory: [false],
canRead: [false]

21-Jun-2018 09:35:56.105 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/usr/share/tomcat8/server/classes], exists: [false],
isDirectory: [false], canRead: [false]

21-Jun-2018 09:35:56.105 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/usr/share/tomcat8/server], exists: [false], isDirectory:
[false], canRead: [false]

21-Jun-2018 09:35:56.105 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/var/lib/tomcat8/shared/classes], exists: [false], isDirectory:
[false], canRead: [false]

21-Jun-2018 09:35:56.105 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/var/lib/tomcat8/shared], exists: [false], isDirectory: [false],
canRead: [false]

21-Jun-2018 09:35:56.106 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/usr/share/tomcat8/shared/classes], exists: [false],
isDirectory: [false], canRead: [false]

21-Jun-2018 09:35:56.106 WARNING [main]
org.apache.catalina.startup.ClassLoaderFactory.validateFile Problem with
directory [/usr/share/tomcat8/shared], exists: [false], isDirectory:
[false], canRead: [false]

21-Jun-2018 09:35:56.795 WARNING [main]
org.apache.tomcat.util.digester.SetPropertiesRule.begin
[SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property
'debug' to '0' did not find a matching property.

21-Jun-2018 09:35:56.806 WARNING [main]
org.apache.tomcat.util.digester.SetPropertiesRule.begin
[SetPropertiesRule]{Server/Service/Engine/Host/Context/Realm/Realm} Setting
property 'debug' to '99' did not find a matching property.

21-Jun-2018 09:35:56.807 WARNING [main]
org.apache.tomcat.util.digester.SetPropertiesRule.begin
[SetPropertiesRule]{Server/Service/Engine/Host/Context/Realm/Realm} Setting
property 'digest' to 'MD5' did not find a matching property.

21-Jun-2018 09:35:56.809 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Server version:
  Apache Tomcat/8.5.30 (Ubuntu)

21-Jun-2018 09:35:56.810 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Server built:
  May 30 2018 13:37:13 UTC

21-Jun-2018 09:35:56.810 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Server number:
  8.5.30.0

21-Jun-2018 09:35:56.810 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log OS Name:
  Linux

21-Jun-2018 09:35:56.810 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log OS Version:
  4.15.13-x86_64-linode106

21-Jun-2018 09:35:56.811 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Architecture:
  amd64

21-Jun-2018 09:35:56.811 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Java Home:
  /usr/lib/jvm/java-11-openjdk-amd64

21-Jun-2018 09:35:56.811 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log JVM Version:
  10.0.1+10-Ubuntu-3ubuntu1

21-Jun-2018 09:35:56.811 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:
  Oracle Corporation

21-Jun-2018 09:35:56.811 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:
  /var/lib/tomcat8

21-Jun-2018 09:35:56.812 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:
  /usr/share/tomcat8

21-Jun-2018 09:35:56.812 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: --add-opens=java.base/java.lang=ALL-UNNAMED

21-Jun-2018 09:35:56.812 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: --add-opens=java.base/java.io=ALL-UNNAMED

21-Jun-2018 09:35:56.812 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED

21-Jun-2018 09:35:56.812 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument:
-Djava.util.logging.config.file=/var/lib/tomcat8/conf/logging.properties

21-Jun-2018 09:35:56.813 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager

21-Jun-2018 09:35:56.813 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Djava.awt.headless=true

21-Jun-2018 09:35:56.813 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Xmx8g

21-Jun-2018 09:35:56.813 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Djdk.tls.ephemeralDHKeySize=2048

21-Jun-2018 09:35:56.813 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources

21-Jun-2018 09:35:56.813 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027

21-Jun-2018 09:35:56.813 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Dignore.endorsed.dirs=

21-Jun-2018 09:35:56.814 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Dcatalina.base=/var/lib/tomcat8

21-Jun-2018 09:35:56.814 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Dcatalina.home=/usr/share/tomcat8

21-Jun-2018 09:35:56.814 INFO [main]
org.apache.catalina.startup.VersionLoggerListener.log Command line
argument: -Djava.io.tmpdir=/tmp/tomcat8-tomcat8-tmp

21-Jun-2018 09:35:56.814 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR
based Apache Tomcat Native library [1.2.16] using APR version [1.6.3].

21-Jun-2018 09:35:56.814 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
capabilities: IPv6 [true], sendfile [true], accept filters [false], random
[true].

21-Jun-2018 09:35:56.814 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL
configuration: useAprConnector [false], useOpenSSL [true]

21-Jun-2018 09:35:56.818 INFO [main]
org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL
successfully initialized [OpenSSL 1.1.0g  2 Nov 2017]

21-Jun-2018 09:35:56.858 INFO [main]
org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
["http-nio-8080"]

21-Jun-2018 09:35:56.876 INFO [main]
org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared
selector for servlet write/read

21-Jun-2018 09:35:56.884 INFO [main]
org.apache.coyote.http11.AbstractHttp11Protocol.configureUpgradeProtocol
The ["https-openssl-apr-8443"] connector has been configured to support
negotiation to [h2] via ALPN

21-Jun-2018 09:35:56.884 INFO [main]
org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
["https-openssl-apr-8443"]

21-Jun-2018 09:35:56.926 INFO [main]
org.apache.catalina.startup.Catalina.load Initialization processed in 773 ms

21-Jun-2018 09:35:56.997 INFO [main]
org.apache.catalina.core.StandardService.startInternal Starting service
[Catalina]

21-Jun-2018 09:35:56.998 INFO [main]
org.apache.catalina.core.StandardEngine.startInternal Starting Servlet
Engine: Apache Tomcat/8.5.30 (Ubuntu)

On 21 June 2018 at 15:31:12, Thomas Stüfe ([hidden email]) wrote:

Whereever you got the "free(): corrupted unsorted chunks" from should
contain actually more information. For an example, see:

https://www.unix.com/red-hat/233292-free-corrupted-unsorted-chunks.html

Thanks, Thomas

On Thu, Jun 21, 2018 at 4:27 PM, Oliver Kohll <[hidden email]>
wrote:
> Not that I can see I'm afraid - there's nothing else in either
> /var/log/tomcat8/catalina.out (or any of the other files in that
directory)
> or /var/log/syslog. I also can't see anything likely in /tmp
>
> I'm Googling how to get tomcat to create a stack trace to a file on a
crash,
> but the consensus seems to be they should be in catalina.out by default.
>
> On 21 June 2018 at 13:28:46, Thomas Stüfe ([hidden email])
wrote:

>
> Hi Oliver,
>
> Is there a stack trace on stderr?
>
> Thanks, Thomas
>
>
>
>
>
> On Thu, Jun 21, 2018 at 1:59 PM, Oliver Kohll <[hidden email]>
> wrote:
>> Hi,
>>
>> I'm a Java developer getting a crash and error message 'free():
corrupted
>> unsorted chunks' in the log. this sounds like it could be a low level
>> issue
>> in the core Java libraries so I'm posting here, but if it would be
better
>> in another list (or even a different community) please let me know.
>>
>> So I'm requesting help because I don't know how to debug this further.
In

>> summary, the issue is
>>
>> We're getting a crash nearly once a day on our production system, at a
>> time
>> of day when it's most busy (mid morning). This crash hasn't been seen on
>> our development environment, which is the same apart from the server has
>> less memory and CPU resources.
>>
>> The environment is:
>> * A VPS host
>> * Ubuntu LTS 18 (Bionic)
>> * The default Java 10 packaged as 11 I believe (
>>
>>
https://askubuntu.com/questions/1037646/why-is-openjdk-10-packaged-as-openjdk-11
>> )
>> * However, javac seems to be from JDK8:
>> root@server:~# file /etc/alternatives/java /etc/alternatives/javac
>> /etc/alternatives/java: symbolic link to
>> /usr/lib/jvm/java-11-openjdk-amd64/bin/java
>> /etc/alternatives/javac: symbolic link to
>> /usr/lib/jvm/java-8-openjdk-amd64/bin/javac
>>
>> We're running our webapp on Tomcat 8 (also the Ubuntu default). When
this
>> happens, the server stops responding to HTTP requests (in fact the java
>> process has quit) and needs to be restarted.
>>
>> The crash doesn't seem to correspond to a particular activity on the
>> server
>> as far as I can see, though it's hard to tell as many people are logged
in
>> at the same time.
>>
>> Does anyone have any pointers of
>> a) how to track the bug further so I can report it, ideally without a
big
>> impact on our live server
>> b) more immediately, any workarounds to try
>>
>> In the meantime I will see if I can downgrade to JDK8 on the test
server,
>> but I'm not sure if that's possible or whether it would make a
difference.
>>
>> Regards
>> Oliver
Reply | Threaded
Open this post in threaded view
|

Re: free(): corrupted unsorted chunks

Bernd Eckenfels-4
In reply to this post by Oliver Kohll
Are you using any native libraries in this VM, is there a hs_err or System core file? Can you run „ldd“ on the java launcher binary. Can you try a OpenJDK binary distribution independent of the OS compiled version?

Gruss
Bernd
--
http://bernd.eckenfels.net
________________________________
From: core-libs-dev <[hidden email]> on behalf of Oliver Kohll <[hidden email]>
Sent: Thursday, June 21, 2018 1:59:27 PM
To: [hidden email]
Subject: free(): corrupted unsorted chunks

Hi,

I'm a Java developer getting a crash and error message 'free(): corrupted
unsorted chunks' in the log. this sounds like it could be a low level issue
in the core Java libraries so I'm posting here, but if it would be better
in another list (or even a different community) please let me know.

So I'm requesting help because I don't know how to debug this further. In
summary, the issue is

We're getting a crash nearly once a day on our production system, at a time
of day when it's most busy (mid morning). This crash hasn't been seen on
our development environment, which is the same apart from the server has
less memory and CPU resources.

The environment is:
* A VPS host
* Ubuntu LTS 18 (Bionic)
* The default Java 10 packaged as 11 I believe (
https://askubuntu.com/questions/1037646/why-is-openjdk-10-packaged-as-openjdk-11
)
* However, javac seems to be from JDK8:
root@server:~# file /etc/alternatives/java /etc/alternatives/javac
/etc/alternatives/java:  symbolic link to
/usr/lib/jvm/java-11-openjdk-amd64/bin/java
/etc/alternatives/javac: symbolic link to
/usr/lib/jvm/java-8-openjdk-amd64/bin/javac

We're running our webapp on Tomcat 8 (also the Ubuntu default). When this
happens, the server stops responding to HTTP requests (in fact the java
process has quit) and needs to be restarted.

The crash doesn't seem to correspond to a particular activity on the server
as far as I can see, though it's hard to tell as many people are logged in
at the same time.

Does anyone have any pointers of
a) how to track the bug further so I can report it, ideally without a big
impact on our live server
b) more immediately, any workarounds to try

In the meantime I will see if I can downgrade to JDK8 on the test server,
but I'm not sure if that's possible or whether it would make a difference.

Regards
Oliver
Reply | Threaded
Open this post in threaded view
|

Re: free(): corrupted unsorted chunks

Oliver Kohll
Aha, I wonder if it's configured with the APR native library for SSL. I
will check out that and those things.

Thanks
Oliver

On 21 June 2018 at 21:43:24, Bernd Eckenfels ([hidden email]) wrote:

Are you using any native libraries in this VM, is there a hs_err or System
core file? Can you run „ldd“ on the java launcher binary. Can you try a
OpenJDK binary distribution independent of the OS compiled version?

Gruss
Bernd
--
http://bernd.eckenfels.net
Reply | Threaded
Open this post in threaded view
|

Re: free(): corrupted unsorted chunks

Oliver Kohll
Hi Bernd,

Many thanks for your suggestions. It looks like the native APR library was
the issue, at least it hasn't crashed yet after removing that library

https://tomcat.apache.org/tomcat-8.0-doc/apr.html (the libapr1 package in
Ubuntu)

and re-configuring the /etc/tomcat8/server.xml to use NIO for SSL,
replacing org.apache.coyote.http11.Http11AprProtocol with
org.apache.coyote.http11.Http11NioProtocol.

I appreciate your help, it saved my bacon!

Oliver

On 21 June 2018 at 23:34:44, Oliver Kohll ([hidden email]) wrote:

Aha, I wonder if it's configured with the APR native library for SSL. I
will check out that and those things.

Thanks
Oliver

On 21 June 2018 at 21:43:24, Bernd Eckenfels ([hidden email]) wrote:

Are you using any native libraries in this VM, is there a hs_err or System
core file? Can you run „ldd“ on the java launcher binary. Can you try a
OpenJDK binary distribution independent of the OS compiled version?

Gruss
Bernd
--
http://bernd.eckenfels.net