Additional JEPS 8182070 requirement: avoiding container OOM

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Additional JEPS 8182070 requirement: avoiding container OOM

Glyn Normington
I would like to mention an additional requirement for JEPS 8182070 (
http://openjdk.java.net/jeps/8182070): avoid the JVM hitting container OOM
by strictly bounding the amount of (physical) memory that the JVM consumes.
This may be implicit in the document, but I think it should be made an
explicit goal.

If a java application hits container OOM, no detailed diagnostics, such as
those associated with an OutOfMemoryError or a JVMTI resource exhaustion
event, are presented to the user, so the user finds it very difficult to
know how to fix the problem.

The Cloud Foundry OSS project has done quite a bit of work on this problem
and provides a couple of utilities which help when running a JVM in a
container:

* Java memory calculator ([1], [2]) to determine JVM memory settings,
* jvmkill JVMTI agent ([3]) to report detailed diagnostics on a resource
exhaustion event.

Regards,
Glyn

[1] https://github.com/cloudfoundry/java-buildpack-memory-calculator

[2] Design doc:
https://docs.google.com/document/d/1vlXBiwRIjwiVcbvUGYMrxx2Aw1RVAtxq3iuZ3UK2vXA/edit#heading=h.uy41ishpv9zc

[3] https://github.com/cloudfoundry/jvmkill
Reply | Threaded
Open this post in threaded view
|

Re: Additional JEPS 8182070 requirement: avoiding container OOM

Glyn Normington
I wonder if someone involved in JEPS 8182070 (Container aware Java) would
care to comment on the additional requirement described below?

On Mon, Nov 13, 2017 at 9:30 AM, Glyn Normington <[hidden email]>
wrote:

> I would like to mention an additional requirement for JEPS 8182070 (
> http://openjdk.java.net/jeps/8182070): avoid the JVM hitting container
> OOM by strictly bounding the amount of (physical) memory that the JVM
> consumes. This may be implicit in the document, but I think it should be
> made an explicit goal.
>
> If a java application hits container OOM, no detailed diagnostics, such as
> those associated with an OutOfMemoryError or a JVMTI resource exhaustion
> event, are presented to the user, so the user finds it very difficult to
> know how to fix the problem.
>
> The Cloud Foundry OSS project has done quite a bit of work on this problem
> and provides a couple of utilities which help when running a JVM in a
> container:
>
> * Java memory calculator ([1], [2]) to determine JVM memory settings,
> * jvmkill JVMTI agent ([3]) to report detailed diagnostics on a resource
> exhaustion event.
>
> Regards,
> Glyn
>
> [1] https://github.com/cloudfoundry/java-buildpack-memory-calculator
>
> [2] Design doc: https://docs.google.com/document/d/
> 1vlXBiwRIjwiVcbvUGYMrxx2Aw1RVAtxq3iuZ3UK2vXA/edit#heading=h.uy41ishpv9zc
>
> [3] https://github.com/cloudfoundry/jvmkill
>



--
Regards,
Glyn
Reply | Threaded
Open this post in threaded view
|

Re: Additional JEPS 8182070 requirement: avoiding container OOM

Bob Vandette
Hi Glyn,

The JEP you mention is going to be updated in a week or so to refocus it’s goals
on monitoring of the JVM running in containers.  This is due to the fact that I was
able to split out the work of dynamically configuring the VM based on container
settings and integrate this work into JDK 10.  This work was integrated under jira issue
https://bugs.openjdk.java.net/browse/JDK-8146115 <https://bugs.openjdk.java.net/browse/JDK-8146115>.

Although this change does not completely address your requirement, it should help.
This change causes the JVM to correctly configure the number of threads and memory
allocations based on the resources configured in the container.  This change also
introduces a new JVM option (-XX:ActiveProcessorCount=xx) which allows the user
to specify how many processors the JVM should use.  We will now honor cpu shares, quotas
in addition to cpusets for determining the number of threads that the JVM will use.

Prior to this change, I also added https://bugs.openjdk.java.net/browse/JDK-8186315 <https://bugs.openjdk.java.net/browse/JDK-8186315>.
This change allows the user to specify the percentage of container memory that
should be used by the Java Heap.  This allows the users to reserve memory outside
of the Java heap for other classes of memory allocations (C-Heap for example).

Both of these changes have been integrated into JDK 10 and are available in the
latest early access release of Linux x64 (http://jdk.java.net/10/ <http://jdk.java.net/10/>).

I will talk to the Hotspot runtime folks about your requirement but other than these
options, have you considered configuring your containers to use some swap space
in addition to memory configurations so that temporary spikes in memory consumption
won’t cause an OOM ?  There are also ways of disabling the OOM killer in containers.
If we get a failed memory allocation, we will try to deliver a Java OutOfMemory
exception which would at least allow you to figure out what went wrong.  I haven’t looked
into this yet but it is something worth considering.

Bob.



> On Dec 13, 2017, at 4:13 AM, Glyn Normington <[hidden email]> wrote:
>
> I wonder if someone involved in JEPS 8182070 (Container aware Java) would
> care to comment on the additional requirement described below?
>
> On Mon, Nov 13, 2017 at 9:30 AM, Glyn Normington <[hidden email]>
> wrote:
>
>> I would like to mention an additional requirement for JEPS 8182070 (
>> http://openjdk.java.net/jeps/8182070): avoid the JVM hitting container
>> OOM by strictly bounding the amount of (physical) memory that the JVM
>> consumes. This may be implicit in the document, but I think it should be
>> made an explicit goal.
>>
>> If a java application hits container OOM, no detailed diagnostics, such as
>> those associated with an OutOfMemoryError or a JVMTI resource exhaustion
>> event, are presented to the user, so the user finds it very difficult to
>> know how to fix the problem.
>>
>> The Cloud Foundry OSS project has done quite a bit of work on this problem
>> and provides a couple of utilities which help when running a JVM in a
>> container:
>>
>> * Java memory calculator ([1], [2]) to determine JVM memory settings,
>> * jvmkill JVMTI agent ([3]) to report detailed diagnostics on a resource
>> exhaustion event.
>>
>> Regards,
>> Glyn
>>
>> [1] https://github.com/cloudfoundry/java-buildpack-memory-calculator
>>
>> [2] Design doc: https://docs.google.com/document/d/
>> 1vlXBiwRIjwiVcbvUGYMrxx2Aw1RVAtxq3iuZ3UK2vXA/edit#heading=h.uy41ishpv9zc
>>
>> [3] https://github.com/cloudfoundry/jvmkill
>>
>
>
>
> --
> Regards,
> Glyn

Reply | Threaded
Open this post in threaded view
|

Re: Additional JEPS 8182070 requirement: avoiding container OOM

Glyn Normington
Hi Bob

Thanks very much for that information. Some comments inline. This one could
be relevant to the refocussed JEP: the JVM should monitor its approach to
the container memory limit and trigger "out of memory" processing at some
threshold close to the limit.

Regards,
Glyn

On Wed, Dec 13, 2017 at 2:58 PM, Bob Vandette <[hidden email]>
wrote:

> Hi Glyn,
>
> The JEP you mention is going to be updated in a week or so to refocus it’s
> goals
> on monitoring of the JVM running in containers.  This is due to the fact
> that I was
> able to split out the work of dynamically configuring the VM based on
> container
> settings and integrate this work into JDK 10.  This work was integrated
> under jira issue
> https://bugs.openjdk.java.net/browse/JDK-8146115.
>
> Although this change does not completely address your requirement, it
> should help.
> This change causes the JVM to correctly configure the number of threads
> and memory
> allocations based on the resources configured in the container.  This
> change also
> introduces a new JVM option (-XX:ActiveProcessorCount=xx) which allows
> the user
> to specify how many processors the JVM should use.  We will now honor cpu
> shares, quotas
> in addition to cpusets for determining the number of threads that the JVM
> will use.
>

Yes, that does sound like a useful set of changes.


>
> Prior to this change, I also added https://bugs.openjdk.jav
> a.net/browse/JDK-8186315.
> This change allows the user to specify the percentage of container memory
> that
> should be used by the Java Heap.  This allows the users to reserve memory
> outside
> of the Java heap for other classes of memory allocations (C-Heap for
> example).
>
> Both of these changes have been integrated into JDK 10 and are available
> in the
> latest early access release of Linux x64 (http://jdk.java.net/10/).
>

How interesting! The previous version of our Java memory calculator allowed
the heap size to be specified as a proportion of container memory, but we
could only guess at a default proportion and users found it quite hard to
know what value to use when they needed to increase or decrease heap size.

The current version adopts a different approach: we try to calculate JVM
memory consumption (apart from heap) from the JVM options and then subtract
that value from the container memory to get the maximum heap size. This
approach suffers from the difficulty of predicting the memory consumption
of the JVM with any accuracy (see my recent thread entitled 'Excessive "GC"
memory area size'). The up-side of the new approach is that the user can
override the behaviour using standard JVM options rather than proportions.


>
> I will talk to the Hotspot runtime folks about your requirement
>

Thanks very much.


> but other than these
> options, have you considered configuring your containers to use some swap
> space
> in addition to memory configurations so that temporary spikes in memory
> consumption
> won’t cause an OOM ?
>

We talked about this but we need to bound the amount of swap space because
our environment is a PaaS and we can't allow users to consume more than
their fair share of system resources. If we did enable a bounded swap
space, I think we'd hit the same issue when the RAM+swap limit was exceeded.


> There are also ways of disabling the OOM killer in containers.
>

We actually tried that. We started the container with the OOM killer
disabled and when the container hit OOM, we gathered diagnostics and
re-enabled the OOM killer to terminate the container. The main issue with
that is the container-level diagnostics (number of committed pages of RAM
etc.) don't tell the Java programmer what JVM options need changing to
avoid the problem in future.

A more sophisticated approach along those lines would be to disable the OOM
killer and when the container *nears* its limit somehow trigger the JVM to
do "out of memory" processing. Then our JVMTI agent would be driven and
produce plenty of diagnostics suitable for a Java programmer. This is a bit
speculative though as it requires fairly deep changes to the container
configuration as well as solving the problem of how to trigger the JVM to
do its "out of memory" processing. Maybe a more practical solution there is
for the JVM itself to monitor its approach to the container memory limit
and trigger "out of memory" processing at some threshold close to the
limit. This could be relevant to the JEP.


> If we get a failed memory allocation, we will try to deliver a Java
> OutOfMemory
> exception which would at least allow you to figure out what went wrong.  I
> haven’t looked
> into this yet but it is something worth considering.
>

Yeah, OutOfMemory exceptions provide some useful information, although we
found it was insufficiently detailed which is why we wrote a JVMTI agent to
dump out detailed diagnostics.


>
> Bob.
>
>
>
> On Dec 13, 2017, at 4:13 AM, Glyn Normington <[hidden email]>
> wrote:
>
> I wonder if someone involved in JEPS 8182070 (Container aware Java) would
> care to comment on the additional requirement described below?
>
> On Mon, Nov 13, 2017 at 9:30 AM, Glyn Normington <[hidden email]>
> wrote:
>
> I would like to mention an additional requirement for JEPS 8182070 (
> http://openjdk.java.net/jeps/8182070): avoid the JVM hitting container
> OOM by strictly bounding the amount of (physical) memory that the JVM
> consumes. This may be implicit in the document, but I think it should be
> made an explicit goal.
>
> If a java application hits container OOM, no detailed diagnostics, such as
> those associated with an OutOfMemoryError or a JVMTI resource exhaustion
> event, are presented to the user, so the user finds it very difficult to
> know how to fix the problem.
>
> The Cloud Foundry OSS project has done quite a bit of work on this problem
> and provides a couple of utilities which help when running a JVM in a
> container:
>
> * Java memory calculator ([1], [2]) to determine JVM memory settings,
> * jvmkill JVMTI agent ([3]) to report detailed diagnostics on a resource
> exhaustion event.
>
> Regards,
> Glyn
>
> [1] https://github.com/cloudfoundry/java-buildpack-memory-calculator
>
> [2] Design doc: https://docs.google.com/document/d/
> 1vlXBiwRIjwiVcbvUGYMrxx2Aw1RVAtxq3iuZ3UK2vXA/edit#heading=h.uy41ishpv9zc
>
> [3] https://github.com/cloudfoundry/jvmkill
>
>
>
>
> --
> Regards,
> Glyn
>
>
>


--
Regards,
Glyn
Reply | Threaded
Open this post in threaded view
|

Re: Additional JEPS 8182070 requirement: avoiding container OOM

Bob Vandette

> On Dec 13, 2017, at 11:01 AM, Glyn Normington <[hidden email]> wrote:
>
> Hi Bob
>
> Thanks very much for that information. Some comments inline. This one could be relevant to the refocussed JEP: the JVM should monitor its approach to the container memory limit and trigger "out of memory" processing at some threshold close to the limit.

Memory allocations in hotspot are difficult to track since there are many different allocators
and we don’t control them all.  Thread stacks get dynamically allocated upon use, the C++ runtime allocates
memory for native code and third party JNI library could be calling malloc.  These are all in addition
to the allocators we do control - Java Heap and internal Memory Arenas for runtime metadata.

One possible solution is to at least attempt threshold checking during allocations we
do control and to do OOM processing at that time.  This wouldn’t catch every case but could
increase the robustness of the Java runtime.  With the change I introduced in JDK-8146155,
we now at least have a way of finding out how much free memory is available in the container.

Bob.

>
> Regards,
> Glyn
>
> On Wed, Dec 13, 2017 at 2:58 PM, Bob Vandette <[hidden email] <mailto:[hidden email]>> wrote:
> Hi Glyn,
>
> The JEP you mention is going to be updated in a week or so to refocus it’s goals
> on monitoring of the JVM running in containers.  This is due to the fact that I was
> able to split out the work of dynamically configuring the VM based on container
> settings and integrate this work into JDK 10.  This work was integrated under jira issue
> https://bugs.openjdk.java.net/browse/JDK-8146115 <https://bugs.openjdk.java.net/browse/JDK-8146115>.
>
> Although this change does not completely address your requirement, it should help.
> This change causes the JVM to correctly configure the number of threads and memory
> allocations based on the resources configured in the container.  This change also
> introduces a new JVM option (-XX:ActiveProcessorCount=xx) which allows the user
> to specify how many processors the JVM should use.  We will now honor cpu shares, quotas
> in addition to cpusets for determining the number of threads that the JVM will use.
>
> Yes, that does sound like a useful set of changes.
>  
>
> Prior to this change, I also added https://bugs.openjdk.java.net/browse/JDK-8186315 <https://bugs.openjdk.java.net/browse/JDK-8186315>.
> This change allows the user to specify the percentage of container memory that
> should be used by the Java Heap.  This allows the users to reserve memory outside
> of the Java heap for other classes of memory allocations (C-Heap for example).
>
> Both of these changes have been integrated into JDK 10 and are available in the
> latest early access release of Linux x64 (http://jdk.java.net/10/ <http://jdk.java.net/10/>).
>
> How interesting! The previous version of our Java memory calculator allowed the heap size to be specified as a proportion of container memory, but we could only guess at a default proportion and users found it quite hard to know what value to use when they needed to increase or decrease heap size.
>
> The current version adopts a different approach: we try to calculate JVM memory consumption (apart from heap) from the JVM options and then subtract that value from the container memory to get the maximum heap size. This approach suffers from the difficulty of predicting the memory consumption of the JVM with any accuracy (see my recent thread entitled 'Excessive "GC" memory area size'). The up-side of the new approach is that the user can override the behaviour using standard JVM options rather than proportions.
>  
>
> I will talk to the Hotspot runtime folks about your requirement
>
> Thanks very much.
>  
> but other than these
> options, have you considered configuring your containers to use some swap space
> in addition to memory configurations so that temporary spikes in memory consumption
> won’t cause an OOM ?
>
> We talked about this but we need to bound the amount of swap space because our environment is a PaaS and we can't allow users to consume more than their fair share of system resources. If we did enable a bounded swap space, I think we'd hit the same issue when the RAM+swap limit was exceeded.
>  
> There are also ways of disabling the OOM killer in containers.
>
> We actually tried that. We started the container with the OOM killer disabled and when the container hit OOM, we gathered diagnostics and re-enabled the OOM killer to terminate the container. The main issue with that is the container-level diagnostics (number of committed pages of RAM etc.) don't tell the Java programmer what JVM options need changing to avoid the problem in future.
>
> A more sophisticated approach along those lines would be to disable the OOM killer and when the container *nears* its limit somehow trigger the JVM to do "out of memory" processing. Then our JVMTI agent would be driven and produce plenty of diagnostics suitable for a Java programmer. This is a bit speculative though as it requires fairly deep changes to the container configuration as well as solving the problem of how to trigger the JVM to do its "out of memory" processing. Maybe a more practical solution there is for the JVM itself to monitor its approach to the container memory limit and trigger "out of memory" processing at some threshold close to the limit. This could be relevant to the JEP.
>  
> If we get a failed memory allocation, we will try to deliver a Java OutOfMemory
> exception which would at least allow you to figure out what went wrong.  I haven’t looked
> into this yet but it is something worth considering.
>
> Yeah, OutOfMemory exceptions provide some useful information, although we found it was insufficiently detailed which is why we wrote a JVMTI agent to dump out detailed diagnostics.
>  
>
> Bob.
>
>
>
>> On Dec 13, 2017, at 4:13 AM, Glyn Normington <[hidden email] <mailto:[hidden email]>> wrote:
>>
>> I wonder if someone involved in JEPS 8182070 (Container aware Java) would
>> care to comment on the additional requirement described below?
>>
>> On Mon, Nov 13, 2017 at 9:30 AM, Glyn Normington <[hidden email] <mailto:[hidden email]>>
>> wrote:
>>
>>> I would like to mention an additional requirement for JEPS 8182070 (
>>> http://openjdk.java.net/jeps/8182070 <http://openjdk.java.net/jeps/8182070>): avoid the JVM hitting container
>>> OOM by strictly bounding the amount of (physical) memory that the JVM
>>> consumes. This may be implicit in the document, but I think it should be
>>> made an explicit goal.
>>>
>>> If a java application hits container OOM, no detailed diagnostics, such as
>>> those associated with an OutOfMemoryError or a JVMTI resource exhaustion
>>> event, are presented to the user, so the user finds it very difficult to
>>> know how to fix the problem.
>>>
>>> The Cloud Foundry OSS project has done quite a bit of work on this problem
>>> and provides a couple of utilities which help when running a JVM in a
>>> container:
>>>
>>> * Java memory calculator ([1], [2]) to determine JVM memory settings,
>>> * jvmkill JVMTI agent ([3]) to report detailed diagnostics on a resource
>>> exhaustion event.
>>>
>>> Regards,
>>> Glyn
>>>
>>> [1] https://github.com/cloudfoundry/java-buildpack-memory-calculator <https://github.com/cloudfoundry/java-buildpack-memory-calculator>
>>>
>>> [2] Design doc: https://docs.google.com/document/d/ <https://docs.google.com/document/d/>
>>> 1vlXBiwRIjwiVcbvUGYMrxx2Aw1RVAtxq3iuZ3UK2vXA/edit#heading=h.uy41ishpv9zc
>>>
>>> [3] https://github.com/cloudfoundry/jvmkill <https://github.com/cloudfoundry/jvmkill>
>>>
>>
>>
>>
>> --
>> Regards,
>> Glyn
>
>
>
>
> --
> Regards,
> Glyn

Reply | Threaded
Open this post in threaded view
|

Re: Additional JEPS 8182070 requirement: avoiding container OOM

Glyn Normington
On Wed, Dec 13, 2017 at 4:51 PM, Bob Vandette <[hidden email]>
wrote:

>
> On Dec 13, 2017, at 11:01 AM, Glyn Normington <[hidden email]>
> wrote:
>
> Hi Bob
>
> Thanks very much for that information. Some comments inline. This one
> could be relevant to the refocussed JEP: the JVM should monitor its
> approach to the container memory limit and trigger "out of memory"
> processing at some threshold close to the limit.
>
>
> Memory allocations in hotspot are difficult to track since there are many
> different allocators
> and we don’t control them all.  Thread stacks get dynamically allocated
> upon use, the C++ runtime allocates
> memory for native code and third party JNI library could be calling
> malloc.  These are all in addition
> to the allocators we do control - Java Heap and internal Memory Arenas for
> runtime metadata.
>
> One possible solution is to at least attempt threshold checking during
> allocations we
> do control and to do OOM processing at that time.  This wouldn’t catch
> every case but could
> increase the robustness of the Java runtime.  With the change I introduced
> in JDK-8146155,
> we now at least have a way of finding out how much free memory is
> available in the container.
>

Another solution you might like to mull over is for the JVM to poll the
memory cgroup value memory.usage_in_bytes (assuming the memory cgroup is
mounted read-only inside the container) and trigger "out of memory" on some
threshold.


>
> Bob.
>
>
> Regards,
> Glyn
>
> On Wed, Dec 13, 2017 at 2:58 PM, Bob Vandette <[hidden email]>
> wrote:
>
>> Hi Glyn,
>>
>> The JEP you mention is going to be updated in a week or so to refocus
>> it’s goals
>> on monitoring of the JVM running in containers.  This is due to the fact
>> that I was
>> able to split out the work of dynamically configuring the VM based on
>> container
>> settings and integrate this work into JDK 10.  This work was integrated
>> under jira issue
>> https://bugs.openjdk.java.net/browse/JDK-8146115.
>>
>> Although this change does not completely address your requirement, it
>> should help.
>> This change causes the JVM to correctly configure the number of threads
>> and memory
>> allocations based on the resources configured in the container.  This
>> change also
>> introduces a new JVM option (-XX:ActiveProcessorCount=xx) which allows
>> the user
>> to specify how many processors the JVM should use.  We will now honor cpu
>> shares, quotas
>> in addition to cpusets for determining the number of threads that the JVM
>> will use.
>>
>
> Yes, that does sound like a useful set of changes.
>
>
>>
>> Prior to this change, I also added https://bugs.openjdk.jav
>> a.net/browse/JDK-8186315.
>> This change allows the user to specify the percentage of container memory
>> that
>> should be used by the Java Heap.  This allows the users to reserve memory
>> outside
>> of the Java heap for other classes of memory allocations (C-Heap for
>> example).
>>
>> Both of these changes have been integrated into JDK 10 and are available
>> in the
>> latest early access release of Linux x64 (http://jdk.java.net/10/).
>>
>
> How interesting! The previous version of our Java memory calculator
> allowed the heap size to be specified as a proportion of container memory,
> but we could only guess at a default proportion and users found it quite
> hard to know what value to use when they needed to increase or decrease
> heap size.
>
> The current version adopts a different approach: we try to calculate JVM
> memory consumption (apart from heap) from the JVM options and then subtract
> that value from the container memory to get the maximum heap size. This
> approach suffers from the difficulty of predicting the memory consumption
> of the JVM with any accuracy (see my recent thread entitled 'Excessive "GC"
> memory area size'). The up-side of the new approach is that the user can
> override the behaviour using standard JVM options rather than proportions.
>
>
>>
>> I will talk to the Hotspot runtime folks about your requirement
>>
>
> Thanks very much.
>
>
>> but other than these
>> options, have you considered configuring your containers to use some swap
>> space
>> in addition to memory configurations so that temporary spikes in memory
>> consumption
>> won’t cause an OOM ?
>>
>
> We talked about this but we need to bound the amount of swap space because
> our environment is a PaaS and we can't allow users to consume more than
> their fair share of system resources. If we did enable a bounded swap
> space, I think we'd hit the same issue when the RAM+swap limit was exceeded.
>
>
>> There are also ways of disabling the OOM killer in containers.
>>
>
> We actually tried that. We started the container with the OOM killer
> disabled and when the container hit OOM, we gathered diagnostics and
> re-enabled the OOM killer to terminate the container. The main issue with
> that is the container-level diagnostics (number of committed pages of RAM
> etc.) don't tell the Java programmer what JVM options need changing to
> avoid the problem in future.
>
> A more sophisticated approach along those lines would be to disable the
> OOM killer and when the container *nears* its limit somehow trigger the JVM
> to do "out of memory" processing. Then our JVMTI agent would be driven and
> produce plenty of diagnostics suitable for a Java programmer. This is a bit
> speculative though as it requires fairly deep changes to the container
> configuration as well as solving the problem of how to trigger the JVM to
> do its "out of memory" processing. Maybe a more practical solution there is
> for the JVM itself to monitor its approach to the container memory limit
> and trigger "out of memory" processing at some threshold close to the
> limit. This could be relevant to the JEP.
>
>
>> If we get a failed memory allocation, we will try to deliver a Java
>> OutOfMemory
>> exception which would at least allow you to figure out what went wrong.
>> I haven’t looked
>> into this yet but it is something worth considering.
>>
>
> Yeah, OutOfMemory exceptions provide some useful information, although we
> found it was insufficiently detailed which is why we wrote a JVMTI agent to
> dump out detailed diagnostics.
>
>
>>
>> Bob.
>>
>>
>>
>> On Dec 13, 2017, at 4:13 AM, Glyn Normington <[hidden email]>
>> wrote:
>>
>> I wonder if someone involved in JEPS 8182070 (Container aware Java) would
>> care to comment on the additional requirement described below?
>>
>> On Mon, Nov 13, 2017 at 9:30 AM, Glyn Normington <[hidden email]>
>> wrote:
>>
>> I would like to mention an additional requirement for JEPS 8182070 (
>> http://openjdk.java.net/jeps/8182070): avoid the JVM hitting container
>> OOM by strictly bounding the amount of (physical) memory that the JVM
>> consumes. This may be implicit in the document, but I think it should be
>> made an explicit goal.
>>
>> If a java application hits container OOM, no detailed diagnostics, such as
>> those associated with an OutOfMemoryError or a JVMTI resource exhaustion
>> event, are presented to the user, so the user finds it very difficult to
>> know how to fix the problem.
>>
>> The Cloud Foundry OSS project has done quite a bit of work on this problem
>> and provides a couple of utilities which help when running a JVM in a
>> container:
>>
>> * Java memory calculator ([1], [2]) to determine JVM memory settings,
>> * jvmkill JVMTI agent ([3]) to report detailed diagnostics on a resource
>> exhaustion event.
>>
>> Regards,
>> Glyn
>>
>> [1] https://github.com/cloudfoundry/java-buildpack-memory-calculator
>>
>> [2] Design doc: https://docs.google.com/document/d/
>> 1vlXBiwRIjwiVcbvUGYMrxx2Aw1RVAtxq3iuZ3UK2vXA/edit#heading=h.uy41ishpv9zc
>>
>> [3] https://github.com/cloudfoundry/jvmkill
>>
>>
>>
>>
>> --
>> Regards,
>> Glyn
>>
>>
>>
>
>
> --
> Regards,
> Glyn
>
>
>


--
Regards,
Glyn
Reply | Threaded
Open this post in threaded view
|

Re: Additional JEPS 8182070 requirement: avoiding container OOM

Bob Vandette

> On Dec 13, 2017, at 12:00 PM, Glyn Normington <[hidden email]> wrote:
>
> On Wed, Dec 13, 2017 at 4:51 PM, Bob Vandette <[hidden email] <mailto:[hidden email]>> wrote:
>
>> On Dec 13, 2017, at 11:01 AM, Glyn Normington <[hidden email] <mailto:[hidden email]>> wrote:
>>
>> Hi Bob
>>
>> Thanks very much for that information. Some comments inline. This one could be relevant to the refocussed JEP: the JVM should monitor its approach to the container memory limit and trigger "out of memory" processing at some threshold close to the limit.
>
> Memory allocations in hotspot are difficult to track since there are many different allocators
> and we don’t control them all.  Thread stacks get dynamically allocated upon use, the C++ runtime allocates
> memory for native code and third party JNI library could be calling malloc.  These are all in addition
> to the allocators we do control - Java Heap and internal Memory Arenas for runtime metadata.
>
> One possible solution is to at least attempt threshold checking during allocations we
> do control and to do OOM processing at that time.  This wouldn’t catch every case but could
> increase the robustness of the Java runtime.  With the change I introduced in JDK-8146155,
> we now at least have a way of finding out how much free memory is available in the container.
>
> Another solution you might like to mull over is for the JVM to poll the memory cgroup value memory.usage_in_bytes (assuming the memory cgroup is mounted read-only inside the container) and trigger "out of memory" on some threshold.

That is exactly what I was referring to in "we now at least have a way of finding out how much free memory is available in the container."

I added several cgroup access functions inside hotspot (http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp>)
including these two.

/* memory_usage_in_bytes <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l456>
 * <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l457>
 * Return the amount of used memory for this process. <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l458>
 * <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l459>
 * return: <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l460>
 *    memory usage in bytes or <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l461>
 *    -1 for unlimited <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l462>
 *    OSCONTAINER_ERROR for not supported <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l463>
 */ <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l464>
jlong OSContainer::memory_usage_in_bytes() { <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l465>
  GET_CONTAINER_INFO(jlong, memory, "/memory.usage_in_bytes", <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l466>
                     "Memory Usage is: " JLONG_FORMAT, JLONG_FORMAT, memusage); <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l467>
  return memusage; <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l468>
} <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l469>
 <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l470>
/* memory_max_usage_in_bytes <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l471>
 * <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l472>
 * Return the maximum amount of used memory for this process. <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l473>
 * <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l474>
 * return: <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l475>
 *    max memory usage in bytes or <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l476>
 *    OSCONTAINER_ERROR for not supported <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l477>
 */ <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l478>
jlong OSContainer::memory_max_usage_in_bytes() { <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l479>
  GET_CONTAINER_INFO(jlong, memory, "/memory.max_usage_in_bytes", <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l480>
                     "Maximum Memory Usage is: " JLONG_FORMAT, JLONG_FORMAT, memmaxusage); <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l481>
  return memmaxusage; <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l482>
}

Bob.

>  
>
> Bob.
>
>>
>> Regards,
>> Glyn
>>
>> On Wed, Dec 13, 2017 at 2:58 PM, Bob Vandette <[hidden email] <mailto:[hidden email]>> wrote:
>> Hi Glyn,
>>
>> The JEP you mention is going to be updated in a week or so to refocus it’s goals
>> on monitoring of the JVM running in containers.  This is due to the fact that I was
>> able to split out the work of dynamically configuring the VM based on container
>> settings and integrate this work into JDK 10.  This work was integrated under jira issue
>> https://bugs.openjdk.java.net/browse/JDK-8146115 <https://bugs.openjdk.java.net/browse/JDK-8146115>.
>>
>> Although this change does not completely address your requirement, it should help.
>> This change causes the JVM to correctly configure the number of threads and memory
>> allocations based on the resources configured in the container.  This change also
>> introduces a new JVM option (-XX:ActiveProcessorCount=xx) which allows the user
>> to specify how many processors the JVM should use.  We will now honor cpu shares, quotas
>> in addition to cpusets for determining the number of threads that the JVM will use.
>>
>> Yes, that does sound like a useful set of changes.
>>  
>>
>> Prior to this change, I also added https://bugs.openjdk.java.net/browse/JDK-8186315 <https://bugs.openjdk.java.net/browse/JDK-8186315>.
>> This change allows the user to specify the percentage of container memory that
>> should be used by the Java Heap.  This allows the users to reserve memory outside
>> of the Java heap for other classes of memory allocations (C-Heap for example).
>>
>> Both of these changes have been integrated into JDK 10 and are available in the
>> latest early access release of Linux x64 (http://jdk.java.net/10/ <http://jdk.java.net/10/>).
>>
>> How interesting! The previous version of our Java memory calculator allowed the heap size to be specified as a proportion of container memory, but we could only guess at a default proportion and users found it quite hard to know what value to use when they needed to increase or decrease heap size.
>>
>> The current version adopts a different approach: we try to calculate JVM memory consumption (apart from heap) from the JVM options and then subtract that value from the container memory to get the maximum heap size. This approach suffers from the difficulty of predicting the memory consumption of the JVM with any accuracy (see my recent thread entitled 'Excessive "GC" memory area size'). The up-side of the new approach is that the user can override the behaviour using standard JVM options rather than proportions.
>>  
>>
>> I will talk to the Hotspot runtime folks about your requirement
>>
>> Thanks very much.
>>  
>> but other than these
>> options, have you considered configuring your containers to use some swap space
>> in addition to memory configurations so that temporary spikes in memory consumption
>> won’t cause an OOM ?
>>
>> We talked about this but we need to bound the amount of swap space because our environment is a PaaS and we can't allow users to consume more than their fair share of system resources. If we did enable a bounded swap space, I think we'd hit the same issue when the RAM+swap limit was exceeded.
>>  
>> There are also ways of disabling the OOM killer in containers.
>>
>> We actually tried that. We started the container with the OOM killer disabled and when the container hit OOM, we gathered diagnostics and re-enabled the OOM killer to terminate the container. The main issue with that is the container-level diagnostics (number of committed pages of RAM etc.) don't tell the Java programmer what JVM options need changing to avoid the problem in future.
>>
>> A more sophisticated approach along those lines would be to disable the OOM killer and when the container *nears* its limit somehow trigger the JVM to do "out of memory" processing. Then our JVMTI agent would be driven and produce plenty of diagnostics suitable for a Java programmer. This is a bit speculative though as it requires fairly deep changes to the container configuration as well as solving the problem of how to trigger the JVM to do its "out of memory" processing. Maybe a more practical solution there is for the JVM itself to monitor its approach to the container memory limit and trigger "out of memory" processing at some threshold close to the limit. This could be relevant to the JEP.
>>  
>> If we get a failed memory allocation, we will try to deliver a Java OutOfMemory
>> exception which would at least allow you to figure out what went wrong.  I haven’t looked
>> into this yet but it is something worth considering.
>>
>> Yeah, OutOfMemory exceptions provide some useful information, although we found it was insufficiently detailed which is why we wrote a JVMTI agent to dump out detailed diagnostics.
>>  
>>
>> Bob.
>>
>>
>>
>>> On Dec 13, 2017, at 4:13 AM, Glyn Normington <[hidden email] <mailto:[hidden email]>> wrote:
>>>
>>> I wonder if someone involved in JEPS 8182070 (Container aware Java) would
>>> care to comment on the additional requirement described below?
>>>
>>> On Mon, Nov 13, 2017 at 9:30 AM, Glyn Normington <[hidden email] <mailto:[hidden email]>>
>>> wrote:
>>>
>>>> I would like to mention an additional requirement for JEPS 8182070 (
>>>> http://openjdk.java.net/jeps/8182070 <http://openjdk.java.net/jeps/8182070>): avoid the JVM hitting container
>>>> OOM by strictly bounding the amount of (physical) memory that the JVM
>>>> consumes. This may be implicit in the document, but I think it should be
>>>> made an explicit goal.
>>>>
>>>> If a java application hits container OOM, no detailed diagnostics, such as
>>>> those associated with an OutOfMemoryError or a JVMTI resource exhaustion
>>>> event, are presented to the user, so the user finds it very difficult to
>>>> know how to fix the problem.
>>>>
>>>> The Cloud Foundry OSS project has done quite a bit of work on this problem
>>>> and provides a couple of utilities which help when running a JVM in a
>>>> container:
>>>>
>>>> * Java memory calculator ([1], [2]) to determine JVM memory settings,
>>>> * jvmkill JVMTI agent ([3]) to report detailed diagnostics on a resource
>>>> exhaustion event.
>>>>
>>>> Regards,
>>>> Glyn
>>>>
>>>> [1] https://github.com/cloudfoundry/java-buildpack-memory-calculator <https://github.com/cloudfoundry/java-buildpack-memory-calculator>
>>>>
>>>> [2] Design doc: https://docs.google.com/document/d/ <https://docs.google.com/document/d/>
>>>> 1vlXBiwRIjwiVcbvUGYMrxx2Aw1RVAtxq3iuZ3UK2vXA/edit#heading=h.uy41ishpv9zc
>>>>
>>>> [3] https://github.com/cloudfoundry/jvmkill <https://github.com/cloudfoundry/jvmkill>
>>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Glyn
>>
>>
>>
>>
>> --
>> Regards,
>> Glyn
>
>
>
>
> --
> Regards,
> Glyn

Reply | Threaded
Open this post in threaded view
|

Re: Additional JEPS 8182070 requirement: avoiding container OOM

Glyn Normington
On Wed, Dec 13, 2017 at 5:34 PM, Bob Vandette <[hidden email]>
wrote:

>
> On Dec 13, 2017, at 12:00 PM, Glyn Normington <[hidden email]>
> wrote:
>
> On Wed, Dec 13, 2017 at 4:51 PM, Bob Vandette <[hidden email]>
> wrote:
>
>>
>> On Dec 13, 2017, at 11:01 AM, Glyn Normington <[hidden email]>
>> wrote:
>>
>> Hi Bob
>>
>> Thanks very much for that information. Some comments inline. This one
>> could be relevant to the refocussed JEP: the JVM should monitor its
>> approach to the container memory limit and trigger "out of memory"
>> processing at some threshold close to the limit.
>>
>>
>> Memory allocations in hotspot are difficult to track since there are many
>> different allocators
>> and we don’t control them all.  Thread stacks get dynamically allocated
>> upon use, the C++ runtime allocates
>> memory for native code and third party JNI library could be calling
>> malloc.  These are all in addition
>> to the allocators we do control - Java Heap and internal Memory Arenas
>> for runtime metadata.
>>
>> One possible solution is to at least attempt threshold checking during
>> allocations we
>> do control and to do OOM processing at that time.  This wouldn’t catch
>> every case but could
>> increase the robustness of the Java runtime.  With the change I
>> introduced in JDK-8146155,
>> we now at least have a way of finding out how much free memory is
>> available in the container.
>>
>
> Another solution you might like to mull over is for the JVM to poll the
> memory cgroup value memory.usage_in_bytes (assuming the memory cgroup is
> mounted read-only inside the container) and trigger "out of memory" on some
> threshold.
>
>
> That is exactly what I was referring to in "we now at least have a way of
> finding out how much free memory is available in the container."
>

Ah, thanks for clarifying. That's great.


>
> I added several cgroup access functions inside hotspot (
> http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/
> hotspot/os/linux/osContainer_linux.cpp)
> including these two.
>
> /* memory_usage_in_bytes <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l456> * <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l457> * Return the amount of used memory for this process. <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l458> * <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l459> * return: <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l460> *    memory usage in bytes or <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l461> *    -1 for unlimited <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l462> *    OSCONTAINER_ERROR for not supported <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l463> */ <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l464>jlong OSContainer::memory_usage_in_bytes() { <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l465>  GET_CONTAINER_INFO(jlong, memory, "/memory.usage_in_bytes", <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l466>                     "Memory Usage is: " JLONG_FORMAT, JLONG_FORMAT, memusage); <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l467>  return memusage; <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l468>} <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l469> <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l470>/* memory_max_usage_in_bytes <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l471> * <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l472> * Return the maximum amount of used memory for this process. <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l473> * <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l474> * return: <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l475> *    max memory usage in bytes or <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l476> *    OSCONTAINER_ERROR for not supported <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l477> */ <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l478>jlong OSContainer::memory_max_usage_in_bytes() { <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l479>  GET_CONTAINER_INFO(jlong, memory, "/memory.max_usage_in_bytes", <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l480>                     "Maximum Memory Usage is: " JLONG_FORMAT, JLONG_FORMAT, memmaxusage); <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l481>  return memmaxusage; <http://hg.openjdk.java.net/jdk/jdk/file/a559b7cd1dea/src/hotspot/os/linux/osContainer_linux.cpp#l482>}
>
>
> Bob.
>
>
>
>>
>> Bob.
>>
>>
>> Regards,
>> Glyn
>>
>> On Wed, Dec 13, 2017 at 2:58 PM, Bob Vandette <[hidden email]>
>> wrote:
>>
>>> Hi Glyn,
>>>
>>> The JEP you mention is going to be updated in a week or so to refocus
>>> it’s goals
>>> on monitoring of the JVM running in containers.  This is due to the fact
>>> that I was
>>> able to split out the work of dynamically configuring the VM based on
>>> container
>>> settings and integrate this work into JDK 10.  This work was integrated
>>> under jira issue
>>> https://bugs.openjdk.java.net/browse/JDK-8146115.
>>>
>>> Although this change does not completely address your requirement, it
>>> should help.
>>> This change causes the JVM to correctly configure the number of threads
>>> and memory
>>> allocations based on the resources configured in the container.  This
>>> change also
>>> introduces a new JVM option (-XX:ActiveProcessorCount=xx) which allows
>>> the user
>>> to specify how many processors the JVM should use.  We will now honor
>>> cpu shares, quotas
>>> in addition to cpusets for determining the number of threads that the
>>> JVM will use.
>>>
>>
>> Yes, that does sound like a useful set of changes.
>>
>>
>>>
>>> Prior to this change, I also added https://bugs.openjdk.jav
>>> a.net/browse/JDK-8186315.
>>> This change allows the user to specify the percentage of container
>>> memory that
>>> should be used by the Java Heap.  This allows the users to reserve
>>> memory outside
>>> of the Java heap for other classes of memory allocations (C-Heap for
>>> example).
>>>
>>> Both of these changes have been integrated into JDK 10 and are available
>>> in the
>>> latest early access release of Linux x64 (http://jdk.java.net/10/).
>>>
>>
>> How interesting! The previous version of our Java memory calculator
>> allowed the heap size to be specified as a proportion of container memory,
>> but we could only guess at a default proportion and users found it quite
>> hard to know what value to use when they needed to increase or decrease
>> heap size.
>>
>> The current version adopts a different approach: we try to calculate JVM
>> memory consumption (apart from heap) from the JVM options and then subtract
>> that value from the container memory to get the maximum heap size. This
>> approach suffers from the difficulty of predicting the memory consumption
>> of the JVM with any accuracy (see my recent thread entitled 'Excessive "GC"
>> memory area size'). The up-side of the new approach is that the user can
>> override the behaviour using standard JVM options rather than proportions.
>>
>>
>>>
>>> I will talk to the Hotspot runtime folks about your requirement
>>>
>>
>> Thanks very much.
>>
>>
>>> but other than these
>>> options, have you considered configuring your containers to use some
>>> swap space
>>> in addition to memory configurations so that temporary spikes in memory
>>> consumption
>>> won’t cause an OOM ?
>>>
>>
>> We talked about this but we need to bound the amount of swap space
>> because our environment is a PaaS and we can't allow users to consume more
>> than their fair share of system resources. If we did enable a bounded swap
>> space, I think we'd hit the same issue when the RAM+swap limit was exceeded.
>>
>>
>>> There are also ways of disabling the OOM killer in containers.
>>>
>>
>> We actually tried that. We started the container with the OOM killer
>> disabled and when the container hit OOM, we gathered diagnostics and
>> re-enabled the OOM killer to terminate the container. The main issue with
>> that is the container-level diagnostics (number of committed pages of RAM
>> etc.) don't tell the Java programmer what JVM options need changing to
>> avoid the problem in future.
>>
>> A more sophisticated approach along those lines would be to disable the
>> OOM killer and when the container *nears* its limit somehow trigger the JVM
>> to do "out of memory" processing. Then our JVMTI agent would be driven and
>> produce plenty of diagnostics suitable for a Java programmer. This is a bit
>> speculative though as it requires fairly deep changes to the container
>> configuration as well as solving the problem of how to trigger the JVM to
>> do its "out of memory" processing. Maybe a more practical solution there is
>> for the JVM itself to monitor its approach to the container memory limit
>> and trigger "out of memory" processing at some threshold close to the
>> limit. This could be relevant to the JEP.
>>
>>
>>> If we get a failed memory allocation, we will try to deliver a Java
>>> OutOfMemory
>>> exception which would at least allow you to figure out what went wrong.
>>> I haven’t looked
>>> into this yet but it is something worth considering.
>>>
>>
>> Yeah, OutOfMemory exceptions provide some useful information, although we
>> found it was insufficiently detailed which is why we wrote a JVMTI agent to
>> dump out detailed diagnostics.
>>
>>
>>>
>>> Bob.
>>>
>>>
>>>
>>> On Dec 13, 2017, at 4:13 AM, Glyn Normington <[hidden email]>
>>> wrote:
>>>
>>> I wonder if someone involved in JEPS 8182070 (Container aware Java) would
>>> care to comment on the additional requirement described below?
>>>
>>> On Mon, Nov 13, 2017 at 9:30 AM, Glyn Normington <[hidden email]
>>> >
>>> wrote:
>>>
>>> I would like to mention an additional requirement for JEPS 8182070 (
>>> http://openjdk.java.net/jeps/8182070): avoid the JVM hitting container
>>> OOM by strictly bounding the amount of (physical) memory that the JVM
>>> consumes. This may be implicit in the document, but I think it should be
>>> made an explicit goal.
>>>
>>> If a java application hits container OOM, no detailed diagnostics, such
>>> as
>>> those associated with an OutOfMemoryError or a JVMTI resource exhaustion
>>> event, are presented to the user, so the user finds it very difficult to
>>> know how to fix the problem.
>>>
>>> The Cloud Foundry OSS project has done quite a bit of work on this
>>> problem
>>> and provides a couple of utilities which help when running a JVM in a
>>> container:
>>>
>>> * Java memory calculator ([1], [2]) to determine JVM memory settings,
>>> * jvmkill JVMTI agent ([3]) to report detailed diagnostics on a resource
>>> exhaustion event.
>>>
>>> Regards,
>>> Glyn
>>>
>>> [1] https://github.com/cloudfoundry/java-buildpack-memory-calculator
>>>
>>> [2] Design doc: https://docs.google.com/document/d/
>>> 1vlXBiwRIjwiVcbvUGYMrxx2Aw1RVAtxq3iuZ3UK2vXA/edit#heading=h.uy41ishpv9zc
>>>
>>> [3] https://github.com/cloudfoundry/jvmkill
>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Glyn
>>>
>>>
>>>
>>
>>
>> --
>> Regards,
>> Glyn
>>
>>
>>
>
>
> --
> Regards,
> Glyn
>
>
>


--
Regards,
Glyn