Heap Size Ergonomics in docker containers

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Heap Size Ergonomics in docker containers

Bob Vandette
I’m trying to test the container support that I’m adding to 
JDK 18.3 and I’m running into an issue with the Heap size calculation.

If I create a docker container that has 100MB of total memory on a
64-bit Linux box, I’m ending up with a 126MB Max Heap which results
in the process getting killed by the OOM killer.

This is due to the fact that MaxHeapSize is 126MB and phys_mem (100MB) is
not less than 50% of MinRAMPercentage of MaxHeapSize.

I would think in a small memory system, you don’t ever want the Heap 
size to be more than somewhere around 70% of total RAM.  

A quick fix for my problem might be this, but this still
leaves some pathological cases where phys_mem is greater than but
close to MaxHeapSize.

if phys_mem < MaxHeapSize || phys_mem < MaxHeapSize * MinRAMPercentage
   use phys_mem * MinRAMPercentage

I could improve on this with:

if MaxHeapSize > 70% of phys_mem || phys_mem < MaxHeapSize * MinRAMPercentage
   use phys_mem * MinRAMPercentage

Here’s the code from arguments.cpp

  // If the maximum heap size has not been set with -Xmx,
  // then set it as fraction of the size of physical memory,
  // respecting the maximum and minimum sizes of the heap.
  if (FLAG_IS_DEFAULT(MaxHeapSize)) {
    julong reasonable_max = (julong)((phys_mem * MaxRAMPercentage) / 100);
    if (phys_mem <= (julong)((MaxHeapSize * MinRAMPercentage) / 100)) {
      // Small physical memory, so use a minimum fraction of it for the heap
      reasonable_max = (julong)((phys_mem * MinRAMPercentage) / 100);
    } else {
      // Not-small physical memory, so require a heap at least
      // as large as MaxHeapSize
      reasonable_max = MAX2(reasonable_max, (julong)MaxHeapSize);
    }

Let me know what you think.  
Bob.


Reply | Threaded
Open this post in threaded view
|

Re: Heap Size Ergonomics in docker containers

Thomas Schatzl
On Tue, 2017-10-24 at 14:43 -0400, Bob Vandette wrote:

> I’m trying to test the container support that I’m adding to 
> JDK 18.3 and I’m running into an issue with the Heap size
> calculation.
>
> If I create a docker container that has 100MB of total memory on a
> 64-bit Linux box, I’m ending up with a 126MB Max Heap which results
> in the process getting killed by the OOM killer.
>
> This is due to the fact that MaxHeapSize is 126MB and phys_mem
> (100MB) is
> not less than 50% of MinRAMPercentage of MaxHeapSize.
>
> I would think in a small memory system, you don’t ever want the Heap 
> size to be more than somewhere around 70% of total RAM.  
>
> A quick fix for my problem might be this, but this still
> leaves some pathological cases where phys_mem is greater than but
> close to MaxHeapSize.
>
> if phys_mem < MaxHeapSize || phys_mem < MaxHeapSize *
> MinRAMPercentage
>    use phys_mem * MinRAMPercentage
>
> I could improve on this with:
>
> if MaxHeapSize > 70% of phys_mem || phys_mem < MaxHeapSize *
> MinRAMPercentage
>    use phys_mem * MinRAMPercentage
>
>  [...]
> Let me know what you think.  
>

  that seems to be almost the same as making the default value of
MaxHeapSize 70% smaller. Also that 70% only seems to serve the same
purpose as MaxRAMPercentage, i.e. imo MaxRAMPercentage should have
covered that.

I kind of question this line in the code:

      // Not-small physical memory, so require a heap at least
      // as large as MaxHeapSize
      reasonable_max = MAX2(reasonable_max, (julong)MaxHeapSize);

So we just found out a "reasonable max" as a part of the calculation,
and then we ignore it...

The problem seems to be the default value of MaxHeapSize being 126M for
64 bit machines. I guess this has been made to provide a "good Java
experience" with machines with only a few 100 MBs (overriding
MaxRAMPercentage). From that point of view, I am not sure why the
default value is that large given typical (PC) memory sizes nowadays.

So one option could be reducing the default MaxHeapSize to some lower,
other random value (E.g. the absolute minimum it needs to run a Hello
World program plus some buffer for good measure)

Then again, the given 126MB as absolute minimum seems to be as good as
any other value (96MB for 32 bit systems) - everyone else just needs to
set -Xmx. I mean, if the next person wants to run the VM in a 50M
docker container, and defaults fail, the same complaints will start
again, so as long as the MAX2() calculation is there any random minimum
heap size will cause this issue.

So another option would be to remove that MAX2 calculation...

  Thomas

Reply | Threaded
Open this post in threaded view
|

Re: Heap Size Ergonomics in docker containers

Bob Vandette

> On Oct 25, 2017, at 5:36 AM, Thomas Schatzl <[hidden email]> wrote:
>
> On Tue, 2017-10-24 at 14:43 -0400, Bob Vandette wrote:
>> I’m trying to test the container support that I’m adding to
>> JDK 18.3 and I’m running into an issue with the Heap size
>> calculation.
>>
>> If I create a docker container that has 100MB of total memory on a
>> 64-bit Linux box, I’m ending up with a 126MB Max Heap which results
>> in the process getting killed by the OOM killer.
>>
>> This is due to the fact that MaxHeapSize is 126MB and phys_mem
>> (100MB) is
>> not less than 50% of MinRAMPercentage of MaxHeapSize.
>>
>> I would think in a small memory system, you don’t ever want the Heap
>> size to be more than somewhere around 70% of total RAM.  
>>
>> A quick fix for my problem might be this, but this still
>> leaves some pathological cases where phys_mem is greater than but
>> close to MaxHeapSize.
>>
>> if phys_mem < MaxHeapSize || phys_mem < MaxHeapSize *
>> MinRAMPercentage
>>    use phys_mem * MinRAMPercentage
>>
>> I could improve on this with:
>>
>> if MaxHeapSize > 70% of phys_mem || phys_mem < MaxHeapSize *
>> MinRAMPercentage
>>    use phys_mem * MinRAMPercentage
>>
>> [...]
>> Let me know what you think.  
>>
>
>  that seems to be almost the same as making the default value of
> MaxHeapSize 70% smaller.

Not really.  That just creates the same problem if the containers memory is set
to some number near the lower value.

> Also that 70% only seems to serve the same
> purpose as MaxRAMPercentage, i.e. imo MaxRAMPercentage should have
> covered that.

I think it makes sense to have a MaxRAMPercentage with a MinRAMPercentage.
You don’t want to use the same % for large GB’s of RAM as you’d use with 128M so
I don’t agree that we should cover all cases with MaxRAMPercentage.

>
> I kind of question this line in the code:
>
>       // Not-small physical memory, so require a heap at least
>       // as large as MaxHeapSize
>       reasonable_max = MAX2(reasonable_max, (julong)MaxHeapSize);
>
> So we just found out a "reasonable max" as a part of the calculation,
> and then we ignore it…

I don’t think it’s ignored.  reasonable_max will be set to 25% of phys_mem
which could be larger than MaxHeapSize.

>
> The problem seems to be the default value of MaxHeapSize being 126M for
> 64 bit machines. I guess this has been made to provide a "good Java
> experience" with machines with only a few 100 MBs (overriding
> MaxRAMPercentage). From that point of view, I am not sure why the
> default value is that large given typical (PC) memory sizes nowadays.
>
> So one option could be reducing the default MaxHeapSize to some lower,
> other random value (E.g. the absolute minimum it needs to run a Hello
> World program plus some buffer for good measure)
Not going there.  This would cause too many regressions for existing
deployments that upgrade to JDK 18.3.
>
> Then again, the given 126MB as absolute minimum seems to be as good as
> any other value (96MB for 32 bit systems) - everyone else just needs to
> set -Xmx. I mean, if the next person wants to run the VM in a 50M
> docker container, and defaults fail, the same complaints will start
> again, so as long as the MAX2() calculation is there any random minimum
> heap size will cause this issue.
>
> So another option would be to remove that MAX2 calculation…

Without changing anything else, this would cause us to select 25% of total available
which I believe is too small.

I still think we need to find a way of determining when MaxHeapSize is set too close or
higher than physical memory in order to know when to use MinRAMPercentage.  My test
for 70% is the best solution I’ve been able to come up with so far.  


Bob.

>
>  Thomas
>

Reply | Threaded
Open this post in threaded view
|

Re: Heap Size Ergonomics in docker containers

Thomas Schatzl
Hi Bob,

 

On Wed, 2017-10-25 at 08:44 -0400, Bob Vandette wrote:

> > On Oct 25, 2017, at 5:36 AM, Thomas Schatzl <thomas.schatzl@oracle.
> > com> wrote:
> >
> > On Tue, 2017-10-24 at 14:43 -0400, Bob Vandette wrote:
> > > I’m trying to test the container support that I’m adding to 
> > > JDK 18.3 and I’m running into an issue with the Heap size
> > > calculation.
> > >
> > > If I create a docker container that has 100MB of total memory on
> > > a
> > > 64-bit Linux box, I’m ending up with a 126MB Max Heap which
> > > results
> > > in the process getting killed by the OOM killer.
> > >
> > > This is due to the fact that MaxHeapSize is 126MB and phys_mem
> > > (100MB) is
> > > not less than 50% of MinRAMPercentage of MaxHeapSize.
> > >
> > > I would think in a small memory system, you don’t ever want the
> > > Heap 
> > > size to be more than somewhere around 70% of total RAM.  
> > >
> > > A quick fix for my problem might be this, but this still
> > > leaves some pathological cases where phys_mem is greater than but
> > > close to MaxHeapSize.
> > >
> > > if phys_mem < MaxHeapSize || phys_mem < MaxHeapSize *
> > > MinRAMPercentage
> > >    use phys_mem * MinRAMPercentage
> > >
> > > I could improve on this with:
> > >
> > > if MaxHeapSize > 70% of phys_mem || phys_mem < MaxHeapSize *
> > > MinRAMPercentage
> > >    use phys_mem * MinRAMPercentage
> > >
> > > [...]
> > > Let me know what you think.  
> > >
> >
> >  that seems to be almost the same as making the default value of
> > MaxHeapSize 70% smaller.
>
> Not really.  That just creates the same problem if the containers
> memory is set to some number near the lower value.
>
> > Also that 70% only seems to serve the same
> > purpose as MaxRAMPercentage, i.e. imo MaxRAMPercentage should have
> > covered that.
>
> I think it makes sense to have a MaxRAMPercentage with a
> MinRAMPercentage. You don’t want to use the same % for large GB’s of
> RAM as you’d use with 128M so I don’t agree that we should cover all
> cases with MaxRAMPercentage.

The general consensus within in the gc team is that the less
complicated any such heuristic is, the less surprising it is, the
easier to explain, with less maintenance and so more favorable from our
point of view. :)

The reasons are simple: this acknowledges the fact that you can't
capture the needs of billions of users in any kind of manageable
heuristic; there is also a limit on how complicated you want to make
this particular heuristic, because the more complicated to make it,
most likely you will get more corner cases for which you might want to
expose knobs to the user; and at that point the ultimate fix to this
issue is to tell to "set -Xmx" anyway.

A user is already going through the trouble of setting a max memory
size limit for your container...

Also, we have never seen anyone setting Min/MaxRAMFraction ever except
in error.

I do not think for this issue we should go down this road, making the
heuristics even more complicated.

> >
> > I kind of question this line in the code:
> >
> >       // Not-small physical memory, so require a heap at least
> >       // as large as MaxHeapSize
> >       reasonable_max = MAX2(reasonable_max, (julong)MaxHeapSize);
> >
> > So we just found out a "reasonable max" as a part of the
> > calculation, and then we ignore it…
>
> I don’t think it’s ignored.  reasonable_max will be set to 25% of
> phys_mem which could be larger than MaxHeapSize.
>

I filed JDK-8190283 [0], as this is a bug however you look at it.

The suggested fix for this is to use physmem*MinRAMFraction while it is
below the default MaxHeapSize value, then switch over to
physmem*MaxRAMFraction. Since physmem*MaxRAMFraction is at that point
smaller than MaxHeapSize, keep using MaxHeapSize as long as
physmem*MaxRAMFraction is smaller.
As such, default MaxHeapSize would act as kind of switchover threshold
between the two heuristics (that switchover point is pretty random as
before) - that should ensure that the selected heap size is always
smaller than the amount of physical memory, and fix the problem.

This would make the heap 50% (MinRAMFraction) of physical memory for
machines with < 252M, 126M for machines with physical memory < 504M,
and 25% (MaxRAMFraction) of physical memory otherwise.

A prototype patch is available at http://cr.openjdk.java.net/~tschatzl/
8190283/webrev/ (did not test that, I simply tried to implement the
formula from a spreadsheet I used, but I think it shows the idea) -
it's also very small.

What do you think?

------

Now there still may be the question whether the use case you presented,
i.e. "with a "small heap" with 100MB use 70% of the available memory"
is worth changing some additional default options for.

We think that we should not go that direction: the VM does not give
outright bad values for heap size any more (50MB with default
MinRAMFraction), providing safe and fairly good settings. There has
been no indication on whether a 100MB container is that pervasive
either.

Particularly a MinRAMFraction of 70% seems a bit too tight as default
for the current default collector, i.e. G1. As we have seen it can, in
extreme cases as a ballpark figure use another 20% of Java heap size
just for its remembered sets (theoretical max is even higher), and
other subsystems also need memory (compiler, runtime).

Note that I think that maybe G1 isn't the correct choice for such small
heaps at the moment - serial or parallel would probably run as fine if
not better with less overhead(*) - but that's also something we would
not like to change haphazardly; and it is possible and might be better
to improve G1 here in the future instead of changing defaults. (And if
the user is already selecting the GC, he might as well set max heap
size).

Thanks,
  Thomas

[0] https://bugs.openjdk.java.net/browse/JDK-8190283
(*) haven't tried lately.


Reply | Threaded
Open this post in threaded view
|

Re: Heap Size Ergonomics in docker containers

Bob Vandette

> On Oct 27, 2017, at 9:54 AM, Thomas Schatzl <[hidden email]> wrote:
>
> Hi Bob,
>
>
>
> On Wed, 2017-10-25 at 08:44 -0400, Bob Vandette wrote:
>>> On Oct 25, 2017, at 5:36 AM, Thomas Schatzl <thomas.schatzl@oracle.
>>> com> wrote:
>>>
>>> On Tue, 2017-10-24 at 14:43 -0400, Bob Vandette wrote:
>>>> I’m trying to test the container support that I’m adding to
>>>> JDK 18.3 and I’m running into an issue with the Heap size
>>>> calculation.
>>>>
>>>> If I create a docker container that has 100MB of total memory on
>>>> a
>>>> 64-bit Linux box, I’m ending up with a 126MB Max Heap which
>>>> results
>>>> in the process getting killed by the OOM killer.
>>>>
>>>> This is due to the fact that MaxHeapSize is 126MB and phys_mem
>>>> (100MB) is
>>>> not less than 50% of MinRAMPercentage of MaxHeapSize.
>>>>
>>>> I would think in a small memory system, you don’t ever want the
>>>> Heap
>>>> size to be more than somewhere around 70% of total RAM.  
>>>>
>>>> A quick fix for my problem might be this, but this still
>>>> leaves some pathological cases where phys_mem is greater than but
>>>> close to MaxHeapSize.
>>>>
>>>> if phys_mem < MaxHeapSize || phys_mem < MaxHeapSize *
>>>> MinRAMPercentage
>>>>    use phys_mem * MinRAMPercentage
>>>>
>>>> I could improve on this with:
>>>>
>>>> if MaxHeapSize > 70% of phys_mem || phys_mem < MaxHeapSize *
>>>> MinRAMPercentage
>>>>    use phys_mem * MinRAMPercentage
>>>>
>>>> [...]
>>>> Let me know what you think.  
>>>>
>>>
>>>  that seems to be almost the same as making the default value of
>>> MaxHeapSize 70% smaller.
>>
>> Not really.  That just creates the same problem if the containers
>> memory is set to some number near the lower value.
>>
>>> Also that 70% only seems to serve the same
>>> purpose as MaxRAMPercentage, i.e. imo MaxRAMPercentage should have
>>> covered that.
>>
>> I think it makes sense to have a MaxRAMPercentage with a
>> MinRAMPercentage. You don’t want to use the same % for large GB’s of
>> RAM as you’d use with 128M so I don’t agree that we should cover all
>> cases with MaxRAMPercentage.
>
> The general consensus within in the gc team is that the less
> complicated any such heuristic is, the less surprising it is, the
> easier to explain, with less maintenance and so more favorable from our
> point of view. :)
>
> The reasons are simple: this acknowledges the fact that you can't
> capture the needs of billions of users in any kind of manageable
> heuristic; there is also a limit on how complicated you want to make
> this particular heuristic, because the more complicated to make it,
> most likely you will get more corner cases for which you might want to
> expose knobs to the user; and at that point the ultimate fix to this
> issue is to tell to "set -Xmx" anyway.
>
> A user is already going through the trouble of setting a max memory
> size limit for your container...
>
> Also, we have never seen anyone setting Min/MaxRAMFraction ever except
> in error.
>
> I do not think for this issue we should go down this road, making the
> heuristics even more complicated.
>
>>>
>>> I kind of question this line in the code:
>>>
>>>       // Not-small physical memory, so require a heap at least
>>>       // as large as MaxHeapSize
>>>       reasonable_max = MAX2(reasonable_max, (julong)MaxHeapSize);
>>>
>>> So we just found out a "reasonable max" as a part of the
>>> calculation, and then we ignore it…
>>
>> I don’t think it’s ignored.  reasonable_max will be set to 25% of
>> phys_mem which could be larger than MaxHeapSize.
>>
>
> I filed JDK-8190283 [0], as this is a bug however you look at it.
>
> The suggested fix for this is to use physmem*MinRAMFraction while it is
> below the default MaxHeapSize value, then switch over to
> physmem*MaxRAMFraction. Since physmem*MaxRAMFraction is at that point
> smaller than MaxHeapSize, keep using MaxHeapSize as long as
> physmem*MaxRAMFraction is smaller.
> As such, default MaxHeapSize would act as kind of switchover threshold
> between the two heuristics (that switchover point is pretty random as
> before) - that should ensure that the selected heap size is always
> smaller than the amount of physical memory, and fix the problem.
>
> This would make the heap 50% (MinRAMFraction) of physical memory for
> machines with < 252M, 126M for machines with physical memory < 504M,
> and 25% (MaxRAMFraction) of physical memory otherwise.
>
> A prototype patch is available at http://cr.openjdk.java.net/~tschatzl/
> 8190283/webrev/ (did not test that, I simply tried to implement the
> formula from a spreadsheet I used, but I think it shows the idea) -
> it's also very small.
>
> What do you think?

You suggested fix looks fine to me.  According to my spreadsheat,
It results in more even and gradual increases in heap size over my proposal.

I’ll do some testing with different sizes and send the change out for review.

>
> ------
>
> Now there still may be the question whether the use case you presented,
> i.e. "with a "small heap" with 100MB use 70% of the available memory"
> is worth changing some additional default options for.
I would actually like to suggest that we change MinRAMPercentage to a value
a bit higher (possible 60%).  I’ve had complaints from the Container team
that they felt Java was not able to make use of available memory.  I gave them
the Percentage flavors of these flag so they could push the envelope a bit
but I think 50% is a bit too conservative.  Having said that, I don’t think
we should change that with this fix.

>
> We think that we should not go that direction: the VM does not give
> outright bad values for heap size any more (50MB with default
> MinRAMFraction), providing safe and fairly good settings. There has
> been no indication on whether a 100MB container is that pervasive
> either.
>
> Particularly a MinRAMFraction of 70% seems a bit too tight as default
> for the current default collector, i.e. G1. As we have seen it can, in
> extreme cases as a ballpark figure use another 20% of Java heap size
> just for its remembered sets (theoretical max is even higher), and
> other subsystems also need memory (compiler, runtime).
In my proposal, there was only one phys mem size where we selected
70% of memory for the heap (at 180MB) and that left 54MB avail.
If you look at the 100MB selection, we only have 50MB free.

In any case, your proposal works.

I also don’t think 100MB is something we should optimize for.  I just
wanted to keep us from picking a heap size too large for all low
numbers.  256MB is probably a better low end for the JFaas folks.

>
> Note that I think that maybe G1 isn't the correct choice for such small
> heaps at the moment - serial or parallel would probably run as fine if
> not better with less overhead(*) - but that's also something we would
> not like to change haphazardly; and it is possible and might be better
> to improve G1 here in the future instead of changing defaults. (And if
> the user is already selecting the GC, he might as well set max heap
> size).

I agree that SerialGC is better for serverless.  This is what Amazon is using
with Lambda.


Thanks for spending time on this issue,
Bob.

>
> Thanks,
>  Thomas
>
> [0] https://bugs.openjdk.java.net/browse/JDK-8190283
> (*) haven't tried lately.
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Heap Size Ergonomics in docker containers

Ruslan Synytsky
In reply to this post by Bob Vandette
Hi guys, I'm interested in improvements in this direction as well. Let me share the statistical experience from Jelastic customers who run Java stacks in containers for a long time.

We set the minimum Xmx to 32m by default as we believe majority wants to start from the minimum, specifically in the "micro" world (services, containers). G1 GC provides good enough elasticity of memory allocation. You can grow up quickly, and important to scale down later when the load goes away.

By default, Xmx is 80% of available RAM with container limits. It works great for the majority of our users. We track OOM Kill events and send alerts to the users. So the numbers are based on the real end users applications.

We keep 80% from max by default always, even if users allocate a bigger amount of RAM, for two reasons: 1 - it's easier to remember the defaults and 2 - unused resources are not reserved with containers, so if Java does not use it - no problem, they are free, and at the same time additional memory available for native non-heap data or for other processes, just like an insurance.     

Big memory and big data solutions - everything like this requires a fine-tuning, the customers who run such stacks definitely do not rely on the default heuristic. You can always find instructions and recommended RAM usage configs for these stacks. Some solutions even use direct memory access, a small heap size is preferable for them. In general, it means you have to do a customization if you want to get a stable solution. 

To make it simple, we came up with a supervisor script that applies custom memory configs. As result, customers can define the needed parameters with env variables in a container if they do not want to rely on the heuristic.  

Btw, is there any guaranteed way to pass Xmx and Xms to java process w/o any external scripts? I mean, it will be cool if java can check env variable like XMX and if it exists apply.    

Hope my feedback can help to find a better solution.  
Regards  

--
Ruslan
CEO @ Jelastic



Reply | Threaded
Open this post in threaded view
|

Re: Heap Size Ergonomics in docker containers

Thomas Schatzl
Hi Ruslan,

On Mon, 2017-10-30 at 19:06 +0100, Ruslan Synytsky wrote:

> Hi guys, I'm interested in improvements in this direction as well.
> Let me share the statistical experience from Jelastic customers who
> run Java stacks in containers for a long time.
>
> We set the minimum Xmx to 32m by default as we believe majority wants
> to start from the minimum, specifically in the "micro" world
> (services, containers). G1 GC provides good enough elasticity of
> memory allocation. You can grow up quickly, and important to scale
> down later when the load goes away.
>
> By default, Xmx is 80% of available RAM with container limits. It
> works great for the majority of our users. We track OOM Kill events
> and send alerts to the users. So the numbers are based on the real
> end users applications.
>
> We keep 80% from max by default always, even if users allocate a
> bigger amount of RAM, for two reasons: 1 - it's easier to remember
> the defaults and 2 - unused resources are not reserved with
> containers, so if Java does not use it - no problem, they are free,
> and at the same time additional memory available for native non-heap
> data or for other processes, just like an insurance.

Thanks for your insights into real-world usage. So maybe the suggested
50% as default for small heaps is indeed a bit too conservative.

> Big memory and big data solutions - everything like this requires a
> fine-tuning, the customers who run such stacks definitely do not rely
> on the default heuristic. You can always find instructions and
> recommended RAM usage configs for these stacks. Some solutions even
> use direct memory access, a small heap size is preferable for them.
> In general, it means you have to do a customization if you want to
> get a stable solution. 
>
> To make it simple, we came up with a supervisor script that
> applies custom memory configs. As result, customers can define the
> needed parameters with env variables in a container if they do not
> want to rely on the heuristic.  
>
> Btw, is there any guaranteed way to pass Xmx and Xms to java process
> w/o any external scripts? I mean, it will be cool if java can check
> env variable like XMX and if it exists apply.    
>

There is JAVA_TOOL_OPTIONS, which are always prepended to the java VM
command line. See the description [0] for more information.

I think this is what you were looking for.

Thanks,
  Thomas

[0] https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot
/envvars002.html
Reply | Threaded
Open this post in threaded view
|

Re: Heap Size Ergonomics in docker containers

Ruslan Synytsky
Hi Thomas, thank you for the feedback. Please see inline. 

On 31 October 2017 at 16:22, Thomas Schatzl <[hidden email]> wrote:
Hi Ruslan,

On Mon, 2017-10-30 at 19:06 +0100, Ruslan Synytsky wrote:
> Hi guys, I'm interested in improvements in this direction as well.
> Let me share the statistical experience from Jelastic customers who
> run Java stacks in containers for a long time.
>
> We set the minimum Xmx to 32m by default as we believe majority wants
> to start from the minimum, specifically in the "micro" world
> (services, containers). G1 GC provides good enough elasticity of
> memory allocation. You can grow up quickly, and important to scale
> down later when the load goes away.
>
> By default, Xmx is 80% of available RAM with container limits. It
> works great for the majority of our users. We track OOM Kill events
> and send alerts to the users. So the numbers are based on the real
> end users applications.
>
> We keep 80% from max by default always, even if users allocate a
> bigger amount of RAM, for two reasons: 1 - it's easier to remember
> the defaults and 2 - unused resources are not reserved with
> containers, so if Java does not use it - no problem, they are free,
> and at the same time additional memory available for native non-heap
> data or for other processes, just like an insurance.

Thanks for your insights into real-world usage. So maybe the suggested
50% as default for small heaps is indeed a bit too conservative.
Understanding of the situations when people rely on the default heuristic will be really helpful to make a final decision... From my personal experience, if you do not specify Xmx you always get troubles at the end. 


> Big memory and big data solutions - everything like this requires a
> fine-tuning, the customers who run such stacks definitely do not rely
> on the default heuristic. You can always find instructions and
> recommended RAM usage configs for these stacks. Some solutions even
> use direct memory access, a small heap size is preferable for them.
> In general, it means you have to do a customization if you want to
> get a stable solution. 
>
> To make it simple, we came up with a supervisor script that
> applies custom memory configs. As result, customers can define the
> needed parameters with env variables in a container if they do not
> want to rely on the heuristic.  
>
> Btw, is there any guaranteed way to pass Xmx and Xms to java process
> w/o any external scripts? I mean, it will be cool if java can check
> env variable like XMX and if it exists apply.    
>

There is JAVA_TOOL_OPTIONS, which are always prepended to the java VM
command line. See the description [0] for more information.
Indeed this option is close to what I was looking for! But seems JAVA_TOOL_OPTIONS does not work for Xmx. Just a quick test with Java HotSpot(TM) 64-Bit Server VM (build 9+181, mixed mode)

Default
$ java -XX:+PrintFlagsFinal -version | grep MaxHeapSize
   size_t MaxHeapSize                              = 643825664                                {product} {command line}

Define Xmx = 2GB via JAVA_TOOL_OPTIONS 
$ export JAVA_TOOL_OPTIONS="-Xmx2g"
$ java -XX:+PrintFlagsFinal -version | grep MaxHeapSize
Picked up JAVA_TOOL_OPTIONS: -Xmx2g
   size_t MaxHeapSize                              = 643825664                                {product} {command line}

It says that Xmx has been picked up, but in reality it has not been applied.  

Define Xmx = 3GB via _JAVA_OPTIONS 
$ export _JAVA_OPTIONS="-Xmx3g"
$ java -XX:+PrintFlagsFinal -version | grep MaxHeapSize
Picked up JAVA_TOOL_OPTIONS: -Xmx2g
Picked up _JAVA_OPTIONS: -Xmx3g
   size_t MaxHeapSize                              = 3221225472                               {product} {command line}

Now it works as expected. However, another issue is that _JAVA_OPTIONS is undocumented and hotspot-specific?...

Also important to mention that _JAVA_OPTIONS trumps command-line arguments, and command-line arguments trump JAVA_TOOL_OPTIONS. That gives a quite good flexibility. Because sometimes you may want to override command line args and sometimes you need just to provide better deafults, but if users specify command line args use them. In other words, I believe, if we can a) improve JAVA_TOOL_OPTIONS (or introduce a new name like JAVA_OPTIONS_DEFAULT) for enabling people to specify Xmx via this env var and b) make _JAVA_OPTIONS as a standard/documented option, then this will solve the existig dificulties and provide enough flexibility.  
 
Thanks 
 

I think this is what you were looking for.

Thanks,
  Thomas

[0] https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot
/envvars002.html




--
Ruslan
CEO @ Jelastic



Reply | Threaded
Open this post in threaded view
|

Passing command line options to Java [Was: Re: Heap Size Ergonomics in docker containers]

Thomas Schatzl
Hi Ruslan,

On Wed, 2017-11-01 at 11:30 +0100, Ruslan Synytsky wrote:

> Hi Thomas, thank you for the feedback. Please see inline. 
>
> [...]
>
> > There is JAVA_TOOL_OPTIONS, which are always prepended to the java
> > VM command line. See the description [0] for more information.
>
> Indeed this option is close to what I was looking for! But seems
> JAVA_TOOL_OPTIONS does not work for Xmx. Just a quick test with Java
> HotSpot(TM) 64-Bit Server VM (build 9+181, mixed mode)
>
> Default
> $ java -XX:+PrintFlagsFinal -version | grep MaxHeapSize
>    size_t MaxHeapSize                              = 643825664      
>                          {product} {command line}
>
> Define Xmx = 2GB via JAVA_TOOL_OPTIONS 
> $ export JAVA_TOOL_OPTIONS="-Xmx2g" 
> $ java -XX:+PrintFlagsFinal -version | grep MaxHeapSize
> Picked up JAVA_TOOL_OPTIONS: -Xmx2g
>    size_t MaxHeapSize                              = 643825664      
>                          {product} {command line}
>
> It says that Xmx has been picked up, but in reality it has not been
> applied.  

Hmm, it seems to work here.

E.g.

$ java -XX:+PrintFlagsFinal HelloWorld | grep MaxHeap
   size_t MaxHeapSize                              = 3072327680
                                     {product} {ergonomic}
$ JAVA_TOOL_OPTIONS=-Xmx100m java -XX:+PrintFlagsFinal HelloWorld |
grep MaxHeap
Picked up JAVA_TOOL_OPTIONS: -Xmx100m
   size_t MaxHeapSize                              = 104857600
                                     {product} {command line}
$ echo $JAVA_TOOL_OPTIONS

$ export JAVA_TOOL_OPTIONS=-Xmx10m
$ java -XX:+PrintFlagsFinal HelloWorld | grep MaxHeap
Picked up JAVA_TOOL_OPTIONS: -Xmx10m
   size_t MaxHeapSize                              = 10485760
                                 {product} {command line}

(linux-x64, same hotspot build)
(Also tried with surrounding quotes)
(It does not matter whether you specify a java class on the command
line or not).

There is a note in the documentation that prevents use of it under
particular circumstances, but then according to the code the VM would
not print the "Picked up" line.

I haven't seen an issue reported with that anywhere either.

>
> Define Xmx = 3GB via _JAVA_OPTIONS 
> $ export _JAVA_OPTIONS="-Xmx3g" 
> $ java -XX:+PrintFlagsFinal -version | grep MaxHeapSize
> Picked up JAVA_TOOL_OPTIONS: -Xmx2g
> Picked up _JAVA_OPTIONS: -Xmx3g
>    size_t MaxHeapSize                              = 3221225472      
>                         {product} {command line}
>
> Now it works as expected. However, another issue is that
> _JAVA_OPTIONS is undocumented and hotspot-specific?...

I did not mention _JAVA_OPTIONS because I thought it wouldn't fit your
requirements and overwrites existing options. Also it's undocumented as
you noted.

There is another way to specify multiple java arguments via argument
files using  the "@" parameter which in some cases may be more useful
than the other options.

Thanks,
  Thomas
Reply | Threaded
Open this post in threaded view
|

Re: Passing command line options to Java [Was: Re: Heap Size Ergonomics in docker containers]

Mandy Chung


On 11/2/17 2:09 AM, Thomas Schatzl wrote:

        
There is JAVA_TOOL_OPTIONS, which are always prepended to the java
VM command line. See the description [0] for more information.


You may want to check out the JDK_JAVA_OPTIONS environment variable and @argfile support introduced in JDK 9 java launcher [1].

Mandy

[1] https://docs.oracle.com/javase/9/tools/java.htm#GUID-3B1CE181-CD30-4178-9602-230B800D4FAE__USINGTHEJDK_JAVA_OPTIONSLAUNCHERENV-F3C0E3BA