Excessive "GC" memory area size [was: Re: Determining "GC" memory area size]

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Excessive "GC" memory area size [was: Re: Determining "GC" memory area size]

Glyn Normington
On the basis of the reply below, we have been thinking of estimating the
size of the "GC" memory area as a little more than (1/32) * heapSize.
However, running `java -version` with a 350m heap results in "GC" memory as
reported by NMT of over 17% of the heap size, so this estimate is way out.

If the analysis below is accurate, then given the number of GC threads is
10 (ParallelGCThreads=8, ConcGCThreads=2), can there be many more
taskqueues than GC threads?

Or is there another explanation for the additional "GC" size?

Thanks,
Glyn

[1] https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe

On Fri, Nov 10, 2017 at 7:05 PM, Zhengyu Gu <[hidden email]> wrote:

> Hi Glyn,
>
>
>> Can anyone here tell me how the GC memory area size is determined? If
>> there
>> is documentation, so much the better as we'd prefer not to depend on code
>> details that might flux arbitrarily.
>>
>>
> GC memory is mainly data structures used by GC runtime. It can be varied
> by collector used, size of the Java heap, the number of GC threads and etc.
> and, of course, the application itself.
>
> Some are *fixed* costs, which can be estimated. E.g. two marking bitmaps
> used by G1, each costs 1/64 of heap size (assuming default object
> alignment).
>
> Some are *semi-fixed*, e.g. taskqueue's fixed cost is about 1M for each
> queue on 64-bits VM, but it can overflow. And the number of task queues is
> proportional to the number of GC threads.
>
> Then there are factors from application itself,  such as object mutation
> rate, inter-generation/region references, etc.
>
>
> I don't see a single formula for estimating GC memory size. If you are
> using G1, the biggest overhead comes from 2 bitmaps (1/32 * heap size).
>
> Thanks,
>
> -Zhengyu
>
>
>
>


--
Regards,
Glyn
Reply | Threaded
Open this post in threaded view
|

Re: Excessive "GC" memory area size [was: Re: Determining "GC" memory area size]

Zhengyu Gu-2
Hi Glyn,

You can run NMT with detail tracking (-XX:NativeMemoryTracking=detail)
then use jcmd (jcmd <pid> VM.native_memory detail) to find out all
allocations for GC.

For example:

[0x00007f9fba000000 - 0x00007f9fbc000000] reserved 32768KB for *GC* from
     [0x00007fa0130260ef] G1CMMarkStack::resize(unsigned long)+0x8f
     [0x00007fa01302cf62]
G1ConcurrentMark::G1ConcurrentMark(G1CollectedHeap*,
G1RegionToSpaceMapper*, G1RegionToSpaceMapper*)+0x782
     [0x00007fa01300e94b] G1CollectedHeap::initialize()+0x62b
     [0x00007fa013ca26af] universe_init()+0x6f

         [0x00007f9fba000000 - 0x00007f9fbc000000] committed 32768KB from
             [0x00007fa01302610d] G1CMMarkStack::resize(unsigned long)+0xad
             [0x00007fa01302cf62]
G1ConcurrentMark::G1ConcurrentMark(G1CollectedHeap*,
G1RegionToSpaceMapper*, G1RegionToSpaceMapper*)+0x782
             [0x00007fa01300e94b] G1CollectedHeap::initialize()+0x62b
             [0x00007fa013ca26af] universe_init()+0x6f


[0x00007fa012ac5395] ArrayAllocator<unsigned long>::allocate(unsigned
long, MemoryType)+0x245
[0x00007fa012ac5448] unsigned long*
BitMap::reallocate<CHeapBitMapAllocator>(CHeapBitMapAllocator const&,
unsigned long*, unsigned long, unsigned long)+0x38
[0x00007fa012ac509f] CHeapBitMap::initialize(unsigned long)+0x6f
[0x00007fa013093b9c]
G1PageBasedVirtualSpace::G1PageBasedVirtualSpace(ReservedSpace, unsigned
long, unsigned long)+0x13c
                              (malloc=129KB *type=GC* #6)

Thanks,

-Zhengyu



On 12/13/2017 04:39 AM, Glyn Normington wrote:

> On the basis of the reply below, we have been thinking of estimating the
> size of the "GC" memory area as a little more than (1/32) * heapSize.
> However, running `java -version` with a 350m heap results in "GC" memory as
> reported by NMT of over 17% of the heap size, so this estimate is way out.
>
> If the analysis below is accurate, then given the number of GC threads is
> 10 (ParallelGCThreads=8, ConcGCThreads=2), can there be many more
> taskqueues than GC threads?
>
> Or is there another explanation for the additional "GC" size?
>
> Thanks,
> Glyn
>
> [1] https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe
>
> On Fri, Nov 10, 2017 at 7:05 PM, Zhengyu Gu <[hidden email]> wrote:
>
>> Hi Glyn,
>>
>>
>>> Can anyone here tell me how the GC memory area size is determined? If
>>> there
>>> is documentation, so much the better as we'd prefer not to depend on code
>>> details that might flux arbitrarily.
>>>
>>>
>> GC memory is mainly data structures used by GC runtime. It can be varied
>> by collector used, size of the Java heap, the number of GC threads and etc.
>> and, of course, the application itself.
>>
>> Some are *fixed* costs, which can be estimated. E.g. two marking bitmaps
>> used by G1, each costs 1/64 of heap size (assuming default object
>> alignment).
>>
>> Some are *semi-fixed*, e.g. taskqueue's fixed cost is about 1M for each
>> queue on 64-bits VM, but it can overflow. And the number of task queues is
>> proportional to the number of GC threads.
>>
>> Then there are factors from application itself,  such as object mutation
>> rate, inter-generation/region references, etc.
>>
>>
>> I don't see a single formula for estimating GC memory size. If you are
>> using G1, the biggest overhead comes from 2 bitmaps (1/32 * heap size).
>>
>> Thanks,
>>
>> -Zhengyu
>>
>>
>>
>>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Excessive "GC" memory area size [was: Re: Determining "GC" memory area size]

Glyn Normington
Thanks Zhengyu! That got me quite a bit further.

The NMT virtual memory map showed the mmap portion of GC space consisting
of:
* the two marking bitmaps you mentioned previously, each 1/64 of heap size.
* a mark stack of 32 MB which is 8* MarkStackSize (but which could grow to
8* MarkStackSizeMax or 128 MB).
* three 700 KB areas, although I'm not sure what these are used for or how
to predict their total size.

This leaves the malloc portion of GC space "17615KB #2312" which I think
was due 2312 calls to malloc. Again, not sure what these are used for or
how to predict their total size except that I presume they include the task
queues you mentioned previously.

Outstanding questions:
1. Is the mmap space which consumed three 700 KB areas above fixed,
proportional to heap size, or what?
2. How many task queues are there for a given number of GC threads?
3. Task queues can overflow, so what is the maximum space each one can
occupy?
4. Of the malloc space not consumed by task queues, how much is fixed and
how much proportional to heap size, number of GC threads, etc.?

The aim is to calculate a reasonable estimate of an upper bound for GC
space consumption.

Raw data for anyone who's interested:
https://gist.github.com/glyn/a3bd944eb2943f4f5a98b5ac712efeb6

Regards,
Glyn

On Wed, Dec 13, 2017 at 12:27 PM, Zhengyu Gu <[hidden email]> wrote:

> Hi Glyn,
>
> You can run NMT with detail tracking (-XX:NativeMemoryTracking=detail)
> then use jcmd (jcmd <pid> VM.native_memory detail) to find out all
> allocations for GC.
>
> For example:
>
> [0x00007f9fba000000 - 0x00007f9fbc000000] reserved 32768KB for *GC* from
>     [0x00007fa0130260ef] G1CMMarkStack::resize(unsigned long)+0x8f
>     [0x00007fa01302cf62] G1ConcurrentMark::G1ConcurrentMark(G1CollectedHeap*,
> G1RegionToSpaceMapper*, G1RegionToSpaceMapper*)+0x782
>     [0x00007fa01300e94b] G1CollectedHeap::initialize()+0x62b
>     [0x00007fa013ca26af] universe_init()+0x6f
>
>         [0x00007f9fba000000 - 0x00007f9fbc000000] committed 32768KB from
>             [0x00007fa01302610d] G1CMMarkStack::resize(unsigned long)+0xad
>             [0x00007fa01302cf62] G1ConcurrentMark::G1ConcurrentMark(G1CollectedHeap*,
> G1RegionToSpaceMapper*, G1RegionToSpaceMapper*)+0x782
>             [0x00007fa01300e94b] G1CollectedHeap::initialize()+0x62b
>             [0x00007fa013ca26af] universe_init()+0x6f
>
>
> [0x00007fa012ac5395] ArrayAllocator<unsigned long>::allocate(unsigned
> long, MemoryType)+0x245
> [0x00007fa012ac5448] unsigned long* BitMap::reallocate<CHeapBitMap
> Allocator>(CHeapBitMapAllocator const&, unsigned long*, unsigned long,
> unsigned long)+0x38
> [0x00007fa012ac509f] CHeapBitMap::initialize(unsigned long)+0x6f
> [0x00007fa013093b9c] G1PageBasedVirtualSpace::G1Pag
> eBasedVirtualSpace(ReservedSpace, unsigned long, unsigned long)+0x13c
>                              (malloc=129KB *type=GC* #6)
>
> Thanks,
>
> -Zhengyu
>
>
>
>
> On 12/13/2017 04:39 AM, Glyn Normington wrote:
>
>> On the basis of the reply below, we have been thinking of estimating the
>> size of the "GC" memory area as a little more than (1/32) * heapSize.
>> However, running `java -version` with a 350m heap results in "GC" memory
>> as
>> reported by NMT of over 17% of the heap size, so this estimate is way out.
>>
>> If the analysis below is accurate, then given the number of GC threads is
>> 10 (ParallelGCThreads=8, ConcGCThreads=2), can there be many more
>> taskqueues than GC threads?
>>
>> Or is there another explanation for the additional "GC" size?
>>
>> Thanks,
>> Glyn
>>
>> [1] https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe
>>
>> On Fri, Nov 10, 2017 at 7:05 PM, Zhengyu Gu <[hidden email]> wrote:
>>
>> Hi Glyn,
>>>
>>>
>>> Can anyone here tell me how the GC memory area size is determined? If
>>>> there
>>>> is documentation, so much the better as we'd prefer not to depend on
>>>> code
>>>> details that might flux arbitrarily.
>>>>
>>>>
>>>> GC memory is mainly data structures used by GC runtime. It can be varied
>>> by collector used, size of the Java heap, the number of GC threads and
>>> etc.
>>> and, of course, the application itself.
>>>
>>> Some are *fixed* costs, which can be estimated. E.g. two marking bitmaps
>>> used by G1, each costs 1/64 of heap size (assuming default object
>>> alignment).
>>>
>>> Some are *semi-fixed*, e.g. taskqueue's fixed cost is about 1M for each
>>> queue on 64-bits VM, but it can overflow. And the number of task queues
>>> is
>>> proportional to the number of GC threads.
>>>
>>> Then there are factors from application itself,  such as object mutation
>>> rate, inter-generation/region references, etc.
>>>
>>>
>>> I don't see a single formula for estimating GC memory size. If you are
>>> using G1, the biggest overhead comes from 2 bitmaps (1/32 * heap size).
>>>
>>> Thanks,
>>>
>>> -Zhengyu
>>>
>>>
>>>
>>>
>>>
>>
>>


--
Regards,
Glyn
Reply | Threaded
Open this post in threaded view
|

Re: Excessive "GC" memory area size [was: Re: Determining "GC" memory area size]

Zhengyu Gu-2
Hi Glyn,

On 12/13/2017 09:35 AM, Glyn Normington wrote:

> Thanks Zhengyu! That got me quite a bit further.
>
> The NMT virtual memory map showed the mmap portion of GC space
> consisting of:
> * the two marking bitmaps you mentioned previously, each 1/64 of heap size.
> * a mark stack of 32 MB which is 8* MarkStackSize (but which could grow
> to 8* MarkStackSizeMax or 128 MB).
> * three 700 KB areas, although I'm not sure what these are used for or
> how to predict their total size.
>
> This leaves the malloc portion of GC space "17615KB #2312" which I think
> was due 2312 calls to malloc. Again, not sure what these are used for or
> how to predict their total size except that I presume they include the
> task queues you mentioned previously.

Can you try on JDK10? it has enhancement [1] which can show what memory
allocation is for, on each allocation site.


[1] https://bugs.openjdk.java.net/browse/JDK-8139673

Thanks,

-Zhengyu

>
> Outstanding questions:
> 1. Is the mmap space which consumed three 700 KB areas above fixed,
> proportional to heap size, or what?
> 2. How many task queues are there for a given number of GC threads?
> 3. Task queues can overflow, so what is the maximum space each one can
> occupy?
> 4. Of the malloc space not consumed by task queues, how much is fixed
> and how much proportional to heap size, number of GC threads, etc.?
>
> The aim is to calculate a reasonable estimate of an upper bound for GC
> space consumption.
>
> Raw data for anyone who's interested:
> https://gist.github.com/glyn/a3bd944eb2943f4f5a98b5ac712efeb6
>
> Regards,
> Glyn
>
> On Wed, Dec 13, 2017 at 12:27 PM, Zhengyu Gu <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi Glyn,
>
>     You can run NMT with detail tracking
>     (-XX:NativeMemoryTracking=detail) then use jcmd (jcmd <pid>
>     VM.native_memory detail) to find out all allocations for GC.
>
>     For example:
>
>     [0x00007f9fba000000 - 0x00007f9fbc000000] reserved 32768KB for *GC* from
>          [0x00007fa0130260ef] G1CMMarkStack::resize(unsigned long)+0x8f
>          [0x00007fa01302cf62]
>     G1ConcurrentMark::G1ConcurrentMark(G1CollectedHeap*,
>     G1RegionToSpaceMapper*, G1RegionToSpaceMapper*)+0x782
>          [0x00007fa01300e94b] G1CollectedHeap::initialize()+0x62b
>          [0x00007fa013ca26af] universe_init()+0x6f
>
>              [0x00007f9fba000000 - 0x00007f9fbc000000] committed 32768KB
>     from
>                  [0x00007fa01302610d] G1CMMarkStack::resize(unsigned
>     long)+0xad
>                  [0x00007fa01302cf62]
>     G1ConcurrentMark::G1ConcurrentMark(G1CollectedHeap*,
>     G1RegionToSpaceMapper*, G1RegionToSpaceMapper*)+0x782
>                  [0x00007fa01300e94b] G1CollectedHeap::initialize()+0x62b
>                  [0x00007fa013ca26af] universe_init()+0x6f
>
>
>     [0x00007fa012ac5395] ArrayAllocator<unsigned
>     long>::allocate(unsigned long, MemoryType)+0x245
>     [0x00007fa012ac5448] unsigned long*
>     BitMap::reallocate<CHeapBitMapAllocator>(CHeapBitMapAllocator
>     const&, unsigned long*, unsigned long, unsigned long)+0x38
>     [0x00007fa012ac509f] CHeapBitMap::initialize(unsigned long)+0x6f
>     [0x00007fa013093b9c]
>     G1PageBasedVirtualSpace::G1PageBasedVirtualSpace(ReservedSpace,
>     unsigned long, unsigned long)+0x13c
>                                   (malloc=129KB *type=GC* #6)
>
>     Thanks,
>
>     -Zhengyu
>
>
>
>
>     On 12/13/2017 04:39 AM, Glyn Normington wrote:
>
>         On the basis of the reply below, we have been thinking of
>         estimating the
>         size of the "GC" memory area as a little more than (1/32) *
>         heapSize.
>         However, running `java -version` with a 350m heap results in
>         "GC" memory as
>         reported by NMT of over 17% of the heap size, so this estimate
>         is way out.
>
>         If the analysis below is accurate, then given the number of GC
>         threads is
>         10 (ParallelGCThreads=8, ConcGCThreads=2), can there be many more
>         taskqueues than GC threads?
>
>         Or is there another explanation for the additional "GC" size?
>
>         Thanks,
>         Glyn
>
>         [1]
>         https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe
>         <https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe>
>
>         On Fri, Nov 10, 2017 at 7:05 PM, Zhengyu Gu <[hidden email]
>         <mailto:[hidden email]>> wrote:
>
>             Hi Glyn,
>
>
>                 Can anyone here tell me how the GC memory area size is
>                 determined? If
>                 there
>                 is documentation, so much the better as we'd prefer not
>                 to depend on code
>                 details that might flux arbitrarily.
>
>
>             GC memory is mainly data structures used by GC runtime. It
>             can be varied
>             by collector used, size of the Java heap, the number of GC
>             threads and etc.
>             and, of course, the application itself.
>
>             Some are *fixed* costs, which can be estimated. E.g. two
>             marking bitmaps
>             used by G1, each costs 1/64 of heap size (assuming default
>             object
>             alignment).
>
>             Some are *semi-fixed*, e.g. taskqueue's fixed cost is about
>             1M for each
>             queue on 64-bits VM, but it can overflow. And the number of
>             task queues is
>             proportional to the number of GC threads.
>
>             Then there are factors from application itself,  such as
>             object mutation
>             rate, inter-generation/region references, etc.
>
>
>             I don't see a single formula for estimating GC memory size.
>             If you are
>             using G1, the biggest overhead comes from 2 bitmaps (1/32 *
>             heap size).
>
>             Thanks,
>
>             -Zhengyu
>
>
>
>
>
>
>
>
>
> --
> Regards,
> Glyn
Reply | Threaded
Open this post in threaded view
|

Re: Excessive "GC" memory area size [was: Re: Determining "GC" memory area size]

Glyn Normington
Hi Zhengyu

Thanks again for a great suggestion. JDK 10 gave more insights such as this
one showing a task queue-related malloc of 8 MB:

0x000000010d128e58]
_ZN20MallocArrayAllocatorI16G1TaskQueueEntryE8allocateEm10MemoryType+0x5c
[0x000000010d12372e]
_ZN16G1ConcurrentMarkC2EP15G1CollectedHeapP21G1RegionToSpaceMapperS3_+0x8e4
[0x000000010d1198af] _ZN15G1CollectedHeap10initializeEv+0x561
[0x000000010d559029] _ZN8Universe15initialize_heapEv+0x4f
                             (malloc=8192KB type=GC #8)

The malloc details don't seem to add up to the total, so maybe some
allocation sites got missed, but it's definitely better than with JDK 8.

Regards,
Glyn


On Wed, Dec 13, 2017 at 3:16 PM, Zhengyu Gu <[hidden email]> wrote:

> Hi Glyn,
>
> On 12/13/2017 09:35 AM, Glyn Normington wrote:
>
>> Thanks Zhengyu! That got me quite a bit further.
>>
>> The NMT virtual memory map showed the mmap portion of GC space consisting
>> of:
>> * the two marking bitmaps you mentioned previously, each 1/64 of heap
>> size.
>> * a mark stack of 32 MB which is 8* MarkStackSize (but which could grow
>> to 8* MarkStackSizeMax or 128 MB).
>> * three 700 KB areas, although I'm not sure what these are used for or
>> how to predict their total size.
>>
>> This leaves the malloc portion of GC space "17615KB #2312" which I think
>> was due 2312 calls to malloc. Again, not sure what these are used for or
>> how to predict their total size except that I presume they include the task
>> queues you mentioned previously.
>>
>
> Can you try on JDK10? it has enhancement [1] which can show what memory
> allocation is for, on each allocation site.
>
>
> [1] https://bugs.openjdk.java.net/browse/JDK-8139673
>
> Thanks,
>
> -Zhengyu
>
>
>> Outstanding questions:
>> 1. Is the mmap space which consumed three 700 KB areas above fixed,
>> proportional to heap size, or what?
>> 2. How many task queues are there for a given number of GC threads?
>> 3. Task queues can overflow, so what is the maximum space each one can
>> occupy?
>> 4. Of the malloc space not consumed by task queues, how much is fixed and
>> how much proportional to heap size, number of GC threads, etc.?
>>
>> The aim is to calculate a reasonable estimate of an upper bound for GC
>> space consumption.
>>
>> Raw data for anyone who's interested: https://gist.github.com/glyn/a
>> 3bd944eb2943f4f5a98b5ac712efeb6
>>
>> Regards,
>> Glyn
>>
>> On Wed, Dec 13, 2017 at 12:27 PM, Zhengyu Gu <[hidden email] <mailto:
>> [hidden email]>> wrote:
>>
>>     Hi Glyn,
>>
>>     You can run NMT with detail tracking
>>     (-XX:NativeMemoryTracking=detail) then use jcmd (jcmd <pid>
>>     VM.native_memory detail) to find out all allocations for GC.
>>
>>     For example:
>>
>>     [0x00007f9fba000000 - 0x00007f9fbc000000] reserved 32768KB for *GC*
>> from
>>          [0x00007fa0130260ef] G1CMMarkStack::resize(unsigned long)+0x8f
>>          [0x00007fa01302cf62]
>>     G1ConcurrentMark::G1ConcurrentMark(G1CollectedHeap*,
>>     G1RegionToSpaceMapper*, G1RegionToSpaceMapper*)+0x782
>>          [0x00007fa01300e94b] G1CollectedHeap::initialize()+0x62b
>>          [0x00007fa013ca26af] universe_init()+0x6f
>>
>>              [0x00007f9fba000000 - 0x00007f9fbc000000] committed 32768KB
>>     from
>>                  [0x00007fa01302610d] G1CMMarkStack::resize(unsigned
>>     long)+0xad
>>                  [0x00007fa01302cf62]
>>     G1ConcurrentMark::G1ConcurrentMark(G1CollectedHeap*,
>>     G1RegionToSpaceMapper*, G1RegionToSpaceMapper*)+0x782
>>                  [0x00007fa01300e94b] G1CollectedHeap::initialize()+0x62b
>>                  [0x00007fa013ca26af] universe_init()+0x6f
>>
>>
>>     [0x00007fa012ac5395] ArrayAllocator<unsigned
>>     long>::allocate(unsigned long, MemoryType)+0x245
>>     [0x00007fa012ac5448] unsigned long*
>>     BitMap::reallocate<CHeapBitMapAllocator>(CHeapBitMapAllocator
>>     const&, unsigned long*, unsigned long, unsigned long)+0x38
>>     [0x00007fa012ac509f] CHeapBitMap::initialize(unsigned long)+0x6f
>>     [0x00007fa013093b9c]
>>     G1PageBasedVirtualSpace::G1PageBasedVirtualSpace(ReservedSpace,
>>     unsigned long, unsigned long)+0x13c
>>                                   (malloc=129KB *type=GC* #6)
>>
>>     Thanks,
>>
>>     -Zhengyu
>>
>>
>>
>>
>>     On 12/13/2017 04:39 AM, Glyn Normington wrote:
>>
>>         On the basis of the reply below, we have been thinking of
>>         estimating the
>>         size of the "GC" memory area as a little more than (1/32) *
>>         heapSize.
>>         However, running `java -version` with a 350m heap results in
>>         "GC" memory as
>>         reported by NMT of over 17% of the heap size, so this estimate
>>         is way out.
>>
>>         If the analysis below is accurate, then given the number of GC
>>         threads is
>>         10 (ParallelGCThreads=8, ConcGCThreads=2), can there be many more
>>         taskqueues than GC threads?
>>
>>         Or is there another explanation for the additional "GC" size?
>>
>>         Thanks,
>>         Glyn
>>
>>         [1]
>>         https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe
>>         <https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe>
>>
>>         On Fri, Nov 10, 2017 at 7:05 PM, Zhengyu Gu <[hidden email]
>>         <mailto:[hidden email]>> wrote:
>>
>>             Hi Glyn,
>>
>>
>>                 Can anyone here tell me how the GC memory area size is
>>                 determined? If
>>                 there
>>                 is documentation, so much the better as we'd prefer not
>>                 to depend on code
>>                 details that might flux arbitrarily.
>>
>>
>>             GC memory is mainly data structures used by GC runtime. It
>>             can be varied
>>             by collector used, size of the Java heap, the number of GC
>>             threads and etc.
>>             and, of course, the application itself.
>>
>>             Some are *fixed* costs, which can be estimated. E.g. two
>>             marking bitmaps
>>             used by G1, each costs 1/64 of heap size (assuming default
>>             object
>>             alignment).
>>
>>             Some are *semi-fixed*, e.g. taskqueue's fixed cost is about
>>             1M for each
>>             queue on 64-bits VM, but it can overflow. And the number of
>>             task queues is
>>             proportional to the number of GC threads.
>>
>>             Then there are factors from application itself,  such as
>>             object mutation
>>             rate, inter-generation/region references, etc.
>>
>>
>>             I don't see a single formula for estimating GC memory size.
>>             If you are
>>             using G1, the biggest overhead comes from 2 bitmaps (1/32 *
>>             heap size).
>>
>>             Thanks,
>>
>>             -Zhengyu
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> Regards,
>> Glyn
>>
>


--
Regards,
Glyn
Reply | Threaded
Open this post in threaded view
|

Re: Excessive "GC" memory area size [was: Re: Determining "GC" memory area size]

Zhengyu Gu-2
Hi Glyn,

On 12/13/2017 11:38 AM, Glyn Normington wrote:

> Hi Zhengyu
>
> Thanks again for a great suggestion. JDK 10 gave more insights such as
> this one showing a task queue-related malloc of 8 MB:
>
> 0x000000010d128e58]
> _ZN20MallocArrayAllocatorI16G1TaskQueueEntryE8allocateEm10MemoryType+0x5c
> [0x000000010d12372e]
> _ZN16G1ConcurrentMarkC2EP15G1CollectedHeapP21G1RegionToSpaceMapperS3_+0x8e4
> [0x000000010d1198af] _ZN15G1CollectedHeap10initializeEv+0x561
> [0x000000010d559029] _ZN8Universe15initialize_heapEv+0x4f
>                               (malloc=8192KB type=GC #8)
>
> The malloc details don't seem to add up to the total, so maybe some
> allocation sites got missed, but it's definitely better than with JDK 8.

Detail tracking does discard allocation sites with small enough
allocation, e.g. < 1K when scale = KB. But I will be surprised if the
disparity is significant.

Thanks,

-Zhengyu



>
> Regards,
> Glyn
>
>
> On Wed, Dec 13, 2017 at 3:16 PM, Zhengyu Gu <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi Glyn,
>
>     On 12/13/2017 09:35 AM, Glyn Normington wrote:
>
>         Thanks Zhengyu! That got me quite a bit further.
>
>         The NMT virtual memory map showed the mmap portion of GC space
>         consisting of:
>         * the two marking bitmaps you mentioned previously, each 1/64 of
>         heap size.
>         * a mark stack of 32 MB which is 8* MarkStackSize (but which
>         could grow to 8* MarkStackSizeMax or 128 MB).
>         * three 700 KB areas, although I'm not sure what these are used
>         for or how to predict their total size.
>
>         This leaves the malloc portion of GC space "17615KB #2312" which
>         I think was due 2312 calls to malloc. Again, not sure what these
>         are used for or how to predict their total size except that I
>         presume they include the task queues you mentioned previously.
>
>
>     Can you try on JDK10? it has enhancement [1] which can show what
>     memory allocation is for, on each allocation site.
>
>
>     [1] https://bugs.openjdk.java.net/browse/JDK-8139673
>     <https://bugs.openjdk.java.net/browse/JDK-8139673>
>
>     Thanks,
>
>     -Zhengyu
>
>
>         Outstanding questions:
>         1. Is the mmap space which consumed three 700 KB areas above
>         fixed, proportional to heap size, or what?
>         2. How many task queues are there for a given number of GC threads?
>         3. Task queues can overflow, so what is the maximum space each
>         one can occupy?
>         4. Of the malloc space not consumed by task queues, how much is
>         fixed and how much proportional to heap size, number of GC
>         threads, etc.?
>
>         The aim is to calculate a reasonable estimate of an upper bound
>         for GC space consumption.
>
>         Raw data for anyone who's interested:
>         https://gist.github.com/glyn/a3bd944eb2943f4f5a98b5ac712efeb6
>         <https://gist.github.com/glyn/a3bd944eb2943f4f5a98b5ac712efeb6>
>
>         Regards,
>         Glyn
>
>         On Wed, Dec 13, 2017 at 12:27 PM, Zhengyu Gu <[hidden email]
>         <mailto:[hidden email]> <mailto:[hidden email]
>         <mailto:[hidden email]>>> wrote:
>
>              Hi Glyn,
>
>              You can run NMT with detail tracking
>              (-XX:NativeMemoryTracking=detail) then use jcmd (jcmd <pid>
>              VM.native_memory detail) to find out all allocations for GC.
>
>              For example:
>
>              [0x00007f9fba000000 - 0x00007f9fbc000000] reserved 32768KB
>         for *GC* from
>                   [0x00007fa0130260ef] G1CMMarkStack::resize(unsigned
>         long)+0x8f
>                   [0x00007fa01302cf62]
>              G1ConcurrentMark::G1ConcurrentMark(G1CollectedHeap*,
>              G1RegionToSpaceMapper*, G1RegionToSpaceMapper*)+0x782
>                   [0x00007fa01300e94b] G1CollectedHeap::initialize()+0x62b
>                   [0x00007fa013ca26af] universe_init()+0x6f
>
>                       [0x00007f9fba000000 - 0x00007f9fbc000000]
>         committed 32768KB
>              from
>                           [0x00007fa01302610d]
>         G1CMMarkStack::resize(unsigned
>              long)+0xad
>                           [0x00007fa01302cf62]
>              G1ConcurrentMark::G1ConcurrentMark(G1CollectedHeap*,
>              G1RegionToSpaceMapper*, G1RegionToSpaceMapper*)+0x782
>                           [0x00007fa01300e94b]
>         G1CollectedHeap::initialize()+0x62b
>                           [0x00007fa013ca26af] universe_init()+0x6f
>
>
>              [0x00007fa012ac5395] ArrayAllocator<unsigned
>              long>::allocate(unsigned long, MemoryType)+0x245
>              [0x00007fa012ac5448] unsigned long*
>              BitMap::reallocate<CHeapBitMapAllocator>(CHeapBitMapAllocator
>              const&, unsigned long*, unsigned long, unsigned long)+0x38
>              [0x00007fa012ac509f] CHeapBitMap::initialize(unsigned
>         long)+0x6f
>              [0x00007fa013093b9c]
>              G1PageBasedVirtualSpace::G1PageBasedVirtualSpace(ReservedSpace,
>              unsigned long, unsigned long)+0x13c
>                                            (malloc=129KB *type=GC* #6)
>
>              Thanks,
>
>              -Zhengyu
>
>
>
>
>              On 12/13/2017 04:39 AM, Glyn Normington wrote:
>
>                  On the basis of the reply below, we have been thinking of
>                  estimating the
>                  size of the "GC" memory area as a little more than (1/32) *
>                  heapSize.
>                  However, running `java -version` with a 350m heap
>         results in
>                  "GC" memory as
>                  reported by NMT of over 17% of the heap size, so this
>         estimate
>                  is way out.
>
>                  If the analysis below is accurate, then given the
>         number of GC
>                  threads is
>                  10 (ParallelGCThreads=8, ConcGCThreads=2), can there be
>         many more
>                  taskqueues than GC threads?
>
>                  Or is there another explanation for the additional "GC"
>         size?
>
>                  Thanks,
>                  Glyn
>
>                  [1]
>         https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe
>         <https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe>
>                
>         <https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe
>         <https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe>>
>
>                  On Fri, Nov 10, 2017 at 7:05 PM, Zhengyu Gu
>         <[hidden email] <mailto:[hidden email]>
>                  <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
>
>                      Hi Glyn,
>
>
>                          Can anyone here tell me how the GC memory area
>         size is
>                          determined? If
>                          there
>                          is documentation, so much the better as we'd
>         prefer not
>                          to depend on code
>                          details that might flux arbitrarily.
>
>
>                      GC memory is mainly data structures used by GC
>         runtime. It
>                      can be varied
>                      by collector used, size of the Java heap, the
>         number of GC
>                      threads and etc.
>                      and, of course, the application itself.
>
>                      Some are *fixed* costs, which can be estimated.
>         E.g. two
>                      marking bitmaps
>                      used by G1, each costs 1/64 of heap size (assuming
>         default
>                      object
>                      alignment).
>
>                      Some are *semi-fixed*, e.g. taskqueue's fixed cost
>         is about
>                      1M for each
>                      queue on 64-bits VM, but it can overflow. And the
>         number of
>                      task queues is
>                      proportional to the number of GC threads.
>
>                      Then there are factors from application itself,
>         such as
>                      object mutation
>                      rate, inter-generation/region references, etc.
>
>
>                      I don't see a single formula for estimating GC
>         memory size.
>                      If you are
>                      using G1, the biggest overhead comes from 2 bitmaps
>         (1/32 *
>                      heap size).
>
>                      Thanks,
>
>                      -Zhengyu
>
>
>
>
>
>
>
>
>
>         --
>         Regards,
>         Glyn
>
>
>
>
> --
> Regards,
> Glyn
Reply | Threaded
Open this post in threaded view
|

Re: Excessive "GC" memory area size [was: Re: Determining "GC" memory area size]

Glyn Normington
On Wed, Dec 13, 2017 at 5:18 PM, Zhengyu Gu <[hidden email]> wrote:

> Hi Glyn,
>
> On 12/13/2017 11:38 AM, Glyn Normington wrote:
>
>> Hi Zhengyu
>>
>> Thanks again for a great suggestion. JDK 10 gave more insights such as
>> this one showing a task queue-related malloc of 8 MB:
>>
>> 0x000000010d128e58] _ZN20MallocArrayAllocatorI16G1
>> TaskQueueEntryE8allocateEm10MemoryType+0x5c
>> [0x000000010d12372e] _ZN16G1ConcurrentMarkC2EP15G1C
>> ollectedHeapP21G1RegionToSpaceMapperS3_+0x8e4
>> [0x000000010d1198af] _ZN15G1CollectedHeap10initializeEv+0x561
>> [0x000000010d559029] _ZN8Universe15initialize_heapEv+0x4f
>>                               (malloc=8192KB type=GC #8)
>>
>> The malloc details don't seem to add up to the total, so maybe some
>> allocation sites got missed, but it's definitely better than with JDK 8.
>>
>
> Detail tracking does discard allocation sites with small enough
> allocation, e.g. < 1K when scale = KB. But I will be surprised if the
> disparity is significant.
>

You're right. The malloc details add up to 17524 KB which is almost equal
to the reported total of 17544 KB.

There was another large malloc, which I guess might also be task
queue-related:

[0x000000010d11dbc4]
_ZN20MallocArrayAllocatorI8StarTaskE8allocateEm10MemoryType+0x5c
[0x000000010d119124] _ZN15G1CollectedHeapC2EP17G1CollectorPolicy+0x82c
[0x000000010d11274a]
_ZN11GCArguments23create_heap_with_policyI15G1CollectedHeap17G1CollectorPolicyEEP13CollectedHeapv+0x44
[0x000000010d559019] _ZN8Universe15initialize_heapEv+0x3f
                             (malloc=8192KB type=GC #8)



>
> Thanks,
>
> -Zhengyu
>
>
>
>
>> Regards,
>> Glyn
>>
>>
>> On Wed, Dec 13, 2017 at 3:16 PM, Zhengyu Gu <[hidden email] <mailto:
>> [hidden email]>> wrote:
>>
>>     Hi Glyn,
>>
>>     On 12/13/2017 09:35 AM, Glyn Normington wrote:
>>
>>         Thanks Zhengyu! That got me quite a bit further.
>>
>>         The NMT virtual memory map showed the mmap portion of GC space
>>         consisting of:
>>         * the two marking bitmaps you mentioned previously, each 1/64 of
>>         heap size.
>>         * a mark stack of 32 MB which is 8* MarkStackSize (but which
>>         could grow to 8* MarkStackSizeMax or 128 MB).
>>         * three 700 KB areas, although I'm not sure what these are used
>>         for or how to predict their total size.
>>
>>         This leaves the malloc portion of GC space "17615KB #2312" which
>>         I think was due 2312 calls to malloc. Again, not sure what these
>>         are used for or how to predict their total size except that I
>>         presume they include the task queues you mentioned previously.
>>
>>
>>     Can you try on JDK10? it has enhancement [1] which can show what
>>     memory allocation is for, on each allocation site.
>>
>>
>>     [1] https://bugs.openjdk.java.net/browse/JDK-8139673
>>     <https://bugs.openjdk.java.net/browse/JDK-8139673>
>>
>>     Thanks,
>>
>>     -Zhengyu
>>
>>
>>         Outstanding questions:
>>         1. Is the mmap space which consumed three 700 KB areas above
>>         fixed, proportional to heap size, or what?
>>         2. How many task queues are there for a given number of GC
>> threads?
>>         3. Task queues can overflow, so what is the maximum space each
>>         one can occupy?
>>         4. Of the malloc space not consumed by task queues, how much is
>>         fixed and how much proportional to heap size, number of GC
>>         threads, etc.?
>>
>>         The aim is to calculate a reasonable estimate of an upper bound
>>         for GC space consumption.
>>
>>         Raw data for anyone who's interested:
>>         https://gist.github.com/glyn/a3bd944eb2943f4f5a98b5ac712efeb6
>>         <https://gist.github.com/glyn/a3bd944eb2943f4f5a98b5ac712efeb6>
>>
>>         Regards,
>>         Glyn
>>
>>         On Wed, Dec 13, 2017 at 12:27 PM, Zhengyu Gu <[hidden email]
>>         <mailto:[hidden email]> <mailto:[hidden email]
>>
>>         <mailto:[hidden email]>>> wrote:
>>
>>              Hi Glyn,
>>
>>              You can run NMT with detail tracking
>>              (-XX:NativeMemoryTracking=detail) then use jcmd (jcmd <pid>
>>              VM.native_memory detail) to find out all allocations for GC.
>>
>>              For example:
>>
>>              [0x00007f9fba000000 - 0x00007f9fbc000000] reserved 32768KB
>>         for *GC* from
>>                   [0x00007fa0130260ef] G1CMMarkStack::resize(unsigned
>>         long)+0x8f
>>                   [0x00007fa01302cf62]
>>              G1ConcurrentMark::G1ConcurrentMark(G1CollectedHeap*,
>>              G1RegionToSpaceMapper*, G1RegionToSpaceMapper*)+0x782
>>                   [0x00007fa01300e94b] G1CollectedHeap::initialize()+
>> 0x62b
>>                   [0x00007fa013ca26af] universe_init()+0x6f
>>
>>                       [0x00007f9fba000000 - 0x00007f9fbc000000]
>>         committed 32768KB
>>              from
>>                           [0x00007fa01302610d]
>>         G1CMMarkStack::resize(unsigned
>>              long)+0xad
>>                           [0x00007fa01302cf62]
>>              G1ConcurrentMark::G1ConcurrentMark(G1CollectedHeap*,
>>              G1RegionToSpaceMapper*, G1RegionToSpaceMapper*)+0x782
>>                           [0x00007fa01300e94b]
>>         G1CollectedHeap::initialize()+0x62b
>>                           [0x00007fa013ca26af] universe_init()+0x6f
>>
>>
>>              [0x00007fa012ac5395] ArrayAllocator<unsigned
>>              long>::allocate(unsigned long, MemoryType)+0x245
>>              [0x00007fa012ac5448] unsigned long*
>>              BitMap::reallocate<CHeapBitMapAllocator>(CHeapBitMapAllocat
>> or
>>              const&, unsigned long*, unsigned long, unsigned long)+0x38
>>              [0x00007fa012ac509f] CHeapBitMap::initialize(unsigned
>>         long)+0x6f
>>              [0x00007fa013093b9c]
>>              G1PageBasedVirtualSpace::G1PageBasedVirtualSpace(ReservedSp
>> ace,
>>              unsigned long, unsigned long)+0x13c
>>                                            (malloc=129KB *type=GC* #6)
>>
>>              Thanks,
>>
>>              -Zhengyu
>>
>>
>>
>>
>>              On 12/13/2017 04:39 AM, Glyn Normington wrote:
>>
>>                  On the basis of the reply below, we have been thinking of
>>                  estimating the
>>                  size of the "GC" memory area as a little more than
>> (1/32) *
>>                  heapSize.
>>                  However, running `java -version` with a 350m heap
>>         results in
>>                  "GC" memory as
>>                  reported by NMT of over 17% of the heap size, so this
>>         estimate
>>                  is way out.
>>
>>                  If the analysis below is accurate, then given the
>>         number of GC
>>                  threads is
>>                  10 (ParallelGCThreads=8, ConcGCThreads=2), can there be
>>         many more
>>                  taskqueues than GC threads?
>>
>>                  Or is there another explanation for the additional "GC"
>>         size?
>>
>>                  Thanks,
>>                  Glyn
>>
>>                  [1]
>>         https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe
>>         <https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe>
>>                         <https://gist.github.com/glyn/
>> f10b7dea77994ad0f5589a416fbc77fe
>>         <https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe>>
>>
>>                  On Fri, Nov 10, 2017 at 7:05 PM, Zhengyu Gu
>>         <[hidden email] <mailto:[hidden email]>
>>                  <mailto:[hidden email] <mailto:[hidden email]>>> wrote:
>>
>>                      Hi Glyn,
>>
>>
>>                          Can anyone here tell me how the GC memory area
>>         size is
>>                          determined? If
>>                          there
>>                          is documentation, so much the better as we'd
>>         prefer not
>>                          to depend on code
>>                          details that might flux arbitrarily.
>>
>>
>>                      GC memory is mainly data structures used by GC
>>         runtime. It
>>                      can be varied
>>                      by collector used, size of the Java heap, the
>>         number of GC
>>                      threads and etc.
>>                      and, of course, the application itself.
>>
>>                      Some are *fixed* costs, which can be estimated.
>>         E.g. two
>>                      marking bitmaps
>>                      used by G1, each costs 1/64 of heap size (assuming
>>         default
>>                      object
>>                      alignment).
>>
>>                      Some are *semi-fixed*, e.g. taskqueue's fixed cost
>>         is about
>>                      1M for each
>>                      queue on 64-bits VM, but it can overflow. And the
>>         number of
>>                      task queues is
>>                      proportional to the number of GC threads.
>>
>>                      Then there are factors from application itself,
>>    such as
>>                      object mutation
>>                      rate, inter-generation/region references, etc.
>>
>>
>>                      I don't see a single formula for estimating GC
>>         memory size.
>>                      If you are
>>                      using G1, the biggest overhead comes from 2 bitmaps
>>         (1/32 *
>>                      heap size).
>>
>>                      Thanks,
>>
>>                      -Zhengyu
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>         --         Regards,
>>         Glyn
>>
>>
>>
>>
>> --
>> Regards,
>> Glyn
>>
>


--
Regards,
Glyn
Reply | Threaded
Open this post in threaded view
|

Re: Excessive "GC" memory area size [was: Re: Determining "GC" memory area size]

Zhengyu Gu-2


On 12/14/2017 06:19 AM, Glyn Normington wrote:

> On Wed, Dec 13, 2017 at 5:18 PM, Zhengyu Gu <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi Glyn,
>
>     On 12/13/2017 11:38 AM, Glyn Normington wrote:
>
>         Hi Zhengyu
>
>         Thanks again for a great suggestion. JDK 10 gave more insights
>         such as this one showing a task queue-related malloc of 8 MB:
>
>         0x000000010d128e58]
>         _ZN20MallocArrayAllocatorI16G1TaskQueueEntryE8allocateEm10MemoryType+0x5c
>         [0x000000010d12372e]
>         _ZN16G1ConcurrentMarkC2EP15G1CollectedHeapP21G1RegionToSpaceMapperS3_+0x8e4
>         [0x000000010d1198af] _ZN15G1CollectedHeap10initializeEv+0x561
>         [0x000000010d559029] _ZN8Universe15initialize_heapEv+0x4f
>                                        (malloc=8192KB type=GC #8)
>
>         The malloc details don't seem to add up to the total, so maybe
>         some allocation sites got missed, but it's definitely better
>         than with JDK 8.
>
>
>     Detail tracking does discard allocation sites with small enough
>     allocation, e.g. < 1K when scale = KB. But I will be surprised if
>     the disparity is significant.
>
>
> You're right. The malloc details add up to 17524 KB which is almost
> equal to the reported total of 17544 KB.
>
> There was another large malloc, which I guess might also be task
> queue-related:
>
> [0x000000010d11dbc4]
> _ZN20MallocArrayAllocatorI8StarTaskE8allocateEm10MemoryType+0x5c
> [0x000000010d119124] _ZN15G1CollectedHeapC2EP17G1CollectorPolicy+0x82c
> [0x000000010d11274a]
> _ZN11GCArguments23create_heap_with_policyI15G1CollectedHeap17G1CollectorPolicyEEP13CollectedHeapv+0x44
> [0x000000010d559019] _ZN8Universe15initialize_heapEv+0x3f
>                               (malloc=8192KB type=GC #8)
>
>
Yes, looks like creation of task queues.


-Zhengyu

>     Thanks,
>
>     -Zhengyu
>
>
>
>
>         Regards,
>         Glyn
>
>
>         On Wed, Dec 13, 2017 at 3:16 PM, Zhengyu Gu <[hidden email]
>         <mailto:[hidden email]> <mailto:[hidden email]
>         <mailto:[hidden email]>>> wrote:
>
>              Hi Glyn,
>
>              On 12/13/2017 09:35 AM, Glyn Normington wrote:
>
>                  Thanks Zhengyu! That got me quite a bit further.
>
>                  The NMT virtual memory map showed the mmap portion of
>         GC space
>                  consisting of:
>                  * the two marking bitmaps you mentioned previously,
>         each 1/64 of
>                  heap size.
>                  * a mark stack of 32 MB which is 8* MarkStackSize (but
>         which
>                  could grow to 8* MarkStackSizeMax or 128 MB).
>                  * three 700 KB areas, although I'm not sure what these
>         are used
>                  for or how to predict their total size.
>
>                  This leaves the malloc portion of GC space "17615KB
>         #2312" which
>                  I think was due 2312 calls to malloc. Again, not sure
>         what these
>                  are used for or how to predict their total size except
>         that I
>                  presume they include the task queues you mentioned
>         previously.
>
>
>              Can you try on JDK10? it has enhancement [1] which can show
>         what
>              memory allocation is for, on each allocation site.
>
>
>              [1] https://bugs.openjdk.java.net/browse/JDK-8139673
>         <https://bugs.openjdk.java.net/browse/JDK-8139673>
>              <https://bugs.openjdk.java.net/browse/JDK-8139673
>         <https://bugs.openjdk.java.net/browse/JDK-8139673>>
>
>              Thanks,
>
>              -Zhengyu
>
>
>                  Outstanding questions:
>                  1. Is the mmap space which consumed three 700 KB areas
>         above
>                  fixed, proportional to heap size, or what?
>                  2. How many task queues are there for a given number of
>         GC threads?
>                  3. Task queues can overflow, so what is the maximum
>         space each
>                  one can occupy?
>                  4. Of the malloc space not consumed by task queues, how
>         much is
>                  fixed and how much proportional to heap size, number of GC
>                  threads, etc.?
>
>                  The aim is to calculate a reasonable estimate of an
>         upper bound
>                  for GC space consumption.
>
>                  Raw data for anyone who's interested:
>         https://gist.github.com/glyn/a3bd944eb2943f4f5a98b5ac712efeb6
>         <https://gist.github.com/glyn/a3bd944eb2943f4f5a98b5ac712efeb6>
>                
>         <https://gist.github.com/glyn/a3bd944eb2943f4f5a98b5ac712efeb6
>         <https://gist.github.com/glyn/a3bd944eb2943f4f5a98b5ac712efeb6>>
>
>                  Regards,
>                  Glyn
>
>                  On Wed, Dec 13, 2017 at 12:27 PM, Zhengyu Gu
>         <[hidden email] <mailto:[hidden email]>
>                  <mailto:[hidden email] <mailto:[hidden email]>>
>         <mailto:[hidden email] <mailto:[hidden email]>
>
>                  <mailto:[hidden email] <mailto:[hidden email]>>>> wrote:
>
>                       Hi Glyn,
>
>                       You can run NMT with detail tracking
>                       (-XX:NativeMemoryTracking=detail) then use jcmd
>         (jcmd <pid>
>                       VM.native_memory detail) to find out all
>         allocations for GC.
>
>                       For example:
>
>                       [0x00007f9fba000000 - 0x00007f9fbc000000] reserved
>         32768KB
>                  for *GC* from
>                            [0x00007fa0130260ef]
>         G1CMMarkStack::resize(unsigned
>                  long)+0x8f
>                            [0x00007fa01302cf62]
>                       G1ConcurrentMark::G1ConcurrentMark(G1CollectedHeap*,
>                       G1RegionToSpaceMapper*, G1RegionToSpaceMapper*)+0x782
>                            [0x00007fa01300e94b]
>         G1CollectedHeap::initialize()+0x62b
>                            [0x00007fa013ca26af] universe_init()+0x6f
>
>                                [0x00007f9fba000000 - 0x00007f9fbc000000]
>                  committed 32768KB
>                       from
>                                    [0x00007fa01302610d]
>                  G1CMMarkStack::resize(unsigned
>                       long)+0xad
>                                    [0x00007fa01302cf62]
>                       G1ConcurrentMark::G1ConcurrentMark(G1CollectedHeap*,
>                       G1RegionToSpaceMapper*, G1RegionToSpaceMapper*)+0x782
>                                    [0x00007fa01300e94b]
>                  G1CollectedHeap::initialize()+0x62b
>                                    [0x00007fa013ca26af] universe_init()+0x6f
>
>
>                       [0x00007fa012ac5395] ArrayAllocator<unsigned
>                       long>::allocate(unsigned long, MemoryType)+0x245
>                       [0x00007fa012ac5448] unsigned long*
>                    
>           BitMap::reallocate<CHeapBitMapAllocator>(CHeapBitMapAllocator
>                       const&, unsigned long*, unsigned long, unsigned
>         long)+0x38
>                       [0x00007fa012ac509f] CHeapBitMap::initialize(unsigned
>                  long)+0x6f
>                       [0x00007fa013093b9c]
>                    
>           G1PageBasedVirtualSpace::G1PageBasedVirtualSpace(ReservedSpace,
>                       unsigned long, unsigned long)+0x13c
>                                                     (malloc=129KB
>         *type=GC* #6)
>
>                       Thanks,
>
>                       -Zhengyu
>
>
>
>
>                       On 12/13/2017 04:39 AM, Glyn Normington wrote:
>
>                           On the basis of the reply below, we have been
>         thinking of
>                           estimating the
>                           size of the "GC" memory area as a little more
>         than (1/32) *
>                           heapSize.
>                           However, running `java -version` with a 350m heap
>                  results in
>                           "GC" memory as
>                           reported by NMT of over 17% of the heap size,
>         so this
>                  estimate
>                           is way out.
>
>                           If the analysis below is accurate, then given the
>                  number of GC
>                           threads is
>                           10 (ParallelGCThreads=8, ConcGCThreads=2), can
>         there be
>                  many more
>                           taskqueues than GC threads?
>
>                           Or is there another explanation for the
>         additional "GC"
>                  size?
>
>                           Thanks,
>                           Glyn
>
>                           [1]
>         https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe
>         <https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe>
>                
>         <https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe
>         <https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe>>
>                                
>         <https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe
>         <https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe>
>                
>         <https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe
>         <https://gist.github.com/glyn/f10b7dea77994ad0f5589a416fbc77fe>>>
>
>                           On Fri, Nov 10, 2017 at 7:05 PM, Zhengyu Gu
>                  <[hidden email] <mailto:[hidden email]>
>         <mailto:[hidden email] <mailto:[hidden email]>>
>                           <mailto:[hidden email] <mailto:[hidden email]>
>         <mailto:[hidden email] <mailto:[hidden email]>>>> wrote:
>
>                               Hi Glyn,
>
>
>                                   Can anyone here tell me how the GC
>         memory area
>                  size is
>                                   determined? If
>                                   there
>                                   is documentation, so much the better
>         as we'd
>                  prefer not
>                                   to depend on code
>                                   details that might flux arbitrarily.
>
>
>                               GC memory is mainly data structures used by GC
>                  runtime. It
>                               can be varied
>                               by collector used, size of the Java heap, the
>                  number of GC
>                               threads and etc.
>                               and, of course, the application itself.
>
>                               Some are *fixed* costs, which can be
>         estimated.
>                  E.g. two
>                               marking bitmaps
>                               used by G1, each costs 1/64 of heap size
>         (assuming
>                  default
>                               object
>                               alignment).
>
>                               Some are *semi-fixed*, e.g. taskqueue's
>         fixed cost
>                  is about
>                               1M for each
>                               queue on 64-bits VM, but it can overflow.
>         And the
>                  number of
>                               task queues is
>                               proportional to the number of GC threads.
>
>                               Then there are factors from application
>         itself,         such as
>                               object mutation
>                               rate, inter-generation/region references, etc.
>
>
>                               I don't see a single formula for estimating GC
>                  memory size.
>                               If you are
>                               using G1, the biggest overhead comes from
>         2 bitmaps
>                  (1/32 *
>                               heap size).
>
>                               Thanks,
>
>                               -Zhengyu
>
>
>
>
>
>
>
>
>
>                  --         Regards,
>                  Glyn
>
>
>
>
>         --
>         Regards,
>         Glyn
>
>
>
>
> --
> Regards,
> Glyn