OperatingSystemMXBean unaware of container memory limits

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

OperatingSystemMXBean unaware of container memory limits

Andrew Azores
Hi all,

Apologies if this is not the most appropriate list, in which case
please direct me where to go.

I've noticed a surprising result from the
com.sun.management.OperatingSystemMXBean implementation when running in
a containerized (specifically, using Docker on Linux) environment. The
bean appears to be container-aware for processors, in that running with
Docker option `--cpus 1.0` for example, on a multicore system, will
cause both java.lang.Runtime#availableProcessors and
java.lang.management.OperatingSystemMXBean#getAvailableProcessors /
com.sun.management.OperatingSystemMXBean#getAvailableProcessors to
return 1. However, the Docker option `--memory 100M` (or any other
limit value) is not reflected in the value returned by
com.sun.management.OperatingSystemMXBeam#getTotalPhysicalMemorySize,
and instead the returned value is still the total physical memory of
the host machine - of which only a small portion may actually be
available to the "Operating System" of the JVM. Similarly for the
methods regarding free physical memory, total swap, and free swap.

I have attached a patch which adds a small reproducer to the existing
MemoryAwareness test.

This seems like a bug to me, since if the imposed container limit on
processors as a resource is included as part of the "Operating System"
resource reporting, then surely memory resources should be reported the
same way. As I said, I found the current behaviour quite surprising.

--
Andrew Azores
Software Engineer, OpenJDK Team
Red Hat

jdk-osmxbean-container-memory-test-01.patch (7K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: OperatingSystemMXBean unaware of container memory limits

Bob Vandette
Hi Andrew,

I am aware of the limitations of the OperatingSystemMXBean and was
hoping to address these limitations during the implementation of

It would be helpful if you feel this is important to add some votes to this issue.

Bob.


On Jun 20, 2019, at 9:43 AM, Andrew Azores <[hidden email]> wrote:

Hi all,

Apologies if this is not the most appropriate list, in which case
please direct me where to go.

I've noticed a surprising result from the
com.sun.management.OperatingSystemMXBean implementation when running in
a containerized (specifically, using Docker on Linux) environment. The
bean appears to be container-aware for processors, in that running with
Docker option `--cpus 1.0` for example, on a multicore system, will
cause both java.lang.Runtime#availableProcessors and
java.lang.management.OperatingSystemMXBean#getAvailableProcessors /
com.sun.management.OperatingSystemMXBean#getAvailableProcessors to
return 1. However, the Docker option `--memory 100M` (or any other
limit value) is not reflected in the value returned by
com.sun.management.OperatingSystemMXBeam#getTotalPhysicalMemorySize,
and instead the returned value is still the total physical memory of
the host machine - of which only a small portion may actually be
available to the "Operating System" of the JVM. Similarly for the
methods regarding free physical memory, total swap, and free swap.

I have attached a patch which adds a small reproducer to the existing
MemoryAwareness test.

This seems like a bug to me, since if the imposed container limit on
processors as a resource is included as part of the "Operating System"
resource reporting, then surely memory resources should be reported the
same way. As I said, I found the current behaviour quite surprising.

--
Andrew Azores
Software Engineer, OpenJDK Team
Red Hat
<jdk-osmxbean-container-memory-test-01.patch>

Reply | Threaded
Open this post in threaded view
|

Re: OperatingSystemMXBean unaware of container memory limits

Severin Gehwolf
Hi Bob,

On Thu, 2019-06-20 at 10:16 -0400, Bob Vandette wrote:
> Hi Andrew,
>
> I am aware of the limitations of the OperatingSystemMXBean and was
> hoping to address these limitations during the implementation of
> https://bugs.openjdk.java.net/browse/JDK-8199944.
>
> It would be helpful if you feel this is important to add some votes
> to this issue.

It seems strange that the getAvailableProcessors() returns the
container limit, while the memory limits are for the physical host. If
anything, shouldn't they agree (both physical host or both container
limits)?

When I briefly looked into it initially it seems to be a side-effect of
what is being used by the JDK code implementation-wise. IIRC
getAvailableProcessors() uses a runtime call. Memory reporting has it's
own logic[1], thus not reporting the container limit.

This seems weird from a consistency perspective. Thoughts?

If I understand you correctly, you are arguing that container reporting
should go into its own MX bean. On the other hand, CPU reporting for
the OS MX bean is container aware already. That makes me believe we
should rather make this consistent before evaluating a new MX bean.

Thanks,
Severin

[1] http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365


> Bob.
>
>
> > On Jun 20, 2019, at 9:43 AM, Andrew Azores <[hidden email]>
> > wrote:
> >
> > Hi all,
> >
> > Apologies if this is not the most appropriate list, in which case
> > please direct me where to go.
> >
> > I've noticed a surprising result from the
> > com.sun.management.OperatingSystemMXBean implementation when
> > running in
> > a containerized (specifically, using Docker on Linux) environment.
> > The
> > bean appears to be container-aware for processors, in that running
> > with
> > Docker option `--cpus 1.0` for example, on a multicore system, will
> > cause both java.lang.Runtime#availableProcessors and
> > java.lang.management.OperatingSystemMXBean#getAvailableProcessors /
> > com.sun.management.OperatingSystemMXBean#getAvailableProcessors to
> > return 1. However, the Docker option `--memory 100M` (or any other
> > limit value) is not reflected in the value returned by
> > com.sun.management.OperatingSystemMXBeam#getTotalPhysicalMemorySize
> > ,
> > and instead the returned value is still the total physical memory
> > of
> > the host machine - of which only a small portion may actually be
> > available to the "Operating System" of the JVM. Similarly for the
> > methods regarding free physical memory, total swap, and free swap.
> >
> > I have attached a patch which adds a small reproducer to the
> > existing
> > MemoryAwareness test.
> >
> > This seems like a bug to me, since if the imposed container limit
> > on
> > processors as a resource is included as part of the "Operating
> > System"
> > resource reporting, then surely memory resources should be reported
> > the
> > same way. As I said, I found the current behaviour quite
> > surprising.
> >
> > --
> > Andrew Azores
> > Software Engineer, OpenJDK Team
> > Red Hat
> > <jdk-osmxbean-container-memory-test-01.patch>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: OperatingSystemMXBean unaware of container memory limits

Mario Torre-5
The way I understood the bug report is a two fold approach, i.e. fix the os bean and *possibly* add a container one or extend it to add more data.

I agree with you and Andrew that the current OS Bean returns wrong information, this should be fixed in any case I think.

Bob, do you have some design ideas to share regarding the new interface? I think this may be an interesting project to pick up, even for students wanting to write a thesis, as it seems time is a limiting factor here for you to work on that?

Cheers,
Mario

On Fri 21. Jun 2019 at 10:53, Severin Gehwolf <[hidden email]> wrote:
Hi Bob,

On Thu, 2019-06-20 at 10:16 -0400, Bob Vandette wrote:
> Hi Andrew,
>
> I am aware of the limitations of the OperatingSystemMXBean and was
> hoping to address these limitations during the implementation of
> https://bugs.openjdk.java.net/browse/JDK-8199944.
>
> It would be helpful if you feel this is important to add some votes
> to this issue.

It seems strange that the getAvailableProcessors() returns the
container limit, while the memory limits are for the physical host. If
anything, shouldn't they agree (both physical host or both container
limits)?

When I briefly looked into it initially it seems to be a side-effect of
what is being used by the JDK code implementation-wise. IIRC
getAvailableProcessors() uses a runtime call. Memory reporting has it's
own logic[1], thus not reporting the container limit.

This seems weird from a consistency perspective. Thoughts?

If I understand you correctly, you are arguing that container reporting
should go into its own MX bean. On the other hand, CPU reporting for
the OS MX bean is container aware already. That makes me believe we
should rather make this consistent before evaluating a new MX bean.

Thanks,
Severin

[1] http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365


> Bob.
>
>
> > On Jun 20, 2019, at 9:43 AM, Andrew Azores <[hidden email]>
> > wrote:
> >
> > Hi all,
> >
> > Apologies if this is not the most appropriate list, in which case
> > please direct me where to go.
> >
> > I've noticed a surprising result from the
> > com.sun.management.OperatingSystemMXBean implementation when
> > running in
> > a containerized (specifically, using Docker on Linux) environment.
> > The
> > bean appears to be container-aware for processors, in that running
> > with
> > Docker option `--cpus 1.0` for example, on a multicore system, will
> > cause both java.lang.Runtime#availableProcessors and
> > java.lang.management.OperatingSystemMXBean#getAvailableProcessors /
> > com.sun.management.OperatingSystemMXBean#getAvailableProcessors to
> > return 1. However, the Docker option `--memory 100M` (or any other
> > limit value) is not reflected in the value returned by
> > com.sun.management.OperatingSystemMXBeam#getTotalPhysicalMemorySize
> > ,
> > and instead the returned value is still the total physical memory
> > of
> > the host machine - of which only a small portion may actually be
> > available to the "Operating System" of the JVM. Similarly for the
> > methods regarding free physical memory, total swap, and free swap.
> >
> > I have attached a patch which adds a small reproducer to the
> > existing
> > MemoryAwareness test.
> >
> > This seems like a bug to me, since if the imposed container limit
> > on
> > processors as a resource is included as part of the "Operating
> > System"
> > resource reporting, then surely memory resources should be reported
> > the
> > same way. As I said, I found the current behaviour quite
> > surprising.
> >
> > --
> > Andrew Azores
> > Software Engineer, OpenJDK Team
> > Red Hat
> > <jdk-osmxbean-container-memory-test-01.patch>
>
>

--
pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF
Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF

Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens
Proud GNU Classpath developer: http://www.classpath.org/
OpenJDK: http://openjdk.java.net/projects/caciocavallo/

Please, support open standards:
http://endsoftpatents.org/
Reply | Threaded
Open this post in threaded view
|

Re: OperatingSystemMXBean unaware of container memory limits

Kirk Pepperdine-2
Hi Mario,

I don’t believe the MBean returns the wrong information. Is it not that the calls by-pass the container? Would it not be more appropriate to add a container aware mean? From a tooling perspective it’s a mistake to completely seal the JVM away from the hardware as it makes certain diagnostics more difficult to perform.

Kind regards,
Kirk


On Jun 21, 2019, at 6:06 AM, Mario Torre <[hidden email]> wrote:

The way I understood the bug report is a two fold approach, i.e. fix the os bean and *possibly* add a container one or extend it to add more data.

I agree with you and Andrew that the current OS Bean returns wrong information, this should be fixed in any case I think.

Bob, do you have some design ideas to share regarding the new interface? I think this may be an interesting project to pick up, even for students wanting to write a thesis, as it seems time is a limiting factor here for you to work on that?

Cheers,
Mario

On Fri 21. Jun 2019 at 10:53, Severin Gehwolf <[hidden email]> wrote:
Hi Bob,

On Thu, 2019-06-20 at 10:16 -0400, Bob Vandette wrote:
> Hi Andrew,
>
> I am aware of the limitations of the OperatingSystemMXBean and was
> hoping to address these limitations during the implementation of
> https://bugs.openjdk.java.net/browse/JDK-8199944.
>
> It would be helpful if you feel this is important to add some votes
> to this issue.

It seems strange that the getAvailableProcessors() returns the
container limit, while the memory limits are for the physical host. If
anything, shouldn't they agree (both physical host or both container
limits)?

When I briefly looked into it initially it seems to be a side-effect of
what is being used by the JDK code implementation-wise. IIRC
getAvailableProcessors() uses a runtime call. Memory reporting has it's
own logic[1], thus not reporting the container limit.

This seems weird from a consistency perspective. Thoughts?

If I understand you correctly, you are arguing that container reporting
should go into its own MX bean. On the other hand, CPU reporting for
the OS MX bean is container aware already. That makes me believe we
should rather make this consistent before evaluating a new MX bean.

Thanks,
Severin

[1] http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365


> Bob.
>
>
> > On Jun 20, 2019, at 9:43 AM, Andrew Azores <[hidden email]>
> > wrote:
> >
> > Hi all,
> >
> > Apologies if this is not the most appropriate list, in which case
> > please direct me where to go.
> >
> > I've noticed a surprising result from the
> > com.sun.management.OperatingSystemMXBean implementation when
> > running in
> > a containerized (specifically, using Docker on Linux) environment.
> > The
> > bean appears to be container-aware for processors, in that running
> > with
> > Docker option `--cpus 1.0` for example, on a multicore system, will
> > cause both java.lang.Runtime#availableProcessors and
> > java.lang.management.OperatingSystemMXBean#getAvailableProcessors /
> > com.sun.management.OperatingSystemMXBean#getAvailableProcessors to
> > return 1. However, the Docker option `--memory 100M` (or any other
> > limit value) is not reflected in the value returned by
> > com.sun.management.OperatingSystemMXBeam#getTotalPhysicalMemorySize
> > ,
> > and instead the returned value is still the total physical memory
> > of
> > the host machine - of which only a small portion may actually be
> > available to the "Operating System" of the JVM. Similarly for the
> > methods regarding free physical memory, total swap, and free swap.
> >
> > I have attached a patch which adds a small reproducer to the
> > existing
> > MemoryAwareness test.
> >
> > This seems like a bug to me, since if the imposed container limit
> > on
> > processors as a resource is included as part of the "Operating
> > System"
> > resource reporting, then surely memory resources should be reported
> > the
> > same way. As I said, I found the current behaviour quite
> > surprising.
> >
> > --
> > Andrew Azores
> > Software Engineer, OpenJDK Team
> > Red Hat
> > <jdk-osmxbean-container-memory-test-01.patch>
>
>

--
pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF
Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF

Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens
Proud GNU Classpath developer: http://www.classpath.org/
OpenJDK: http://openjdk.java.net/projects/caciocavallo/

Please, support open standards:
http://endsoftpatents.org/

Reply | Threaded
Open this post in threaded view
|

Re: OperatingSystemMXBean unaware of container memory limits

Mario Torre-5
Hi Kirk,

I think I understand what you mean, however then the OS Bean should be consistent regarding CPU information as well.

I think I remember why this was fixed the way it is now was because of incorrect behavior during GC configuration, or something along those lines, so I guess we would need two APIs after all to give tooling a way to sneak into the actual hardware properties.

I would guess, however, that from the point of view of a containerised VM its OS is the container not the bare metal (or virtualized metal perhaps), so tooling would need to check specifically for a separate bean.

That could be indicated via a property in the OS Bean (if it’s not the case already).

Nevertheless, I think consistency in the OS Bean should be achieved.

Cheers,
Mario 

On Fri 21. Jun 2019 at 13:23, Kirk Pepperdine <[hidden email]> wrote:
Hi Mario,

I don’t believe the MBean returns the wrong information. Is it not that the calls by-pass the container? Would it not be more appropriate to add a container aware mean? From a tooling perspective it’s a mistake to completely seal the JVM away from the hardware as it makes certain diagnostics more difficult to perform.

Kind regards,
Kirk


On Jun 21, 2019, at 6:06 AM, Mario Torre <[hidden email]> wrote:

The way I understood the bug report is a two fold approach, i.e. fix the os bean and *possibly* add a container one or extend it to add more data.

I agree with you and Andrew that the current OS Bean returns wrong information, this should be fixed in any case I think.

Bob, do you have some design ideas to share regarding the new interface? I think this may be an interesting project to pick up, even for students wanting to write a thesis, as it seems time is a limiting factor here for you to work on that?

Cheers,
Mario

On Fri 21. Jun 2019 at 10:53, Severin Gehwolf <[hidden email]> wrote:
Hi Bob,

On Thu, 2019-06-20 at 10:16 -0400, Bob Vandette wrote:
> Hi Andrew,
>
> I am aware of the limitations of the OperatingSystemMXBean and was
> hoping to address these limitations during the implementation of
> https://bugs.openjdk.java.net/browse/JDK-8199944.
>
> It would be helpful if you feel this is important to add some votes
> to this issue.

It seems strange that the getAvailableProcessors() returns the
container limit, while the memory limits are for the physical host. If
anything, shouldn't they agree (both physical host or both container
limits)?

When I briefly looked into it initially it seems to be a side-effect of
what is being used by the JDK code implementation-wise. IIRC
getAvailableProcessors() uses a runtime call. Memory reporting has it's
own logic[1], thus not reporting the container limit.

This seems weird from a consistency perspective. Thoughts?

If I understand you correctly, you are arguing that container reporting
should go into its own MX bean. On the other hand, CPU reporting for
the OS MX bean is container aware already. That makes me believe we
should rather make this consistent before evaluating a new MX bean.

Thanks,
Severin

[1] http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365


> Bob.
>
>
> > On Jun 20, 2019, at 9:43 AM, Andrew Azores <[hidden email]>
> > wrote:
> >
> > Hi all,
> >
> > Apologies if this is not the most appropriate list, in which case
> > please direct me where to go.
> >
> > I've noticed a surprising result from the
> > com.sun.management.OperatingSystemMXBean implementation when
> > running in
> > a containerized (specifically, using Docker on Linux) environment.
> > The
> > bean appears to be container-aware for processors, in that running
> > with
> > Docker option `--cpus 1.0` for example, on a multicore system, will
> > cause both java.lang.Runtime#availableProcessors and
> > java.lang.management.OperatingSystemMXBean#getAvailableProcessors /
> > com.sun.management.OperatingSystemMXBean#getAvailableProcessors to
> > return 1. However, the Docker option `--memory 100M` (or any other
> > limit value) is not reflected in the value returned by
> > com.sun.management.OperatingSystemMXBeam#getTotalPhysicalMemorySize
> > ,
> > and instead the returned value is still the total physical memory
> > of
> > the host machine - of which only a small portion may actually be
> > available to the "Operating System" of the JVM. Similarly for the
> > methods regarding free physical memory, total swap, and free swap.
> >
> > I have attached a patch which adds a small reproducer to the
> > existing
> > MemoryAwareness test.
> >
> > This seems like a bug to me, since if the imposed container limit
> > on
> > processors as a resource is included as part of the "Operating
> > System"
> > resource reporting, then surely memory resources should be reported
> > the
> > same way. As I said, I found the current behaviour quite
> > surprising.
> >
> > --
> > Andrew Azores
> > Software Engineer, OpenJDK Team
> > Red Hat
> > <jdk-osmxbean-container-memory-test-01.patch>
>
>

--
pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF
Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF

Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens
Proud GNU Classpath developer: http://www.classpath.org/
OpenJDK: http://openjdk.java.net/projects/caciocavallo/

Please, support open standards:
http://endsoftpatents.org/

--
pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF
Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF

Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens
Proud GNU Classpath developer: http://www.classpath.org/
OpenJDK: http://openjdk.java.net/projects/caciocavallo/

Please, support open standards:
http://endsoftpatents.org/
Reply | Threaded
Open this post in threaded view
|

Re: OperatingSystemMXBean unaware of container memory limits

Bob Vandette
I agree with you Mario.  When I originally designed the jdk.internal.platform.Metrics
API, I was considering providing both a Host and Container Metrics implementation.
The problem is that the primary goal of containers is to provide isolation (hiding access to
the host) and it would be very difficult to provide reliable forms of both metrics from within
a container.  The Linux kernel does leak some host metric information but not everything.
This is why I decided to focus on the container configuration and metric data.  If Host
metrics are of interest you can always just run Java outside of a container.

Bob.

On Jun 21, 2019, at 8:27 AM, Mario Torre <[hidden email]> wrote:

Hi Kirk,

I think I understand what you mean, however then the OS Bean should be consistent regarding CPU information as well.

I think I remember why this was fixed the way it is now was because of incorrect behavior during GC configuration, or something along those lines, so I guess we would need two APIs after all to give tooling a way to sneak into the actual hardware properties.

I would guess, however, that from the point of view of a containerised VM its OS is the container not the bare metal (or virtualized metal perhaps), so tooling would need to check specifically for a separate bean.

That could be indicated via a property in the OS Bean (if it’s not the case already).

Nevertheless, I think consistency in the OS Bean should be achieved.

Cheers,
Mario 

On Fri 21. Jun 2019 at 13:23, Kirk Pepperdine <[hidden email]> wrote:
Hi Mario,

I don’t believe the MBean returns the wrong information. Is it not that the calls by-pass the container? Would it not be more appropriate to add a container aware mean? From a tooling perspective it’s a mistake to completely seal the JVM away from the hardware as it makes certain diagnostics more difficult to perform.

Kind regards,
Kirk


On Jun 21, 2019, at 6:06 AM, Mario Torre <[hidden email]> wrote:

The way I understood the bug report is a two fold approach, i.e. fix the os bean and *possibly* add a container one or extend it to add more data.

I agree with you and Andrew that the current OS Bean returns wrong information, this should be fixed in any case I think.

Bob, do you have some design ideas to share regarding the new interface? I think this may be an interesting project to pick up, even for students wanting to write a thesis, as it seems time is a limiting factor here for you to work on that?

Cheers,
Mario

On Fri 21. Jun 2019 at 10:53, Severin Gehwolf <[hidden email]> wrote:
Hi Bob,

On Thu, 2019-06-20 at 10:16 -0400, Bob Vandette wrote:
> Hi Andrew,
>
> I am aware of the limitations of the OperatingSystemMXBean and was
> hoping to address these limitations during the implementation of
> https://bugs.openjdk.java.net/browse/JDK-8199944.
>
> It would be helpful if you feel this is important to add some votes
> to this issue.

It seems strange that the getAvailableProcessors() returns the
container limit, while the memory limits are for the physical host. If
anything, shouldn't they agree (both physical host or both container
limits)?

When I briefly looked into it initially it seems to be a side-effect of
what is being used by the JDK code implementation-wise. IIRC
getAvailableProcessors() uses a runtime call. Memory reporting has it's
own logic[1], thus not reporting the container limit.

This seems weird from a consistency perspective. Thoughts?

If I understand you correctly, you are arguing that container reporting
should go into its own MX bean. On the other hand, CPU reporting for
the OS MX bean is container aware already. That makes me believe we
should rather make this consistent before evaluating a new MX bean.

Thanks,
Severin

[1] http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365


> Bob.
>
>
> > On Jun 20, 2019, at 9:43 AM, Andrew Azores <[hidden email]>
> > wrote:
> >
> > Hi all,
> >
> > Apologies if this is not the most appropriate list, in which case
> > please direct me where to go.
> >
> > I've noticed a surprising result from the
> > com.sun.management.OperatingSystemMXBean implementation when
> > running in
> > a containerized (specifically, using Docker on Linux) environment.
> > The
> > bean appears to be container-aware for processors, in that running
> > with
> > Docker option `--cpus 1.0` for example, on a multicore system, will
> > cause both java.lang.Runtime#availableProcessors and
> > java.lang.management.OperatingSystemMXBean#getAvailableProcessors /
> > com.sun.management.OperatingSystemMXBean#getAvailableProcessors to
> > return 1. However, the Docker option `--memory 100M` (or any other
> > limit value) is not reflected in the value returned by
> > com.sun.management.OperatingSystemMXBeam#getTotalPhysicalMemorySize
> > ,
> > and instead the returned value is still the total physical memory
> > of
> > the host machine - of which only a small portion may actually be
> > available to the "Operating System" of the JVM. Similarly for the
> > methods regarding free physical memory, total swap, and free swap.
> >
> > I have attached a patch which adds a small reproducer to the
> > existing
> > MemoryAwareness test.
> >
> > This seems like a bug to me, since if the imposed container limit
> > on
> > processors as a resource is included as part of the "Operating
> > System"
> > resource reporting, then surely memory resources should be reported
> > the
> > same way. As I said, I found the current behaviour quite
> > surprising.
> >
> > --
> > Andrew Azores
> > Software Engineer, OpenJDK Team
> > Red Hat
> > <jdk-osmxbean-container-memory-test-01.patch>
>
>

--
pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF
Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF

Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens
Proud GNU Classpath developer: http://www.classpath.org/
OpenJDK: http://openjdk.java.net/projects/caciocavallo/

Please, support open standards:
http://endsoftpatents.org/

--
pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF
Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF

Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens
Proud GNU Classpath developer: http://www.classpath.org/
OpenJDK: http://openjdk.java.net/projects/caciocavallo/

Please, support open standards:
http://endsoftpatents.org/

Reply | Threaded
Open this post in threaded view
|

Re: OperatingSystemMXBean unaware of container memory limits

Bob Vandette
In reply to this post by Mario Torre-5

> On Jun 21, 2019, at 6:06 AM, Mario Torre <[hidden email]> wrote:
>
> The way I understood the bug report is a two fold approach, i.e. fix the os bean and *possibly* add a container one or extend it to add more data.
Correct.  We could file a separate bug to just correct the OS Bean but I was hoping to add access to the new Container Metrics information
at the same time.

>
> I agree with you and Andrew that the current OS Bean returns wrong information, this should be fixed in any case I think.
Yes.

>
> Bob, do you have some design ideas to share regarding the new interface? I think this may be an interesting project to pick up, even for students wanting to write a thesis, as it seems time is a limiting factor here for you to work on that?

Fixing the existing OS Mbean is pretty straight forward.  Just have each method call the new Metrics API, check for error returns
in case this API is not supported on this platform and if success, return result, otherwise call existing logic.

Adding new OS access or a new Container Mbean is a bit more involved.

Since this is a public API, we’d need to take a look at at least one more OS (possibly Windows) and
decide which of the Metrics we want to expose.  I focused on Linux since this is the primary OS used
in container runtimes (even on Windows).  

Once you have the list of Metrics and Configuration data, Add a new Container MBean modeled
after the accessors in the Metrics API.

    jdk/open/src/java.base/share/classes/jdk/internal/platform/Metrics.java

You’d need to ensure proper javadocs are available and jtreg tests would need to be written to
validate the new APIs.

Bob.


>
> Cheers,
> Mario
>
> On Fri 21. Jun 2019 at 10:53, Severin Gehwolf <[hidden email]> wrote:
> Hi Bob,
>
> On Thu, 2019-06-20 at 10:16 -0400, Bob Vandette wrote:
> > Hi Andrew,
> >
> > I am aware of the limitations of the OperatingSystemMXBean and was
> > hoping to address these limitations during the implementation of
> > https://bugs.openjdk.java.net/browse/JDK-8199944.
> >
> > It would be helpful if you feel this is important to add some votes
> > to this issue.
>
> It seems strange that the getAvailableProcessors() returns the
> container limit, while the memory limits are for the physical host. If
> anything, shouldn't they agree (both physical host or both container
> limits)?
>
> When I briefly looked into it initially it seems to be a side-effect of
> what is being used by the JDK code implementation-wise. IIRC
> getAvailableProcessors() uses a runtime call. Memory reporting has it's
> own logic[1], thus not reporting the container limit.
>
> This seems weird from a consistency perspective. Thoughts?
>
> If I understand you correctly, you are arguing that container reporting
> should go into its own MX bean. On the other hand, CPU reporting for
> the OS MX bean is container aware already. That makes me believe we
> should rather make this consistent before evaluating a new MX bean.
>
> Thanks,
> Severin
>
> [1] http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365
>
>
> > Bob.
> >
> >
> > > On Jun 20, 2019, at 9:43 AM, Andrew Azores <[hidden email]>
> > > wrote:
> > >
> > > Hi all,
> > >
> > > Apologies if this is not the most appropriate list, in which case
> > > please direct me where to go.
> > >
> > > I've noticed a surprising result from the
> > > com.sun.management.OperatingSystemMXBean implementation when
> > > running in
> > > a containerized (specifically, using Docker on Linux) environment.
> > > The
> > > bean appears to be container-aware for processors, in that running
> > > with
> > > Docker option `--cpus 1.0` for example, on a multicore system, will
> > > cause both java.lang.Runtime#availableProcessors and
> > > java.lang.management.OperatingSystemMXBean#getAvailableProcessors /
> > > com.sun.management.OperatingSystemMXBean#getAvailableProcessors to
> > > return 1. However, the Docker option `--memory 100M` (or any other
> > > limit value) is not reflected in the value returned by
> > > com.sun.management.OperatingSystemMXBeam#getTotalPhysicalMemorySize
> > > ,
> > > and instead the returned value is still the total physical memory
> > > of
> > > the host machine - of which only a small portion may actually be
> > > available to the "Operating System" of the JVM. Similarly for the
> > > methods regarding free physical memory, total swap, and free swap.
> > >
> > > I have attached a patch which adds a small reproducer to the
> > > existing
> > > MemoryAwareness test.
> > >
> > > This seems like a bug to me, since if the imposed container limit
> > > on
> > > processors as a resource is included as part of the "Operating
> > > System"
> > > resource reporting, then surely memory resources should be reported
> > > the
> > > same way. As I said, I found the current behaviour quite
> > > surprising.
> > >
> > > --
> > > Andrew Azores
> > > Software Engineer, OpenJDK Team
> > > Red Hat
> > > <jdk-osmxbean-container-memory-test-01.patch>
> >
> >
>
> --
> pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF
> Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF
>
> Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens
> Proud GNU Classpath developer: http://www.classpath.org/
> OpenJDK: http://openjdk.java.net/projects/caciocavallo/
>
> Please, support open standards:
> http://endsoftpatents.org/

Reply | Threaded
Open this post in threaded view
|

Re: OperatingSystemMXBean unaware of container memory limits

Bob Vandette
In reply to this post by Severin Gehwolf

> On Jun 21, 2019, at 4:22 AM, Severin Gehwolf <[hidden email]> wrote:
>
> Hi Bob,
>
> On Thu, 2019-06-20 at 10:16 -0400, Bob Vandette wrote:
>> Hi Andrew,
>>
>> I am aware of the limitations of the OperatingSystemMXBean and was
>> hoping to address these limitations during the implementation of
>> https://bugs.openjdk.java.net/browse/JDK-8199944.
>>
>> It would be helpful if you feel this is important to add some votes
>> to this issue.
>
> It seems strange that the getAvailableProcessors() returns the
> container limit, while the memory limits are for the physical host. If
> anything, shouldn't they agree (both physical host or both container
> limits)?
>
> When I briefly looked into it initially it seems to be a side-effect of
> what is being used by the JDK code implementation-wise. IIRC
> getAvailableProcessors() uses a runtime call. Memory reporting has it's
> own logic[1], thus not reporting the container limit.
>
> This seems weird from a consistency perspective. Thoughts?
>
> If I understand you correctly, you are arguing that container reporting
> should go into its own MX bean. On the other hand, CPU reporting for
> the OS MX bean is container aware already. That makes me believe we
> should rather make this consistent before evaluating a new MX bean.

You make a good point.  I’ll split the enhancement and add a bug to fix the
current MX Bean since this is pretty easy to do.

Bob.

>
> Thanks,
> Severin
>
> [1] http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365
>
>
>> Bob.
>>
>>
>>> On Jun 20, 2019, at 9:43 AM, Andrew Azores <[hidden email]>
>>> wrote:
>>>
>>> Hi all,
>>>
>>> Apologies if this is not the most appropriate list, in which case
>>> please direct me where to go.
>>>
>>> I've noticed a surprising result from the
>>> com.sun.management.OperatingSystemMXBean implementation when
>>> running in
>>> a containerized (specifically, using Docker on Linux) environment.
>>> The
>>> bean appears to be container-aware for processors, in that running
>>> with
>>> Docker option `--cpus 1.0` for example, on a multicore system, will
>>> cause both java.lang.Runtime#availableProcessors and
>>> java.lang.management.OperatingSystemMXBean#getAvailableProcessors /
>>> com.sun.management.OperatingSystemMXBean#getAvailableProcessors to
>>> return 1. However, the Docker option `--memory 100M` (or any other
>>> limit value) is not reflected in the value returned by
>>> com.sun.management.OperatingSystemMXBeam#getTotalPhysicalMemorySize
>>> ,
>>> and instead the returned value is still the total physical memory
>>> of
>>> the host machine - of which only a small portion may actually be
>>> available to the "Operating System" of the JVM. Similarly for the
>>> methods regarding free physical memory, total swap, and free swap.
>>>
>>> I have attached a patch which adds a small reproducer to the
>>> existing
>>> MemoryAwareness test.
>>>
>>> This seems like a bug to me, since if the imposed container limit
>>> on
>>> processors as a resource is included as part of the "Operating
>>> System"
>>> resource reporting, then surely memory resources should be reported
>>> the
>>> same way. As I said, I found the current behaviour quite
>>> surprising.
>>>
>>> --
>>> Andrew Azores
>>> Software Engineer, OpenJDK Team
>>> Red Hat
>>> <jdk-osmxbean-container-memory-test-01.patch>
>>
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: OperatingSystemMXBean unaware of container memory limits

Severin Gehwolf
Hi Bob,

On Fri, 2019-06-21 at 08:56 -0400, Bob Vandette wrote:

> > On Jun 21, 2019, at 4:22 AM, Severin Gehwolf <[hidden email]> wrote:
> >
> > Hi Bob,
> >
> > On Thu, 2019-06-20 at 10:16 -0400, Bob Vandette wrote:
> > > Hi Andrew,
> > >
> > > I am aware of the limitations of the OperatingSystemMXBean and was
> > > hoping to address these limitations during the implementation of
> > > https://bugs.openjdk.java.net/browse/JDK-8199944.
> > >
> > > It would be helpful if you feel this is important to add some votes
> > > to this issue.
> >
> > It seems strange that the getAvailableProcessors() returns the
> > container limit, while the memory limits are for the physical host. If
> > anything, shouldn't they agree (both physical host or both container
> > limits)?
> >
> > When I briefly looked into it initially it seems to be a side-effect of
> > what is being used by the JDK code implementation-wise. IIRC
> > getAvailableProcessors() uses a runtime call. Memory reporting has it's
> > own logic[1], thus not reporting the container limit.
> >
> > This seems weird from a consistency perspective. Thoughts?
> >
> > If I understand you correctly, you are arguing that container reporting
> > should go into its own MX bean. On the other hand, CPU reporting for
> > the OS MX bean is container aware already. That makes me believe we
> > should rather make this consistent before evaluating a new MX bean.
>
> You make a good point.  I’ll split the enhancement and add a bug to fix the
> current MX Bean since this is pretty easy to do.

Sounds great! Please let me know once the OS MX bean bug has been
created. I'll then assign it to myself and help Andrew through the
process of getting it fixed.

Thanks,
Severin

> Bob.
>
> > Thanks,
> > Severin
> >
> > [1] http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365
> >
> >
> > > Bob.
> > >
> > >
> > > > On Jun 20, 2019, at 9:43 AM, Andrew Azores <[hidden email]>
> > > > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > Apologies if this is not the most appropriate list, in which case
> > > > please direct me where to go.
> > > >
> > > > I've noticed a surprising result from the
> > > > com.sun.management.OperatingSystemMXBean implementation when
> > > > running in
> > > > a containerized (specifically, using Docker on Linux) environment.
> > > > The
> > > > bean appears to be container-aware for processors, in that running
> > > > with
> > > > Docker option `--cpus 1.0` for example, on a multicore system, will
> > > > cause both java.lang.Runtime#availableProcessors and
> > > > java.lang.management.OperatingSystemMXBean#getAvailableProcessors /
> > > > com.sun.management.OperatingSystemMXBean#getAvailableProcessors to
> > > > return 1. However, the Docker option `--memory 100M` (or any other
> > > > limit value) is not reflected in the value returned by
> > > > com.sun.management.OperatingSystemMXBeam#getTotalPhysicalMemorySize
> > > > ,
> > > > and instead the returned value is still the total physical memory
> > > > of
> > > > the host machine - of which only a small portion may actually be
> > > > available to the "Operating System" of the JVM. Similarly for the
> > > > methods regarding free physical memory, total swap, and free swap.
> > > >
> > > > I have attached a patch which adds a small reproducer to the
> > > > existing
> > > > MemoryAwareness test.
> > > >
> > > > This seems like a bug to me, since if the imposed container limit
> > > > on
> > > > processors as a resource is included as part of the "Operating
> > > > System"
> > > > resource reporting, then surely memory resources should be reported
> > > > the
> > > > same way. As I said, I found the current behaviour quite
> > > > surprising.
> > > >
> > > > --
> > > > Andrew Azores
> > > > Software Engineer, OpenJDK Team
> > > > Red Hat
> > > > <jdk-osmxbean-container-memory-test-01.patch>

Reply | Threaded
Open this post in threaded view
|

Re: OperatingSystemMXBean unaware of container memory limits

Bob Vandette
Here you go.

https://bugs.openjdk.java.net/browse/JDK-8226575

Thanks,
Bob.

On Jun 21, 2019, at 9:08 AM, Severin Gehwolf <[hidden email]> wrote:

Hi Bob,

On Fri, 2019-06-21 at 08:56 -0400, Bob Vandette wrote:
On Jun 21, 2019, at 4:22 AM, Severin Gehwolf <[hidden email]> wrote:

Hi Bob,

On Thu, 2019-06-20 at 10:16 -0400, Bob Vandette wrote:
Hi Andrew,

I am aware of the limitations of the OperatingSystemMXBean and was
hoping to address these limitations during the implementation of
https://bugs.openjdk.java.net/browse/JDK-8199944.

It would be helpful if you feel this is important to add some votes
to this issue.

It seems strange that the getAvailableProcessors() returns the
container limit, while the memory limits are for the physical host. If
anything, shouldn't they agree (both physical host or both container
limits)?

When I briefly looked into it initially it seems to be a side-effect of
what is being used by the JDK code implementation-wise. IIRC
getAvailableProcessors() uses a runtime call. Memory reporting has it's
own logic[1], thus not reporting the container limit.

This seems weird from a consistency perspective. Thoughts?

If I understand you correctly, you are arguing that container reporting
should go into its own MX bean. On the other hand, CPU reporting for
the OS MX bean is container aware already. That makes me believe we
should rather make this consistent before evaluating a new MX bean.

You make a good point.  I’ll split the enhancement and add a bug to fix the
current MX Bean since this is pretty easy to do.

Sounds great! Please let me know once the OS MX bean bug has been
created. I'll then assign it to myself and help Andrew through the
process of getting it fixed.

Thanks,
Severin

Bob.

Thanks,
Severin

[1] http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365


Bob.


On Jun 20, 2019, at 9:43 AM, Andrew Azores <[hidden email]>
wrote:

Hi all,

Apologies if this is not the most appropriate list, in which case
please direct me where to go.

I've noticed a surprising result from the
com.sun.management.OperatingSystemMXBean implementation when
running in
a containerized (specifically, using Docker on Linux) environment.
The
bean appears to be container-aware for processors, in that running
with
Docker option `--cpus 1.0` for example, on a multicore system, will
cause both java.lang.Runtime#availableProcessors and
java.lang.management.OperatingSystemMXBean#getAvailableProcessors /
com.sun.management.OperatingSystemMXBean#getAvailableProcessors to
return 1. However, the Docker option `--memory 100M` (or any other
limit value) is not reflected in the value returned by
com.sun.management.OperatingSystemMXBeam#getTotalPhysicalMemorySize
,
and instead the returned value is still the total physical memory
of
the host machine - of which only a small portion may actually be
available to the "Operating System" of the JVM. Similarly for the
methods regarding free physical memory, total swap, and free swap.

I have attached a patch which adds a small reproducer to the
existing
MemoryAwareness test.

This seems like a bug to me, since if the imposed container limit
on
processors as a resource is included as part of the "Operating
System"
resource reporting, then surely memory resources should be reported
the
same way. As I said, I found the current behaviour quite
surprising.

--
Andrew Azores
Software Engineer, OpenJDK Team
Red Hat
<jdk-osmxbean-container-memory-test-01.patch>


Reply | Threaded
Open this post in threaded view
|

Re: OperatingSystemMXBean unaware of container memory limits

Mario Torre-5
Thanks!

Cheers,
Mario 

On Fri 21. Jun 2019 at 15:54, Bob Vandette <[hidden email]> wrote:

On Jun 21, 2019, at 9:08 AM, Severin Gehwolf <[hidden email]> wrote:

Hi Bob,

On Fri, 2019-06-21 at 08:56 -0400, Bob Vandette wrote:
On Jun 21, 2019, at 4:22 AM, Severin Gehwolf <[hidden email]> wrote:

Hi Bob,

On Thu, 2019-06-20 at 10:16 -0400, Bob Vandette wrote:
Hi Andrew,

I am aware of the limitations of the OperatingSystemMXBean and was
hoping to address these limitations during the implementation of
https://bugs.openjdk.java.net/browse/JDK-8199944.

It would be helpful if you feel this is important to add some votes
to this issue.

It seems strange that the getAvailableProcessors() returns the
container limit, while the memory limits are for the physical host. If
anything, shouldn't they agree (both physical host or both container
limits)?

When I briefly looked into it initially it seems to be a side-effect of
what is being used by the JDK code implementation-wise. IIRC
getAvailableProcessors() uses a runtime call. Memory reporting has it's
own logic[1], thus not reporting the container limit.

This seems weird from a consistency perspective. Thoughts?

If I understand you correctly, you are arguing that container reporting
should go into its own MX bean. On the other hand, CPU reporting for
the OS MX bean is container aware already. That makes me believe we
should rather make this consistent before evaluating a new MX bean.

You make a good point.  I’ll split the enhancement and add a bug to fix the
current MX Bean since this is pretty easy to do.

Sounds great! Please let me know once the OS MX bean bug has been
created. I'll then assign it to myself and help Andrew through the
process of getting it fixed.

Thanks,
Severin

Bob.

Thanks,
Severin

[1] http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365


Bob.


On Jun 20, 2019, at 9:43 AM, Andrew Azores <[hidden email]>
wrote:

Hi all,

Apologies if this is not the most appropriate list, in which case
please direct me where to go.

I've noticed a surprising result from the
com.sun.management.OperatingSystemMXBean implementation when
running in
a containerized (specifically, using Docker on Linux) environment.
The
bean appears to be container-aware for processors, in that running
with
Docker option `--cpus 1.0` for example, on a multicore system, will
cause both java.lang.Runtime#availableProcessors and
java.lang.management.OperatingSystemMXBean#getAvailableProcessors /
com.sun.management.OperatingSystemMXBean#getAvailableProcessors to
return 1. However, the Docker option `--memory 100M` (or any other
limit value) is not reflected in the value returned by
com.sun.management.OperatingSystemMXBeam#getTotalPhysicalMemorySize
,
and instead the returned value is still the total physical memory
of
the host machine - of which only a small portion may actually be
available to the "Operating System" of the JVM. Similarly for the
methods regarding free physical memory, total swap, and free swap.

I have attached a patch which adds a small reproducer to the
existing
MemoryAwareness test.

This seems like a bug to me, since if the imposed container limit
on
processors as a resource is included as part of the "Operating
System"
resource reporting, then surely memory resources should be reported
the
same way. As I said, I found the current behaviour quite
surprising.

--
Andrew Azores
Software Engineer, OpenJDK Team
Red Hat
<jdk-osmxbean-container-memory-test-01.patch>


--
pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF
Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF

Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens
Proud GNU Classpath developer: http://www.classpath.org/
OpenJDK: http://openjdk.java.net/projects/caciocavallo/

Please, support open standards:
http://endsoftpatents.org/
Reply | Threaded
Open this post in threaded view
|

Re: OperatingSystemMXBean unaware of container memory limits

Kirk Pepperdine-2
In reply to this post by Bob Vandette
Hi Bob,

100% isolate reduces visibility and from a diagnostic perspective there are a number of issues that can only be seen if you have access to the host information.

Also one of the things that gets forgotten is how this change will affect or break tooling. I would suggest a better solution would be to introduce a new MXBean that was “container aware”.

Kind regards,
Kikr Pepperdine


On Jun 21, 2019, at 5:40 AM, Bob Vandette <[hidden email]> wrote:

I agree with you Mario.  When I originally designed the jdk.internal.platform.Metrics
API, I was considering providing both a Host and Container Metrics implementation.
The problem is that the primary goal of containers is to provide isolation (hiding access to
the host) and it would be very difficult to provide reliable forms of both metrics from within
a container.  The Linux kernel does leak some host metric information but not everything.
This is why I decided to focus on the container configuration and metric data.  If Host
metrics are of interest you can always just run Java outside of a container.

Bob.

On Jun 21, 2019, at 8:27 AM, Mario Torre <[hidden email]> wrote:

Hi Kirk,

I think I understand what you mean, however then the OS Bean should be consistent regarding CPU information as well.

I think I remember why this was fixed the way it is now was because of incorrect behavior during GC configuration, or something along those lines, so I guess we would need two APIs after all to give tooling a way to sneak into the actual hardware properties.

I would guess, however, that from the point of view of a containerised VM its OS is the container not the bare metal (or virtualized metal perhaps), so tooling would need to check specifically for a separate bean.

That could be indicated via a property in the OS Bean (if it’s not the case already).

Nevertheless, I think consistency in the OS Bean should be achieved.

Cheers,
Mario 

On Fri 21. Jun 2019 at 13:23, Kirk Pepperdine <[hidden email]> wrote:
Hi Mario,

I don’t believe the MBean returns the wrong information. Is it not that the calls by-pass the container? Would it not be more appropriate to add a container aware mean? From a tooling perspective it’s a mistake to completely seal the JVM away from the hardware as it makes certain diagnostics more difficult to perform.

Kind regards,
Kirk


On Jun 21, 2019, at 6:06 AM, Mario Torre <[hidden email]> wrote:

The way I understood the bug report is a two fold approach, i.e. fix the os bean and *possibly* add a container one or extend it to add more data.

I agree with you and Andrew that the current OS Bean returns wrong information, this should be fixed in any case I think.

Bob, do you have some design ideas to share regarding the new interface? I think this may be an interesting project to pick up, even for students wanting to write a thesis, as it seems time is a limiting factor here for you to work on that?

Cheers,
Mario

On Fri 21. Jun 2019 at 10:53, Severin Gehwolf <[hidden email]> wrote:
Hi Bob,

On Thu, 2019-06-20 at 10:16 -0400, Bob Vandette wrote:
> Hi Andrew,
>
> I am aware of the limitations of the OperatingSystemMXBean and was
> hoping to address these limitations during the implementation of
> https://bugs.openjdk.java.net/browse/JDK-8199944.
>
> It would be helpful if you feel this is important to add some votes
> to this issue.

It seems strange that the getAvailableProcessors() returns the
container limit, while the memory limits are for the physical host. If
anything, shouldn't they agree (both physical host or both container
limits)?

When I briefly looked into it initially it seems to be a side-effect of
what is being used by the JDK code implementation-wise. IIRC
getAvailableProcessors() uses a runtime call. Memory reporting has it's
own logic[1], thus not reporting the container limit.

This seems weird from a consistency perspective. Thoughts?

If I understand you correctly, you are arguing that container reporting
should go into its own MX bean. On the other hand, CPU reporting for
the OS MX bean is container aware already. That makes me believe we
should rather make this consistent before evaluating a new MX bean.

Thanks,
Severin

[1] http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365


> Bob.
>
>
> > On Jun 20, 2019, at 9:43 AM, Andrew Azores <[hidden email]>
> > wrote:
> >
> > Hi all,
> >
> > Apologies if this is not the most appropriate list, in which case
> > please direct me where to go.
> >
> > I've noticed a surprising result from the
> > com.sun.management.OperatingSystemMXBean implementation when
> > running in
> > a containerized (specifically, using Docker on Linux) environment.
> > The
> > bean appears to be container-aware for processors, in that running
> > with
> > Docker option `--cpus 1.0` for example, on a multicore system, will
> > cause both java.lang.Runtime#availableProcessors and
> > java.lang.management.OperatingSystemMXBean#getAvailableProcessors /
> > com.sun.management.OperatingSystemMXBean#getAvailableProcessors to
> > return 1. However, the Docker option `--memory 100M` (or any other
> > limit value) is not reflected in the value returned by
> > com.sun.management.OperatingSystemMXBeam#getTotalPhysicalMemorySize
> > ,
> > and instead the returned value is still the total physical memory
> > of
> > the host machine - of which only a small portion may actually be
> > available to the "Operating System" of the JVM. Similarly for the
> > methods regarding free physical memory, total swap, and free swap.
> >
> > I have attached a patch which adds a small reproducer to the
> > existing
> > MemoryAwareness test.
> >
> > This seems like a bug to me, since if the imposed container limit
> > on
> > processors as a resource is included as part of the "Operating
> > System"
> > resource reporting, then surely memory resources should be reported
> > the
> > same way. As I said, I found the current behaviour quite
> > surprising.
> >
> > --
> > Andrew Azores
> > Software Engineer, OpenJDK Team
> > Red Hat
> > <jdk-osmxbean-container-memory-test-01.patch>
>
>

--
pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF
Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF

Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens
Proud GNU Classpath developer: http://www.classpath.org/
OpenJDK: http://openjdk.java.net/projects/caciocavallo/

Please, support open standards:
http://endsoftpatents.org/

--
pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF
Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF

Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens
Proud GNU Classpath developer: http://www.classpath.org/
OpenJDK: http://openjdk.java.net/projects/caciocavallo/

Please, support open standards:
http://endsoftpatents.org/


Reply | Threaded
Open this post in threaded view
|

Re: OperatingSystemMXBean unaware of container memory limits

Kirk Pepperdine-2
In reply to this post by Mario Torre-5
Hi Mario,

I completely agree, the OS Bean should be consistent with the memory. That said I favor introducing new MXBeans to achieve the desired effects. If there is no container than the container MXBean could simply not be instantiated.

I’m looking for more visibility especially for the purposes of supporting diagnostic tooling. If visibility is shut down it’s just makes it that much harder to perform any reasonable level of high level diagnostics.

Kind regards,
Kirk


On Jun 21, 2019, at 5:27 AM, Mario Torre <[hidden email]> wrote:

Hi Kirk,

I think I understand what you mean, however then the OS Bean should be consistent regarding CPU information as well.

I think I remember why this was fixed the way it is now was because of incorrect behavior during GC configuration, or something along those lines, so I guess we would need two APIs after all to give tooling a way to sneak into the actual hardware properties.

I would guess, however, that from the point of view of a containerised VM its OS is the container not the bare metal (or virtualized metal perhaps), so tooling would need to check specifically for a separate bean.

That could be indicated via a property in the OS Bean (if it’s not the case already).

Nevertheless, I think consistency in the OS Bean should be achieved.

Cheers,
Mario 

On Fri 21. Jun 2019 at 13:23, Kirk Pepperdine <[hidden email]> wrote:
Hi Mario,

I don’t believe the MBean returns the wrong information. Is it not that the calls by-pass the container? Would it not be more appropriate to add a container aware mean? From a tooling perspective it’s a mistake to completely seal the JVM away from the hardware as it makes certain diagnostics more difficult to perform.

Kind regards,
Kirk


On Jun 21, 2019, at 6:06 AM, Mario Torre <[hidden email]> wrote:

The way I understood the bug report is a two fold approach, i.e. fix the os bean and *possibly* add a container one or extend it to add more data.

I agree with you and Andrew that the current OS Bean returns wrong information, this should be fixed in any case I think.

Bob, do you have some design ideas to share regarding the new interface? I think this may be an interesting project to pick up, even for students wanting to write a thesis, as it seems time is a limiting factor here for you to work on that?

Cheers,
Mario

On Fri 21. Jun 2019 at 10:53, Severin Gehwolf <[hidden email]> wrote:
Hi Bob,

On Thu, 2019-06-20 at 10:16 -0400, Bob Vandette wrote:
> Hi Andrew,
>
> I am aware of the limitations of the OperatingSystemMXBean and was
> hoping to address these limitations during the implementation of
> https://bugs.openjdk.java.net/browse/JDK-8199944.
>
> It would be helpful if you feel this is important to add some votes
> to this issue.

It seems strange that the getAvailableProcessors() returns the
container limit, while the memory limits are for the physical host. If
anything, shouldn't they agree (both physical host or both container
limits)?

When I briefly looked into it initially it seems to be a side-effect of
what is being used by the JDK code implementation-wise. IIRC
getAvailableProcessors() uses a runtime call. Memory reporting has it's
own logic[1], thus not reporting the container limit.

This seems weird from a consistency perspective. Thoughts?

If I understand you correctly, you are arguing that container reporting
should go into its own MX bean. On the other hand, CPU reporting for
the OS MX bean is container aware already. That makes me believe we
should rather make this consistent before evaluating a new MX bean.

Thanks,
Severin

[1] http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365


> Bob.
>
>
> > On Jun 20, 2019, at 9:43 AM, Andrew Azores <[hidden email]>
> > wrote:
> >
> > Hi all,
> >
> > Apologies if this is not the most appropriate list, in which case
> > please direct me where to go.
> >
> > I've noticed a surprising result from the
> > com.sun.management.OperatingSystemMXBean implementation when
> > running in
> > a containerized (specifically, using Docker on Linux) environment.
> > The
> > bean appears to be container-aware for processors, in that running
> > with
> > Docker option `--cpus 1.0` for example, on a multicore system, will
> > cause both java.lang.Runtime#availableProcessors and
> > java.lang.management.OperatingSystemMXBean#getAvailableProcessors /
> > com.sun.management.OperatingSystemMXBean#getAvailableProcessors to
> > return 1. However, the Docker option `--memory 100M` (or any other
> > limit value) is not reflected in the value returned by
> > com.sun.management.OperatingSystemMXBeam#getTotalPhysicalMemorySize
> > ,
> > and instead the returned value is still the total physical memory
> > of
> > the host machine - of which only a small portion may actually be
> > available to the "Operating System" of the JVM. Similarly for the
> > methods regarding free physical memory, total swap, and free swap.
> >
> > I have attached a patch which adds a small reproducer to the
> > existing
> > MemoryAwareness test.
> >
> > This seems like a bug to me, since if the imposed container limit
> > on
> > processors as a resource is included as part of the "Operating
> > System"
> > resource reporting, then surely memory resources should be reported
> > the
> > same way. As I said, I found the current behaviour quite
> > surprising.
> >
> > --
> > Andrew Azores
> > Software Engineer, OpenJDK Team
> > Red Hat
> > <jdk-osmxbean-container-memory-test-01.patch>
>
>

--
pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF
Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF

Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens
Proud GNU Classpath developer: http://www.classpath.org/
OpenJDK: http://openjdk.java.net/projects/caciocavallo/

Please, support open standards:
http://endsoftpatents.org/

--
pgp key: http://subkeys.pgp.net/ PGP Key ID: 80F240CF
Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF

Java Champion - Blog: http://neugens.wordpress.com - Twitter: @neugens
Proud GNU Classpath developer: http://www.classpath.org/
OpenJDK: http://openjdk.java.net/projects/caciocavallo/

Please, support open standards:
http://endsoftpatents.org/

Reply | Threaded
Open this post in threaded view
|

Re: OperatingSystemMXBean unaware of container memory limits

Andrew Azores
In reply to this post by Bob Vandette
Hi Bob (and all),

On Fri, 2019-06-21 at 08:50 -0400, Bob Vandette wrote:
> >
> Fixing the existing OS Mbean is pretty straight forward.  Just have
> each method call the new Metrics API, check for error returns
> in case this API is not supported on this platform and if success,
> return result, otherwise call existing logic.
>

I've had free time to circle back to this issue lately, so I've
prepared a patch which modifies the OperatingSystemMXBean as you've
described here. This seems to work perfectly well for
getTotalPhysicalMemorySize, getFreePhysicalMemorySize, and
getTotalSwapSpaceSize. getFreeSwapSpaceSize looks at a glance to be
implementable by comparing Metrics' getMemoryAndSwapUsage,
getMemoryUsage, and getMemoryAndSwap limit, but the implementation
would only provide an approximation (and not necessarily accurately)
since the amount of memory/swap used would likely change in between or
due to subsequent calls to the Metrics API.

Also, I'm not sure that there are reasonable equivalents for all of the
other metrics implemented in
jdk.management/unix/classes/com/sun/management/internal/OperatingSystem
Impl.java available from Metrics or elsewhere which make sense in a
containerized environment (ex. getCommittedVirtualMemorySize).

So, before I proceed much further with this patch, does anyone have
input on how these issues should be handled such that the
OperatingSystemMXBean is self-consistent wrt its container awareness?

--
Andrew Azores
Software Engineer, OpenJDK Team
Red Hat

Reply | Threaded
Open this post in threaded view
|

Re: OperatingSystemMXBean unaware of container memory limits

Bob Vandette

> On Jul 11, 2019, at 1:18 PM, Andrew Azores <[hidden email]> wrote:
>
> Hi Bob (and all),
>
> On Fri, 2019-06-21 at 08:50 -0400, Bob Vandette wrote:
>>>
>> Fixing the existing OS Mbean is pretty straight forward.  Just have
>> each method call the new Metrics API, check for error returns
>> in case this API is not supported on this platform and if success,
>> return result, otherwise call existing logic.
>>
>
> I've had free time to circle back to this issue lately, so I've
> prepared a patch which modifies the OperatingSystemMXBean as you've
> described here. This seems to work perfectly well for
> getTotalPhysicalMemorySize, getFreePhysicalMemorySize, and
> getTotalSwapSpaceSize.


> getFreeSwapSpaceSize looks at a glance to be
> implementable by comparing Metrics' getMemoryAndSwapUsage,
> getMemoryUsage, and getMemoryAndSwap limit, but the implementation
> would only provide an approximation (and not necessarily accurately)
> since the amount of memory/swap used would likely change in between or
> due to subsequent calls to the Metrics API.

That one appears a bit tricky to get right.

Assuming memory and swap limits are set, returning getMemoryAndSwapLimit -  getMemoryLimit - getMemoryAndSwapUsage.
Since the limits are constant, there’s no issue with variability between the two calls.

However, if the Swap limit is unlimited then I’d revert to the original logic since the limit is
the hosts when swap is unlimited.

One problem I see if that in cgroups, memory.swappiness can alter or disable swapping.
I didn’t provide a Metrics API that returns this value or take this into account in
the result of getMemoryAndSwapLimit.

>
> Also, I'm not sure that there are reasonable equivalents for all of the
> other metrics implemented in
> jdk.management/unix/classes/com/sun/management/internal/OperatingSystem
> Impl.java available from Metrics or elsewhere which make sense in a
> containerized environment (ex. getCommittedVirtualMemorySize).

We’d have to experiment with this but since this is actual committed memory for
the current process, I would expect the existing logic to be accurate.

>
> So, before I proceed much further with this patch, does anyone have> input on how these issues should be handled such that the
> OperatingSystemMXBean is self-consistent wrt its container awareness?
>

Looking at the list of native methods in OperatingSystemImpl.java,
it looks like the ones that are process based can be handled by the existing
logic.  The APIs that are system wide need to be forwarded to the Metrics
API.  The only problematic one I see is getSystemCpuLoad0.  This one needs
some thought.   I don’t see a way of determining the total containers
Cpu load assuming there are multiple process running in a single container
and the fact that Cgroups only provides cpu usage (in time) and not load.


Fall Thru:

    private native long getCommittedVirtualMemorySize0();
    private native long getMaxFileDescriptorCount0();
    private native long getOpenFileDescriptorCount0();
    private native long getProcessCpuTime0();
    private native double getProcessCpuLoad0();

Metrics:

    private native long getFreePhysicalMemorySize0();
    private native long getFreeSwapSpaceSize0();
    private native long getTotalPhysicalMemorySize0();
    private native long getTotalSwapSpaceSize0()

???:

    private native double getSystemCpuLoad0();

Bob.

> --
> Andrew Azores
> Software Engineer, OpenJDK Team
> Red Hat
>