What actions are allowed in an JVMTI ResourceExhausted event handler?

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

What actions are allowed in an JVMTI ResourceExhausted event handler?

Thomas Stüfe-2
Hi all,

We have a client using CloudFoundry and its "jvmkill" agent. That is a
tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill) which
subscribes to the JVMTI ResourceExhausted Event. In the handler it
then does call JVMTI FollowReferences() to produce a heap histogram.

The thing is, at our client we seem to run out of Metaspace in a
compiler thread. That thread normally would swallow the Metaspace OOM
and just bailout from the compilation. But as part of the metaspace
OOME handling the ResourceExhausted event gets posted, the handler
then uses JVMTI FollowReferences() and attempts to print out the heap
histogram, then runs into a guarantee since the compiler thread cannot
call java methods.

My question is: are there any limitations about what one can do inside
a ResourceExhausted event handler?

I checked the https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
documentation, but I cannot find any mentioning of limitations in that
case.

Thanks and Best Regards, Thomas
Reply | Threaded
Open this post in threaded view
|

Re: What actions are allowed in an JVMTI ResourceExhausted event handler?

David Holmes
Hi Thomas,

On 14/11/2018 6:50 am, Thomas Stüfe wrote:

> Hi all,
>
> We have a client using CloudFoundry and its "jvmkill" agent. That is a
> tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill) which
> subscribes to the JVMTI ResourceExhausted Event. In the handler it
> then does call JVMTI FollowReferences() to produce a heap histogram.
>
> The thing is, at our client we seem to run out of Metaspace in a
> compiler thread. That thread normally would swallow the Metaspace OOM
> and just bailout from the compilation. But as part of the metaspace
> OOME handling the ResourceExhausted event gets posted, the handler
> then uses JVMTI FollowReferences() and attempts to print out the heap
> histogram, then runs into a guarantee since the compiler thread cannot
> call java methods.
>
> My question is: are there any limitations about what one can do inside
> a ResourceExhausted event handler?

Not specified no. But the reality of JVM TI is that you can't anticipate
every execution context and there are times when there are implicit
constraints imposed by the implementation.

In this case I think we have a mismatch between the fact we post the
event from the compiler thread, but that a compiler thread is not a true
"Java thread" and so can not execute arbitrary JNI or JVM TI code, or in
particular can not lead to executing Java code. I think we should not be
posting the event from the compiler thread in this case.

Cheers,
David

> I checked the https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
> documentation, but I cannot find any mentioning of limitations in that
> case.
>
> Thanks and Best Regards, Thomas
>
Reply | Threaded
Open this post in threaded view
|

Re: What actions are allowed in an JVMTI ResourceExhausted event handler?

Chris Plummer
On 11/13/18 9:32 PM, David Holmes wrote:

> Hi Thomas,
>
> On 14/11/2018 6:50 am, Thomas Stüfe wrote:
>> Hi all,
>>
>> We have a client using CloudFoundry and its "jvmkill" agent. That is a
>> tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill) which
>> subscribes to the JVMTI ResourceExhausted Event. In the handler it
>> then does call JVMTI FollowReferences() to produce a heap histogram.
>>
>> The thing is, at our client we seem to run out of Metaspace in a
>> compiler thread. That thread normally would swallow the Metaspace OOM
>> and just bailout from the compilation. But as part of the metaspace
>> OOME handling the ResourceExhausted event gets posted, the handler
>> then uses JVMTI FollowReferences() and attempts to print out the heap
>> histogram, then runs into a guarantee since the compiler thread cannot
>> call java methods.
>>
>> My question is: are there any limitations about what one can do inside
>> a ResourceExhausted event handler?
>
> Not specified no. But the reality of JVM TI is that you can't
> anticipate every execution context and there are times when there are
> implicit constraints imposed by the implementation.
>
> In this case I think we have a mismatch between the fact we post the
> event from the compiler thread, but that a compiler thread is not a
> true "Java thread" and so can not execute arbitrary JNI or JVM TI
> code, or in particular can not lead to executing Java code. I think we
> should not be posting the event from the compiler thread in this case.
>
Does the recent fix for https://bugs.openjdk.java.net/browse/JDK-8193126 
address the metaspace failure you are seeing?

Chris

> Cheers,
> David
>
>> I checked the
>> https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
>> documentation, but I cannot find any mentioning of limitations in that
>> case.
>>
>> Thanks and Best Regards, Thomas
>>


Reply | Threaded
Open this post in threaded view
|

Re: What actions are allowed in an JVMTI ResourceExhausted event handler?

Thomas Stüfe-2
In reply to this post by David Holmes


On Wed, Nov 14, 2018, 06:32 David Holmes <[hidden email] wrote:
Hi Thomas,

On 14/11/2018 6:50 am, Thomas Stüfe wrote:
> Hi all,
>
> We have a client using CloudFoundry and its "jvmkill" agent. That is a
> tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill) which
> subscribes to the JVMTI ResourceExhausted Event. In the handler it
> then does call JVMTI FollowReferences() to produce a heap histogram.
>
> The thing is, at our client we seem to run out of Metaspace in a
> compiler thread. That thread normally would swallow the Metaspace OOM
> and just bailout from the compilation. But as part of the metaspace
> OOME handling the ResourceExhausted event gets posted, the handler
> then uses JVMTI FollowReferences() and attempts to print out the heap
> histogram, then runs into a guarantee since the compiler thread cannot
> call java methods.
>
> My question is: are there any limitations about what one can do inside
> a ResourceExhausted event handler?

Not specified no. But the reality of JVM TI is that you can't anticipate
every execution context and there are times when there are implicit
constraints imposed by the implementation.

In this case I think we have a mismatch between the fact we post the
event from the compiler thread, but that a compiler thread is not a true
"Java thread" and so can not execute arbitrary JNI or JVM TI code, or in
particular can not lead to executing Java code. I think we should not be
posting the event from the compiler thread in this case.

Cheers,
David

Hi David, 

Yes I thought so too. I'll prepare a fix. 

Thanks, Thomas 


> I checked the https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
> documentation, but I cannot find any mentioning of limitations in that
> case.
>
> Thanks and Best Regards, Thomas
>
Reply | Threaded
Open this post in threaded view
|

Re: What actions are allowed in an JVMTI ResourceExhausted event handler?

David Holmes
On 14/11/2018 3:37 pm, Thomas Stüfe wrote:

> On Wed, Nov 14, 2018, 06:32 David Holmes <[hidden email]
> <mailto:[hidden email]> wrote:
>
>     Hi Thomas,
>
>     On 14/11/2018 6:50 am, Thomas Stüfe wrote:
>      > Hi all,
>      >
>      > We have a client using CloudFoundry and its "jvmkill" agent. That
>     is a
>      > tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill) which
>      > subscribes to the JVMTI ResourceExhausted Event. In the handler it
>      > then does call JVMTI FollowReferences() to produce a heap histogram.
>      >
>      > The thing is, at our client we seem to run out of Metaspace in a
>      > compiler thread. That thread normally would swallow the Metaspace OOM
>      > and just bailout from the compilation. But as part of the metaspace
>      > OOME handling the ResourceExhausted event gets posted, the handler
>      > then uses JVMTI FollowReferences() and attempts to print out the heap
>      > histogram, then runs into a guarantee since the compiler thread
>     cannot
>      > call java methods.
>      >
>      > My question is: are there any limitations about what one can do
>     inside
>      > a ResourceExhausted event handler?
>
>     Not specified no. But the reality of JVM TI is that you can't
>     anticipate
>     every execution context and there are times when there are implicit
>     constraints imposed by the implementation.
>
>     In this case I think we have a mismatch between the fact we post the
>     event from the compiler thread, but that a compiler thread is not a
>     true
>     "Java thread" and so can not execute arbitrary JNI or JVM TI code,
>     or in
>     particular can not lead to executing Java code. I think we should
>     not be
>     posting the event from the compiler thread in this case.
>
>     Cheers,
>     David
>
>
> Hi David,
>
> Yes I thought so too. I'll prepare a fix.

My thought on the fix is that we need to check if
Thread::current()->can_call_java(). And that should probably be inside
the JvmtiExport::should_post_xxx implementation.

Cheers,
David

> Thanks, Thomas
>
>
>      > I checked the
>     https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
>      > documentation, but I cannot find any mentioning of limitations in
>     that
>      > case.
>      >
>      > Thanks and Best Regards, Thomas
>      >
>
Reply | Threaded
Open this post in threaded view
|

Re: What actions are allowed in an JVMTI ResourceExhausted event handler?

Thomas Stüfe-2
In reply to this post by Chris Plummer
Hi Chris,

I do not think so. The MetaspaceSize we see when we fail has the
expected value, it is not downgraded somehow.

Also, would that bug apply in a normal VM? Is UseJVMCICompiler not off
by default?

I think with this bug the chance for this error to happen may increase
but the bug itself (the fact that we post ResourceExhausted from
Compiler Thread) is older and also predates JVMCI. We see this also
with older VMs.

Thanks Thomas
On Wed, Nov 14, 2018 at 6:37 AM Chris Plummer <[hidden email]> wrote:

>
> On 11/13/18 9:32 PM, David Holmes wrote:
> > Hi Thomas,
> >
> > On 14/11/2018 6:50 am, Thomas Stüfe wrote:
> >> Hi all,
> >>
> >> We have a client using CloudFoundry and its "jvmkill" agent. That is a
> >> tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill) which
> >> subscribes to the JVMTI ResourceExhausted Event. In the handler it
> >> then does call JVMTI FollowReferences() to produce a heap histogram.
> >>
> >> The thing is, at our client we seem to run out of Metaspace in a
> >> compiler thread. That thread normally would swallow the Metaspace OOM
> >> and just bailout from the compilation. But as part of the metaspace
> >> OOME handling the ResourceExhausted event gets posted, the handler
> >> then uses JVMTI FollowReferences() and attempts to print out the heap
> >> histogram, then runs into a guarantee since the compiler thread cannot
> >> call java methods.
> >>
> >> My question is: are there any limitations about what one can do inside
> >> a ResourceExhausted event handler?
> >
> > Not specified no. But the reality of JVM TI is that you can't
> > anticipate every execution context and there are times when there are
> > implicit constraints imposed by the implementation.
> >
> > In this case I think we have a mismatch between the fact we post the
> > event from the compiler thread, but that a compiler thread is not a
> > true "Java thread" and so can not execute arbitrary JNI or JVM TI
> > code, or in particular can not lead to executing Java code. I think we
> > should not be posting the event from the compiler thread in this case.
> >
> Does the recent fix for https://bugs.openjdk.java.net/browse/JDK-8193126
> address the metaspace failure you are seeing?
>
> Chris
> > Cheers,
> > David
> >
> >> I checked the
> >> https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
> >> documentation, but I cannot find any mentioning of limitations in that
> >> case.
> >>
> >> Thanks and Best Regards, Thomas
> >>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: What actions are allowed in an JVMTI ResourceExhausted event handler?

Thomas Stüfe-2
In reply to this post by David Holmes
On Wed, Nov 14, 2018 at 6:58 AM David Holmes <[hidden email]> wrote:

>
> On 14/11/2018 3:37 pm, Thomas Stüfe wrote:
> > On Wed, Nov 14, 2018, 06:32 David Holmes <[hidden email]
> > <mailto:[hidden email]> wrote:
> >
> >     Hi Thomas,
> >
> >     On 14/11/2018 6:50 am, Thomas Stüfe wrote:
> >      > Hi all,
> >      >
> >      > We have a client using CloudFoundry and its "jvmkill" agent. That
> >     is a
> >      > tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill) which
> >      > subscribes to the JVMTI ResourceExhausted Event. In the handler it
> >      > then does call JVMTI FollowReferences() to produce a heap histogram.
> >      >
> >      > The thing is, at our client we seem to run out of Metaspace in a
> >      > compiler thread. That thread normally would swallow the Metaspace OOM
> >      > and just bailout from the compilation. But as part of the metaspace
> >      > OOME handling the ResourceExhausted event gets posted, the handler
> >      > then uses JVMTI FollowReferences() and attempts to print out the heap
> >      > histogram, then runs into a guarantee since the compiler thread
> >     cannot
> >      > call java methods.
> >      >
> >      > My question is: are there any limitations about what one can do
> >     inside
> >      > a ResourceExhausted event handler?
> >
> >     Not specified no. But the reality of JVM TI is that you can't
> >     anticipate
> >     every execution context and there are times when there are implicit
> >     constraints imposed by the implementation.
> >
> >     In this case I think we have a mismatch between the fact we post the
> >     event from the compiler thread, but that a compiler thread is not a
> >     true
> >     "Java thread" and so can not execute arbitrary JNI or JVM TI code,
> >     or in
> >     particular can not lead to executing Java code. I think we should
> >     not be
> >     posting the event from the compiler thread in this case.
> >
> >     Cheers,
> >     David
> >
> >
> > Hi David,
> >
> > Yes I thought so too. I'll prepare a fix.
>
> My thought on the fix is that we need to check if
> Thread::current()->can_call_java(). And that should probably be inside
> the JvmtiExport::should_post_xxx implementation.
>

I wonder whether that may be too harsh. JVMTI agents may not
necessarily call into java as reaction to ResourceExhausted. I would
have limited this to !CompilerThread, and only in Metaspace.

Also, looking at CompilerThread::can_call_java(), I see that we return
true for jvmci compilers. Still do we want to post it there?

But I am not sure. What do you think?

> Cheers,
> David
>
> > Thanks, Thomas
> >
> >
> >      > I checked the
> >     https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
> >      > documentation, but I cannot find any mentioning of limitations in
> >     that
> >      > case.
> >      >
> >      > Thanks and Best Regards, Thomas
> >      >
> >
Reply | Threaded
Open this post in threaded view
|

Re: What actions are allowed in an JVMTI ResourceExhausted event handler?

David Holmes
On 14/11/2018 4:38 pm, Thomas Stüfe wrote:

> On Wed, Nov 14, 2018 at 6:58 AM David Holmes <[hidden email]> wrote:
>>
>> On 14/11/2018 3:37 pm, Thomas Stüfe wrote:
>>> On Wed, Nov 14, 2018, 06:32 David Holmes <[hidden email]
>>> <mailto:[hidden email]> wrote:
>>>
>>>      Hi Thomas,
>>>
>>>      On 14/11/2018 6:50 am, Thomas Stüfe wrote:
>>>       > Hi all,
>>>       >
>>>       > We have a client using CloudFoundry and its "jvmkill" agent. That
>>>      is a
>>>       > tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill) which
>>>       > subscribes to the JVMTI ResourceExhausted Event. In the handler it
>>>       > then does call JVMTI FollowReferences() to produce a heap histogram.
>>>       >
>>>       > The thing is, at our client we seem to run out of Metaspace in a
>>>       > compiler thread. That thread normally would swallow the Metaspace OOM
>>>       > and just bailout from the compilation. But as part of the metaspace
>>>       > OOME handling the ResourceExhausted event gets posted, the handler
>>>       > then uses JVMTI FollowReferences() and attempts to print out the heap
>>>       > histogram, then runs into a guarantee since the compiler thread
>>>      cannot
>>>       > call java methods.
>>>       >
>>>       > My question is: are there any limitations about what one can do
>>>      inside
>>>       > a ResourceExhausted event handler?
>>>
>>>      Not specified no. But the reality of JVM TI is that you can't
>>>      anticipate
>>>      every execution context and there are times when there are implicit
>>>      constraints imposed by the implementation.
>>>
>>>      In this case I think we have a mismatch between the fact we post the
>>>      event from the compiler thread, but that a compiler thread is not a
>>>      true
>>>      "Java thread" and so can not execute arbitrary JNI or JVM TI code,
>>>      or in
>>>      particular can not lead to executing Java code. I think we should
>>>      not be
>>>      posting the event from the compiler thread in this case.
>>>
>>>      Cheers,
>>>      David
>>>
>>>
>>> Hi David,
>>>
>>> Yes I thought so too. I'll prepare a fix.
>>
>> My thought on the fix is that we need to check if
>> Thread::current()->can_call_java(). And that should probably be inside
>> the JvmtiExport::should_post_xxx implementation.
>>
>
> I wonder whether that may be too harsh. JVMTI agents may not
> necessarily call into java as reaction to ResourceExhausted. I would
> have limited this to !CompilerThread, and only in Metaspace.
>
> Also, looking at CompilerThread::can_call_java(), I see that we return
> true for jvmci compilers. Still do we want to post it there?
>
> But I am not sure. What do you think?

The JVM TI spec is really only aware of "Java threads", though it does
allow for an implementation specific thread to be used in places. So
posting events from a thread that may not be able to execute
JNI/JVM-TI/Java code successfully seems a very risky proposition. So I'd
say "don't do that".

But if we want to deal only with the callsite in question there are
various ugly hacks we could put in there.

But the basic constraint is the ability to run Java code. JVMCI compiler
threads can run Java code so we don't need to exclude them unnecessarily
- hence the check for can_call_Java().

Cheers,
David

>> Cheers,
>> David
>>
>>> Thanks, Thomas
>>>
>>>
>>>       > I checked the
>>>      https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
>>>       > documentation, but I cannot find any mentioning of limitations in
>>>      that
>>>       > case.
>>>       >
>>>       > Thanks and Best Regards, Thomas
>>>       >
>>>
Reply | Threaded
Open this post in threaded view
|

Re: What actions are allowed in an JVMTI ResourceExhausted event handler?

Thomas Stüfe-2
In reply to this post by Thomas Stüfe-2
I did open a bug to track this: https://bugs.openjdk.java.net/browse/JDK-8213834
On Wed, Nov 14, 2018 at 7:38 AM Thomas Stüfe <[hidden email]> wrote:

>
> On Wed, Nov 14, 2018 at 6:58 AM David Holmes <[hidden email]> wrote:
> >
> > On 14/11/2018 3:37 pm, Thomas Stüfe wrote:
> > > On Wed, Nov 14, 2018, 06:32 David Holmes <[hidden email]
> > > <mailto:[hidden email]> wrote:
> > >
> > >     Hi Thomas,
> > >
> > >     On 14/11/2018 6:50 am, Thomas Stüfe wrote:
> > >      > Hi all,
> > >      >
> > >      > We have a client using CloudFoundry and its "jvmkill" agent. That
> > >     is a
> > >      > tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill) which
> > >      > subscribes to the JVMTI ResourceExhausted Event. In the handler it
> > >      > then does call JVMTI FollowReferences() to produce a heap histogram.
> > >      >
> > >      > The thing is, at our client we seem to run out of Metaspace in a
> > >      > compiler thread. That thread normally would swallow the Metaspace OOM
> > >      > and just bailout from the compilation. But as part of the metaspace
> > >      > OOME handling the ResourceExhausted event gets posted, the handler
> > >      > then uses JVMTI FollowReferences() and attempts to print out the heap
> > >      > histogram, then runs into a guarantee since the compiler thread
> > >     cannot
> > >      > call java methods.
> > >      >
> > >      > My question is: are there any limitations about what one can do
> > >     inside
> > >      > a ResourceExhausted event handler?
> > >
> > >     Not specified no. But the reality of JVM TI is that you can't
> > >     anticipate
> > >     every execution context and there are times when there are implicit
> > >     constraints imposed by the implementation.
> > >
> > >     In this case I think we have a mismatch between the fact we post the
> > >     event from the compiler thread, but that a compiler thread is not a
> > >     true
> > >     "Java thread" and so can not execute arbitrary JNI or JVM TI code,
> > >     or in
> > >     particular can not lead to executing Java code. I think we should
> > >     not be
> > >     posting the event from the compiler thread in this case.
> > >
> > >     Cheers,
> > >     David
> > >
> > >
> > > Hi David,
> > >
> > > Yes I thought so too. I'll prepare a fix.
> >
> > My thought on the fix is that we need to check if
> > Thread::current()->can_call_java(). And that should probably be inside
> > the JvmtiExport::should_post_xxx implementation.
> >
>
> I wonder whether that may be too harsh. JVMTI agents may not
> necessarily call into java as reaction to ResourceExhausted. I would
> have limited this to !CompilerThread, and only in Metaspace.
>
> Also, looking at CompilerThread::can_call_java(), I see that we return
> true for jvmci compilers. Still do we want to post it there?
>
> But I am not sure. What do you think?
>
> > Cheers,
> > David
> >
> > > Thanks, Thomas
> > >
> > >
> > >      > I checked the
> > >     https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
> > >      > documentation, but I cannot find any mentioning of limitations in
> > >     that
> > >      > case.
> > >      >
> > >      > Thanks and Best Regards, Thomas
> > >      >
> > >
Reply | Threaded
Open this post in threaded view
|

Re: What actions are allowed in an JVMTI ResourceExhausted event handler?

JC Beyler
It seems what we do with other events that might have this type of "risk" is to defer the event to the ServiceThread, which is a Java thread, no? But perhaps for a resource exhausted just ignoring it for the compiler thread and letting another "Java thread" be aware of it and posting is a better choice?

Thanks,
Jc

On Tue, Nov 13, 2018 at 11:03 PM Thomas Stüfe <[hidden email]> wrote:
I did open a bug to track this: https://bugs.openjdk.java.net/browse/JDK-8213834
On Wed, Nov 14, 2018 at 7:38 AM Thomas Stüfe <[hidden email]> wrote:
>
> On Wed, Nov 14, 2018 at 6:58 AM David Holmes <[hidden email]> wrote:
> >
> > On 14/11/2018 3:37 pm, Thomas Stüfe wrote:
> > > On Wed, Nov 14, 2018, 06:32 David Holmes <[hidden email]
> > > <mailto:[hidden email]> wrote:
> > >
> > >     Hi Thomas,
> > >
> > >     On 14/11/2018 6:50 am, Thomas Stüfe wrote:
> > >      > Hi all,
> > >      >
> > >      > We have a client using CloudFoundry and its "jvmkill" agent. That
> > >     is a
> > >      > tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill) which
> > >      > subscribes to the JVMTI ResourceExhausted Event. In the handler it
> > >      > then does call JVMTI FollowReferences() to produce a heap histogram.
> > >      >
> > >      > The thing is, at our client we seem to run out of Metaspace in a
> > >      > compiler thread. That thread normally would swallow the Metaspace OOM
> > >      > and just bailout from the compilation. But as part of the metaspace
> > >      > OOME handling the ResourceExhausted event gets posted, the handler
> > >      > then uses JVMTI FollowReferences() and attempts to print out the heap
> > >      > histogram, then runs into a guarantee since the compiler thread
> > >     cannot
> > >      > call java methods.
> > >      >
> > >      > My question is: are there any limitations about what one can do
> > >     inside
> > >      > a ResourceExhausted event handler?
> > >
> > >     Not specified no. But the reality of JVM TI is that you can't
> > >     anticipate
> > >     every execution context and there are times when there are implicit
> > >     constraints imposed by the implementation.
> > >
> > >     In this case I think we have a mismatch between the fact we post the
> > >     event from the compiler thread, but that a compiler thread is not a
> > >     true
> > >     "Java thread" and so can not execute arbitrary JNI or JVM TI code,
> > >     or in
> > >     particular can not lead to executing Java code. I think we should
> > >     not be
> > >     posting the event from the compiler thread in this case.
> > >
> > >     Cheers,
> > >     David
> > >
> > >
> > > Hi David,
> > >
> > > Yes I thought so too. I'll prepare a fix.
> >
> > My thought on the fix is that we need to check if
> > Thread::current()->can_call_java(). And that should probably be inside
> > the JvmtiExport::should_post_xxx implementation.
> >
>
> I wonder whether that may be too harsh. JVMTI agents may not
> necessarily call into java as reaction to ResourceExhausted. I would
> have limited this to !CompilerThread, and only in Metaspace.
>
> Also, looking at CompilerThread::can_call_java(), I see that we return
> true for jvmci compilers. Still do we want to post it there?
>
> But I am not sure. What do you think?
>
> > Cheers,
> > David
> >
> > > Thanks, Thomas
> > >
> > >
> > >      > I checked the
> > >     https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
> > >      > documentation, but I cannot find any mentioning of limitations in
> > >     that
> > >      > case.
> > >      >
> > >      > Thanks and Best Regards, Thomas
> > >      >
> > >


--

Thanks,
Jc
Reply | Threaded
Open this post in threaded view
|

Re: What actions are allowed in an JVMTI ResourceExhausted event handler?

Thomas Stüfe-2
Hi JC,

On Wed, Nov 14, 2018 at 3:56 PM JC Beyler <[hidden email]> wrote:
>
> It seems what we do with other events that might have this type of "risk" is to defer the event to the ServiceThread, which is a Java thread, no? But perhaps for a resource exhausted just ignoring it for the compiler thread and letting another "Java thread" be aware of it and posting is a better choice?
>
> Thanks,
> Jc

in case of ResourceExhausted I think this is safe to be ignored for
the compiler thread. If the problem persists, surely a java thread
will hit it next.

...Thomas

>
> On Tue, Nov 13, 2018 at 11:03 PM Thomas Stüfe <[hidden email]> wrote:
>>
>> I did open a bug to track this: https://bugs.openjdk.java.net/browse/JDK-8213834
>> On Wed, Nov 14, 2018 at 7:38 AM Thomas Stüfe <[hidden email]> wrote:
>> >
>> > On Wed, Nov 14, 2018 at 6:58 AM David Holmes <[hidden email]> wrote:
>> > >
>> > > On 14/11/2018 3:37 pm, Thomas Stüfe wrote:
>> > > > On Wed, Nov 14, 2018, 06:32 David Holmes <[hidden email]
>> > > > <mailto:[hidden email]> wrote:
>> > > >
>> > > >     Hi Thomas,
>> > > >
>> > > >     On 14/11/2018 6:50 am, Thomas Stüfe wrote:
>> > > >      > Hi all,
>> > > >      >
>> > > >      > We have a client using CloudFoundry and its "jvmkill" agent. That
>> > > >     is a
>> > > >      > tiny JVMTI agent (see https://github.com/cloudfoundry/jvmkill) which
>> > > >      > subscribes to the JVMTI ResourceExhausted Event. In the handler it
>> > > >      > then does call JVMTI FollowReferences() to produce a heap histogram.
>> > > >      >
>> > > >      > The thing is, at our client we seem to run out of Metaspace in a
>> > > >      > compiler thread. That thread normally would swallow the Metaspace OOM
>> > > >      > and just bailout from the compilation. But as part of the metaspace
>> > > >      > OOME handling the ResourceExhausted event gets posted, the handler
>> > > >      > then uses JVMTI FollowReferences() and attempts to print out the heap
>> > > >      > histogram, then runs into a guarantee since the compiler thread
>> > > >     cannot
>> > > >      > call java methods.
>> > > >      >
>> > > >      > My question is: are there any limitations about what one can do
>> > > >     inside
>> > > >      > a ResourceExhausted event handler?
>> > > >
>> > > >     Not specified no. But the reality of JVM TI is that you can't
>> > > >     anticipate
>> > > >     every execution context and there are times when there are implicit
>> > > >     constraints imposed by the implementation.
>> > > >
>> > > >     In this case I think we have a mismatch between the fact we post the
>> > > >     event from the compiler thread, but that a compiler thread is not a
>> > > >     true
>> > > >     "Java thread" and so can not execute arbitrary JNI or JVM TI code,
>> > > >     or in
>> > > >     particular can not lead to executing Java code. I think we should
>> > > >     not be
>> > > >     posting the event from the compiler thread in this case.
>> > > >
>> > > >     Cheers,
>> > > >     David
>> > > >
>> > > >
>> > > > Hi David,
>> > > >
>> > > > Yes I thought so too. I'll prepare a fix.
>> > >
>> > > My thought on the fix is that we need to check if
>> > > Thread::current()->can_call_java(). And that should probably be inside
>> > > the JvmtiExport::should_post_xxx implementation.
>> > >
>> >
>> > I wonder whether that may be too harsh. JVMTI agents may not
>> > necessarily call into java as reaction to ResourceExhausted. I would
>> > have limited this to !CompilerThread, and only in Metaspace.
>> >
>> > Also, looking at CompilerThread::can_call_java(), I see that we return
>> > true for jvmci compilers. Still do we want to post it there?
>> >
>> > But I am not sure. What do you think?
>> >
>> > > Cheers,
>> > > David
>> > >
>> > > > Thanks, Thomas
>> > > >
>> > > >
>> > > >      > I checked the
>> > > >     https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html
>> > > >      > documentation, but I cannot find any mentioning of limitations in
>> > > >     that
>> > > >      > case.
>> > > >      >
>> > > >      > Thanks and Best Regards, Thomas
>> > > >      >
>> > > >
>
>
>
> --
>
> Thanks,
> Jc