Quantcast

Question regarding JVM crashes in GC

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Question regarding JVM crashes in GC

Michael Rasmussen
Hi

We are currently in the middle of making JRebel ready for Java 9, and in
that process came across some issues with the concurrent GCs, and was
wondering if anyone here could help shed some insight into what's going
wrong?

Basically, running with the current internal build of JRebel, I can
reproducible crash the JVM by calling System.gc(), if using a concurrent GC.

Output from 9-ea+160-jigsaw-nightly-h6207-20170316 build on Windows x64:

-XX:+UseG1GC: EXCEPTION_ACCESS_VIOLATION # UpdateRSOopClosure::do_oop+0x47
-XX:+UseConcMarkSweepGC: Internal Error (.../synchronizer.cpp:1574),
pid=12588, tid=6176 # guarantee(obj->mark() == markOopDesc::encode(mid))
failed: invariant
-XX:+UseParallelOldGC: Internal Error (.../synchronizer.cpp:1574),
pid=13792, tid=12284 # guarantee(obj->mark() == markOopDesc::encode(mid))
failed: invariant
-XX:+UseParallelGC: EXCEPTION_ACCESS_VIOLATION #
ServiceUtil::visible_oop+0x32
-XX:+UseSerialGC works

With -XX:+VerifyBeforeGC -XX:+VerifyAfterGC
-XX:+UseParallelOldGC: EXCEPTION_ACCESS_VIOLATION # oopDesc::verify+0x2b
-XX:+UseParallelGC: EXCEPTION_ACCESS_VIOLATION # oopDesc::verify+0x38
Rest have same outcome as without the verification.

Also ran on build 9-ea+163 (Windows x64), with similar results (though less
symbol info):
G1: EXCEPTION_ACCESS_VIOLATION  [jvm.dll+0x2ac0b7]
CMS: EXCEPTION_ACCESS_VIOLATION  [jvm.dll+0x45f1d2]
ParallelOld: Internal Error (.../synchronizer.cpp:1574), pid=10472,
tid=13212 #  guarantee(obj->mark() == markOopDesc::encode(mid)) failed:
invariant
Parallel: EXCEPTION_ACCESS_VIOLATION [jvm.dll+0x45f1d2]


JRebel basically instruments every class in the system, including the
classes in java.base. We also set native prefix, so it is likely that we
trigger something somewhere.
Though, at this point, I don't know what. Perhaps the GCs make some
assumptions about some of the classes we touch (like hard-coded method or
field offsets etc?) ?

If anyone here has any idea where to start looking, it would be greatly
appreciated!

Kind regards
Michael Rasmussen
Product Manager, JRebel
ZeroTurnaround

<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Virus-free.
www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question regarding JVM crashes in GC

Michael Rasmussen
Ok

Found Shipilev's fastdebug builds, and tried running with that
(openjdk-jdk9-dev-fastdebug-2017-04-04.tar.gz).

Using the fastdebug build and running with G1, the JVM crashes when
calling System.gc(), with one of the following assertions:

Internal Error (hotspot/src/share/vm/gc/serial/markSweep.inline.hpp:61)
assert(is_archive_object(obj) || new_obj != __null || obj->mark() ==
markOopDesc::prototype() || (UseBiasedLocking &&
obj->mark()->has_bias_pattern())) failed: should be forwarded

Internal Error (hotspot/src/share/vm/gc/serial/markSweep.inline.hpp:53)
assert(Universe::heap()->is_in(obj)) failed: should be in heap

Internal Error (hotspot/src/share/vm/gc/serial/markSweep.inline.hpp:65)
assert(Universe::heap()->is_in_reserved(new_obj)) failed: should be in
object space

/Michael
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question regarding JVM crashes in GC

Mikael Gerdin
Hi Michael,

On 2017-04-05 09:11, Michael Rasmussen wrote:
> Ok
>
> Found Shipilev's fastdebug builds, and tried running with that
> (openjdk-jdk9-dev-fastdebug-2017-04-04.tar.gz).
>
> Using the fastdebug build and running with G1, the JVM crashes when
> calling System.gc(), with one of the following assertions:

Did you try out -XX:+Verify{Before/After}GC with G1? G1 has the most
complex verification code so that might provide some more information.

/Mikael

>
> Internal Error (hotspot/src/share/vm/gc/serial/markSweep.inline.hpp:61)
> assert(is_archive_object(obj) || new_obj != __null || obj->mark() ==
> markOopDesc::prototype() || (UseBiasedLocking &&
> obj->mark()->has_bias_pattern())) failed: should be forwarded
>
> Internal Error (hotspot/src/share/vm/gc/serial/markSweep.inline.hpp:53)
> assert(Universe::heap()->is_in(obj)) failed: should be in heap
>
> Internal Error (hotspot/src/share/vm/gc/serial/markSweep.inline.hpp:65)
> assert(Universe::heap()->is_in_reserved(new_obj)) failed: should be in
> object space
>
> /Michael
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question regarding JVM crashes in GC

Michael Rasmussen
On 5 April 2017 at 11:00, Mikael Gerdin <[hidden email]> wrote:
> Did you try out -XX:+Verify{Before/After}GC with G1? G1 has the most complex
> verification code so that might provide some more information.
>

Adding -XX:+VerifyBeforeGC -XX:+VerifyAfterGC didn't produce any
different result, still get one of the 3 before mentioned assertions.

Adding -Xlog:gc*=debug, I see that "Verifying Before GC" complete
without anything, and the last log line I get is:
[info ][gc,phases,start      ] GC(16) Phase 3: Adjust pointers


/Michael
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question regarding JVM crashes in GC

Stefan Johansson
In reply to this post by Michael Rasmussen
Hi Michael,

On 2017-04-05 09:11, Michael Rasmussen wrote:

> Ok
>
> Found Shipilev's fastdebug builds, and tried running with that
> (openjdk-jdk9-dev-fastdebug-2017-04-04.tar.gz).
>
> Using the fastdebug build and running with G1, the JVM crashes when
> calling System.gc(), with one of the following assertions:
>
> Internal Error (hotspot/src/share/vm/gc/serial/markSweep.inline.hpp:61)
> assert(is_archive_object(obj) || new_obj != __null || obj->mark() ==
> markOopDesc::prototype() || (UseBiasedLocking &&
> obj->mark()->has_bias_pattern())) failed: should be forwarded
>
> Internal Error (hotspot/src/share/vm/gc/serial/markSweep.inline.hpp:53)
> assert(Universe::heap()->is_in(obj)) failed: should be in heap
>
> Internal Error (hotspot/src/share/vm/gc/serial/markSweep.inline.hpp:65)
> assert(Universe::heap()->is_in_reserved(new_obj)) failed: should be in
> object space
All these assertions are in MarkSweep::adjust_pointer and they suggest
that you are visiting an object with bad references. What's causing this
is really hard to say without more information. It sounds like this is
easy for you to reproduce, could we run the same tests ourselves? Or
could you construct a reproducer that we can run?

Thanks,
Stefan

> /Michael

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question regarding JVM crashes in GC

Michael Rasmussen
On 5 April 2017 at 15:41, Stefan Johansson <[hidden email]> wrote:
> All these assertions are in MarkSweep::adjust_pointer and they suggest that
> you are visiting an object with bad references. What's causing this is
> really hard to say without more information. It sounds like this is easy for
> you to reproduce, could we run the same tests ourselves? Or could you
> construct a reproducer that we can run?

Yes, it reproduces every single time :)

At the moment, I assume it has something to do with
java.lang.ref.Reference, as I see
InstanceRefKlass::oop_ms_adjust_pointers(oop)+0x281 in the stacktrace
in the core file. And I know we use a lot of weak references.

Unfortunately, currently I cannot share the test, but I'll try to see
if can make a small standalone test case that I can share.

/Michael
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question regarding JVM crashes in GC

Stefan Johansson


On 2017-04-05 16:34, Michael Rasmussen wrote:

> On 5 April 2017 at 15:41, Stefan Johansson <[hidden email]> wrote:
>> All these assertions are in MarkSweep::adjust_pointer and they suggest that
>> you are visiting an object with bad references. What's causing this is
>> really hard to say without more information. It sounds like this is easy for
>> you to reproduce, could we run the same tests ourselves? Or could you
>> construct a reproducer that we can run?
> Yes, it reproduces every single time :)
>
> At the moment, I assume it has something to do with
> java.lang.ref.Reference, as I see
> InstanceRefKlass::oop_ms_adjust_pointers(oop)+0x281 in the stacktrace
> in the core file. And I know we use a lot of weak references.
Interesting, the GC (or rather reference processing) actually do have
some assumptions when it comes to the fields in reference objects. We
expect the referent, discovered and next fields to be at certain
offsets. You mentioned earlier that such assumptions might be
problematic, are you doing any kind of instrumentation that could change
those offsets?

> Unfortunately, currently I cannot share the test, but I'll try to see
> if can make a small standalone test case that I can share.
That would be really helpful to understand what's going on.

Stefan

> /Michael

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question regarding JVM crashes in GC

Michael Rasmussen
On 6 April 2017 at 12:58, Stefan Johansson <[hidden email]> wrote:
> Interesting, the GC (or rather reference processing) actually do have some
> assumptions when it comes to the fields in reference objects. We expect the
> referent, discovered and next fields to be at certain offsets. You mentioned
> earlier that such assumptions might be problematic, are you doing any kind
> of instrumentation that could change those offsets?

Generally, we are doing instrumentation that add fields to classes, so
can change those offsets, but it doesn't seem to affect the Reference
classes.
I ran some tests with and without JRebel, printing out the results of
unsafe::objectFieldOffset and unsafe::staticFieldOffset for all
declared fields on PhantomReference, WeakReference, SoftReference and
Reference. The offsets were identical. So assuming those unsafe
methods return the equivalent of what's used internally, then we don't
modify the offset.

We are using a JVMTI agent for instrumenting the early classes
(classes loaded before premain is called), I'm currently trying to
investigate if this is causing some side effects, since when doing the
equivalent using --patch-module, I cannot reproduce it with a simple
main() { System.gc(); } test class.

/Michael
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question regarding JVM crashes in GC

Stefan Johansson


On 2017-04-06 13:42, Michael Rasmussen wrote:

> On 6 April 2017 at 12:58, Stefan Johansson <[hidden email]> wrote:
>> Interesting, the GC (or rather reference processing) actually do have some
>> assumptions when it comes to the fields in reference objects. We expect the
>> referent, discovered and next fields to be at certain offsets. You mentioned
>> earlier that such assumptions might be problematic, are you doing any kind
>> of instrumentation that could change those offsets?
> Generally, we are doing instrumentation that add fields to classes, so
> can change those offsets, but it doesn't seem to affect the Reference
> classes.
> I ran some tests with and without JRebel, printing out the results of
> unsafe::objectFieldOffset and unsafe::staticFieldOffset for all
> declared fields on PhantomReference, WeakReference, SoftReference and
> Reference. The offsets were identical. So assuming those unsafe
> methods return the equivalent of what's used internally, then we don't
> modify the offset.
Sounds good, but still suspicious since we seem to be crashing on
references.
> We are using a JVMTI agent for instrumenting the early classes
> (classes loaded before premain is called), I'm currently trying to
> investigate if this is causing some side effects, since when doing the
> equivalent using --patch-module, I cannot reproduce it with a simple
> main() { System.gc(); } test class.
One thing that is a bit strange is that you don't reproduce this with
SerialGC, G1 and Serial currently share a lot of the Full GC code, but
there is a optimization only used for serial that skips objects in the
beginning of the heap, could you try running with -XX:+UseSerialGC
-XX:MarkSweepDeadRatio=0 -XX:MarkSweepAlwaysCompactCount=1 and see if
you run into the same issues with Serial as the other collectors?

Thanks,
Stefan
>
> /Michael

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question regarding JVM crashes in GC

Michael Rasmussen
On 6 April 2017 at 15:26, Stefan Johansson <[hidden email]> wrote:
> could you try running with -XX:+UseSerialGC
> -XX:MarkSweepDeadRatio=0 -XX:MarkSweepAlwaysCompactCount=1 and see if you
> run into the same issues with Serial as the other collectors?

With that, it fails on the very first GC:
# Internal Error (genOopClosures.inline.hpp:110), pid=28567, tid=28569
# assert(!_g->to()->is_in_reserved(obj)) failed: Scanning field twice?

With the following stack trace in hs_err (before report_vm_error is called):
V  [libjvm.so+0xcd37a6]  void FastScanClosure::do_oop_work<unsigned
int>(unsigned int*)+0x1d6
V  [libjvm.so+0xcd6fce]  void
InstanceRefKlass::oop_oop_iterate_ref_processing<true,
FastScanClosure>(oop, FastScanClosure*)+0x37e
V  [libjvm.so+0xcdade7]  void InstanceRefKlass::oop_oop_iterate<true,
FastScanClosure>(oop, FastScanClosure*)+0x187
V  [libjvm.so+0xcceff2]  InstanceRefKlass::oop_oop_iterate_nv(oop,
FastScanClosure*)+0x32


I'm still trying to debug if it's somewhere in our native code that
triggers this, although I don't understand how/if we can touch
something this deep in the GC.

/Michael
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question regarding JVM crashes in GC

Michael Rasmussen
On 6 April 2017 at 18:36, Michael Rasmussen
<[hidden email]> wrote:
> I'm still trying to debug if it's somewhere in our native code that
> triggers this, although I don't understand how/if we can touch
> something this deep in the GC.

Ok, think I've found the likely culprit. I think some of the
instrumentation we're doing to these early classes are causing the
Reference class to be loaded too soon!
Would that explain this behavior?

/Michael
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question regarding JVM crashes in GC

Stefan Johansson


On 2017-04-06 17:51, Michael Rasmussen wrote:
> On 6 April 2017 at 18:36, Michael Rasmussen
> <[hidden email]> wrote:
>> I'm still trying to debug if it's somewhere in our native code that
>> triggers this, although I don't understand how/if we can touch
>> something this deep in the GC.
> Ok, think I've found the likely culprit. I think some of the
> instrumentation we're doing to these early classes are causing the
> Reference class to be loaded too soon!
> Would that explain this behavior?
I'm not sure I follow what too soon mean. For the GC we should be fine
as long as the offsets are correct, and you seem to have verified this.

I guess there could be an issue where loading the Reference class
earlier than expected could cause us to write bad oops into the
reference fields somehow and that would then crash in the GC. But I
would have expected that to have been caught by running with
-XX:+VerifyBeforeGC.

Stefan

> /Michael

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question regarding JVM crashes in GC

Michael Rasmussen
On 10 April 2017 at 15:31, Stefan Johansson <[hidden email]> wrote:
> I'm not sure I follow what too soon mean. For the GC we should be fine as
> long as the offsets are correct, and you seem to have verified this.

Here is a minimal test case, with a JVMTI agent that defines a subtype
of WeakReference "too soon", causing the issue:
https://gist.github.com/anonymous/d6fbe111082b0059cbeb36b90ba87558

/Michael
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question regarding JVM crashes in GC

Stefan Johansson


On 2017-04-10 15:31, Michael Rasmussen wrote:
> On 10 April 2017 at 15:31, Stefan Johansson <[hidden email]> wrote:
>> I'm not sure I follow what too soon mean. For the GC we should be fine as
>> long as the offsets are correct, and you seem to have verified this.
> Here is a minimal test case, with a JVMTI agent that defines a subtype
> of WeakReference "too soon", causing the issue:
> https://gist.github.com/anonymous/d6fbe111082b0059cbeb36b90ba87558
Thanks Michael,

This sounds great. Could you attach the test case to the mailing list in
case we want to add this to the OpenJDK testing going forward?

Thanks,
Stefan

> /Michael

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question regarding JVM crashes in GC

Michael Rasmussen
On 10 April 2017 at 17:48, Stefan Johansson <[hidden email]> wrote:
> This sounds great. Could you attach the test case to the mailing list in
> case we want to add this to the OpenJDK testing going forward?

Attached inline below

/Michael

--- agent.c ---

#include <string.h>

#include "jni.h"
#include "jvmti.h"

/*
  public class java.lang.ref.MyWeakReference
    extends java.lang.ref.WeakReference {};
*/
jbyte MyWeakReference_bytes[] = {
  -54,  -2, -70, -66,   0,   0,   0,  49,
    0,   5,   1,   0,  29, 106,  97, 118,
   97,  47, 108,  97, 110, 103,  47, 114,
  101, 102,  47,  77, 121,  87, 101,  97,
  107,  82, 101, 102, 101, 114, 101, 110,
   99, 101,   7,   0,   1,   1,   0,  27,
  106,  97, 118,  97,  47, 108,  97, 110,
  103,  47, 114, 101, 102,  47,  87, 101,
   97, 107,  82, 101, 102, 101, 114, 101,
  110,  99, 101,   7,   0,   3,   0,   1,
    0,   2,   0,   4,   0,   0,   0,   0,
    0,   0,   0,   0
};
jsize MyWeakReference_len = 0x5C;

void JNICALL callback_ClassFileLoadHook(jvmtiEnv *jvmti, JNIEnv* env,
  jclass class_being_redefined, jobject loader, const char* name,
  jobject protection_domain,
  jint class_data_len, const unsigned char* class_data,
  jint* new_class_data_len, unsigned char** new_class_data) {

  if (name && !strcmp(name, "java/lang/System")) {
    (*env)->DefineClass(env, "java/lang/ref/MyWeakReference", NULL,
      MyWeakReference_bytes, MyWeakReference_len);
  }
}

JNIEXPORT jint JNICALL Agent_OnLoad(JavaVM *vm,
  char *options, void *reserved) {

  jvmtiEventCallbacks callbacks;
  jvmtiCapabilities caps;
  jvmtiEnv *jvmti;

  (*vm)->GetEnv(vm, (void **)&jvmti, JVMTI_VERSION_9);

  memset(&caps, 0, sizeof(caps));
  memset(&callbacks, 0, sizeof(callbacks));

  caps.can_generate_early_vmstart = 1;
  caps.can_generate_all_class_hook_events = 1;
  caps.can_generate_early_class_hook_events = 1;

  callbacks.ClassFileLoadHook = &callback_ClassFileLoadHook;

  (*jvmti)->AddCapabilities(jvmti, &caps);
  (*jvmti)->SetEventCallbacks(jvmti, &callbacks, sizeof(callbacks));
  (*jvmti)->SetEventNotificationMode(jvmti, JVMTI_ENABLE,
    JVMTI_EVENT_CLASS_FILE_LOAD_HOOK, NULL);

  return 0;
}

--- MiscTest9.java ---

package app1;

import java.lang.ref.WeakReference;

public class MiscTest9 {
  public static void main(String[] args) throws Exception {
    WeakReference<?> weak = new WeakReference<>(new MiscTest9());

    boolean present = true;
    while (present) {
      System.out.println("Running GC");
      System.gc();
      present = weak.get() != null;
    }
  }
}
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question regarding JVM crashes in GC

Volker Simonis
Hi Michael,

I think the problem is the following:

In normal operation (i.e. without your agent) classes get loaded in the
following order:

...
[0.046s][info][class,load] java.lang.ClassLoader source: jrt:/java.base
[0.046s][info][class,load] java.lang.System source: jrt:/java.base
...
[0.047s][info][class,load] java.lang.ref.Reference source: jrt:/java.base
[0.047s][info][class,load] java.lang.ref.SoftReference source:
jrt:/java.base
[0.047s][info][class,load] java.lang.ref.WeakReference source:
jrt:/java.base
[0.047s][info][class,load] java.lang.ref.FinalReference source:
jrt:/java.base
[0.047s][info][class,load] java.lang.ref.PhantomReference source:
jrt:/java.base

This order is specified in systemDictionary.hpp:

// The order of these definitions is significant; it is the order in which
// preloading is actually performed by initialize_preloaded_classes.

#define WK_KLASSES_DO(do_klass)
                                             \
  /* well-known classes */
                                              \
  do_klass(Object_klass,                                java_lang_Object,
                       Pre                 ) \
  do_klass(String_klass,                                java_lang_String,
                       Pre                 ) \
  do_klass(Class_klass,                                 java_lang_Class,
                        Pre                 ) \
  do_klass(Cloneable_klass,
java_lang_Cloneable,                       Pre                 ) \
  do_klass(ClassLoader_klass,
java_lang_ClassLoader,                     Pre                 ) \
  do_klass(Serializable_klass,
 java_io_Serializable,                      Pre                 ) \
  do_klass(System_klass,                                java_lang_System,
                       Pre                 ) \
...
  do_klass(Reference_klass,
java_lang_ref_Reference,                   Pre                 ) \
  /* Preload ref klasses and set reference types */
                                             \
  do_klass(SoftReference_klass,
java_lang_ref_SoftReference,               Pre                 ) \
  do_klass(WeakReference_klass,
java_lang_ref_WeakReference,               Pre                 ) \
  do_klass(FinalReference_klass,
 java_lang_ref_FinalReference,              Pre                 ) \
  do_klass(PhantomReference_klass,
 java_lang_ref_PhantomReference,            Pre                 ) \

The method 'SystemDictionary::initialize_preloaded_classes(TRAPS)' in
systemDictionary.cpp not only loads the classes in this order, it also does
some initialization stuff in between (i.e. specific initializations after
some specific class have been loaded):

   initialize_wk_klasses_through(WK_KLASS_ENUM_NAME(Reference_klass), scan,
CHECK);

  // Preload ref klasses and set reference types

InstanceKlass::cast(WK_KLASS(Reference_klass))->set_reference_type(REF_OTHER);
  InstanceRefKlass::update_nonstatic_oop_maps(WK_KLASS(Reference_klass));

  initialize_wk_klasses_through(WK_KLASS_ENUM_NAME(PhantomReference_klass),
scan, CHECK);

InstanceKlass::cast(WK_KLASS(SoftReference_klass))->set_reference_type(REF_SOFT);

InstanceKlass::cast(WK_KLASS(WeakReference_klass))->set_reference_type(REF_WEAK);

InstanceKlass::cast(WK_KLASS(FinalReference_klass))->set_reference_type(REF_FINAL);

InstanceKlass::cast(WK_KLASS(PhantomReference_klass))->set_reference_type(REF_PHANTOM);

As you can see, first all the classes up to java.lang.ref.Reference are
loaded. Then the reference type of java.lang.ref.Reference is set to
"REF_OTHER". After that the other reference class (from SoftReference up to
PhantomReference) are loaded and their corresponding reference type is set.

For all the other classes which get loaded later, the reference type will
be set to the reference type of their parent class (see
'ClassFileParser::post_process_parsed_stream()' in classFileParser.cpp):

  // Compute reference typ
  _rt = (NULL ==_super_klass) ? REF_NONE : _super_klass->reference_type();

So when you derive a class from WeakReference, its reference type will be
"REF_WEAK".

Now with your agent, you change the loading of classes as follows:

...
[0.046s][info][class,load] java.lang.ClassLoader source: jrt:/java.base
[0.046s][info][class,load] java.lang.ref.Reference source: jrt:/java.base
[0.046s][info][class,load] java.lang.ref.WeakReference source:
jrt:/java.base
[0.046s][info][class,load] java.lang.ref.MyWeakReference
[0.047s][info][class,load] java.lang.System source: jrt:/java.base
...
[0.048s][info][class,load] java.lang.ref.SoftReference source:
jrt:/java.base
[0.048s][info][class,load] java.lang.ref.FinalReference source:
jrt:/java.base
[0.048s][info][class,load] java.lang.ref.PhantomReference source:
jrt:/java.base
...

This means that your class 'java.lang.ref.MyWeakReference' will get
reference type "REF_NONE", because you implicitly triggered the loading of
WeakReference and Reference without updating their reference type
accordingly. Later on in the game, the reference type of WeakReference and
Reference will be corrected, but your custom reference class will remain
with its initial reference type "REF_NONE".

I think this confuses the GC in the end and leads to the crashes observed
by you. Funny enough, I don't see any crash with a product build of the
latest jdk9-dev sources on Linux, but I can reproduce "should be in heap"
assertion observed by you:

#  Internal Error
(/usr/work/d046063/OpenJDK/jdk9-dev/hotspot/src/share/vm/gc/serial/markSweep.inline.hpp:53),
pid=18985, tid=19058
#  assert(Universe::heap()->is_in(obj)) failed: should be in heap

So to keep a long story short, I don't think it is a good idea to change
the class loading and initialization order of the "well-known" classes as
described in systemDictionary.hpp. That order is carefully handcrafted and
extremely sensitive to changes and can have all kind of unforeseeable side
effects (as you've painfully detected yourself :)

Regards,
Volker


On Mon, Apr 10, 2017 at 5:37 PM, Michael Rasmussen <
[hidden email]> wrote:

> On 10 April 2017 at 17:48, Stefan Johansson <[hidden email]>
> wrote:
> > This sounds great. Could you attach the test case to the mailing list in
> > case we want to add this to the OpenJDK testing going forward?
>
> Attached inline below
>
> /Michael
>
> --- agent.c ---
>
> #include <string.h>
>
> #include "jni.h"
> #include "jvmti.h"
>
> /*
>   public class java.lang.ref.MyWeakReference
>     extends java.lang.ref.WeakReference {};
> */
> jbyte MyWeakReference_bytes[] = {
>   -54,  -2, -70, -66,   0,   0,   0,  49,
>     0,   5,   1,   0,  29, 106,  97, 118,
>    97,  47, 108,  97, 110, 103,  47, 114,
>   101, 102,  47,  77, 121,  87, 101,  97,
>   107,  82, 101, 102, 101, 114, 101, 110,
>    99, 101,   7,   0,   1,   1,   0,  27,
>   106,  97, 118,  97,  47, 108,  97, 110,
>   103,  47, 114, 101, 102,  47,  87, 101,
>    97, 107,  82, 101, 102, 101, 114, 101,
>   110,  99, 101,   7,   0,   3,   0,   1,
>     0,   2,   0,   4,   0,   0,   0,   0,
>     0,   0,   0,   0
> };
> jsize MyWeakReference_len = 0x5C;
>
> void JNICALL callback_ClassFileLoadHook(jvmtiEnv *jvmti, JNIEnv* env,
>   jclass class_being_redefined, jobject loader, const char* name,
>   jobject protection_domain,
>   jint class_data_len, const unsigned char* class_data,
>   jint* new_class_data_len, unsigned char** new_class_data) {
>
>   if (name && !strcmp(name, "java/lang/System")) {
>     (*env)->DefineClass(env, "java/lang/ref/MyWeakReference", NULL,
>       MyWeakReference_bytes, MyWeakReference_len);
>   }
> }
>
> JNIEXPORT jint JNICALL Agent_OnLoad(JavaVM *vm,
>   char *options, void *reserved) {
>
>   jvmtiEventCallbacks callbacks;
>   jvmtiCapabilities caps;
>   jvmtiEnv *jvmti;
>
>   (*vm)->GetEnv(vm, (void **)&jvmti, JVMTI_VERSION_9);
>
>   memset(&caps, 0, sizeof(caps));
>   memset(&callbacks, 0, sizeof(callbacks));
>
>   caps.can_generate_early_vmstart = 1;
>   caps.can_generate_all_class_hook_events = 1;
>   caps.can_generate_early_class_hook_events = 1;
>
>   callbacks.ClassFileLoadHook = &callback_ClassFileLoadHook;
>
>   (*jvmti)->AddCapabilities(jvmti, &caps);
>   (*jvmti)->SetEventCallbacks(jvmti, &callbacks, sizeof(callbacks));
>   (*jvmti)->SetEventNotificationMode(jvmti, JVMTI_ENABLE,
>     JVMTI_EVENT_CLASS_FILE_LOAD_HOOK, NULL);
>
>   return 0;
> }
>
> --- MiscTest9.java ---
>
> package app1;
>
> import java.lang.ref.WeakReference;
>
> public class MiscTest9 {
>   public static void main(String[] args) throws Exception {
>     WeakReference<?> weak = new WeakReference<>(new MiscTest9());
>
>     boolean present = true;
>     while (present) {
>       System.out.println("Running GC");
>       System.gc();
>       present = weak.get() != null;
>     }
>   }
> }
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question regarding JVM crashes in GC

Michael Rasmussen
On 10 April 2017 at 21:14, Volker Simonis <[hidden email]> wrote:
> So to keep a long story short, I don't think it is a good idea to change the
> class loading and initialization order of the "well-known" classes as
> described in systemDictionary.hpp. That order is carefully handcrafted and
> extremely sensitive to changes and can have all kind of unforeseeable side
> effects (as you've painfully detected yourself :)

Hi Volker

Thank you very much for the in-depth explanation.

I'm aware that the order is a bit fragile, and the approach we've been
using up till now hasn't changed that order (we've only caused extra
classes to be loaded, basically a class and an interface loaded very
early, and then the rest later as needed) -- unfortunately that
approach doesn't work with JDK9 and the restrictions added by the
module system, where what we can load before the module system is up
and running has been severely limited.

Usually though, when we've tinkered with something like that, we've
seen crashes immediately, which was why this one was kind of tricky,
since basically all our integration tests were working, except a few
here and there, which we narrowed down to those that did a full GC
(and then with help from Stefan, I figured out that it was because of
our Reference types being loaded).

/Michael
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question regarding JVM crashes in GC

Andrew Haley
On 10/04/17 20:15, Michael Rasmussen wrote:
> I'm aware that the order is a bit fragile, and the approach we've been
> using up till now hasn't changed that order (we've only caused extra
> classes to be loaded, basically a class and an interface loaded very
> early, and then the rest later as needed) -- unfortunately that
> approach doesn't work with JDK9 and the restrictions added by the
> module system, where what we can load before the module system is up
> and running has been severely limited.

Right, but if you initialize the classes in the correct order then
your program won't be fragile at all.  You've then got a solid and
repeatable solution.

Andrew.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question regarding JVM crashes in GC

Michael Rasmussen
On 11 April 2017 at 11:00, Andrew Haley <[hidden email]> wrote:
> Right, but if you initialize the classes in the correct order then
> your program won't be fragile at all.  You've then got a solid and
> repeatable solution.

Yeah, based on information from here we are working on a solution
right now, trying to defer loading as many of our classes as possible
until after VMStart.
I feel confident we'll get it to work, without crashing the JVM :D

/Michael
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Question regarding JVM crashes in GC

Stefan Johansson
Thanks Volker for digging down and identifying the root cause.

On 2017-04-11 10:07, Michael Rasmussen wrote:
> On 11 April 2017 at 11:00, Andrew Haley <[hidden email]> wrote:
>> Right, but if you initialize the classes in the correct order then
>> your program won't be fragile at all.  You've then got a solid and
>> repeatable solution.
> Yeah, based on information from here we are working on a solution
> right now, trying to defer loading as many of our classes as possible
> until after VMStart.
> I feel confident we'll get it to work, without crashing the JVM :D
Sounds like you have a way forward Michael. Good luck =)

Stefan
> /Michael

12
Loading...