Problems Excluding Specific Tests

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Problems Excluding Specific Tests

Ben Walsh
I am experiencing weird problems when using jtreg ...

I want to exclude the test https://hg.openjdk.java.net/jdk-updates/jdk11u/file/2c53a52abfee/test/jdk/java/util/concurrent/LinkedTransferQueue/WhiteBox.java

Adding this line identifying this test ...
java/util/concurrent/LinkedTransferQueue/WhiteBox.java https://example.com/issues/1 generic-all


... to an exclude file and using the "-exclude:<the exclude file>" parameter, I'm NOT seeing this test excluded in any of my runs.

Generally, there is no problem excluding other tests, including others under test/jdk/java/util/concurrent, using this method.

These sets of commands and the responses illustrate the problems (I have added comments in [ *** ... *** ] to clarify) and how it seems to be worse than just a specific test not being excluded when it should be ...

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

walshbp@bendev:~/jtreg_testing$ pwd
/home/walshbp/jtreg_testing
walshbp@bendev:~/jtreg_testing$ cat excludeFile
java/util/concurrent/LinkedTransferQueue/WhiteBox.java https://example.com/issues/1 generic-all
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/LinkedTransferQueue
Directory "JTwork" not found: creating
Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk
java/util/concurrent/LinkedTransferQueue/SpliteratorTraverseAddRemoveTest.java
Tests found: 1
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/LinkedTransferQueue
Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk
java/util/concurrent/LinkedTransferQueue/SpliteratorTraverseAddRemoveTest.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]
Tests found: 2
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent | tail
java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java
java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java
java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java
java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java
java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java
java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java
java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java
java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java
java/util/concurrent/TimeUnit/Basic.java
Tests found: 108
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent | tail
java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java
java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java
java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java
java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java
java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java
java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java
java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java
java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java
java/util/concurrent/TimeUnit/Basic.java
Tests found: 108
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 821
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 821
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 5,275
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 5,275
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]
java/util/concurrent/LinkedTransferQueue/WhiteBox.java
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail
vm/runtime/MonitorCacheMaybeExpand_DeadLock.java
vm/runtime/ReflectStackOverflow.java
vm/runtime/ShiftTest.java
vm/runtime/WideStrictInline.java
vm/verifier/defaultMethods/DefaultMethodRegressionTests.java
vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java
vm/verifier/TestStaticIF.java
vm/verifier/VerifyProtectedConstructor.java
vm/verifier/VerifyStackForExceptionHandlers.java
Tests found: 9,478
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail
vm/runtime/MonitorCacheMaybeExpand_DeadLock.java
vm/runtime/ReflectStackOverflow.java
vm/runtime/ShiftTest.java
vm/runtime/WideStrictInline.java
vm/verifier/defaultMethods/DefaultMethodRegressionTests.java
vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java
vm/verifier/TestStaticIF.java
vm/verifier/VerifyProtectedConstructor.java
vm/verifier/VerifyStackForExceptionHandlers.java
Tests found: 9,478

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

walshbp@bendev:~/jtreg_testing$ pwd
/home/walshbp/jtreg_testing
walshbp@bendev:~/jtreg_testing$ cat excludeFile
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java https://example.com/issues/1 generic-all
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/ArrayBlockingQueue
Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk
Tests found: 0
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/ArrayBlockingQueue
Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]
Tests found: 1
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent | grep WhiteBox
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent | tail
java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java
java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java
java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java
java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java
java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java
java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java
java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java
java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java
java/util/concurrent/TimeUnit/Basic.java
Tests found: 104
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent | tail
java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java
java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java
java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java
java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java
java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java
java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java
java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java
java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java
java/util/concurrent/TimeUnit/Basic.java
Tests found: 108
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 817
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 821
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 5,271
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 5,275
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail
vm/runtime/MonitorCacheMaybeExpand_DeadLock.java
vm/runtime/ReflectStackOverflow.java
vm/runtime/ShiftTest.java
vm/runtime/WideStrictInline.java
vm/verifier/defaultMethods/DefaultMethodRegressionTests.java
vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java
vm/verifier/TestStaticIF.java
vm/verifier/VerifyProtectedConstructor.java
vm/verifier/VerifyStackForExceptionHandlers.java
Tests found: 9,474
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail
vm/runtime/MonitorCacheMaybeExpand_DeadLock.java
vm/runtime/ReflectStackOverflow.java
vm/runtime/ShiftTest.java
vm/runtime/WideStrictInline.java
vm/verifier/defaultMethods/DefaultMethodRegressionTests.java
vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java
vm/verifier/TestStaticIF.java
vm/verifier/VerifyProtectedConstructor.java
vm/verifier/VerifyStackForExceptionHandlers.java
Tests found: 9,478

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -version
jtreg, version 4.2 dev 398
Installed in /home/walshbp/jtreg/lib/jtreg.jar
Running on platform version 11 from /home/walshbp/openjdk11-hotspot-binary/jdk-11.
Built with Java(TM) 2 SDK, Version 1.8.0_121-b13 on June 11, 2018.
Copyright (c) 1999, 2016, Oracle and/or its affiliates. All rights reserved.
Use is subject to license terms.
JTHarness: version 5.0
JCov 3.0-2
TestNG (testng.jar): version unknown
TestNG (jcommander.jar): version 1.72
AsmTools: version 7.0

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Please let me know what other information is required to investigate this further.


Thanks,
Ben Walsh

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Reply | Threaded
Open this post in threaded view
|

Re: Problems Excluding Specific Tests

Ben Walsh
Here's a Github Gist which maintains the formatting for the commands and responses in my original email - https://gist.github.com/ben-walsh/25cec1ea7cbfd2c1cedd444cf6c7cd84


Ben Walsh

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Reply | Threaded
Open this post in threaded view
|

Re: Problems Excluding Specific Tests

Martin Buchholz-3
In reply to this post by Ben Walsh
(Author of test is curious ...)
There's a bunch of similar WhiteBox tests.
(They are expected to be brittle as you would expect from the name.)
Maybe it has to do with @run testng

On Wed, Nov 7, 2018 at 7:40 AM, Ben Walsh <[hidden email]> wrote:
I am experiencing weird problems when using jtreg ...

I want to exclude the test https://hg.openjdk.java.net/jdk-updates/jdk11u/file/2c53a52abfee/test/jdk/java/util/concurrent/LinkedTransferQueue/WhiteBox.java

Adding this line identifying this test ...
java/util/concurrent/LinkedTransferQueue/WhiteBox.java https://example.com/issues/1 generic-all


... to an exclude file and using the "-exclude:<the exclude file>" parameter, I'm NOT seeing this test excluded in any of my runs.

Generally, there is no problem excluding other tests, including others under test/jdk/java/util/concurrent, using this method.

These sets of commands and the responses illustrate the problems (I have added comments in [ *** ... *** ] to clarify) and how it seems to be worse than just a specific test not being excluded when it should be ...

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

walshbp@bendev:~/jtreg_testing$ pwd
/home/walshbp/jtreg_testing
walshbp@bendev:~/jtreg_testing$ cat excludeFile
java/util/concurrent/LinkedTransferQueue/WhiteBox.java https://example.com/issues/1 generic-all
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/LinkedTransferQueue
Directory "JTwork" not found: creating
Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk
java/util/concurrent/LinkedTransferQueue/SpliteratorTraverseAddRemoveTest.java
Tests found: 1
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/LinkedTransferQueue
Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk
java/util/concurrent/LinkedTransferQueue/SpliteratorTraverseAddRemoveTest.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]
Tests found: 2
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent | tail
java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java
java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java
java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java
java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java
java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java
java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java
java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java
java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java
java/util/concurrent/TimeUnit/Basic.java
Tests found: 108
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent | tail
java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java
java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java
java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java
java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java
java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java
java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java
java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java
java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java
java/util/concurrent/TimeUnit/Basic.java
Tests found: 108
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 821
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 821
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 5,275
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 5,275
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]
java/util/concurrent/LinkedTransferQueue/WhiteBox.java
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail
vm/runtime/MonitorCacheMaybeExpand_DeadLock.java
vm/runtime/ReflectStackOverflow.java
vm/runtime/ShiftTest.java
vm/runtime/WideStrictInline.java
vm/verifier/defaultMethods/DefaultMethodRegressionTests.java
vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java
vm/verifier/TestStaticIF.java
vm/verifier/VerifyProtectedConstructor.java
vm/verifier/VerifyStackForExceptionHandlers.java
Tests found: 9,478
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail
vm/runtime/MonitorCacheMaybeExpand_DeadLock.java
vm/runtime/ReflectStackOverflow.java
vm/runtime/ShiftTest.java
vm/runtime/WideStrictInline.java
vm/verifier/defaultMethods/DefaultMethodRegressionTests.java
vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java
vm/verifier/TestStaticIF.java
vm/verifier/VerifyProtectedConstructor.java
vm/verifier/VerifyStackForExceptionHandlers.java
Tests found: 9,478

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

walshbp@bendev:~/jtreg_testing$ pwd
/home/walshbp/jtreg_testing
walshbp@bendev:~/jtreg_testing$ cat excludeFile
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java https://example.com/issues/1 generic-all
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/ArrayBlockingQueue
Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk
Tests found: 0
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/ArrayBlockingQueue
Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]
Tests found: 1
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent | grep WhiteBox
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent | tail
java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java
java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java
java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java
java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java
java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java
java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java
java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java
java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java
java/util/concurrent/TimeUnit/Basic.java
Tests found: 104
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent | tail
java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java
java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java
java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java
java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java
java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java
java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java
java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java
java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java
java/util/concurrent/TimeUnit/Basic.java
Tests found: 108
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 817
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 821
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 5,271
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 5,275
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail
vm/runtime/MonitorCacheMaybeExpand_DeadLock.java
vm/runtime/ReflectStackOverflow.java
vm/runtime/ShiftTest.java
vm/runtime/WideStrictInline.java
vm/verifier/defaultMethods/DefaultMethodRegressionTests.java
vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java
vm/verifier/TestStaticIF.java
vm/verifier/VerifyProtectedConstructor.java
vm/verifier/VerifyStackForExceptionHandlers.java
Tests found: 9,474
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail
vm/runtime/MonitorCacheMaybeExpand_DeadLock.java
vm/runtime/ReflectStackOverflow.java
vm/runtime/ShiftTest.java
vm/runtime/WideStrictInline.java
vm/verifier/defaultMethods/DefaultMethodRegressionTests.java
vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java
vm/verifier/TestStaticIF.java
vm/verifier/VerifyProtectedConstructor.java
vm/verifier/VerifyStackForExceptionHandlers.java
Tests found: 9,478

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -version
jtreg, version 4.2 dev 398
Installed in /home/walshbp/jtreg/lib/jtreg.jar
Running on platform version 11 from /home/walshbp/openjdk11-hotspot-binary/jdk-11.
Built with Java(TM) 2 SDK, Version 1.8.0_121-b13 on June 11, 2018.
Copyright (c) 1999, 2016, Oracle and/or its affiliates. All rights reserved.
Use is subject to license terms.
JTHarness: version 5.0
JCov 3.0-2
TestNG (testng.jar): version unknown
TestNG (jcommander.jar): version 1.72
AsmTools: version 7.0

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Please let me know what other information is required to investigate this further.


Thanks,
Ben Walsh

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

Reply | Threaded
Open this post in threaded view
|

Re: Problems Excluding Specific Tests

Jonathan Gibbons
In reply to this post by Ben Walsh

Ben,

The second column should be a number, such as a 7-digit JBS issue number, and not a URL.

Because you have not used a number, jtreg has incorrectly assumed no number, and that the word that follows (your URL) is in fact the platform, meaning that the test will only be excluded on a platform called https://example.com/issues/1 . That is why your test is not being excluded.

The fix is to change the second word on the line to a number. Use a default value like 0 if necessary.

-- Jon


On 11/7/18 7:40 AM, Ben Walsh wrote:
I am experiencing weird problems when using jtreg ...

I want to exclude the test https://hg.openjdk.java.net/jdk-updates/jdk11u/file/2c53a52abfee/test/jdk/java/util/concurrent/LinkedTransferQueue/WhiteBox.java

Adding this line identifying this test ...
java/util/concurrent/LinkedTransferQueue/WhiteBox.java https://example.com/issues/1 generic-all



... to an exclude file and using the "-exclude:<the exclude file>" parameter, I'm NOT seeing this test excluded in any of my runs.

Generally, there is no problem excluding other tests, including others under test/jdk/java/util/concurrent, using this method.

These sets of commands and the responses illustrate the problems (I have added comments in [ *** ... *** ] to clarify) and how it seems to be worse than just a specific test not being excluded when it should be ...

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

walshbp@bendev:~/jtreg_testing$ pwd
/home/walshbp/jtreg_testing
walshbp@bendev:~/jtreg_testing$ cat excludeFile
java/util/concurrent/LinkedTransferQueue/WhiteBox.java https://example.com/issues/1 generic-all
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/LinkedTransferQueue
Directory "JTwork" not found: creating
Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk
java/util/concurrent/LinkedTransferQueue/SpliteratorTraverseAddRemoveTest.java
Tests found: 1
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/LinkedTransferQueue
Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk
java/util/concurrent/LinkedTransferQueue/SpliteratorTraverseAddRemoveTest.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]
Tests found: 2
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| tail
java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java
java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java
java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java
java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java
java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java
java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java
java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java
java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java
java/util/concurrent/TimeUnit/Basic.java
Tests found: 108
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| tail
java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java
java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java
java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java
java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java
java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java
java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java
java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java
java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java
java/util/concurrent/TimeUnit/Basic.java
Tests found: 108
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 821
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 821
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 5,275
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 5,275
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]
java/util/concurrent/LinkedTransferQueue/WhiteBox.java
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail
vm/runtime/MonitorCacheMaybeExpand_DeadLock.java
vm/runtime/ReflectStackOverflow.java
vm/runtime/ShiftTest.java
vm/runtime/WideStrictInline.java
vm/verifier/defaultMethods/DefaultMethodRegressionTests.java
vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java
vm/verifier/TestStaticIF.java
vm/verifier/VerifyProtectedConstructor.java
vm/verifier/VerifyStackForExceptionHandlers.java
Tests found: 9,478
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail
vm/runtime/MonitorCacheMaybeExpand_DeadLock.java
vm/runtime/ReflectStackOverflow.java
vm/runtime/ShiftTest.java
vm/runtime/WideStrictInline.java
vm/verifier/defaultMethods/DefaultMethodRegressionTests.java
vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java
vm/verifier/TestStaticIF.java
vm/verifier/VerifyProtectedConstructor.java
vm/verifier/VerifyStackForExceptionHandlers.java
Tests found: 9,478

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

walshbp@bendev:~/jtreg_testing$ pwd
/home/walshbp/jtreg_testing
walshbp@bendev:~/jtreg_testing$ cat excludeFile
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java https://example.com/issues/1 generic-all
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/ArrayBlockingQueue
Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk
Tests found: 0
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/ArrayBlockingQueue
Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]
Tests found: 1
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| grep WhiteBox
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| tail
java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java
java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java
java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java
java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java
java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java
java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java
java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java
java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java
java/util/concurrent/TimeUnit/Basic.java
Tests found: 104
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| tail
java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java
java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java
java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java
java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java
java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java
java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java
java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java
java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java
java/util/concurrent/TimeUnit/Basic.java
Tests found: 108
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 817
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 821
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 5,271
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail
java/util/zip/TestCRC32.java
java/util/zip/TestCRC32C.java
java/util/zip/TestEmptyZip.java
java/util/zip/TestExtraTime.java
java/util/zip/TestLocalTime.java
java/util/zip/TestZipError.java
java/util/zip/TimeChecksum.java
java/util/zip/TotalInOut.java
java/util/zip/ZipCoding.java
Tests found: 5,275
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail
vm/runtime/MonitorCacheMaybeExpand_DeadLock.java
vm/runtime/ReflectStackOverflow.java
vm/runtime/ShiftTest.java
vm/runtime/WideStrictInline.java
vm/verifier/defaultMethods/DefaultMethodRegressionTests.java
vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java
vm/verifier/TestStaticIF.java
vm/verifier/VerifyProtectedConstructor.java
vm/verifier/VerifyStackForExceptionHandlers.java
Tests found: 9,474
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]
java/util/concurrent/ConcurrentHashMap/WhiteBox.java
java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java
java/util/concurrent/LinkedTransferQueue/WhiteBox.java
java/util/concurrent/PriorityBlockingQueue/WhiteBox.java
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail
vm/runtime/MonitorCacheMaybeExpand_DeadLock.java
vm/runtime/ReflectStackOverflow.java
vm/runtime/ShiftTest.java
vm/runtime/WideStrictInline.java
vm/verifier/defaultMethods/DefaultMethodRegressionTests.java
vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java
vm/verifier/TestStaticIF.java
vm/verifier/VerifyProtectedConstructor.java
vm/verifier/VerifyStackForExceptionHandlers.java
Tests found: 9,478

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -version
jtreg, version 4.2 dev 398
Installed in /home/walshbp/jtreg/lib/jtreg.jar
Running on platform version 11 from /home/walshbp/openjdk11-hotspot-binary/jdk-11.
Built with Java(TM) 2 SDK, Version 1.8.0_121-b13 on June 11, 2018.
Copyright (c) 1999, 2016, Oracle and/or its affiliates. All rights reserved.
Use is subject to license terms.
JTHarness: version 5.0
JCov 3.0-2
TestNG (testng.jar): version unknown
TestNG (jcommander.jar): version 1.72
AsmTools: version 7.0

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Please let me know what other information is required to investigate this further.


Thanks,
Ben Walsh

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

Reply | Threaded
Open this post in threaded view
|

Re: Problems Excluding Specific Tests

Ben Walsh
Unfortunately, that does not seem to be the problem, as these commands and responses show - https://gist.github.com/ben-walsh/6180847c91dcb1d71b5ff807e718b210

First set of commands is an attempt to exclude one of the problematic tests again but this time using your suggestion of replacing the URL with just "0" within the exclude file. There is no change in the responses.

Second set of commands illustrates that such a URL can be used instead of just a number, when instead excluding a randomly chosen alternative test. All responses are as expected - the test continues to be excluded up through the test package hierarchy.

Regards,
Ben Walsh



From:        Jonathan Gibbons <[hidden email]>
To:        Ben Walsh <[hidden email]>, [hidden email]
Date:        07/11/2018 18:53
Subject:        Re: Problems Excluding Specific Tests




Ben,

The second column should be a number, such as a 7-digit JBS issue number, and not a URL.

Because you have not used a number, jtreg has incorrectly assumed no number, and that the word that follows (your URL) is in fact the platform, meaning that the test will only be excluded on a platform called https://example.com/issues/1 . That is why your test is not being excluded.

The fix is to change the second word on the line to a number. Use a default value like 0 if necessary.

-- Jon


On 11/7/18 7:40 AM, Ben Walsh wrote:
I am experiencing weird problems when using jtreg ...

I want to exclude the test
https://hg.openjdk.java.net/jdk-updates/jdk11u/file/2c53a52abfee/test/jdk/java/util/concurrent/LinkedTransferQueue/WhiteBox.java

Adding this line identifying this test ...

java/util/concurrent/LinkedTransferQueue/WhiteBox.java https://example.com/issues/1 generic-all



... to an exclude file and using the "-exclude:<the exclude file>" parameter, I'm NOT seeing this test excluded in any of my runs.


Generally, there is no problem excluding other tests, including others under test/jdk/java/util/concurrent, using this method.


These sets of commands and the responses illustrate the problems (I have added comments in [ *** ... *** ] to clarify) and how it seems to be worse than just a specific test not being excluded when it should be ...


--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


walshbp@bendev:~/jtreg_testing$ pwd

/home/walshbp/jtreg_testing

walshbp@bendev:~/jtreg_testing$ cat excludeFile

java/util/concurrent/LinkedTransferQueue/WhiteBox.java
https://example.com/issues/1 generic-all
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/LinkedTransferQueue
Directory "JTwork" not found: creating

Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk

java/util/concurrent/LinkedTransferQueue/SpliteratorTraverseAddRemoveTest.java

Tests found: 1

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/LinkedTransferQueue
Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk

java/util/concurrent/LinkedTransferQueue/SpliteratorTraverseAddRemoveTest.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]

Tests found: 2

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| tail

java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java

java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java

java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java

java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java

java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java

java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java

java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java

java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java

java/util/concurrent/TimeUnit/Basic.java

Tests found: 108

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| tail

java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java

java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java

java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java

java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java

java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java

java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java

java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java

java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java

java/util/concurrent/TimeUnit/Basic.java

Tests found: 108

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 821

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 821

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 5,275

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 5,275

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]

java/util/concurrent/LinkedTransferQueue/WhiteBox.java

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail

vm/runtime/MonitorCacheMaybeExpand_DeadLock.java

vm/runtime/ReflectStackOverflow.java

vm/runtime/ShiftTest.java

vm/runtime/WideStrictInline.java

vm/verifier/defaultMethods/DefaultMethodRegressionTests.java

vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java

vm/verifier/TestStaticIF.java

vm/verifier/VerifyProtectedConstructor.java

vm/verifier/VerifyStackForExceptionHandlers.java

Tests found: 9,478

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail

vm/runtime/MonitorCacheMaybeExpand_DeadLock.java

vm/runtime/ReflectStackOverflow.java

vm/runtime/ShiftTest.java

vm/runtime/WideStrictInline.java

vm/verifier/defaultMethods/DefaultMethodRegressionTests.java

vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java

vm/verifier/TestStaticIF.java

vm/verifier/VerifyProtectedConstructor.java

vm/verifier/VerifyStackForExceptionHandlers.java

Tests found: 9,478


--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


walshbp@bendev:~/jtreg_testing$ pwd

/home/walshbp/jtreg_testing

walshbp@bendev:~/jtreg_testing$ cat excludeFile
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
https://example.com/issues/1 generic-all
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/ArrayBlockingQueue
Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk

Tests found: 0

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/ArrayBlockingQueue
Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]

Tests found: 1

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| grep WhiteBox

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| tail

java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java

java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java

java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java

java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java

java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java

java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java

java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java

java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java

java/util/concurrent/TimeUnit/Basic.java

Tests found: 104

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| tail

java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java

java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java

java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java

java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java

java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java

java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java

java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java

java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java

java/util/concurrent/TimeUnit/Basic.java

Tests found: 108

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 817

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 821

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 5,271

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 5,275

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail

vm/runtime/MonitorCacheMaybeExpand_DeadLock.java

vm/runtime/ReflectStackOverflow.java

vm/runtime/ShiftTest.java

vm/runtime/WideStrictInline.java

vm/verifier/defaultMethods/DefaultMethodRegressionTests.java

vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java

vm/verifier/TestStaticIF.java

vm/verifier/VerifyProtectedConstructor.java

vm/verifier/VerifyStackForExceptionHandlers.java

Tests found: 9,474

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail

vm/runtime/MonitorCacheMaybeExpand_DeadLock.java

vm/runtime/ReflectStackOverflow.java

vm/runtime/ShiftTest.java

vm/runtime/WideStrictInline.java

vm/verifier/defaultMethods/DefaultMethodRegressionTests.java

vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java

vm/verifier/TestStaticIF.java

vm/verifier/VerifyProtectedConstructor.java

vm/verifier/VerifyStackForExceptionHandlers.java

Tests found: 9,478


--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -version

jtreg, version 4.2 dev 398

Installed in /home/walshbp/jtreg/lib/jtreg.jar

Running on platform version 11 from /home/walshbp/openjdk11-hotspot-binary/jdk-11.

Built with Java(TM) 2 SDK, Version 1.8.0_121-b13 on June 11, 2018.

Copyright (c) 1999, 2016, Oracle and/or its affiliates. All rights reserved.

Use is subject to license terms.

JTHarness: version 5.0

JCov 3.0-2

TestNG (testng.jar): version unknown

TestNG (jcommander.jar): version 1.72

AsmTools: version 7.0


--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------



Please let me know what other information is required to investigate this further.



Thanks,
Ben Walsh

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU




Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Reply | Threaded
Open this post in threaded view
|

Re: Problems Excluding Specific Tests

Jonathan Gibbons

Ben,

My apologies; you have tripped over a known bug in jtreg [1], such that it is failing to correctly match tests against the exclude list in some unusual cases.

Although I didn't read all of your test cases, the first 3 executions were enough to confirm the bug. Note how in the first run of jtreg, the test *is* correctly excluded, but then it it is incorrectly not excluded later on, when other similar tests are not excluded.

I am working on a fix for jtreg, which should be available soon.

The bug is that the content of the test description (the /* @test */) comment is identical for this test and some other tests. If you need a short term workaround, add some unique text, such as a @summary or @bug info into the test you are trying to exclude.

-- Jon

[1] https://bugs.openjdk.java.net/browse/CODETOOLS-7902067



On 11/08/2018 03:42 AM, Ben Walsh wrote:
Unfortunately, that does not seem to be the problem, as these commands and responses show - https://gist.github.com/ben-walsh/6180847c91dcb1d71b5ff807e718b210

First set of commands is an attempt to exclude one of the problematic tests again but this time using your suggestion of replacing the URL with just "0" within the exclude file. There is no change in the responses.

Second set of commands illustrates that such a URL can be used instead of just a number, when instead excluding a randomly chosen alternative test. All responses are as expected - the test continues to be excluded up through the test package hierarchy.

Regards,
Ben Walsh



From:        Jonathan Gibbons [hidden email]
To:        Ben Walsh [hidden email], [hidden email]
Date:        07/11/2018 18:53
Subject:        Re: Problems Excluding Specific Tests




Ben,

The second column should be a number, such as a 7-digit JBS issue number, and not a URL.

Because you have not used a number, jtreg has incorrectly assumed no number, and that the word that follows (your URL) is in fact the platform, meaning that the test will only be excluded on a platform called https://example.com/issues/1 . That is why your test is not being excluded.

The fix is to change the second word on the line to a number. Use a default value like 0 if necessary.

-- Jon


On 11/7/18 7:40 AM, Ben Walsh wrote:
I am experiencing weird problems when using jtreg ...

I want to exclude the test
https://hg.openjdk.java.net/jdk-updates/jdk11u/file/2c53a52abfee/test/jdk/java/util/concurrent/LinkedTransferQueue/WhiteBox.java

Adding this line identifying this test ...

java/util/concurrent/LinkedTransferQueue/WhiteBox.java https://example.com/issues/1 generic-all




... to an exclude file and using the "-exclude:<the exclude file>" parameter, I'm NOT seeing this test excluded in any of my runs.


Generally, there is no problem excluding other tests, including others under test/jdk/java/util/concurrent, using this method.


These sets of commands and the responses illustrate the problems (I have added comments in [ *** ... *** ] to clarify) and how it seems to be worse than just a specific test not being excluded when it should be ...


--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


walshbp@bendev:~/jtreg_testing$ pwd

/home/walshbp/jtreg_testing

walshbp@bendev:~/jtreg_testing$ cat excludeFile

java/util/concurrent/LinkedTransferQueue/WhiteBox.java
https://example.com/issues/1 generic-all
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/LinkedTransferQueue
Directory "JTwork" not found: creating

Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk

java/util/concurrent/LinkedTransferQueue/SpliteratorTraverseAddRemoveTest.java

Tests found: 1

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/LinkedTransferQueue
Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk

java/util/concurrent/LinkedTransferQueue/SpliteratorTraverseAddRemoveTest.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]

Tests found: 2

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| tail

java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java

java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java

java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java

java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java

java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java

java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java

java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java

java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java

java/util/concurrent/TimeUnit/Basic.java

Tests found: 108

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent|grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent|tail

java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java

java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java

java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java

java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java

java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java

java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java

java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java

java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java

java/util/concurrent/TimeUnit/Basic.java

Tests found: 108

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 821

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 821

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 5,275

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 5,275

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]

java/util/concurrent/LinkedTransferQueue/WhiteBox.java

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail

vm/runtime/MonitorCacheMaybeExpand_DeadLock.java

vm/runtime/ReflectStackOverflow.java

vm/runtime/ShiftTest.java

vm/runtime/WideStrictInline.java

vm/verifier/defaultMethods/DefaultMethodRegressionTests.java

vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java

vm/verifier/TestStaticIF.java

vm/verifier/VerifyProtectedConstructor.java

vm/verifier/VerifyStackForExceptionHandlers.java

Tests found: 9,478

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail

vm/runtime/MonitorCacheMaybeExpand_DeadLock.java

vm/runtime/ReflectStackOverflow.java

vm/runtime/ShiftTest.java

vm/runtime/WideStrictInline.java

vm/verifier/defaultMethods/DefaultMethodRegressionTests.java

vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java

vm/verifier/TestStaticIF.java

vm/verifier/VerifyProtectedConstructor.java

vm/verifier/VerifyStackForExceptionHandlers.java

Tests found: 9,478


--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


walshbp@bendev:~/jtreg_testing$ pwd

/home/walshbp/jtreg_testing

walshbp@bendev:~/jtreg_testing$ cat excludeFile
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
https://example.com/issues/1 generic-all
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/ArrayBlockingQueue
Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk

Tests found: 0

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/ArrayBlockingQueue
Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]

Tests found: 1

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| grep WhiteBox

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| tail

java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java

java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java

java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java

java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java

java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java

java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java

java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java

java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java

java/util/concurrent/TimeUnit/Basic.java

Tests found: 104

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent|grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent|tail

java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java

java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java

java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java

java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java

java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java

java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java

java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java

java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java

java/util/concurrent/TimeUnit/Basic.java

Tests found: 108

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 817

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 821

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 5,271

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 5,275

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail

vm/runtime/MonitorCacheMaybeExpand_DeadLock.java

vm/runtime/ReflectStackOverflow.java

vm/runtime/ShiftTest.java

vm/runtime/WideStrictInline.java

vm/verifier/defaultMethods/DefaultMethodRegressionTests.java

vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java

vm/verifier/TestStaticIF.java

vm/verifier/VerifyProtectedConstructor.java

vm/verifier/VerifyStackForExceptionHandlers.java

Tests found: 9,474

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail

vm/runtime/MonitorCacheMaybeExpand_DeadLock.java

vm/runtime/ReflectStackOverflow.java

vm/runtime/ShiftTest.java

vm/runtime/WideStrictInline.java

vm/verifier/defaultMethods/DefaultMethodRegressionTests.java

vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java

vm/verifier/TestStaticIF.java

vm/verifier/VerifyProtectedConstructor.java

vm/verifier/VerifyStackForExceptionHandlers.java

Tests found: 9,478


--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -version

jtreg, version 4.2 dev 398

Installed in /home/walshbp/jtreg/lib/jtreg.jar

Running on platform version 11 from /home/walshbp/openjdk11-hotspot-binary/jdk-11.

Built with Java(TM) 2 SDK, Version 1.8.0_121-b13 on June 11, 2018.

Copyright (c) 1999, 2016, Oracle and/or its affiliates. All rights reserved.

Use is subject to license terms.

JTHarness: version 5.0

JCov 3.0-2

TestNG (testng.jar): version unknown

TestNG (jcommander.jar): version 1.72

AsmTools: version 7.0


--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------



Please let me know what other information is required to investigate this further.



Thanks,
Ben Walsh

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU




Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU


Reply | Threaded
Open this post in threaded view
|

Re: Problems Excluding Specific Tests

Ben Walsh
Hi Jon,

No problem. That's good to know. Thanks for investigating.



Ben




From:        Jonathan Gibbons <[hidden email]>
To:        Ben Walsh <[hidden email]>, [hidden email]
Date:        09/11/2018 01:52
Subject:        Re: Problems Excluding Specific Tests




Ben,

My apologies; you have tripped over a known bug in jtreg [1], such that it is failing to correctly match tests against the exclude list in some unusual cases.

Although I didn't read all of your test cases, the first 3 executions were enough to confirm the bug. Note how in the first run of jtreg, the test *is* correctly excluded, but then it it is incorrectly not excluded later on, when other similar tests are not excluded.

I am working on a fix for jtreg, which should be available soon.

The bug is that the content of the test description (the /* @test */) comment is identical for this test and some other tests. If you need a short term workaround, add some unique text, such as a @summary or @bug info into the test you are trying to exclude.

-- Jon

[1] https://bugs.openjdk.java.net/browse/CODETOOLS-7902067


On 11/08/2018 03:42 AM, Ben Walsh wrote:
Unfortunately, that does not seem to be the problem, as these commands and responses show - https://gist.github.com/ben-walsh/6180847c91dcb1d71b5ff807e718b210

First set of commands is an attempt to exclude one of the problematic tests again but this time using your suggestion of replacing the URL with just "0" within the exclude file. There is no change in the responses.


Second set of commands illustrates that such a URL can be used instead of just a number, when instead excluding a randomly chosen alternative test. All responses are as expected - the test continues to be excluded up through the test package hierarchy.


Regards,
Ben Walsh




From:        
Jonathan Gibbons [hidden email]
To:        
Ben Walsh [hidden email], [hidden email]
Date:        
07/11/2018 18:53
Subject:        
Re: Problems Excluding Specific Tests





Ben,

The second column should be a number, such as a 7-digit JBS issue number, and not a URL.

Because you have not used a number, jtreg has incorrectly assumed no number, and that the word that follows (your URL) is in fact the platform, meaning that the test will only be excluded on a platform called https://example.com/issues/1 . That is why your test is not being excluded.

The fix is to change the second word on the line to a number. Use a default value like 0 if necessary.

-- Jon


On 11/7/18 7:40 AM, Ben Walsh wrote:

I am experiencing weird problems when using jtreg ...


I want to exclude the test
https://hg.openjdk.java.net/jdk-updates/jdk11u/file/2c53a52abfee/test/jdk/java/util/concurrent/LinkedTransferQueue/WhiteBox.java

Adding this line identifying this test ...

java/util/concurrent/LinkedTransferQueue/WhiteBox.java https://example.com/issues/1 generic-all




... to an exclude file and using the "-exclude:<the exclude file>" parameter, I'm NOT seeing this test excluded in any of my runs.


Generally, there is no problem excluding other tests, including others under test/jdk/java/util/concurrent, using this method.


These sets of commands and the responses illustrate the problems (I have added comments in [ *** ... *** ] to clarify) and how it seems to be worse than just a specific test not being excluded when it should be ...


--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


walshbp@bendev:~/jtreg_testing$ pwd

/home/walshbp/jtreg_testing

walshbp@bendev:~/jtreg_testing$ cat excludeFile

java/util/concurrent/LinkedTransferQueue/WhiteBox.java
https://example.com/issues/1 generic-all
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/LinkedTransferQueue
Directory "JTwork" not found: creating

Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk

java/util/concurrent/LinkedTransferQueue/SpliteratorTraverseAddRemoveTest.java

Tests found: 1

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/LinkedTransferQueue
Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk

java/util/concurrent/LinkedTransferQueue/SpliteratorTraverseAddRemoveTest.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]

Tests found: 2

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| tail

java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java

java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java

java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java

java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java

java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java

java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java

java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java

java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java

java/util/concurrent/TimeUnit/Basic.java

Tests found: 108

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent|grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent|tail

java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java

java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java

java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java

java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java

java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java

java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java

java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java

java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java

java/util/concurrent/TimeUnit/Basic.java

Tests found: 108

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 821

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 821

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 5,275

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ *** EXPECTED - NOT USING EXCLUDE PARAMETER *** ]

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 5,275

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java                                [ *** NOT EXPECTED - SHOULD HAVE BEEN EXCLUDED *** ]

java/util/concurrent/LinkedTransferQueue/WhiteBox.java

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail

vm/runtime/MonitorCacheMaybeExpand_DeadLock.java

vm/runtime/ReflectStackOverflow.java

vm/runtime/ShiftTest.java

vm/runtime/WideStrictInline.java

vm/verifier/defaultMethods/DefaultMethodRegressionTests.java

vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java

vm/verifier/TestStaticIF.java

vm/verifier/VerifyProtectedConstructor.java

vm/verifier/VerifyStackForExceptionHandlers.java

Tests found: 9,478

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail

vm/runtime/MonitorCacheMaybeExpand_DeadLock.java

vm/runtime/ReflectStackOverflow.java

vm/runtime/ShiftTest.java

vm/runtime/WideStrictInline.java

vm/verifier/defaultMethods/DefaultMethodRegressionTests.java

vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java

vm/verifier/TestStaticIF.java

vm/verifier/VerifyProtectedConstructor.java

vm/verifier/VerifyStackForExceptionHandlers.java

Tests found: 9,478


--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


walshbp@bendev:~/jtreg_testing$ pwd

/home/walshbp/jtreg_testing

walshbp@bendev:~/jtreg_testing$ cat excludeFile
java/util/concurrent/ArrayBlockingQueue/WhiteBox.java
https://example.com/issues/1 generic-all
walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/ArrayBlockingQueue
Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk

Tests found: 0

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent/ArrayBlockingQueue
Testsuite: /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]

Tests found: 1

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| grep WhiteBox

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent| tail

java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java

java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java

java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java

java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java

java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java

java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java

java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java

java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java

java/util/concurrent/TimeUnit/Basic.java

Tests found: 104

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent|grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util/concurrent|tail

java/util/concurrent/ThreadPoolExecutor/FlakyThreadFactory.java

java/util/concurrent/ThreadPoolExecutor/ModifyCorePoolSize.java

java/util/concurrent/ThreadPoolExecutor/ScheduledTickleService.java

java/util/concurrent/ThreadPoolExecutor/SelfInterrupt.java

java/util/concurrent/ThreadPoolExecutor/ShutdownNowExecuteRace.java

java/util/concurrent/ThreadPoolExecutor/ThreadRestarts.java

java/util/concurrent/ThreadPoolExecutor/ThrowingTasks.java

java/util/concurrent/ThreadPoolExecutor/TimeOutShrink.java

java/util/concurrent/TimeUnit/Basic.java

Tests found: 108

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 817

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java/util | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 821

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 5,271

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk/java | tail

java/util/zip/TestCRC32.java

java/util/zip/TestCRC32C.java

java/util/zip/TestEmptyZip.java

java/util/zip/TestExtraTime.java

java/util/zip/TestLocalTime.java

java/util/zip/TestZipError.java

java/util/zip/TimeChecksum.java

java/util/zip/TotalInOut.java

java/util/zip/ZipCoding.java

Tests found: 5,275

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java                        [ PROBLEM - WHY HAVE FOUR TESTS BEEN EXCLUDED ? THREE MORE THAN EXPECTED ]

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l -exclude:/home/walshbp/jtreg_testing/excludeFile /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail

vm/runtime/MonitorCacheMaybeExpand_DeadLock.java

vm/runtime/ReflectStackOverflow.java

vm/runtime/ShiftTest.java

vm/runtime/WideStrictInline.java

vm/verifier/defaultMethods/DefaultMethodRegressionTests.java

vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java

vm/verifier/TestStaticIF.java

vm/verifier/VerifyProtectedConstructor.java

vm/verifier/VerifyStackForExceptionHandlers.java

Tests found: 9,474

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | grep WhiteBox

java/util/concurrent/ArrayBlockingQueue/WhiteBox.java                                [ EXPECTED - NOT USING EXCLUDE PARAMETER ]

java/util/concurrent/ConcurrentHashMap/WhiteBox.java

java/util/concurrent/ConcurrentLinkedQueue/WhiteBox.java

java/util/concurrent/LinkedTransferQueue/WhiteBox.java

java/util/concurrent/PriorityBlockingQueue/WhiteBox.java

walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -l /home/walshbp/jep_testing/test-openjdk-jdk11/test/jdk | tail

vm/runtime/MonitorCacheMaybeExpand_DeadLock.java

vm/runtime/ReflectStackOverflow.java

vm/runtime/ShiftTest.java

vm/runtime/WideStrictInline.java

vm/verifier/defaultMethods/DefaultMethodRegressionTests.java

vm/verifier/defaultMethods/DefaultMethodRegressionTestsRun.java

vm/verifier/TestStaticIF.java

vm/verifier/VerifyProtectedConstructor.java

vm/verifier/VerifyStackForExceptionHandlers.java

Tests found: 9,478


--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


walshbp@bendev:~/jtreg_testing$ ../jtreg/bin/jtreg -version

jtreg, version 4.2 dev 398

Installed in /home/walshbp/jtreg/lib/jtreg.jar

Running on platform version 11 from /home/walshbp/openjdk11-hotspot-binary/jdk-11.

Built with Java(TM) 2 SDK, Version 1.8.0_121-b13 on June 11, 2018.

Copyright (c) 1999, 2016, Oracle and/or its affiliates. All rights reserved.

Use is subject to license terms.

JTHarness: version 5.0

JCov 3.0-2

TestNG (testng.jar): version unknown

TestNG (jcommander.jar): version 1.72

AsmTools: version 7.0


--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------



Please let me know what other information is required to investigate this further.



Thanks,
Ben Walsh

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU




Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU



Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU