jtreg adds some postfixes into test names

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

jtreg adds some postfixes into test names

Vitaly Provodin
Hi all,

Running JDK regression tests via jtreg I detected that names of some tests can be changed during test execution. Namely, some postfix is added at the end of test name like follows:
===================================8<-------------------------------------------------------------
#Mon May 29 18:56:12 MSK 2017
#-----testdescription-----
$file=/opt/buildAgent/work/9d35cf148e807375/test/java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java
$root=/opt/buildAgent/work/9d35cf148e807375/test
author=anton.tarasov\: area\=awt.focus
id=id0
keywords=bug6187066
library=../../regtesthelpers
run=USER_SPECIFIED build Util\nUSER_SPECIFIED main AutoRequestFocusSetVisibleTest\n
source=AutoRequestFocusSetVisibleTest.java
title=Tests the Window.autoRequestFocus property for the Window.setVisible() method.

#-----environment-----

#-----testresult-----
compileJDK=/Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home
description=file\:/opt/buildAgent/work/9d35cf148e807375/test/java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java\#id0
elapsed=7463 0\:00\:07.463
end=Mon May 29 18\:56\:12 MSK 2017
environment=regtest
===================================8<——————————————————————————————

And jtreg output:
runner starting test: java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java#id0
runner finished test: java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java#id0
Failed. Execution failed: `main' threw exception: TestFailedException: Test failed: the window shouldn't gain focus but it did!


As you can see the test java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java became java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java\#id0 ("\#id0” was added).
Please note the issue was observed intermittently.

The matter was that this test was included into my exclude list, nevertheless it was executed because of this postfix (most probably new test name does not correspond to the name included into exclude list).
I looked through documentation on jtreg but could not find any explanations for this behaviour.
My suspicious was concurrent execution of the tests but I could not find any symptoms confirming that.

My original intentions was to exclude some tests from test runs but I could not do it because of this issue. I tried to use regular expressions in test names adding them into exclude list but it did not help.

Here is command line used to start test runs:

$ jtreg -exclude:test/jbProblemsList.txt -ignore:quiet -v -a -xml -compilejdk:'%env.JDK_18_x64%' -testjdk:'%jtreg.testjdk%' -timeoutFactor:2 test/java/awt || echo 'Execution complete’

Could you please help me in resolving the following questions? why does jtreg add some postfixes? how can I exclude such tests from test runs?


Thanks in advance, 
Vitaly

Reply | Threaded
Open this post in threaded view
|

Re: jtreg adds some postfixes into test names

Jonathan Gibbons

Test names are never changed during test execution.

Postfixes, as you call them, are added when the test is read when you have multiple test descriptions (comments beginning @test) in the same file.

They've always been added for the 2nd and subsequent test descriptions in a file; the recent change is to add them for the first test in a file when the file has multiple tests. This is for uniformity and to be able to disambiguate between specifying "all the tests in a file" vs. "the first test in a file."

Note that putting multiple test descriptions in a single file has never been formally supported, because of the potential ambiguity that was (finally) resolved by the recent change.

-- Jon

On 5/30/17 8:06 PM, Vitaly Provodin wrote:
Hi all,

Running JDK regression tests via jtreg I detected that names of some tests can be changed during test execution. Namely, some postfix is added at the end of test name like follows:
===================================8<-------------------------------------------------------------
#Mon May 29 18:56:12 MSK 2017
#-----testdescription-----
$file=/opt/buildAgent/work/9d35cf148e807375/test/java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java
$root=/opt/buildAgent/work/9d35cf148e807375/test
author=anton.tarasov\: area\=awt.focus
id=id0
keywords=bug6187066
library=../../regtesthelpers
run=USER_SPECIFIED build Util\nUSER_SPECIFIED main AutoRequestFocusSetVisibleTest\n
source=AutoRequestFocusSetVisibleTest.java
title=Tests the Window.autoRequestFocus property for the Window.setVisible() method.

#-----environment-----

#-----testresult-----
compileJDK=/Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home
description=file\:/opt/buildAgent/work/9d35cf148e807375/test/java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java\#id0
elapsed=7463 0\:00\:07.463
end=Mon May 29 18\:56\:12 MSK 2017
environment=regtest
===================================8<——————————————————————————————

And jtreg output:
runner starting test: java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java#id0
runner finished test: java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java#id0
Failed. Execution failed: `main' threw exception: TestFailedException: Test failed: the window shouldn't gain focus but it did!


As you can see the test java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java became java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java\#id0 ("\#id0” was added).
Please note the issue was observed intermittently.

The matter was that this test was included into my exclude list, nevertheless it was executed because of this postfix (most probably new test name does not correspond to the name included into exclude list).
I looked through documentation on jtreg but could not find any explanations for this behaviour.
My suspicious was concurrent execution of the tests but I could not find any symptoms confirming that.

My original intentions was to exclude some tests from test runs but I could not do it because of this issue. I tried to use regular expressions in test names adding them into exclude list but it did not help.

Here is command line used to start test runs:

$ jtreg -exclude:test/jbProblemsList.txt -ignore:quiet -v -a -xml -compilejdk:'%env.JDK_18_x64%' -testjdk:'%jtreg.testjdk%' -timeoutFactor:2 test/java/awt || echo 'Execution complete’

Could you please help me in resolving the following questions? why does jtreg add some postfixes? how can I exclude such tests from test runs?


Thanks in advance, 
Vitaly


Reply | Threaded
Open this post in threaded view
|

Re: jtreg adds some postfixes into test names

Vitaly Provodin
Hi Jon,

Thanks for your reply.

Unfortunately, I am not sure it is that case your wrote.
My particular case is the file containing one test description (one tag @test). The test was taken from http://hg.openjdk.java.net/jdk9/dev/jdk/file/148a2cbe49a4/test/java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java and it was not changed.

$ cat test/java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java | grep "@test"
  @test

I also tried to write in description something like

$ cat test/java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java | grep "@test"
  @test      test description

in both cases there is only one tag @test and both cases produce the same result:

./jtreg/bin/jtreg -exclude:test/jbProblemsList.txt -ignore:quiet -v -a -xml -testjdk:jbsdk/jdk/Contents/Home test/java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java
XML output  to /Users/builduser/9d35cf148e807375/JTwork
runner starting test: java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java#id0
runner finished test: java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java#id0
Failed. Execution failed: `main' threw exception: TestFailedException: Test failed: the window shouldn't gain focus but it did!
Test results: failed: 1
Report written to /Users/builduser/9d35cf148e807375/JTreport/html/report.html
Results written to /Users/builduser/9d35cf148e807375/JTwork
Error: Some tests failed or other problems occurred.


Thanks,
Vitaly


On 31 May 2017, at 22:11, Jonathan Gibbons <[hidden email]> wrote:

Test names are never changed during test execution.

Postfixes, as you call them, are added when the test is read when you have multiple test descriptions (comments beginning @test) in the same file.

They've always been added for the 2nd and subsequent test descriptions in a file; the recent change is to add them for the first test in a file when the file has multiple tests. This is for uniformity and to be able to disambiguate between specifying "all the tests in a file" vs. "the first test in a file."

Note that putting multiple test descriptions in a single file has never been formally supported, because of the potential ambiguity that was (finally) resolved by the recent change.

-- Jon

On 5/30/17 8:06 PM, Vitaly Provodin wrote:
Hi all,

Running JDK regression tests via jtreg I detected that names of some tests can be changed during test execution. Namely, some postfix is added at the end of test name like follows:
===================================8<-------------------------------------------------------------
#Mon May 29 18:56:12 MSK 2017
#-----testdescription-----
$file=/opt/buildAgent/work/9d35cf148e807375/test/java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java
$root=/opt/buildAgent/work/9d35cf148e807375/test
author=anton.tarasov\: area\=awt.focus
id=id0
keywords=bug6187066
library=../../regtesthelpers
run=USER_SPECIFIED build Util\nUSER_SPECIFIED main AutoRequestFocusSetVisibleTest\n
source=AutoRequestFocusSetVisibleTest.java
title=Tests the Window.autoRequestFocus property for the Window.setVisible() method.

#-----environment-----

#-----testresult-----
compileJDK=/Library/Java/JavaVirtualMachines/jdk1.8.0_111.jdk/Contents/Home
description=file\:/opt/buildAgent/work/9d35cf148e807375/test/java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java\#id0
elapsed=7463 0\:00\:07.463
end=Mon May 29 18\:56\:12 MSK 2017
environment=regtest
===================================8<——————————————————————————————

And jtreg output:
runner starting test: java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java#id0
runner finished test: java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java#id0
Failed. Execution failed: `main' threw exception: TestFailedException: Test failed: the window shouldn't gain focus but it did!


As you can see the test java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java became java/awt/Focus/AutoRequestFocusTest/AutoRequestFocusSetVisibleTest.java\#id0 ("\#id0” was added).
Please note the issue was observed intermittently.

The matter was that this test was included into my exclude list, nevertheless it was executed because of this postfix (most probably new test name does not correspond to the name included into exclude list).
I looked through documentation on jtreg but could not find any explanations for this behaviour.
My suspicious was concurrent execution of the tests but I could not find any symptoms confirming that.

My original intentions was to exclude some tests from test runs but I could not do it because of this issue. I tried to use regular expressions in test names adding them into exclude list but it did not help.

Here is command line used to start test runs:

$ jtreg -exclude:test/jbProblemsList.txt -ignore:quiet -v -a -xml -compilejdk:'%env.JDK_18_x64%' -testjdk:'%jtreg.testjdk%' -timeoutFactor:2 test/java/awt || echo 'Execution complete’

Could you please help me in resolving the following questions? why does jtreg add some postfixes? how can I exclude such tests from test runs?


Thanks in advance, 
Vitaly