jtreg FAQ updates

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

jtreg FAQ updates

Jonathan Gibbons
For those of you that run jtreg, I've made some updates to the jtreg FAQ
that is available in builds of jtreg, and on the OpenJDK website. [1]

In addition to some general cleanup, I've reorganized it a bit, to create a
couple of new sections, "3. Using jtreg" and "5. Organizing tests".
Most of section 3 is new, although a few questions were moved from
elsewhere;
Other new questions include:
4.19 Can I (and should I) write shell tests?
5.5  How should I organize tests, libraries, and other test-related files?
5.6  Can I put more than one test in a file?

-- Jon

[1]  http://openjdk.java.net/jtreg/faq.html
(linked under "FAQ" from the main jtreg page
http://openjdk.java.net/jtreg/)
Reply | Threaded
Open this post in threaded view
|

Re: jtreg FAQ updates

Martijn Verburg
Great - thank you!

On Thu, 16 Aug 2018 at 00:46, Jonathan Gibbons <[hidden email]> wrote:
For those of you that run jtreg, I've made some updates to the jtreg FAQ
that is available in builds of jtreg, and on the OpenJDK website. [1]

In addition to some general cleanup, I've reorganized it a bit, to create a
couple of new sections, "3. Using jtreg" and "5. Organizing tests".
Most of section 3 is new, although a few questions were moved from
elsewhere;
Other new questions include:
4.19 Can I (and should I) write shell tests?
5.5  How should I organize tests, libraries, and other test-related files?
5.6  Can I put more than one test in a file?

-- Jon

[1]  http://openjdk.java.net/jtreg/faq.html
(linked under "FAQ" from the main jtreg page
http://openjdk.java.net/jtreg/)
--
Cheers, Martijn (Sent from Gmail Mobile)
Reply | Threaded
Open this post in threaded view
|

Re: jtreg FAQ updates

Chris Hegarty
In reply to this post by Jonathan Gibbons
Jon,

Thanks for this. A question on testng.

> On 16 Aug 2018, at 00:46, Jonathan Gibbons <[hidden email]> wrote:
> ...
> 5.6  Can I put more than one test in a file?

I think that you can put several jtreg-invoked testng tests in a single
file. At least it works for me. For example:

   /*
    * @test
    * @run testng/othervm Test
    */
   /*
    * @test
    * @run testng/othervm Test
    */
 
  , but it is not all that useful.

The reason I enquire about this is that we have several long running, and
verbose, testng tests, that have several hundred combinations. These test
take a long time to run and can generate a lot of data. The approach we’ve
taken to date is to effectively create an abstract class that contains all the
test logic and data providers. Then have several “driver” like subtypes that
each override a single supertypes moral "test" method and add the testng
@Test annotation. Example below.

What we’d really like to do is to add several test descriptions to the main
test specify something like:
   /*
    * @test
    * @run testng:groups="synchronous"/othervm TestAsync
    */
   /*
    * @test
    * @run  testng:groups=“asynchronous"/othervm TestAsync
    */

 , or some such syntax that would allow us to pass in a testng test
group. This would allow each group the same timeout, jtr output file
limit ( if any ), etc, without the need for the extra clutter of the “driver”
files. We have several of test drivers in java/net/httpclient, e.g.
ThrowingXXX.java

I’ve looked a little at jtreg’s support for testng and I think that this is
doable. Before filing an enhancement request, or hacking on the code,
is this something that you think would be accepted? I’m not sure if
others have come across a need for it before, but it is certainly useful
to the work being done by the HTTP Client, and the workaround that
exists today is a little crummy.

-Chris.

$ cat test/jdk/java/net/httpclient/AbstractTest.java

import java.util.concurrent.CompletableFuture;
import org.testng.annotations.*;

public abstract class AbstractTest {

    @DataProvider(name = "values")
    public Object[][] values() {
        return new Object[][] { {1} , {2}, {3} };
    }

    //@Test(dataProvider = "values", groups = "synchronous")
    void testSync(int value) {
        System.out.println(value);
    }

    //@Test(dataProvider = "values", groups = “asynchronous")
    void testAsync(int value) {
        CompletableFuture.runAsync(() -> System.out.println(value)).join();
    }

    @BeforeTest
    public void setup() throws Exception { /* ... */  }

    @AfterTest
    public void teardown() throws Exception { /* ... */  }

    // More common code and infrastructure …
}

$ cat test/jdk/java/net/httpclient/TestSync.java
/*
 * @test
 * @build AbstractTest
 * @run testng/othervm TestSync
 */

import org.testng.annotations.Test;

public class TestSync extends AbstractTest {

    @Override
    @Test(dataProvider = "values", groups = "synchronous")
    void testSync(int value) {
        super.testSync(value);
    }
}

$ cat test/jdk/java/net/httpclient/TestAsync.java
/*
 * @test
 * @build AbstractTest
 * @run testng/othervm TestAsync
 */

import org.testng.annotations.Test;

public class TestAsync extends AbstractTest {

    @Override
    @Test(dataProvider = "values", groups = “asynchronous")
    void testAsync(int value) {
        super.testAsync(value);
    }
}

 

Reply | Threaded
Open this post in threaded view
|

Re: jtreg FAQ updates

Jonathan Gibbons
Chris,

This seems generally reasonable, and worth filing as an enhancement request,
although it might also trigger a Q in the FAQ on "How long should a test
take?"

-- Jon


On 8/16/18 4:30 AM, Chris Hegarty wrote:

> Jon,
>
> Thanks for this. A question on testng.
>
>> On 16 Aug 2018, at 00:46, Jonathan Gibbons <[hidden email]> wrote:
>> ...
>> 5.6  Can I put more than one test in a file?
> I think that you can put several jtreg-invoked testng tests in a single
> file. At least it works for me. For example:
>
>     /*
>      * @test
>      * @run testng/othervm Test
>      */
>     /*
>      * @test
>      * @run testng/othervm Test
>      */
>  
>    , but it is not all that useful.
>
> The reason I enquire about this is that we have several long running, and
> verbose, testng tests, that have several hundred combinations. These test
> take a long time to run and can generate a lot of data. The approach we’ve
> taken to date is to effectively create an abstract class that contains all the
> test logic and data providers. Then have several “driver” like subtypes that
> each override a single supertypes moral "test" method and add the testng
> @Test annotation. Example below.
>
> What we’d really like to do is to add several test descriptions to the main
> test specify something like:
>     /*
>      * @test
>      * @run testng:groups="synchronous"/othervm TestAsync
>      */
>     /*
>      * @test
>      * @run  testng:groups=“asynchronous"/othervm TestAsync
>      */
>
>   , or some such syntax that would allow us to pass in a testng test
> group. This would allow each group the same timeout, jtr output file
> limit ( if any ), etc, without the need for the extra clutter of the “driver”
> files. We have several of test drivers in java/net/httpclient, e.g.
> ThrowingXXX.java
>
> I’ve looked a little at jtreg’s support for testng and I think that this is
> doable. Before filing an enhancement request, or hacking on the code,
> is this something that you think would be accepted? I’m not sure if
> others have come across a need for it before, but it is certainly useful
> to the work being done by the HTTP Client, and the workaround that
> exists today is a little crummy.
>
> -Chris.
>
> $ cat test/jdk/java/net/httpclient/AbstractTest.java
>
> import java.util.concurrent.CompletableFuture;
> import org.testng.annotations.*;
>
> public abstract class AbstractTest {
>
>      @DataProvider(name = "values")
>      public Object[][] values() {
>          return new Object[][] { {1} , {2}, {3} };
>      }
>
>      //@Test(dataProvider = "values", groups = "synchronous")
>      void testSync(int value) {
>          System.out.println(value);
>      }
>
>      //@Test(dataProvider = "values", groups = “asynchronous")
>      void testAsync(int value) {
>          CompletableFuture.runAsync(() -> System.out.println(value)).join();
>      }
>
>      @BeforeTest
>      public void setup() throws Exception { /* ... */  }
>
>      @AfterTest
>      public void teardown() throws Exception { /* ... */  }
>
>      // More common code and infrastructure …
> }
>
> $ cat test/jdk/java/net/httpclient/TestSync.java
> /*
>   * @test
>   * @build AbstractTest
>   * @run testng/othervm TestSync
>   */
>
> import org.testng.annotations.Test;
>
> public class TestSync extends AbstractTest {
>
>      @Override
>      @Test(dataProvider = "values", groups = "synchronous")
>      void testSync(int value) {
>          super.testSync(value);
>      }
> }
>
> $ cat test/jdk/java/net/httpclient/TestAsync.java
> /*
>   * @test
>   * @build AbstractTest
>   * @run testng/othervm TestAsync
>   */
>
> import org.testng.annotations.Test;
>
> public class TestAsync extends AbstractTest {
>
>      @Override
>      @Test(dataProvider = "values", groups = “asynchronous")
>      void testAsync(int value) {
>          super.testAsync(value);
>      }
> }
>
>  
>

Reply | Threaded
Open this post in threaded view
|

Re: jtreg FAQ updates

Martin Buchholz-3
In reply to this post by Chris Hegarty


On Thu, Aug 16, 2018 at 4:30 AM, Chris Hegarty <[hidden email]> wrote:
Jon,

Thanks for this. A question on testng.

> On 16 Aug 2018, at 00:46, Jonathan Gibbons <[hidden email]> wrote:
> ...
> 5.6  Can I put more than one test in a file?

I think that you can put several jtreg-invoked testng tests in a single
file. At least it works for me. For example:

   /*
    * @test
    * @run testng/othervm Test
    */
   /*
    * @test
    * @run testng/othervm Test
    */

  , but it is not all that useful.

I was confused by this as well, but for junit.  The existing JSR166TestCase successfully and usefully defines multiple @run junit tests.

I think the confusion has to do with """jtreg supports TestNG tests in two ways."""
Reply | Threaded
Open this post in threaded view
|

Re: jtreg FAQ updates

Jonathan Gibbons



On 8/16/18 8:18 AM, Martin Buchholz wrote:


On Thu, Aug 16, 2018 at 4:30 AM, Chris Hegarty <[hidden email]> wrote:
Jon,

Thanks for this. A question on testng.

> On 16 Aug 2018, at 00:46, Jonathan Gibbons <[hidden email]> wrote:
> ...
> 5.6  Can I put more than one test in a file?

I think that you can put several jtreg-invoked testng tests in a single
file.
The ability to put more than one test in a file is independent of the actions of the test.
It's always been undocumented but possible to put multiple tests in a file; it was
undocumented (until a while back) because the ability to name the individual tests
was imperfect.

At least it works for me. For example:

   /*
    * @test
    * @run testng/othervm Test
    */
   /*
    * @test
    * @run testng/othervm Test
    */

  , but it is not all that useful.
I agree that putting multiple tests with the same set of actions is not useful. You would normally want to
have different actions in the individual tests, such as with different arguments in some manner.

I was confused by this as well, but for junit.  The existing JSR166TestCase successfully and usefully defines multiple @run junit tests.
Yes, JSR166TestCase provides a number of different tests in that one source file.

I think the confusion has to do with """jtreg supports TestNG tests in two ways."""
I'm confused by your confusion. Can you say more, so that I can try and improve the text in this area?

-- Jon

Reply | Threaded
Open this post in threaded view
|

Re: jtreg FAQ updates

Martin Buchholz-3


On Thu, Aug 16, 2018 at 9:01 AM, Jonathan Gibbons <[hidden email]> wrote:
I was confused by this as well, but for junit.  The existing JSR166TestCase successfully and usefully defines multiple @run junit tests.
Yes, JSR166TestCase provides a number of different tests in that one source file.

I think the confusion has to do with """jtreg supports TestNG tests in two ways."""
I'm confused by your confusion. Can you say more, so that I can try and improve the text in this area?


faq says:

"""The feature is supported in normal Java tests, in shell tests, and in legacy applet tests. (It is not supported in JUnit or TestNG tests, which do not use explicit test descriptions.)"""

But ... Both JUnit and TestNG tests can use """explicit test descriptions""".  Unless they are Type 2 tests as described in 6.2
 
Further confusion - there's documented support for running "implicit" testng tests but I don't see any such support for junit.  E.g. there's no documented junit.dirs I can put into TEST.ROOT.
Reply | Threaded
Open this post in threaded view
|

Re: jtreg FAQ updates

Jonathan Gibbons



On 8/16/18 9:26 AM, Martin Buchholz wrote:


On Thu, Aug 16, 2018 at 9:01 AM, Jonathan Gibbons <[hidden email]> wrote:
I was confused by this as well, but for junit.  The existing JSR166TestCase successfully and usefully defines multiple @run junit tests.
Yes, JSR166TestCase provides a number of different tests in that one source file.

I think the confusion has to do with """jtreg supports TestNG tests in two ways."""
I'm confused by your confusion. Can you say more, so that I can try and improve the text in this area?


faq says:

"""The feature is supported in normal Java tests, in shell tests, and in legacy applet tests. (It is not supported in JUnit or TestNG tests, which do not use explicit test descriptions.)"""

But ... Both JUnit and TestNG tests can use """explicit test descriptions""".  Unless they are Type 2 tests as described in 6.2
 
Further confusion - there's documented support for running "implicit" testng tests but I don't see any such support for junit.  E.g. there's no documented junit.dirs I can put into TEST.ROOT.

OK, thanks for the clarifications;  I will address them in updates to the FAQ.

-- Jon