Fwd: Re: Passing time factor to tests run under jtreg

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Fwd: Re: Passing time factor to tests run under jtreg

gary.adams@oracle.com
I'm investigating the current jdk tests with timeouts to see if we
can make them more reliable for slower machines. As a a first step,
I want to see if the jtreg command line arguments can be made
visible to the individual test.

Second I want to explore the information about the target machine that
can help adjust time limits from the time sensitive tests.

-------- Original Message --------
Subject: Re: Passing time factor to tests run under jtreg
Date: Tue, 15 Nov 2011 22:45:03 +0000
From: Alan Bateman [hidden email]
To: gary Adams [hidden email]
CC: [hidden email]


Gary - this might be something to bring up on the jtreg-use list. 
Ideally the tests wouldn't have any hardcoded timeouts but sometimes 
there isn't any other choice.

-Alan

On 15/11/2011 20:14, Gary Adams wrote:
>  I've been scanning a number of the slow machine test
> bugs that are reported and wanted to check to see if
> anyone has looked into time dependencies in the regression
> tests previously. From what I've been able to learn so far
> individual bugs can use the "timeout" parameter to indicate to
> the test harness an expected time to run.
>
> The test harness has command line arguments where it can
> filter out tests that take too long (timelimit) or can apply a 
> multiplier to
> to the timeout when conditions are known to slow down the process
> (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp
>
> I see that there are some wrappers that can be applied around running
> a particular test to allow processing before main(). Could this mechanism
> be exploited so the harness command line options could be made known
> to the time dependent tests as command line arguments or as system
> properties?
>
> My thought is the current timeout granularity is too large and only 
> applies
> to the full test execution. If a test knew that a timeoutFactor was to 
> be applied,
> it could internally adjust the time dependent delays appropriately. e.g.
> not every sleep(), await(), join() with timeouts  would need the 
> timeoutFactor
> applied.
>
> Before any test could be updated the information would need to be 
> available
> from the test context.
>
> Any feedback/pointers appreciated!
>
>
> See
>  timeoutFactorArg
>     jtreg/src/share/classes/com/sun/javatest/regtest/Main.java
>  runOtherJVM()
>     jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java
>  maxTimeoutValue
>     
> jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Re: Passing time factor to tests run under jtreg

Jonathan Gibbons
Gary,

If command line values were to be made available, it would probably be best to use system properties, but it feels like this is wrong, as a general solution. I could see making specific values available, such as the timeout factor, but even then I think it would be better to a more specific proposal of how this would be used by tests running on slower machines.

-- Jon

On 11/16/2011 05:08 AM, Gary Adams wrote:
I'm investigating the current jdk tests with timeouts to see if we
can make them more reliable for slower machines. As a a first step,
I want to see if the jtreg command line arguments can be made
visible to the individual test.

Second I want to explore the information about the target machine that
can help adjust time limits from the time sensitive tests.

-------- Original Message --------
Subject: Re: Passing time factor to tests run under jtreg
Date: Tue, 15 Nov 2011 22:45:03 +0000
From: Alan Bateman [hidden email]
To: gary Adams [hidden email]
CC: [hidden email]


Gary - this might be something to bring up on the jtreg-use list. 
Ideally the tests wouldn't have any hardcoded timeouts but sometimes 
there isn't any other choice.

-Alan

On 15/11/2011 20:14, Gary Adams wrote:
>  I've been scanning a number of the slow machine test
> bugs that are reported and wanted to check to see if
> anyone has looked into time dependencies in the regression
> tests previously. From what I've been able to learn so far
> individual bugs can use the "timeout" parameter to indicate to
> the test harness an expected time to run.
>
> The test harness has command line arguments where it can
> filter out tests that take too long (timelimit) or can apply a 
> multiplier to
> to the timeout when conditions are known to slow down the process
> (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp
>
> I see that there are some wrappers that can be applied around running
> a particular test to allow processing before main(). Could this mechanism
> be exploited so the harness command line options could be made known
> to the time dependent tests as command line arguments or as system
> properties?
>
> My thought is the current timeout granularity is too large and only 
> applies
> to the full test execution. If a test knew that a timeoutFactor was to 
> be applied,
> it could internally adjust the time dependent delays appropriately. e.g.
> not every sleep(), await(), join() with timeouts  would need the 
> timeoutFactor
> applied.
>
> Before any test could be updated the information would need to be 
> available
> from the test context.
>
> Any feedback/pointers appreciated!
>
>
> See
>  timeoutFactorArg
>     jtreg/src/share/classes/com/sun/javatest/regtest/Main.java
>  runOtherJVM()
>     jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java
>  maxTimeoutValue
>     
> jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Passing time factor to tests run under jtreg

Kelly O'Hair
In reply to this post by gary.adams@oracle.com
The jdk/test/Makefile that runs jtreg for JPRT (and the TL Nightly now?), adds a -timefactor:2
option I think, you could have the test/Makefile detect a slower system and adjust the timefactor on the
command line there.  Or is that what you are suggesting?

I know it's a bit crude to use Makefiles as an interface for testing, but there is a big advantage to
JPRT, Nightly testing, and developers being able to run the tests the exact same way.

-kto

On Nov 16, 2011, at 5:08 AM, Gary Adams wrote:

I'm investigating the current jdk tests with timeouts to see if we
can make them more reliable for slower machines. As a a first step,
I want to see if the jtreg command line arguments can be made
visible to the individual test.

Second I want to explore the information about the target machine that
can help adjust time limits from the time sensitive tests.

-------- Original Message --------
Subject: Re: Passing time factor to tests run under jtreg
Date: Tue, 15 Nov 2011 22:45:03 +0000
From: Alan Bateman [hidden email]
To: gary Adams [hidden email]
CC: [hidden email]


Gary - this might be something to bring up on the jtreg-use list. 
Ideally the tests wouldn't have any hardcoded timeouts but sometimes 
there isn't any other choice.

-Alan

On 15/11/2011 20:14, Gary Adams wrote:
>  I've been scanning a number of the slow machine test
> bugs that are reported and wanted to check to see if
> anyone has looked into time dependencies in the regression
> tests previously. From what I've been able to learn so far
> individual bugs can use the "timeout" parameter to indicate to
> the test harness an expected time to run.
>
> The test harness has command line arguments where it can
> filter out tests that take too long (timelimit) or can apply a 
> multiplier to
> to the timeout when conditions are known to slow down the process
> (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp
>
> I see that there are some wrappers that can be applied around running
> a particular test to allow processing before main(). Could this mechanism
> be exploited so the harness command line options could be made known
> to the time dependent tests as command line arguments or as system
> properties?
>
> My thought is the current timeout granularity is too large and only 
> applies
> to the full test execution. If a test knew that a timeoutFactor was to 
> be applied,
> it could internally adjust the time dependent delays appropriately. e.g.
> not every sleep(), await(), join() with timeouts  would need the 
> timeoutFactor
> applied.
>
> Before any test could be updated the information would need to be 
> available
> from the test context.
>
> Any feedback/pointers appreciated!
>
>
> See
>  timeoutFactorArg
>     jtreg/src/share/classes/com/sun/javatest/regtest/Main.java
>  runOtherJVM()
>     jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java
>  maxTimeoutValue
>     
> jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Passing time factor to tests run under jtreg

gary.adams@oracle.com
I'm not concerned how the timefactor is set.
It could be from makefiles or could be from
developer running jtreg from the command line.

What I'm talking about is the actual test code
being informed that the timefactor is being applied.
e.g. communicated from the test harness to the test case

A typical test using timeouts internally has some expectation
how long particular tasks will take. It might  use 5 seconds for
threads to be set up and then 1 second delay while each
thread is shut down.

An overall timefactor setting will still catch the runaway test,
but in some cases the test will terminate prematurely because
individual steps were not alloted sufficient time.



On 11/16/11 11:21 AM, Kelly O'Hair wrote:
The jdk/test/Makefile that runs jtreg for JPRT (and the TL Nightly now?), adds a -timefactor:2
option I think, you could have the test/Makefile detect a slower system and adjust the timefactor on the
command line there.  Or is that what you are suggesting?

I know it's a bit crude to use Makefiles as an interface for testing, but there is a big advantage to
JPRT, Nightly testing, and developers being able to run the tests the exact same way.

-kto

On Nov 16, 2011, at 5:08 AM, Gary Adams wrote:

I'm investigating the current jdk tests with timeouts to see if we
can make them more reliable for slower machines. As a a first step,
I want to see if the jtreg command line arguments can be made
visible to the individual test.

Second I want to explore the information about the target machine that
can help adjust time limits from the time sensitive tests.

-------- Original Message --------
Subject: Re: Passing time factor to tests run under jtreg
Date: Tue, 15 Nov 2011 22:45:03 +0000
From: Alan Bateman [hidden email]
To: gary Adams [hidden email]
CC: [hidden email]


Gary - this might be something to bring up on the jtreg-use list. 
Ideally the tests wouldn't have any hardcoded timeouts but sometimes 
there isn't any other choice.

-Alan

On 15/11/2011 20:14, Gary Adams wrote:
>  I've been scanning a number of the slow machine test
> bugs that are reported and wanted to check to see if
> anyone has looked into time dependencies in the regression
> tests previously. From what I've been able to learn so far
> individual bugs can use the "timeout" parameter to indicate to
> the test harness an expected time to run.
>
> The test harness has command line arguments where it can
> filter out tests that take too long (timelimit) or can apply a 
> multiplier to
> to the timeout when conditions are known to slow down the process
> (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp
>
> I see that there are some wrappers that can be applied around running
> a particular test to allow processing before main(). Could this mechanism
> be exploited so the harness command line options could be made known
> to the time dependent tests as command line arguments or as system
> properties?
>
> My thought is the current timeout granularity is too large and only 
> applies
> to the full test execution. If a test knew that a timeoutFactor was to 
> be applied,
> it could internally adjust the time dependent delays appropriately. e.g.
> not every sleep(), await(), join() with timeouts  would need the 
> timeoutFactor
> applied.
>
> Before any test could be updated the information would need to be 
> available
> from the test context.
>
> Any feedback/pointers appreciated!
>
>
> See
>  timeoutFactorArg
>     jtreg/src/share/classes/com/sun/javatest/regtest/Main.java
>  runOtherJVM()
>     jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java
>  maxTimeoutValue
>     
> jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java
>
>



Reply | Threaded
Open this post in threaded view
|

Re: Passing time factor to tests run under jtreg

gary.adams@oracle.com
In reply to this post by Kelly O'Hair
  On 11/16/11 11:21 AM, Kelly O'Hair wrote:
>
> I know it's a bit crude to use Makefiles as an interface for testing, but
> there is a big advantage to
> JPRT, Nightly testing, and developers being able to run the tests the exact
> same way.
>
This may not be the best place to discuss JPRT infrastructure, but I'm curious
how the automated testing knows which systems have which prerequisites
available. e.g. which system runs linux/x86 versus linux/arm

Obviously, binaries need to be matched to the systems that can run them.
Would it be possible for that "available systems" mapping to include the
available memory and cpu speeds. e.g. Information from
"cat /proc/{mem,cpu}info" or from "cpufreq-info" command
Reply | Threaded
Open this post in threaded view
|

Re: Passing time factor to tests run under jtreg

Jonathan Gibbons
In reply to this post by gary.adams@oracle.com
Gary,

With respect, my sense is that this is somewhat misdirected micromanagement.  It's not the time factor that the test should know, but the actual allotted time.   Given a certain period of time, a test could breakdown the allotted time into intervals for each step of its processing.

Would that work for what you have in mind?

-- Jon


On 11/16/2011 08:36 AM, Gary Adams wrote:
I'm not concerned how the timefactor is set.
It could be from makefiles or could be from
developer running jtreg from the command line.

What I'm talking about is the actual test code
being informed that the timefactor is being applied.
e.g. communicated from the test harness to the test case

A typical test using timeouts internally has some expectation
how long particular tasks will take. It might  use 5 seconds for
threads to be set up and then 1 second delay while each
thread is shut down.

An overall timefactor setting will still catch the runaway test,
but in some cases the test will terminate prematurely because
individual steps were not alloted sufficient time.



On 11/16/11 11:21 AM, Kelly O'Hair wrote:
The jdk/test/Makefile that runs jtreg for JPRT (and the TL Nightly now?), adds a -timefactor:2
option I think, you could have the test/Makefile detect a slower system and adjust the timefactor on the
command line there.  Or is that what you are suggesting?

I know it's a bit crude to use Makefiles as an interface for testing, but there is a big advantage to
JPRT, Nightly testing, and developers being able to run the tests the exact same way.

-kto

On Nov 16, 2011, at 5:08 AM, Gary Adams wrote:

I'm investigating the current jdk tests with timeouts to see if we
can make them more reliable for slower machines. As a a first step,
I want to see if the jtreg command line arguments can be made
visible to the individual test.

Second I want to explore the information about the target machine that
can help adjust time limits from the time sensitive tests.

-------- Original Message --------
Subject: Re: Passing time factor to tests run under jtreg
Date: Tue, 15 Nov 2011 22:45:03 +0000
From: Alan Bateman [hidden email]
To: gary Adams [hidden email]
CC: [hidden email]


Gary - this might be something to bring up on the jtreg-use list. 
Ideally the tests wouldn't have any hardcoded timeouts but sometimes 
there isn't any other choice.

-Alan

On 15/11/2011 20:14, Gary Adams wrote:
>  I've been scanning a number of the slow machine test
> bugs that are reported and wanted to check to see if
> anyone has looked into time dependencies in the regression
> tests previously. From what I've been able to learn so far
> individual bugs can use the "timeout" parameter to indicate to
> the test harness an expected time to run.
>
> The test harness has command line arguments where it can
> filter out tests that take too long (timelimit) or can apply a 
> multiplier to
> to the timeout when conditions are known to slow down the process
> (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp
>
> I see that there are some wrappers that can be applied around running
> a particular test to allow processing before main(). Could this mechanism
> be exploited so the harness command line options could be made known
> to the time dependent tests as command line arguments or as system
> properties?
>
> My thought is the current timeout granularity is too large and only 
> applies
> to the full test execution. If a test knew that a timeoutFactor was to 
> be applied,
> it could internally adjust the time dependent delays appropriately. e.g.
> not every sleep(), await(), join() with timeouts  would need the 
> timeoutFactor
> applied.
>
> Before any test could be updated the information would need to be 
> available
> from the test context.
>
> Any feedback/pointers appreciated!
>
>
> See
>  timeoutFactorArg
>     jtreg/src/share/classes/com/sun/javatest/regtest/Main.java
>  runOtherJVM()
>     jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java
>  maxTimeoutValue
>     
> jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java
>
>




Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Re: Passing time factor to tests run under jtreg

gary.adams@oracle.com
In reply to this post by gary.adams@oracle.com
Hi Jon,
   I don't think it is as simple as designating a length of time
a test is allowed to run. In the scenario, I mentioned below
a test needed 5 seconds to perform it's main task and 1 second to
clean up. Internally both operations were manged with separate
timeouts.

  The jtreg harness allows for timefactor=2 to allow for 2*(5+1)
or 12 seconds to complete, but the test as written really needs
(2*5) + (2*1) seconds to run correctly. e.g. it has to allow for 2 seconds
in the cleanup on the slower machine.

  If we can provide the (6 seconds * 2 timefactor) to the test it
could divide up the time between subtasks, but that may be a fairly
complicated computation over a large number of tests currently using
hard coded timeouts.

Gary
=======
With respect, my sense is that this is somewhat misdirected 
micromanagement.  It's not the time factor that the test should know, 
but the actual allotted time.   Given a certain period of time, a test 
could breakdown the allotted time into intervals for each step of its 
processing.

Would that work for what you have in mind?

-- Jon
On 11/16/11 8:08 AM, Gary Adams wrote:
I'm investigating the current jdk tests with timeouts to see if we
can make them more reliable for slower machines. As a a first step,
I want to see if the jtreg command line arguments can be made
visible to the individual test.

Second I want to explore the information about the target machine that
can help adjust time limits from the time sensitive tests.

-------- Original Message --------
Subject: Re: Passing time factor to tests run under jtreg
Date: Tue, 15 Nov 2011 22:45:03 +0000
From: Alan Bateman [hidden email]
To: gary Adams [hidden email]
CC: [hidden email]


Gary - this might be something to bring up on the jtreg-use list. 
Ideally the tests wouldn't have any hardcoded timeouts but sometimes 
there isn't any other choice.

-Alan

On 15/11/2011 20:14, Gary Adams wrote:
>  I've been scanning a number of the slow machine test
> bugs that are reported and wanted to check to see if
> anyone has looked into time dependencies in the regression
> tests previously. From what I've been able to learn so far
> individual bugs can use the "timeout" parameter to indicate to
> the test harness an expected time to run.
>
> The test harness has command line arguments where it can
> filter out tests that take too long (timelimit) or can apply a 
> multiplier to
> to the timeout when conditions are known to slow down the process
> (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp
>
> I see that there are some wrappers that can be applied around running
> a particular test to allow processing before main(). Could this mechanism
> be exploited so the harness command line options could be made known
> to the time dependent tests as command line arguments or as system
> properties?
>
> My thought is the current timeout granularity is too large and only 
> applies
> to the full test execution. If a test knew that a timeoutFactor was to 
> be applied,
> it could internally adjust the time dependent delays appropriately. e.g.
> not every sleep(), await(), join() with timeouts  would need the 
> timeoutFactor
> applied.
>
> Before any test could be updated the information would need to be 
> available
> from the test context.
>
> Any feedback/pointers appreciated!
>
>
> See
>  timeoutFactorArg
>     jtreg/src/share/classes/com/sun/javatest/regtest/Main.java
>  runOtherJVM()
>     jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java
>  maxTimeoutValue
>     
> jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Fwd: Re: Passing time factor to tests run under jtreg

Jonathan Gibbons
I don't understand your reasoning/math in your 2nd para below. Did you mean 5 + (2*1) ??    Otherwise, according to my high school
math, 2*(5+1) == (2*5) + (2*1).

Timeouts are not meant to be reached in normal use. They are intended to catch runaway tests, doing stuff like "while(true);". Therefore, I don't think we should be micromanaging time too much.    If the test is written to take 6 seconds on a typical machine, consisting of 5 seconds activity and 1 second cleanup, then on a slower machine it should be acceptable to use timefactor=2 and give it 12 seconds. Internally, the test should not need to know it is running on a slower machine; it should proceed to do the same activity as before, and if it now takes 5 seconds activity + (2*1) cleanup, well,   5 + 2*1 < 2*(5 + 1) and the test will continue to pass.

-- Jon



On 11/16/2011 02:08 PM, Gary Adams wrote:
Hi Jon,
   I don't think it is as simple as designating a length of time
a test is allowed to run. In the scenario, I mentioned below
a test needed 5 seconds to perform it's main task and 1 second to
clean up. Internally both operations were manged with separate
timeouts.

  The jtreg harness allows for timefactor=2 to allow for 2*(5+1)
or 12 seconds to complete, but the test as written really needs
(2*5) + (2*1) seconds to run correctly. e.g. it has to allow for 2 seconds
in the cleanup on the slower machine.

  If we can provide the (6 seconds * 2 timefactor) to the test it
could divide up the time between subtasks, but that may be a fairly
complicated computation over a large number of tests currently using
hard coded timeouts.

Gary
=======
With respect, my sense is that this is somewhat misdirected 
micromanagement.  It's not the time factor that the test should know, 
but the actual allotted time.   Given a certain period of time, a test 
could breakdown the allotted time into intervals for each step of its 
processing.

Would that work for what you have in mind?

-- Jon
On 11/16/11 8:08 AM, Gary Adams wrote:
I'm investigating the current jdk tests with timeouts to see if we
can make them more reliable for slower machines. As a a first step,
I want to see if the jtreg command line arguments can be made
visible to the individual test.

Second I want to explore the information about the target machine that
can help adjust time limits from the time sensitive tests.

-------- Original Message --------
Subject: Re: Passing time factor to tests run under jtreg
Date: Tue, 15 Nov 2011 22:45:03 +0000
From: Alan Bateman [hidden email]
To: gary Adams [hidden email]
CC: [hidden email]


Gary - this might be something to bring up on the jtreg-use list. 
Ideally the tests wouldn't have any hardcoded timeouts but sometimes 
there isn't any other choice.

-Alan

On 15/11/2011 20:14, Gary Adams wrote:
>  I've been scanning a number of the slow machine test
> bugs that are reported and wanted to check to see if
> anyone has looked into time dependencies in the regression
> tests previously. From what I've been able to learn so far
> individual bugs can use the "timeout" parameter to indicate to
> the test harness an expected time to run.
>
> The test harness has command line arguments where it can
> filter out tests that take too long (timelimit) or can apply a 
> multiplier to
> to the timeout when conditions are known to slow down the process
> (timeoutFactor). e.g. 8X for a slow machine or running with -Xcomp
>
> I see that there are some wrappers that can be applied around running
> a particular test to allow processing before main(). Could this mechanism
> be exploited so the harness command line options could be made known
> to the time dependent tests as command line arguments or as system
> properties?
>
> My thought is the current timeout granularity is too large and only 
> applies
> to the full test execution. If a test knew that a timeoutFactor was to 
> be applied,
> it could internally adjust the time dependent delays appropriately. e.g.
> not every sleep(), await(), join() with timeouts  would need the 
> timeoutFactor
> applied.
>
> Before any test could be updated the information would need to be 
> available
> from the test context.
>
> Any feedback/pointers appreciated!
>
>
> See
>  timeoutFactorArg
>     jtreg/src/share/classes/com/sun/javatest/regtest/Main.java
>  runOtherJVM()
>     jtreg/src/share/classes/com/sun/javatest/regtest/MainAction.java
>  maxTimeoutValue
>     
> jtreg/src/share/classes/com/sun/javatest/regtest/RegressionParameters.java
>
>