RFR(S) : 8183337 : hotspot/compiler/aot tests fail due to missed tools

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RFR(S) : 8183337 : hotspot/compiler/aot tests fail due to missed tools

Igor Ignatyev
http://cr.openjdk.java.net/~iignatyev//8183337/webrev.00/index.html
> 226 lines changed: 160 ins; 62 del; 4 mod;


Hi all,

could you please review this small patch which aims to solve the problem w/ missed linker in aot tests?

problem: jaotc tool needs linker to be installed, but at some hosts, esp. windows and mac, it is not always a case. as a results, aot tests which use jaotc fail on such hosts.
fix: on hosts which don't have 'ld' in PATH, we download the devkit and pass the path to the linker from it to jaotc tool.
caveats:
 - oracle implementation of artifacts resolver does not work concurrently, so the tests have to be run in 'exclusiveAccess' mode
 - some devkits (linux, solaris) are applicable only for a specific OS version

bug: https://bugs.openjdk.java.net/browse/JDK-8183337
webrev: http://cr.openjdk.java.net/~iignatyev//8183337/webrev.00/index.html
testing: hotspot/compiler/aot on all platforms including systems w/o installed linker

Thanks,
-- Igor

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RFR(S) : 8183337 : hotspot/compiler/aot tests fail due to missed tools

Vladimir Kozlov
This is ugly. Embedding into tests versions of compilers we use. I see
they are listed in common/conf/jib-profiles.js. Can we extract them from
there or other place?

Such change may be okay for internal use but what about community?

And can you add a simple tests that artifacts resolver exist at all?

Thanks,
Vladimir

On 8/4/17 3:13 PM, Igor Ignatyev wrote:

> http://cr.openjdk.java.net/~iignatyev//8183337/webrev.00/index.html
>> 226 lines changed: 160 ins; 62 del; 4 mod;
>
>
> Hi all,
>
> could you please review this small patch which aims to solve the problem w/ missed linker in aot tests?
>
> problem: jaotc tool needs linker to be installed, but at some hosts, esp. windows and mac, it is not always a case. as a results, aot tests which use jaotc fail on such hosts.
> fix: on hosts which don't have 'ld' in PATH, we download the devkit and pass the path to the linker from it to jaotc tool.
> caveats:
>   - oracle implementation of artifacts resolver does not work concurrently, so the tests have to be run in 'exclusiveAccess' mode
>   - some devkits (linux, solaris) are applicable only for a specific OS version
>
> bug: https://bugs.openjdk.java.net/browse/JDK-8183337
> webrev: http://cr.openjdk.java.net/~iignatyev//8183337/webrev.00/index.html
> testing: hotspot/compiler/aot on all platforms including systems w/o installed linker
>
> Thanks,
> -- Igor
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RFR(S) : 8183337 : hotspot/compiler/aot tests fail due to missed tools

Igor Ignatyev

> On Aug 4, 2017, at 3:37 PM, Vladimir Kozlov <[hidden email]> wrote:
>
> This is ugly. Embedding into tests versions of compilers we use. I see they are listed in common/conf/jib-profiles.js. Can we extract them from there or other place?
I agree that this is not a beauty. but extracting compiler version info from jib-profiles.js will introduce a dependency b/w the tests and source code outside the testbase, we tend to avoid such dependencies as they might break some infra configurations. unfortunately there is no place inside the testbase which has this info. so although I don't like having compiler versions in the tests, I dislike introducing this kind of dependency even more.

> Such change may be okay for internal use but what about community?
the changes don't make it any worse for community. 1st, the test will continue to work as before on hosts which have a linker installed. 2nd, there is a default implementation of artifacts resolver which basically uses java property to specify paths to artifacts. so if someone runs hotspot/compiler/aot tests on a host w/ a linker in PATH, the tests will use this linker, if the tests are run on a host w/o a linker, the tests will try to resolve an artifact and either "install" it using default artifacts resolve or fail w/ a clearer error message. so this patch increases aot test-coverage for community as well.

I also would like to mention that infra team has plans to make it easier for community to use ArtifactResolver and JIB. Hence I believe this won't be much of a problem.

> And can you add a simple tests that artifacts resolver exist at all?
not sure I understand that you mean. the default artifact resolver always exists. and we already have the tests which use an artifact resolver, e.g. hotspot/test/applications/scimark/Scimark.java .

Thanks
-- Igor

>
> Thanks,
> Vladimir
>
> On 8/4/17 3:13 PM, Igor Ignatyev wrote:
>> http://cr.openjdk.java.net/~iignatyev//8183337/webrev.00/index.html
>>> 226 lines changed: 160 ins; 62 del; 4 mod;
>> Hi all,
>> could you please review this small patch which aims to solve the problem w/ missed linker in aot tests?
>> problem: jaotc tool needs linker to be installed, but at some hosts, esp. windows and mac, it is not always a case. as a results, aot tests which use jaotc fail on such hosts.
>> fix: on hosts which don't have 'ld' in PATH, we download the devkit and pass the path to the linker from it to jaotc tool.
>> caveats:
>>  - oracle implementation of artifacts resolver does not work concurrently, so the tests have to be run in 'exclusiveAccess' mode
>>  - some devkits (linux, solaris) are applicable only for a specific OS version
>> bug: https://bugs.openjdk.java.net/browse/JDK-8183337
>> webrev: http://cr.openjdk.java.net/~iignatyev//8183337/webrev.00/index.html
>> testing: hotspot/compiler/aot on all platforms including systems w/o installed linker
>> Thanks,
>> -- Igor

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RFR(S) : 8183337 : hotspot/compiler/aot tests fail due to missed tools

Vladimir Kozlov
On 8/7/17 4:23 PM, Igor Ignatyev wrote:
>
>> On Aug 4, 2017, at 3:37 PM, Vladimir Kozlov <[hidden email]> wrote:
>>
>> This is ugly. Embedding into tests versions of compilers we use. I see they are listed in common/conf/jib-profiles.js. Can we extract them from there or other place?
> I agree that this is not a beauty. but extracting compiler version info from jib-profiles.js will introduce a dependency b/w the tests and source code outside the testbase, we tend to avoid such dependencies as they might break some infra configurations. unfortunately there is no place inside the testbase which has this info. so although I don't like having compiler versions in the tests, I dislike introducing this kind of dependency even more.

:(

Okay.

About next check. Should you also check for link.exe on Windows? Or it is not on PATH usually?

+            if (Files.exists(Paths.get(path).resolve("ld"))) {
+                // there is ld in PATH, jaotc is supposed to find it by its own
+                return null;
+            }

>
>> Such change may be okay for internal use but what about community?
> the changes don't make it any worse for community. 1st, the test will continue to work as before on hosts which have a linker installed. 2nd, there is a default implementation of artifacts resolver which basically uses java property to specify paths to artifacts. so if someone runs hotspot/compiler/aot tests on a host w/ a linker in PATH, the tests will use this linker, if the tests are run on a host w/o a linker, the tests will try to resolve an artifact and either "install" it using default artifacts resolve or fail w/ a clearer error message. so this patch increases aot test-coverage for community as well.
>
> I also would like to mention that infra team has plans to make it easier for community to use ArtifactResolver and JIB. Hence I believe this won't be much of a problem.

I did not know about that. Good.

>
>> And can you add a simple tests that artifacts resolver exist at all?
> not sure I understand that you mean. the default artifact resolver always exists. and we already have the tests which use an artifact resolver, e.g. hotspot/test/applications/scimark/Scimark.java .

I see.

Thanks,
Vladimir

>
> Thanks
> -- Igor
>
>>
>> Thanks,
>> Vladimir
>>
>> On 8/4/17 3:13 PM, Igor Ignatyev wrote:
>>> http://cr.openjdk.java.net/~iignatyev//8183337/webrev.00/index.html
>>>> 226 lines changed: 160 ins; 62 del; 4 mod;
>>> Hi all,
>>> could you please review this small patch which aims to solve the problem w/ missed linker in aot tests?
>>> problem: jaotc tool needs linker to be installed, but at some hosts, esp. windows and mac, it is not always a case. as a results, aot tests which use jaotc fail on such hosts.
>>> fix: on hosts which don't have 'ld' in PATH, we download the devkit and pass the path to the linker from it to jaotc tool.
>>> caveats:
>>>   - oracle implementation of artifacts resolver does not work concurrently, so the tests have to be run in 'exclusiveAccess' mode
>>>   - some devkits (linux, solaris) are applicable only for a specific OS version
>>> bug: https://bugs.openjdk.java.net/browse/JDK-8183337
>>> webrev: http://cr.openjdk.java.net/~iignatyev//8183337/webrev.00/index.html
>>> testing: hotspot/compiler/aot on all platforms including systems w/o installed linker
>>> Thanks,
>>> -- Igor
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RFR(S) : 8183337 : hotspot/compiler/aot tests fail due to missed tools

Igor Ignatyev

> On Aug 17, 2017, at 11:45 AM, Vladimir Kozlov <[hidden email]> wrote:
>
> On 8/7/17 4:23 PM, Igor Ignatyev wrote:
>>> On Aug 4, 2017, at 3:37 PM, Vladimir Kozlov <[hidden email]> wrote:
>>>
>>> This is ugly. Embedding into tests versions of compilers we use. I see they are listed in common/conf/jib-profiles.js. Can we extract them from there or other place?
>> I agree that this is not a beauty. but extracting compiler version info from jib-profiles.js will introduce a dependency b/w the tests and source code outside the testbase, we tend to avoid such dependencies as they might break some infra configurations. unfortunately there is no place inside the testbase which has this info. so although I don't like having compiler versions in the tests, I dislike introducing this kind of dependency even more.
>
> :(
>
> Okay.
>
> About next check. Should you also check for link.exe on Windows? Or it is not on PATH usually?
>
> +            if (Files.exists(Paths.get(path).resolve("ld"))) {
> +                // there is ld in PATH, jaotc is supposed to find it by its own
> +                return null;
> +            }
in some configurations, link.exe is on PATH, but jaotc tool does not use it from PATH, it's trying to find '\\VC\\bin\\amd64\\link.exe' in one of 'VS*COMNTOOLS' env. variables or uses link.exe from "well-known" location in "C:\\Program Files (x86)\\", I'll file an RFE for jatoc to search for link.exe in PATH 1st and then update this webrev.


>>> Such change may be okay for internal use but what about community?
>> the changes don't make it any worse for community. 1st, the test will continue to work as before on hosts which have a linker installed. 2nd, there is a default implementation of artifacts resolver which basically uses java property to specify paths to artifacts. so if someone runs hotspot/compiler/aot tests on a host w/ a linker in PATH, the tests will use this linker, if the tests are run on a host w/o a linker, the tests will try to resolve an artifact and either "install" it using default artifacts resolve or fail w/ a clearer error message. so this patch increases aot test-coverage for community as well.
>> I also would like to mention that infra team has plans to make it easier for community to use ArtifactResolver and JIB. Hence I believe this won't be much of a problem.
>
> I did not know about that. Good.
>
>>> And can you add a simple tests that artifacts resolver exist at all?
>> not sure I understand that you mean. the default artifact resolver always exists. and we already have the tests which use an artifact resolver, e.g. hotspot/test/applications/scimark/Scimark.java .
>
> I see.
>
> Thanks,
> Vladimir
>
>> Thanks
>> -- Igor
>>>
>>> Thanks,
>>> Vladimir
>>>
>>> On 8/4/17 3:13 PM, Igor Ignatyev wrote:
>>>> http://cr.openjdk.java.net/~iignatyev//8183337/webrev.00/index.html
>>>>> 226 lines changed: 160 ins; 62 del; 4 mod;
>>>> Hi all,
>>>> could you please review this small patch which aims to solve the problem w/ missed linker in aot tests?
>>>> problem: jaotc tool needs linker to be installed, but at some hosts, esp. windows and mac, it is not always a case. as a results, aot tests which use jaotc fail on such hosts.
>>>> fix: on hosts which don't have 'ld' in PATH, we download the devkit and pass the path to the linker from it to jaotc tool.
>>>> caveats:
>>>>  - oracle implementation of artifacts resolver does not work concurrently, so the tests have to be run in 'exclusiveAccess' mode
>>>>  - some devkits (linux, solaris) are applicable only for a specific OS version
>>>> bug: https://bugs.openjdk.java.net/browse/JDK-8183337
>>>> webrev: http://cr.openjdk.java.net/~iignatyev//8183337/webrev.00/index.html
>>>> testing: hotspot/compiler/aot on all platforms including systems w/o installed linker
>>>> Thanks,
>>>> -- Igor

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RFR(S) : 8183337 : hotspot/compiler/aot tests fail due to missed tools

Vladimir Kozlov
On 8/18/17 2:34 PM, Igor Ignatyev wrote:

>
>> On Aug 17, 2017, at 11:45 AM, Vladimir Kozlov <[hidden email]> wrote:
>>
>> On 8/7/17 4:23 PM, Igor Ignatyev wrote:
>>>> On Aug 4, 2017, at 3:37 PM, Vladimir Kozlov <[hidden email]> wrote:
>>>>
>>>> This is ugly. Embedding into tests versions of compilers we use. I see they are listed in common/conf/jib-profiles.js. Can we extract them from there or other place?
>>> I agree that this is not a beauty. but extracting compiler version info from jib-profiles.js will introduce a dependency b/w the tests and source code outside the testbase, we tend to avoid such dependencies as they might break some infra configurations. unfortunately there is no place inside the testbase which has this info. so although I don't like having compiler versions in the tests, I dislike introducing this kind of dependency even more.
>>
>> :(
>>
>> Okay.
>>
>> About next check. Should you also check for link.exe on Windows? Or it is not on PATH usually?
>>
>> +            if (Files.exists(Paths.get(path).resolve("ld"))) {
>> +                // there is ld in PATH, jaotc is supposed to find it by its own
>> +                return null;
>> +            }
> in some configurations, link.exe is on PATH, but jaotc tool does not use it from PATH, it's trying to find '\\VC\\bin\\amd64\\link.exe' in one of 'VS*COMNTOOLS' env. variables or uses link.exe from "well-known" location in "C:\\Program Files (x86)\\", I'll file an RFE for jatoc to search for link.exe in PATH 1st and then update this webrev.

Okay. I saw on some JPRT Win client link.exe in the PATH. Yes, file jaotc RFE.

But you don't need to do in this changes which should be pushed first, I think. That RFE can update test code later.

You answered all my questions. Reviewed.

Thanks,
Vladimir

>
>
>>>> Such change may be okay for internal use but what about community?
>>> the changes don't make it any worse for community. 1st, the test will continue to work as before on hosts which have a linker installed. 2nd, there is a default implementation of artifacts resolver which basically uses java property to specify paths to artifacts. so if someone runs hotspot/compiler/aot tests on a host w/ a linker in PATH, the tests will use this linker, if the tests are run on a host w/o a linker, the tests will try to resolve an artifact and either "install" it using default artifacts resolve or fail w/ a clearer error message. so this patch increases aot test-coverage for community as well.
>>> I also would like to mention that infra team has plans to make it easier for community to use ArtifactResolver and JIB. Hence I believe this won't be much of a problem.
>>
>> I did not know about that. Good.
>>
>>>> And can you add a simple tests that artifacts resolver exist at all?
>>> not sure I understand that you mean. the default artifact resolver always exists. and we already have the tests which use an artifact resolver, e.g. hotspot/test/applications/scimark/Scimark.java .
>>
>> I see.
>>
>> Thanks,
>> Vladimir
>>
>>> Thanks
>>> -- Igor
>>>>
>>>> Thanks,
>>>> Vladimir
>>>>
>>>> On 8/4/17 3:13 PM, Igor Ignatyev wrote:
>>>>> http://cr.openjdk.java.net/~iignatyev//8183337/webrev.00/index.html
>>>>>> 226 lines changed: 160 ins; 62 del; 4 mod;
>>>>> Hi all,
>>>>> could you please review this small patch which aims to solve the problem w/ missed linker in aot tests?
>>>>> problem: jaotc tool needs linker to be installed, but at some hosts, esp. windows and mac, it is not always a case. as a results, aot tests which use jaotc fail on such hosts.
>>>>> fix: on hosts which don't have 'ld' in PATH, we download the devkit and pass the path to the linker from it to jaotc tool.
>>>>> caveats:
>>>>>   - oracle implementation of artifacts resolver does not work concurrently, so the tests have to be run in 'exclusiveAccess' mode
>>>>>   - some devkits (linux, solaris) are applicable only for a specific OS version
>>>>> bug: https://bugs.openjdk.java.net/browse/JDK-8183337
>>>>> webrev: http://cr.openjdk.java.net/~iignatyev//8183337/webrev.00/index.html
>>>>> testing: hotspot/compiler/aot on all platforms including systems w/o installed linker
>>>>> Thanks,
>>>>> -- Igor
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RFR(S) : 8183337 : hotspot/compiler/aot tests fail due to missed tools

Igor Ignatyev

On Aug 18, 2017, at 2:50 PM, Vladimir Kozlov <[hidden email]> wrote:

On 8/18/17 2:34 PM, Igor Ignatyev wrote:
On Aug 17, 2017, at 11:45 AM, Vladimir Kozlov <[hidden email]> wrote:

On 8/7/17 4:23 PM, Igor Ignatyev wrote:
On Aug 4, 2017, at 3:37 PM, Vladimir Kozlov <[hidden email]> wrote:

This is ugly. Embedding into tests versions of compilers we use. I see they are listed in common/conf/jib-profiles.js. Can we extract them from there or other place?
I agree that this is not a beauty. but extracting compiler version info from jib-profiles.js will introduce a dependency b/w the tests and source code outside the testbase, we tend to avoid such dependencies as they might break some infra configurations. unfortunately there is no place inside the testbase which has this info. so although I don't like having compiler versions in the tests, I dislike introducing this kind of dependency even more.

:(

Okay.

About next check. Should you also check for link.exe on Windows? Or it is not on PATH usually?

+            if (Files.exists(Paths.get(path).resolve("ld"))) {
+                // there is ld in PATH, jaotc is supposed to find it by its own
+                return null;
+            }
in some configurations, link.exe is on PATH, but jaotc tool does not use it from PATH, it's trying to find '\\VC\\bin\\amd64\\link.exe' in one of 'VS*COMNTOOLS' env. variables or uses link.exe from "well-known" location in "C:\\Program Files (x86)\\", I'll file an RFE for jatoc to search for link.exe in PATH 1st and then update this webrev.

Okay. I saw on some JPRT Win client link.exe in the PATH. Yes, file jaotc RFE.

But you don't need to do in this changes which should be pushed first, I think. That RFE can update test code later.
I've filed https://bugs.openjdk.java.net/browse/JDK-8186463 . ok, I'll fix test code as part of 8186463.

You answered all my questions. Reviewed.
thank you Vladimir.

-- Igor


Such change may be okay for internal use but what about community?
the changes don't make it any worse for community. 1st, the test will continue to work as before on hosts which have a linker installed. 2nd, there is a default implementation of artifacts resolver which basically uses java property to specify paths to artifacts. so if someone runs hotspot/compiler/aot tests on a host w/ a linker in PATH, the tests will use this linker, if the tests are run on a host w/o a linker, the tests will try to resolve an artifact and either "install" it using default artifacts resolve or fail w/ a clearer error message. so this patch increases aot test-coverage for community as well.
I also would like to mention that infra team has plans to make it easier for community to use ArtifactResolver and JIB. Hence I believe this won't be much of a problem.

I did not know about that. Good.

And can you add a simple tests that artifacts resolver exist at all?
not sure I understand that you mean. the default artifact resolver always exists. and we already have the tests which use an artifact resolver, e.g. hotspot/test/applications/scimark/Scimark.java .

I see.

Thanks,
Vladimir

Thanks
-- Igor

Thanks,
Vladimir

On 8/4/17 3:13 PM, Igor Ignatyev wrote:
http://cr.openjdk.java.net/~iignatyev//8183337/webrev.00/index.html
226 lines changed: 160 ins; 62 del; 4 mod;
Hi all,
could you please review this small patch which aims to solve the problem w/ missed linker in aot tests?
problem: jaotc tool needs linker to be installed, but at some hosts, esp. windows and mac, it is not always a case. as a results, aot tests which use jaotc fail on such hosts.
fix: on hosts which don't have 'ld' in PATH, we download the devkit and pass the path to the linker from it to jaotc tool.
caveats:
 - oracle implementation of artifacts resolver does not work concurrently, so the tests have to be run in 'exclusiveAccess' mode
 - some devkits (linux, solaris) are applicable only for a specific OS version
bug: https://bugs.openjdk.java.net/browse/JDK-8183337
webrev: http://cr.openjdk.java.net/~iignatyev//8183337/webrev.00/index.html
testing: hotspot/compiler/aot on all platforms including systems w/o installed linker
Thanks,
-- Igor

Loading...