Quantcast

OpenJDK ARM build not using all the CPU cores

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

OpenJDK ARM build not using all the CPU cores

Mani Sarkar
Hi all,

Whilst build OpenJDK (latest version from
http://hg.openjdk.java.net/jdk8u/jdk8u/), we have noticed, that the build
system does not make use of all the CPU cores available on the machine even
though it detects them all.

(see snippets from the build log below)

Detects *4 cores*

checking for number of cores... 4


But scales down to *1 core* only

Build performance summary:
* Cores to use:   1
* Memory limit:   2021 MB
* ccache status:  installed and in use


We have use the --with-num-cores flag with configure but to no avail.

Has anyone else has noticed this, is this a known issue or expected
behaviour or any config that needs applying in order for this to make use
of all the cores.

Let me know if you need to see other aspects of the build log.

Thanks.

Cheers,
Mani
--
@theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
<http://neomatrix369.wordpress.com/>**  |  *LJC Associate & LJC Advocate
(@adoptopenjdk & @adoptajsr programs)
*Meet-a-Project - *MutabilityDetector
<https://github.com/MutabilityDetector>*  |  **Bitbucket
<https://bitbucket.org/neomatrix369>* * |  **Github
<https://github.com/neomatrix369>* * |  **LinkedIn
<http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
*Come to Devoxx UK 2017:* http://www.devoxx.co.uk/

*Don't chase success, rather aim for "Excellence", and success will come
chasing after you!*
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Fwd: OpenJDK ARM build not using all the CPU cores

Mani Sarkar
+[hidden email] <[hidden email]> to
the discussion

---------- Forwarded message ---------
From: Mani Sarkar <[hidden email]>
Date: Wed, 19 Apr 2017 at 08:42
Subject: OpenJDK ARM build not using all the CPU cores
To: [hidden email] <[hidden email]>,
[hidden email] <[hidden email]>


Hi all,

Whilst build OpenJDK (latest version from
http://hg.openjdk.java.net/jdk8u/jdk8u/), we have noticed, that the build
system does not make use of all the CPU cores available on the machine even
though it detects them all.

(see snippets from the build log below)

Detects *4 cores*

checking for number of cores... 4


But scales down to *1 core* only

Build performance summary:
* Cores to use:   1
* Memory limit:   2021 MB
* ccache status:  installed and in use


We have use the --with-num-cores flag with configure but to no avail.

Has anyone else has noticed this, is this a known issue or expected
behaviour or any config that needs applying in order for this to make use
of all the cores.

Let me know if you need to see other aspects of the build log.

Thanks.

Cheers,
Mani
--
@theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
<http://neomatrix369.wordpress.com/>**  |  *LJC Associate & LJC Advocate
(@adoptopenjdk & @adoptajsr programs)
*Meet-a-Project - *MutabilityDetector
<https://github.com/MutabilityDetector>*  |  **Bitbucket
<https://bitbucket.org/neomatrix369>* * |  **Github
<https://github.com/neomatrix369>* * |  **LinkedIn
<http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
*Come to Devoxx UK 2017:* http://www.devoxx.co.uk/

*Don't chase success, rather aim for "Excellence", and success will come
chasing after you!*
--
@theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
<http://neomatrix369.wordpress.com/>**  |  *LJC Associate & LJC Advocate
(@adoptopenjdk & @adoptajsr programs)
*Meet-a-Project - *MutabilityDetector
<https://github.com/MutabilityDetector>*  |  **Bitbucket
<https://bitbucket.org/neomatrix369>* * |  **Github
<https://github.com/neomatrix369>* * |  **LinkedIn
<http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
*Come to Devoxx UK 2017:* http://www.devoxx.co.uk/

*Don't chase success, rather aim for "Excellence", and success will come
chasing after you!*
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OpenJDK ARM build not using all the CPU cores

Andrew Haley
In reply to this post by Mani Sarkar
On 19/04/17 08:42, Mani Sarkar wrote:
> Whilst build OpenJDK (latest version from
> http://hg.openjdk.java.net/jdk8u/jdk8u/), we have noticed, that the build
> system does not make use of all the CPU cores available on the machine even
> though it detects them all.

It's because you have almost no memory installed.  If you want to
override the number of cores to use you can do so manually, but the
idea is to prevent you from running out of memory during the build.
It's deliberate, not a bug.

Andrew.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OpenJDK ARM build not using all the CPU cores

Mani Sarkar
Thanks Andrew for clarifying.

On Wed, 19 Apr 2017 at 08:56 Andrew Haley <[hidden email]> wrote:

> On 19/04/17 08:42, Mani Sarkar wrote:
> > Whilst build OpenJDK (latest version from
> > http://hg.openjdk.java.net/jdk8u/jdk8u/), we have noticed, that the
> build
> > system does not make use of all the CPU cores available on the machine
> even
> > though it detects them all.
>
> It's because you have almost no memory installed.  If you want to
> override the number of cores to use you can do so manually, but the
> idea is to prevent you from running out of memory during the build.
> It's deliberate, not a bug.
>
> Andrew.
>
> --
@theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
<http://neomatrix369.wordpress.com/>**  |  *LJC Associate & LJC Advocate
(@adoptopenjdk & @adoptajsr programs)
*Meet-a-Project - *MutabilityDetector
<https://github.com/MutabilityDetector>*  |  **Bitbucket
<https://bitbucket.org/neomatrix369>* * |  **Github
<https://github.com/neomatrix369>* * |  **LinkedIn
<http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
*Come to Devoxx UK 2017:* http://www.devoxx.co.uk/

*Don't chase success, rather aim for "Excellence", and success will come
chasing after you!*
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OpenJDK ARM build not using all the CPU cores

Ioi Lam
If you really want, you can override the settings with

make JOBS=4 images

Of course, YMMV due to the low amount of memory.

Ioi

> Mani Sarkar <[hidden email]> 於 2017年4月19日 下午4:07 寫道:
>
> Thanks Andrew for clarifying.
>
>> On Wed, 19 Apr 2017 at 08:56 Andrew Haley <[hidden email]> wrote:
>>
>>> On 19/04/17 08:42, Mani Sarkar wrote:
>>> Whilst build OpenJDK (latest version from
>>> http://hg.openjdk.java.net/jdk8u/jdk8u/), we have noticed, that the
>> build
>>> system does not make use of all the CPU cores available on the machine
>> even
>>> though it detects them all.
>>
>> It's because you have almost no memory installed.  If you want to
>> override the number of cores to use you can do so manually, but the
>> idea is to prevent you from running out of memory during the build.
>> It's deliberate, not a bug.
>>
>> Andrew.
>>
>> --
> @theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
> <http://neomatrix369.wordpress.com/>**  |  *LJC Associate & LJC Advocate
> (@adoptopenjdk & @adoptajsr programs)
> *Meet-a-Project - *MutabilityDetector
> <https://github.com/MutabilityDetector>*  |  **Bitbucket
> <https://bitbucket.org/neomatrix369>* * |  **Github
> <https://github.com/neomatrix369>* * |  **LinkedIn
> <http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
> *Come to Devoxx UK 2017:* http://www.devoxx.co.uk/
>
> *Don't chase success, rather aim for "Excellence", and success will come
> chasing after you!*

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OpenJDK ARM build not using all the CPU cores

Erik Joelsson
If you want to do the override at configure time, the correct configure
parameter is --with-jobs=4. The --with-num-cores isn't all that useful
except for easier testing of the logic for calculating the default
number of jobs (which is why I left it there).

The default limitation of jobs based on memory is rather conservative (1
job per 1024MB IIRC). We don't like to assume that building is the only
thing the system ever does. Also most of the build tasks aren't that
memory hungry, but some are. I recommend you try manually increasing
jobs until you find your optimal level of concurrency for the types of
builds you do.

/Erik


On 2017-04-19 11:07, Ioi Lam wrote:

> If you really want, you can override the settings with
>
> make JOBS=4 images
>
> Of course, YMMV due to the low amount of memory.
>
> Ioi
>
>> Mani Sarkar <[hidden email]> 於 2017年4月19日 下午4:07 寫道:
>>
>> Thanks Andrew for clarifying.
>>
>>> On Wed, 19 Apr 2017 at 08:56 Andrew Haley <[hidden email]> wrote:
>>>
>>>> On 19/04/17 08:42, Mani Sarkar wrote:
>>>> Whilst build OpenJDK (latest version from
>>>> http://hg.openjdk.java.net/jdk8u/jdk8u/), we have noticed, that the
>>> build
>>>> system does not make use of all the CPU cores available on the machine
>>> even
>>>> though it detects them all.
>>> It's because you have almost no memory installed.  If you want to
>>> override the number of cores to use you can do so manually, but the
>>> idea is to prevent you from running out of memory during the build.
>>> It's deliberate, not a bug.
>>>
>>> Andrew.
>>>
>>> --
>> @theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
>> <http://neomatrix369.wordpress.com/>**  |  *LJC Associate & LJC Advocate
>> (@adoptopenjdk & @adoptajsr programs)
>> *Meet-a-Project - *MutabilityDetector
>> <https://github.com/MutabilityDetector>*  |  **Bitbucket
>> <https://bitbucket.org/neomatrix369>* * |  **Github
>> <https://github.com/neomatrix369>* * |  **LinkedIn
>> <http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
>> *Come to Devoxx UK 2017:* http://www.devoxx.co.uk/
>>
>> *Don't chase success, rather aim for "Excellence", and success will come
>> chasing after you!*

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OpenJDK ARM build not using all the CPU cores

Mani Sarkar
Thanks Erik, I have updated our scripts with this flag.

Cheers,
Mani

On Wed, 19 Apr 2017 at 10:29 Erik Joelsson <[hidden email]> wrote:

> If you want to do the override at configure time, the correct configure
> parameter is --with-jobs=4. The --with-num-cores isn't all that useful
> except for easier testing of the logic for calculating the default
> number of jobs (which is why I left it there).
>
> The default limitation of jobs based on memory is rather conservative (1
> job per 1024MB IIRC). We don't like to assume that building is the only
> thing the system ever does. Also most of the build tasks aren't that
> memory hungry, but some are. I recommend you try manually increasing
> jobs until you find your optimal level of concurrency for the types of
> builds you do.
>
> /Erik
>
>
> On 2017-04-19 11:07, Ioi Lam wrote:
> > If you really want, you can override the settings with
> >
> > make JOBS=4 images
> >
> > Of course, YMMV due to the low amount of memory.
> >
> > Ioi
> >
> >> Mani Sarkar <[hidden email]> 於 2017年4月19日 下午4:07 寫道:
> >>
> >> Thanks Andrew for clarifying.
> >>
> >>> On Wed, 19 Apr 2017 at 08:56 Andrew Haley <[hidden email]> wrote:
> >>>
> >>>> On 19/04/17 08:42, Mani Sarkar wrote:
> >>>> Whilst build OpenJDK (latest version from
> >>>> http://hg.openjdk.java.net/jdk8u/jdk8u/), we have noticed, that the
> >>> build
> >>>> system does not make use of all the CPU cores available on the machine
> >>> even
> >>>> though it detects them all.
> >>> It's because you have almost no memory installed.  If you want to
> >>> override the number of cores to use you can do so manually, but the
> >>> idea is to prevent you from running out of memory during the build.
> >>> It's deliberate, not a bug.
> >>>
> >>> Andrew.
> >>>
> >>> --
> >> @theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
> >> <http://neomatrix369.wordpress.com/>**  |  *LJC Associate & LJC
> Advocate
> >> (@adoptopenjdk & @adoptajsr programs)
> >> *Meet-a-Project - *MutabilityDetector
> >> <https://github.com/MutabilityDetector>*  |  **Bitbucket
> >> <https://bitbucket.org/neomatrix369>* * |  **Github
> >> <https://github.com/neomatrix369>* * |  **LinkedIn
> >> <http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
> >> *Come to Devoxx UK 2017:* http://www.devoxx.co.uk/
> >>
> >> *Don't chase success, rather aim for "Excellence", and success will come
> >> chasing after you!*
>
> --
@theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
<http://neomatrix369.wordpress.com/>**  |  *LJC Associate & LJC Advocate
(@adoptopenjdk & @adoptajsr programs)
*Meet-a-Project - *MutabilityDetector
<https://github.com/MutabilityDetector>*  |  **Bitbucket
<https://bitbucket.org/neomatrix369>* * |  **Github
<https://github.com/neomatrix369>* * |  **LinkedIn
<http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
*Come to Devoxx UK 2017:* http://www.devoxx.co.uk/

*Don't chase success, rather aim for "Excellence", and success will come
chasing after you!*
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: OpenJDK ARM build not using all the CPU cores

Mani Sarkar
Thanks Erik we applied that flag and it has picked up the cores, we wll see
when the build finishes how much or how long it took to finish compared to
our previous 1-3 hour builds.

On Wed, 19 Apr 2017 at 11:09 Mani Sarkar <[hidden email]> wrote:

> Thanks Erik, I have updated our scripts with this flag.
>
> Cheers,
> Mani
>
> On Wed, 19 Apr 2017 at 10:29 Erik Joelsson <[hidden email]>
> wrote:
>
>> If you want to do the override at configure time, the correct configure
>> parameter is --with-jobs=4. The --with-num-cores isn't all that useful
>> except for easier testing of the logic for calculating the default
>> number of jobs (which is why I left it there).
>>
>> The default limitation of jobs based on memory is rather conservative (1
>> job per 1024MB IIRC). We don't like to assume that building is the only
>> thing the system ever does. Also most of the build tasks aren't that
>> memory hungry, but some are. I recommend you try manually increasing
>> jobs until you find your optimal level of concurrency for the types of
>> builds you do.
>>
>> /Erik
>>
>>
>> On 2017-04-19 11:07, Ioi Lam wrote:
>> > If you really want, you can override the settings with
>> >
>> > make JOBS=4 images
>> >
>> > Of course, YMMV due to the low amount of memory.
>> >
>> > Ioi
>> >
>> >> Mani Sarkar <[hidden email]> 於 2017年4月19日 下午4:07 寫道:
>> >>
>> >> Thanks Andrew for clarifying.
>> >>
>> >>> On Wed, 19 Apr 2017 at 08:56 Andrew Haley <[hidden email]> wrote:
>> >>>
>> >>>> On 19/04/17 08:42, Mani Sarkar wrote:
>> >>>> Whilst build OpenJDK (latest version from
>> >>>> http://hg.openjdk.java.net/jdk8u/jdk8u/), we have noticed, that the
>> >>> build
>> >>>> system does not make use of all the CPU cores available on the
>> machine
>> >>> even
>> >>>> though it detects them all.
>> >>> It's because you have almost no memory installed.  If you want to
>> >>> override the number of cores to use you can do so manually, but the
>> >>> idea is to prevent you from running out of memory during the build.
>> >>> It's deliberate, not a bug.
>> >>>
>> >>> Andrew.
>> >>>
>> >>> --
>> >> @theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
>> >> <http://neomatrix369.wordpress.com/>**  |  *LJC Associate & LJC
>> Advocate
>> >> (@adoptopenjdk & @adoptajsr programs)
>> >> *Meet-a-Project - *MutabilityDetector
>> >> <https://github.com/MutabilityDetector>*  |  **Bitbucket
>> >> <https://bitbucket.org/neomatrix369>* * |  **Github
>> >> <https://github.com/neomatrix369>* * |  **LinkedIn
>> >> <http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
>> >> *Come to Devoxx UK 2017:* http://www.devoxx.co.uk/
>> >>
>> >> *Don't chase success, rather aim for "Excellence", and success will
>> come
>> >> chasing after you!*
>>
>> --
> @theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
> <http://neomatrix369.wordpress.com/>**  |  *LJC Associate & LJC Advocate
> (@adoptopenjdk & @adoptajsr programs)
> *Meet-a-Project - *MutabilityDetector
> <https://github.com/MutabilityDetector>*  |  **Bitbucket
> <https://bitbucket.org/neomatrix369>* * |  **Github
> <https://github.com/neomatrix369>* * |  **LinkedIn
> <http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
> *Come to Devoxx UK 2017:* http://www.devoxx.co.uk/
>
> *Don't chase success, rather aim for "Excellence", and success will come
> chasing after you!*
>
--
@theNeomatrix369 <http://twitter.com/theNeomatrix369>*  |  **Blog
<http://neomatrix369.wordpress.com/>**  |  *LJC Associate & LJC Advocate
(@adoptopenjdk & @adoptajsr programs)
*Meet-a-Project - *MutabilityDetector
<https://github.com/MutabilityDetector>*  |  **Bitbucket
<https://bitbucket.org/neomatrix369>* * |  **Github
<https://github.com/neomatrix369>* * |  **LinkedIn
<http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>*
*Come to Devoxx UK 2017:* http://www.devoxx.co.uk/

*Don't chase success, rather aim for "Excellence", and success will come
chasing after you!*
Loading...