CMake replacing Autotools?
Locked
CMake replacing Autotools?
 
|
Dear Java devs,
what do you think about the idea to replace Autotools by CMake? Is there any chance anybody would be interested? Would it be worth to write a feature request? Or even a JEP? If somebody is interested I could elaborate more about why I think CMake has advantages over Autotools. Bye Christoph |
Re: CMake replacing Autotools?
|
|
On 2021-03-18 00:21, Christoph Grüninger wrote:
> Dear Java devs, > > what do you think about the idea to replace Autotools by CMake? Is there > any chance anybody would be interested? > Would it be worth to write a feature request? Or even a JEP? > > If somebody is interested I could elaborate more about why I think CMake > has advantages over Autotools. Did you have any specific complaints about the build system, or is this just some general CMake cheerleading? In any way, I don't see how replacing the current build framework with CMake would bring any benefit that even remotely come close to the cost of doing that. But if you want to elaborate on your thinking, by all means, go ahead! How much have you actually looked at the build source code? We are not really what I'd describe as an "autotools" project. It is true that we still use autoconf to create our final configure script, but we have steadily moved away from most of the provided AC_* macros, since they have proven inadequate for our needs. And we have never even considered using automake. In fact, a more proper description would be to say that our configure script consists of a bash script generated using m4, with a few remaining support macros incorporated from the autoconf libraries. At various times, I have dreamed of replacing the configure script with something that is more modern and easy to maintain than this bash/m4 mix. We have a very well-defined API for the configure script: the user calls "bash configure" in the root directory of the project, with a set of --options, and as a result we create a spec.gmk file that defines the configuration. This could easily be replicated in another system. But if I were to rewrite this from scratch, I'd rather write the whole configure logic in Java (apart from some thin shell script logic needed to find the boot jdk), rather than trying to shoehorn in our build model in CMake. /Magnus |
Re: CMake replacing Autotools?
|
|
Hi Magnus,
for sure, I am going to do CMake cheerleading. I think OpenJDK would profit from a change to CMake: 1. Less tools required for building: CMake + gmake/Ninja/msbuild/Xcode instead of Autotools and all the related Linux/Unix tools, that are somewhat alien to Windows and OS X. Visual Studio even provides its own CMake. 2. More choices to actually build the project: Use integrated build tools of IDEs (Visual Studio, Xcode) or use Ninja, which is faster than gmake 3. Today, from what I hear, more people are familiar with CMake compared to Autotools. This argument can be applied to how familiar user are building OpenJDK and to how familiar potential contributors are. And I know, you developers know your code base :-) 4. CMake is better supported by IDEs like Visual Studio, Qt Creator, KDevelop. 5. A lot of code bases were ported to CMake like KDE, Qt, or LLVM. Their arguments apply here, too. Also their trade-offs between investment and benefit. > At various times, I have dreamed of replacing the configure script with > something that is more modern and easy to maintain than this bash/m4 > mix. We have a very well-defined API for the configure script: the user > calls "bash configure" in the root directory of the project, with a set > of --options, and as a result we create a spec.gmk file that defines the > configuration. This could easily be replicated in another system. But if > I were to rewrite this from scratch, I'd rather write the whole > configure logic in Java (apart from some thin shell script logic needed > to find the boot jdk), rather than trying to shoehorn in our build model > in CMake. I understand your temptation, but writing and maintaining all the configure/find logic and quirks will be a burden. I'd try to reduce the build system code to a minimum and rely on a third-party solution to do as much as possible for me. Bye Christoph -- Als wär es nix, leb' ich von [IT] und mach' nur, was ich lieb' Lebe wie im Paradies, womit hab' ich das verdient? Die Wahrheit ist: Hab' ich nicht, ich bin nur reicher beschenkt Als jemand in einem armen Land mit dem gleichen Talent [frei nach Tua von Die Orsons - Oioioiropa] |
|
|
Hi Christoph,
On Fri, Mar 19, 2021 at 12:01 AM Christoph Grüninger <[hidden email]> wrote: > Hi Magnus, > > for sure, I am going to do CMake cheerleading. > > I think OpenJDK would profit from a change to CMake: > > 1. Less tools required for building: CMake + gmake/Ninja/msbuild/Xcode > instead of Autotools and all the related Linux/Unix tools, that are > somewhat alien to Windows and OS X. Visual Studio even provides its own > CMake. > Note that CMake is missing on a number of proprietary Unices (AIX, HPUX) whereas Autotools are universally available and just work; on Windows via Cygwin, on Mac via Fink/homebrew to Mac. So I think the argument is actually the reverse. > > 2. More choices to actually build the project: Use integrated build > tools of IDEs (Visual Studio, Xcode) or use Ninja, which is faster than > gmake > > Is gmake really where we lose time? Did you analyze this or is this just an assumption? I would have thought it's things like single threaded jmod, jlink, and subprocess spawning. Cheers, Thomas |
|
|
On 3/19/21 9:22 AM, Thomas Stüfe wrote:
>> 2. More choices to actually build the project: Use integrated build >> tools of IDEs (Visual Studio, Xcode) or use Ninja, which is faster than >> gmake >> > Is gmake really where we lose time? Did you analyze this or is this just an > assumption? I would have thought it's things like single threaded jmod, > jlink, and subprocess spawning. I'm sure it is. The other slow thing is linking HotSpot. -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. <https://www.redhat.com> https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 |
Re: CMake replacing Autotools?
|
|
Hello
On macmini m1 jdk in default config can be built in about 4 minutes. (make images) I might guess autotools aren't the slowest thing. Regards, Vladimir Andrew Haley <[hidden email]> 19 марта 2021 г. 13:04:38 написал: On 3/19/21 9:22 AM, Thomas Stüfe wrote: 2. More choices to actually build the project: Use integrated build tools of IDEs (Visual Studio, Xcode) or use Ninja, which is faster than gmake Is gmake really where we lose time? Did you analyze this or is this just an assumption? I would have thought it's things like single threaded jmod, jlink, and subprocess spawning. I'm sure it is. The other slow thing is linking HotSpot. -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. <https://www.redhat.com> https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 |
|
|
In reply to this post by Andrew Haley
On Fri, Mar 19, 2021 at 11:04 AM Andrew Haley <[hidden email]> wrote:
> On 3/19/21 9:22 AM, Thomas Stüfe wrote: > >> 2. More choices to actually build the project: Use integrated build > >> tools of IDEs (Visual Studio, Xcode) or use Ninja, which is faster than > >> gmake > >> > > Is gmake really where we lose time? Did you analyze this or is this just > an > > assumption? I would have thought it's things like single threaded jmod, > > jlink, and subprocess spawning. > > I'm sure it is. The other slow thing is linking HotSpot. > What is so slow with gmake? Rule processing? It also depends on the platform, I guess. Eg on Cygwin, the fork emulation is extremely slow. > -- > Andrew Haley (he/him) > Java Platform Lead Engineer > Red Hat UK Ltd. <https://www.redhat.com> > https://keybase.io/andrewhaley > EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 > > |
|
|
On 3/19/21 10:14 AM, Thomas Stüfe wrote:
> On Fri, Mar 19, 2021 at 11:04 AM Andrew Haley <[hidden email]> wrote: > >> On 3/19/21 9:22 AM, Thomas Stüfe wrote: >>>> 2. More choices to actually build the project: Use integrated build >>>> tools of IDEs (Visual Studio, Xcode) or use Ninja, which is faster than >>>> gmake >>>> >>> Is gmake really where we lose time? Did you analyze this or is this just >> an >>> assumption? I would have thought it's things like single threaded jmod, >>> jlink, and subprocess spawning. >> >> I'm sure it is. The other slow thing is linking HotSpot. >> > > What is so slow with gmake? Rule processing? Nothing, I suspect. > It also depends on the platform, I guess. Eg on Cygwin, the fork emulation > is extremely slow. Weren't many Cygwin tools changed to use spawn() instead? It's been a while. -- Andrew Haley (he/him) Java Platform Lead Engineer Red Hat UK Ltd. <https://www.redhat.com> https://keybase.io/andrewhaley EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671 |
|
|
In reply to this post by tstuefe
On 2021-03-19 03:14, Thomas Stüfe wrote:
> On Fri, Mar 19, 2021 at 11:04 AM Andrew Haley <[hidden email]> wrote: > >> On 3/19/21 9:22 AM, Thomas Stüfe wrote: >>>> 2. More choices to actually build the project: Use integrated build >>>> tools of IDEs (Visual Studio, Xcode) or use Ninja, which is faster than >>>> gmake >>>> >>> Is gmake really where we lose time? Did you analyze this or is this just >> an >>> assumption? I would have thought it's things like single threaded jmod, >>> jlink, and subprocess spawning. >> I'm sure it is. The other slow thing is linking HotSpot. >> > What is so slow with gmake? Rule processing? > > It also depends on the platform, I guess. Eg on Cygwin, the fork emulation > is extremely slow. > over the years. There are many ways to measure performance. First we need to establish what kind of build we are even measuring. For a full images build ("make images"), on a reasonably sized machine (8-16 HV threads), we scale pretty well and use most CPUs most of the time. There isn't much additional concurrency to gain here. Obvious single threaded steps are hotspot linking and jlink. In such a build, Hotspot is mostly linked in parallel with all the Java compilation, so not an issue. Jlinking the JDK image does stick out as something we can't do much in parallel with, unless we also build the test or docs image. For a hotspot only build ("make hotspot"), then the hotspot linking will stick out as a single threaded step. Note that cmake/ninja will not help with any of this. Potential speed up from ninja is also very limited as the rules processing of our make scripts does not amount to any significant part of a full build. On Windows specifically, we do have an issue with fork being inefficient. We also have a less efficient file system making file operations more expensive in general. I have two big (though old) identical workstations, one with Windows and one Linux. Very rough numbers are 5 minutes for "make images" on the Linux machine and 10 on the Windows machine. These differences vary wildly on different hardware though. Using Windows native tools here would certainly help to some extent. OTOH, we have WSL, which is already considerably more performant than Cygwin (very rough numbers, maybe 8-9 minutes for the same build). The setup is a bit trickier than Cygwin, but once set up, it works really well in my experience. The area where ninja would provide the most benefit is for incremental builds, especially when very little work is actually needed, as it processes rules much faster than make. We have worked hard at making incremental builds as efficient and fast as possible, but our build is also pretty big so the time it takes is still noticeable, especially on Windows. All this said, when picking a build system, compatibility issues are the number one concern. If the support matrix of CMake does not completely cover the support matrix of OpenJDK, it's a no go to me. I would also be hesitant to be at the mercy of the platform support of a 3rd party when a new port of OpenJDK needs to be made. Regarding IDE integration, our build system is able to produce compile-commands.json which several IDEs know how to consume. Another big objection I have to this is the amount work required to rewrite the build system (again). I would expect a rewrite like this to be several man months, just for the OpenJDK (not counting forced downstream work for custom extensions to the build as well as all system currently interacting with the build, which I'm sure exist in more places than just Oracle). /Erik |
|
|
On Fri, Mar 19, 2021 at 2:39 PM <[hidden email]> wrote:
> On 2021-03-19 03:14, Thomas Stüfe wrote: > > On Fri, Mar 19, 2021 at 11:04 AM Andrew Haley <[hidden email]> wrote: > > > >> On 3/19/21 9:22 AM, Thomas Stüfe wrote: > >>>> 2. More choices to actually build the project: Use integrated build > >>>> tools of IDEs (Visual Studio, Xcode) or use Ninja, which is faster > than > >>>> gmake > >>>> > >>> Is gmake really where we lose time? Did you analyze this or is this > just > >> an > >>> assumption? I would have thought it's things like single threaded jmod, > >>> jlink, and subprocess spawning. > >> I'm sure it is. The other slow thing is linking HotSpot. > >> > > What is so slow with gmake? Rule processing? > > > > It also depends on the platform, I guess. Eg on Cygwin, the fork > emulation > > is extremely slow. > > > I have done pretty extensive work optimizing our build's performance > over the years. There are many ways to measure performance. First we > need to establish what kind of build we are even measuring. > > For a full images build ("make images"), on a reasonably sized machine > (8-16 HV threads), we scale pretty well and use most CPUs most of the > time. There isn't much additional concurrency to gain here. Obvious > single threaded steps are hotspot linking and jlink. In such a build, > Hotspot is mostly linked in parallel with all the Java compilation, so > not an issue. Jlinking the JDK image does stick out as something we > can't do much in parallel with, unless we also build the test or docs > image. For a hotspot only build ("make hotspot"), then the hotspot > linking will stick out as a single threaded step. Note that cmake/ninja > will not help with any of this. Potential speed up from ninja is also > very limited as the rules processing of our make scripts does not amount > to any significant part of a full build. > > On Windows specifically, we do have an issue with fork being > inefficient. We also have a less efficient file system making file > operations more expensive in general. I have two big (though old) > identical workstations, one with Windows and one Linux. Very rough > numbers are 5 minutes for "make images" on the Linux machine and 10 on > the Windows machine. These differences vary wildly on different hardware > though. Using Windows native tools here would certainly help to some > extent. OTOH, we have WSL, which is already considerably more performant > than Cygwin (very rough numbers, maybe 8-9 minutes for the same build). > The setup is a bit trickier than Cygwin, but once set up, it works > really well in my experience. > > The area where ninja would provide the most benefit is for incremental > builds, especially when very little work is actually needed, as it > processes rules much faster than make. We have worked hard at making > incremental builds as efficient and fast as possible, but our build is > also pretty big so the time it takes is still noticeable, especially on > Windows. > > All this said, when picking a build system, compatibility issues are the > number one concern. If the support matrix of CMake does not completely > cover the support matrix of OpenJDK, it's a no go to me. I would also be > hesitant to be at the mercy of the platform support of a 3rd party when > a new port of OpenJDK needs to be made. > > Regarding IDE integration, our build system is able to produce > compile-commands.json which several IDEs know how to consume. > > Another big objection I have to this is the amount work required to > rewrite the build system (again). I would expect a rewrite like this to > be several man months, just for the OpenJDK (not counting forced > downstream work for custom extensions to the build as well as all system > currently interacting with the build, which I'm sure exist in more > places than just Oracle). > > /Erik > Thanks for that extensive explanation! I have nothing to contribute here. But would like to thank you guys for the work on the build system. It's exceptionally stable and works very well. Cheers, Thomas |
Re: CMake replacing Autotools?
|
|
Dear Java developers,
thank you for the great discussion! > Thanks for that extensive explanation! I have nothing to contribute here. > But would like to thank you guys for the work on the build system. It's > exceptionally stable and works very well. You seem to be fine with the build system Java has. This is great! Once Java drops support for AIX and HPUX, I might feel to come back and continue our discussion. Have a nice day, Christoph -- GDB does hate your application, expresses its contempt through the design of its command-line interface. -- Tom Tromey, FOSDEM 2014 |
Re: CMake replacing Autotools?
|
|
In reply to this post by Christoph Grüninger
Hi Christoph,
> On 19 Mar 2021, at 00:00, Christoph Grüninger <[hidden email]> wrote: > > ... > > 2. More choices to actually build the project: Use integrated build > tools of IDEs (Visual Studio, Xcode) or use Ninja, which is faster than > gmake > > ... > > 4. CMake is better supported by IDEs like Visual Studio, Qt Creator, > KDevelop. Regarding the IDE support points, it’s possible to generate a CMakeLists.txt from the compile_commands.json file created when building using the current make system. You could then use CMake to generate a native project file for your IDE of choice, and use that for compiling and debugging HotSpot (final linking etc would still be done by make). I have a prototype for this that worked reasonably well with at least Xcode, CLion and Visual Studio as I remember it. If this sounds interesting to anyone I could perhaps try to make it available somewhere.. :) Best regards, Robin > > 5. A lot of code bases were ported to CMake like KDE, Qt, or LLVM. Their > arguments apply here, too. Also their trade-offs between investment and > benefit. > >> At various times, I have dreamed of replacing the configure script with >> something that is more modern and easy to maintain than this bash/m4 >> mix. We have a very well-defined API for the configure script: the user >> calls "bash configure" in the root directory of the project, with a set >> of --options, and as a result we create a spec.gmk file that defines the >> configuration. This could easily be replicated in another system. But if >> I were to rewrite this from scratch, I'd rather write the whole >> configure logic in Java (apart from some thin shell script logic needed >> to find the boot jdk), rather than trying to shoehorn in our build model >> in CMake. > > I understand your temptation, but writing and maintaining all the > configure/find logic and quirks will be a burden. I'd try to reduce the > build system code to a minimum and rely on a third-party solution to do > as much as possible for me. > > Bye > Christoph > > -- > Als wär es nix, leb' ich von [IT] und mach' nur, was ich lieb' > Lebe wie im Paradies, womit hab' ich das verdient? > Die Wahrheit ist: Hab' ich nicht, ich bin nur reicher beschenkt > Als jemand in einem armen Land mit dem gleichen Talent > [frei nach Tua von Die Orsons - Oioioiropa] |
«
Return to OpenJDK Build Infrastructure
|
1 view|%1 views
| Free forum by Nabble | Edit this page |