<i18n dev> RFR: 8262744: Formatter '%g' conversion uses wrong format for BigDecimal rounding up to limits

[Ian Graves-2]

This fixes a bug where the formatting code for `%g` flags incorrectly tries to round `BigDecimal` after determining whether it should be a decimal numeric format or a scientific numeric format. The solution rounds before determining the correct format.

-------------

Commit messages:
 - Fixing rounding issue with BigDecimal and %g formatting flags

Changes: https://git.openjdk.java.net/jdk/pull/3363/files
 Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=3363&range=00
  Issue: https://bugs.openjdk.java.net/browse/JDK-8262744
  Stats: 59 lines in 2 files changed: 59 ins; 0 del; 0 mod
  Patch: https://git.openjdk.java.net/jdk/pull/3363.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/3363/head:pull/3363

PR: https://git.openjdk.java.net/jdk/pull/3363

[Joe Darcy-2]

On Tue, 6 Apr 2021 20:34:52 GMT, Ian Graves <[hidden email]> wrote:

> This fixes a bug where the formatting code for `%g` flags incorrectly tries to round `BigDecimal` after determining whether it should be a decimal numeric format or a scientific numeric format. The solution rounds before determining the correct format.

test/jdk/java/util/Formatter/BigDecimalRounding.java line 44:

> 42:     public static void testBigDecimalRounding() {
> 43:         var res1 = String.format("%g", 0.00009999999999999995);
> 44:         var res2 = String.format("%g", 0.00009999999f);

To avoid possible vagaries in decimal -> binary conversion of double values, I suggest also testing against a BigDecimal directly rounded to the precision in question.

-------------

PR: https://git.openjdk.java.net/jdk/pull/3363

[Ian Graves-2]

> This fixes a bug where the formatting code for `%g` flags incorrectly tries to round `BigDecimal` after determining whether it should be a decimal numeric format or a scientific numeric format. The solution rounds before determining the correct format.

Ian Graves has updated the pull request incrementally with one additional commit since the last revision:

  Adding BigDecimal equivalents to tests

-------------

Changes:
  - all: https://git.openjdk.java.net/jdk/pull/3363/files
  - new: https://git.openjdk.java.net/jdk/pull/3363/files/8ec00f8f..cf40421e

Webrevs:
 - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=3363&range=01
 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=3363&range=00-01

  Stats: 10 lines in 1 file changed: 5 ins; 1 del; 4 mod
  Patch: https://git.openjdk.java.net/jdk/pull/3363.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/3363/head:pull/3363

PR: https://git.openjdk.java.net/jdk/pull/3363

[Ian Graves-2]

On Tue, 6 Apr 2021 22:56:22 GMT, Joe Darcy <[hidden email]> wrote:

>> Ian Graves has updated the pull request incrementally with one additional commit since the last revision:
>>
>>   Adding BigDecimal equivalents to tests
>
> test/jdk/java/util/Formatter/BigDecimalRounding.java line 44:
>
>> 42:     public static void testBigDecimalRounding() {
>> 43:         var res1 = String.format("%g", 0.00009999999999999995);
>> 44:         var res2 = String.format("%g", 0.00009999999f);
>
> To avoid possible vagaries in decimal -> binary conversion of double values, I suggest also testing against a BigDecimal directly rounded to the precision in question.

Good thought, thanks.

-------------

PR: https://git.openjdk.java.net/jdk/pull/3363