RFR: 8263583: Emoji rendering on macOS

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

RFR: 8263583: Emoji rendering on macOS

Dmitry Batrak-2
This is the implementation used by JetBrains Runtime for the last 4 years, after some cleanup, and with one problem,
found while preparing the pull request, fixed.
Even though typical scenarios for a UI application should be covered, it's not a complete solution. In particular, emoji-s
still won't be rendered for large font sizes (more than 100pt), and for non-trivial composite/painting modes.
Notable implementation details are listed below.

**Glyph image generation**

Deprecated CGContextShowGlyphsAtPoint function, used by JDK on macOS to render text, cannot render emojis,
CTFontDrawGlyphs is used instead. It ignores the scale component of text transformation matrix, so a 'real-sized'
CTFont object should be passed to it. The same sizing procedure is done when calculating glyph metrics, because they
are not scaled proportionally with font size (as they do for vector fonts).

**Glyph image storage**

Existing GlyphInfo structure is used to store color glyph image. Color glyph can be distinguished by having 4 bytes
of storage per pixel. Color components are stored in pre-multiplied alpha format.

**Glyph rendering**

Previously, GlyphList instance always contained glyphs in the same format (solid, grayscale or LCD), determined by the
effective rendering hint. Now the renderers must be prepared to GlyphList having 'normal' glyphs interspersed with
color glyphs (they can appear due to font fallback). This isn't a problem for OpenGL renderer (used for on-screen painting),
but GlyphListLoopPipe-based renderers (used for off-screen painting) needed an adjustment to be able to operate on
specific segments of GlyphList.
As an incidental optimization, calculation of GlyphList bounds ('getBounds' method) is performed now only when needed
(most text renderers don't need this information).
Speaking of the actual rendering of the glyph image, it's done by the straightforward glDrawPixels call in OpenGL renderer,
and by re-using existing Blit primitive in off-screen renderers.

**Testing**

There's no good way to test the new functionality automatically, but I've added a test verifying that 'something' is
rendered for the emoji character, when painting to BufferedImage.

Existing tests pass after the change.

-------------

Commit messages:
 - 8263583: Emoji rendering on macOS

Changes: https://git.openjdk.java.net/jdk/pull/3007/files
 Webrev: https://webrevs.openjdk.java.net/?repo=jdk&pr=3007&range=00
  Issue: https://bugs.openjdk.java.net/browse/JDK-8263583
  Stats: 816 lines in 28 files changed: 677 ins; 29 del; 110 mod
  Patch: https://git.openjdk.java.net/jdk/pull/3007.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/3007/head:pull/3007

PR: https://git.openjdk.java.net/jdk/pull/3007
Reply | Threaded
Open this post in threaded view
|

Re: RFR: 8263583: Emoji rendering on macOS

Alexey Ushakov-3
On Mon, 15 Mar 2021 10:25:50 GMT, Dmitry Batrak <[hidden email]> wrote:

> This is the implementation used by JetBrains Runtime for the last 4 years, after some cleanup, and with one problem,
> found while preparing the pull request, fixed.
> Even though typical scenarios for a UI application should be covered, it's not a complete solution. In particular, emoji-s
> still won't be rendered for large font sizes (more than 100pt), and for non-trivial composite/painting modes.
> Notable implementation details are listed below.
>
> **Glyph image generation**
>
> Deprecated CGContextShowGlyphsAtPoint function, used by JDK on macOS to render text, cannot render emojis,
> CTFontDrawGlyphs is used instead. It ignores the scale component of text transformation matrix, so a 'real-sized'
> CTFont object should be passed to it. The same sizing procedure is done when calculating glyph metrics, because they
> are not scaled proportionally with font size (as they do for vector fonts).
>
> **Glyph image storage**
>
> Existing GlyphInfo structure is used to store color glyph image. Color glyph can be distinguished by having 4 bytes
> of storage per pixel. Color components are stored in pre-multiplied alpha format.
>
> **Glyph rendering**
>
> Previously, GlyphList instance always contained glyphs in the same format (solid, grayscale or LCD), determined by the
> effective rendering hint. Now the renderers must be prepared to GlyphList having 'normal' glyphs interspersed with
> color glyphs (they can appear due to font fallback). This isn't a problem for OpenGL renderer (used for on-screen painting),
> but GlyphListLoopPipe-based renderers (used for off-screen painting) needed an adjustment to be able to operate on
> specific segments of GlyphList.
> As an incidental optimization, calculation of GlyphList bounds ('getBounds' method) is performed now only when needed
> (most text renderers don't need this information).
> Speaking of the actual rendering of the glyph image, it's done by the straightforward glDrawPixels call in OpenGL renderer,
> and by re-using existing Blit primitive in off-screen renderers.
>
> **Testing**
>
> There's no good way to test the new functionality automatically, but I've added a test verifying that 'something' is
> rendered for the emoji character, when painting to BufferedImage.
>
> Existing tests pass after the change.

Looks good

-------------

PR: https://git.openjdk.java.net/jdk/pull/3007
Reply | Threaded
Open this post in threaded view
|

Re: RFR: 8263583: Emoji rendering on macOS

Phil Race
On Mon, 15 Mar 2021 15:59:36 GMT, Alexey Ushakov <[hidden email]> wrote:

>> This is the implementation used by JetBrains Runtime for the last 4 years, after some cleanup, and with one problem,
>> found while preparing the pull request, fixed.
>> Even though typical scenarios for a UI application should be covered, it's not a complete solution. In particular, emoji-s
>> still won't be rendered for large font sizes (more than 100pt), and for non-trivial composite/painting modes.
>> Notable implementation details are listed below.
>>
>> **Glyph image generation**
>>
>> Deprecated CGContextShowGlyphsAtPoint function, used by JDK on macOS to render text, cannot render emojis,
>> CTFontDrawGlyphs is used instead. It ignores the scale component of text transformation matrix, so a 'real-sized'
>> CTFont object should be passed to it. The same sizing procedure is done when calculating glyph metrics, because they
>> are not scaled proportionally with font size (as they do for vector fonts).
>>
>> **Glyph image storage**
>>
>> Existing GlyphInfo structure is used to store color glyph image. Color glyph can be distinguished by having 4 bytes
>> of storage per pixel. Color components are stored in pre-multiplied alpha format.
>>
>> **Glyph rendering**
>>
>> Previously, GlyphList instance always contained glyphs in the same format (solid, grayscale or LCD), determined by the
>> effective rendering hint. Now the renderers must be prepared to GlyphList having 'normal' glyphs interspersed with
>> color glyphs (they can appear due to font fallback). This isn't a problem for OpenGL renderer (used for on-screen painting),
>> but GlyphListLoopPipe-based renderers (used for off-screen painting) needed an adjustment to be able to operate on
>> specific segments of GlyphList.
>> As an incidental optimization, calculation of GlyphList bounds ('getBounds' method) is performed now only when needed
>> (most text renderers don't need this information).
>> Speaking of the actual rendering of the glyph image, it's done by the straightforward glDrawPixels call in OpenGL renderer,
>> and by re-using existing Blit primitive in off-screen renderers.
>>
>> **Testing**
>>
>> There's no good way to test the new functionality automatically, but I've added a test verifying that 'something' is
>> rendered for the emoji character, when painting to BufferedImage.
>>
>> Existing tests pass after the change.
>
> Looks good

Initial testing it looks OK with OpenGL but it renders as garbage with the metal pipeline.
I'd guess metal isn't expecting 4bpp and the stride is wrong.
Since metal is now integrated this should be fixed before this change can be pushed.

-------------

PR: https://git.openjdk.java.net/jdk/pull/3007
Reply | Threaded
Open this post in threaded view
|

Re: RFR: 8263583: Emoji rendering on macOS

Sergey Bylokhov-2
In reply to this post by Dmitry Batrak-2
On Mon, 15 Mar 2021 10:25:50 GMT, Dmitry Batrak <[hidden email]> wrote:

> This is the implementation used by JetBrains Runtime for the last 4 years, after some cleanup, and with one problem,
> found while preparing the pull request, fixed.
> Even though typical scenarios for a UI application should be covered, it's not a complete solution. In particular, emoji-s
> still won't be rendered for large font sizes (more than 100pt), and for non-trivial composite/painting modes.
> Notable implementation details are listed below.
>
> **Glyph image generation**
>
> Deprecated CGContextShowGlyphsAtPoint function, used by JDK on macOS to render text, cannot render emojis,
> CTFontDrawGlyphs is used instead. It ignores the scale component of text transformation matrix, so a 'real-sized'
> CTFont object should be passed to it. The same sizing procedure is done when calculating glyph metrics, because they
> are not scaled proportionally with font size (as they do for vector fonts).
>
> **Glyph image storage**
>
> Existing GlyphInfo structure is used to store color glyph image. Color glyph can be distinguished by having 4 bytes
> of storage per pixel. Color components are stored in pre-multiplied alpha format.
>
> **Glyph rendering**
>
> Previously, GlyphList instance always contained glyphs in the same format (solid, grayscale or LCD), determined by the
> effective rendering hint. Now the renderers must be prepared to GlyphList having 'normal' glyphs interspersed with
> color glyphs (they can appear due to font fallback). This isn't a problem for OpenGL renderer (used for on-screen painting),
> but GlyphListLoopPipe-based renderers (used for off-screen painting) needed an adjustment to be able to operate on
> specific segments of GlyphList.
> As an incidental optimization, calculation of GlyphList bounds ('getBounds' method) is performed now only when needed
> (most text renderers don't need this information).
> Speaking of the actual rendering of the glyph image, it's done by the straightforward glDrawPixels call in OpenGL renderer,
> and by re-using existing Blit primitive in off-screen renderers.
>
> **Testing**
>
> There's no good way to test the new functionality automatically, but I've added a test verifying that 'something' is
> rendered for the emoji character, when painting to BufferedImage.
>
> Existing tests pass after the change.

src/java.desktop/share/classes/sun/java2d/loops/DrawGlyphListColor.java line 71:

> 69:     static {
> 70:         GraphicsPrimitiveMgr.registerGeneral(
> 71:                                 new DrawGlyphListColor(null, null, null));

If it is a complete "General" loop which supports all possible combination of incoming src/comp/dst/transform/etc I suggest to confirm that in the test.

-------------

PR: https://git.openjdk.java.net/jdk/pull/3007
Reply | Threaded
Open this post in threaded view
|

Re: RFR: 8263583: Emoji rendering on macOS

Dmitry Batrak-2
In reply to this post by Phil Race
On Mon, 15 Mar 2021 23:47:16 GMT, Phil Race <[hidden email]> wrote:

> Initial testing it looks OK with OpenGL but it renders as garbage with the metal pipeline.
> I'd guess metal isn't expecting 4bpp and the stride is wrong.
> I see updates in OGLTextRenderer so something similar likely needed for metal.
> Since metal is now integrated this should be fixed before this change can be pushed.

I'm going to add Metal implementation soon.

-------------

PR: https://git.openjdk.java.net/jdk/pull/3007
Reply | Threaded
Open this post in threaded view
|

Re: RFR: 8263583: Emoji rendering on macOS

Dmitry Batrak-2
On Tue, 16 Mar 2021 05:10:05 GMT, Dmitry Batrak <[hidden email]> wrote:

>> Initial testing it looks OK with OpenGL but it renders as garbage with the metal pipeline.
>> I'd guess metal isn't expecting 4bpp and the stride is wrong.
>> I see updates in OGLTextRenderer so something similar likely needed for metal.
>> Since metal is now integrated this should be fixed before this change can be pushed.
>
>> Initial testing it looks OK with OpenGL but it renders as garbage with the metal pipeline.
>> I'd guess metal isn't expecting 4bpp and the stride is wrong.
>> I see updates in OGLTextRenderer so something similar likely needed for metal.
>> Since metal is now integrated this should be fixed before this change can be pushed.
>
> I'm going to add Metal implementation soon.

> If it is a complete "General" loop which supports all possible combination of incoming src/comp/dst/transform/etc I suggest to confirm that in the test.

@mrserb This code is currently only used by GlyphListLoopPipe-based text renderers (SolidTextRenderer, AATextRenderer, LCDTextRenderer), so it won't be invoked for arbitrary composites and paint types. But different destination types (image formats) and transforms are supported.

What kind of test are you having in mind? I can extend the added MacEmoji.java test, that will check that 'something' is painted to different types of BufferedImage-s and/or with transforms applied. Or you want me to add a manual test case?

-------------

PR: https://git.openjdk.java.net/jdk/pull/3007
Reply | Threaded
Open this post in threaded view
|

Re: RFR: 8263583: Emoji rendering on macOS

Sergey Bylokhov-2
On Tue, 16 Mar 2021 08:50:27 GMT, Dmitry Batrak <[hidden email]> wrote:

> What kind of test are you having in mind? I can extend the added MacEmoji.java test, that will check that 'something' is painted to different types of BufferedImage-s and/or with transforms applied. Or you want me to add a manual test case?

The automated tests will be enough, not sure about 'something is painted' however, it is better to check that it works.

-------------

PR: https://git.openjdk.java.net/jdk/pull/3007
Reply | Threaded
Open this post in threaded view
|

Re: RFR: 8263583: Emoji rendering on macOS

Dmitry Batrak-2
On Tue, 16 Mar 2021 19:46:16 GMT, Sergey Bylokhov <[hidden email]> wrote:

>>> If it is a complete "General" loop which supports all possible combination of incoming src/comp/dst/transform/etc I suggest to confirm that in the test.
>>
>> @mrserb This code is currently only used by GlyphListLoopPipe-based text renderers (SolidTextRenderer, AATextRenderer, LCDTextRenderer), so it won't be invoked for arbitrary composites and paint types. But different destination types (image formats) and transforms are supported.
>>
>> What kind of test are you having in mind? I can extend the added MacEmoji.java test, that will check that 'something' is painted to different types of BufferedImage-s and/or with transforms applied. Or you want me to add a manual test case?
>
>> What kind of test are you having in mind? I can extend the added MacEmoji.java test, that will check that 'something' is painted to different types of BufferedImage-s and/or with transforms applied. Or you want me to add a manual test case?
>
> The automated tests will be enough, not sure about 'something is painted' however, it is better to check that it works.

@mrserb I don't know how to check automatically that glyph painting works correctly. Could you please suggest a way to do it? In JetBrains Runtime we have a test that checks that rendered emoji glyph matches one of stored 'golden images', but I don't think that's suitable for OpenJDK, unless someone will volunteer to update that list when the need arises (e.g. when newer version of macOS changes the font, or rendering details). At the moment, we have already 7 golden images for a single glyph in our repository.

-------------

PR: https://git.openjdk.java.net/jdk/pull/3007
Reply | Threaded
Open this post in threaded view
|

Re: RFR: 8263583: Emoji rendering on macOS

Sergey Bylokhov-2
On Wed, 17 Mar 2021 10:45:00 GMT, Dmitry Batrak <[hidden email]> wrote:

>>> What kind of test are you having in mind? I can extend the added MacEmoji.java test, that will check that 'something' is painted to different types of BufferedImage-s and/or with transforms applied. Or you want me to add a manual test case?
>>
>> The automated tests will be enough, not sure about 'something is painted' however, it is better to check that it works.
>
> @mrserb I don't know how to check automatically that glyph painting works correctly. Could you please suggest a way to do it? In JetBrains Runtime we have a test that checks that rendered emoji glyph matches one of stored 'golden images', but I don't think that's suitable for OpenJDK, unless someone will volunteer to update that list when the need arises (e.g. when newer version of macOS changes the font, or rendering details). At the moment, we have already 7 golden images for a single glyph in our repository.

You the current image from the MacEmoji test as a golden image for other formats/transforms/extraalpha/etc. For example, if DST type is changed then it is unlikely the shape of the emoji will be changed as well, or if it is too big/ too small.

BTW It will be good if the test will fail on the current metal implementation.

-------------

PR: https://git.openjdk.java.net/jdk/pull/3007
Reply | Threaded
Open this post in threaded view
|

Re: RFR: 8263583: Emoji rendering on macOS [v2]

Dmitry Batrak-2
In reply to this post by Dmitry Batrak-2
> This is the implementation used by JetBrains Runtime for the last 4 years, after some cleanup, and with one problem,
> found while preparing the pull request, fixed.
> Even though typical scenarios for a UI application should be covered, it's not a complete solution. In particular, emoji-s
> still won't be rendered for large font sizes (more than 100pt), and for non-trivial composite/painting modes.
> Notable implementation details are listed below.
>
> **Glyph image generation**
>
> Deprecated CGContextShowGlyphsAtPoint function, used by JDK on macOS to render text, cannot render emojis,
> CTFontDrawGlyphs is used instead. It ignores the scale component of text transformation matrix, so a 'real-sized'
> CTFont object should be passed to it. The same sizing procedure is done when calculating glyph metrics, because they
> are not scaled proportionally with font size (as they do for vector fonts).
>
> **Glyph image storage**
>
> Existing GlyphInfo structure is used to store color glyph image. Color glyph can be distinguished by having 4 bytes
> of storage per pixel. Color components are stored in pre-multiplied alpha format.
>
> **Glyph rendering**
>
> Previously, GlyphList instance always contained glyphs in the same format (solid, grayscale or LCD), determined by the
> effective rendering hint. Now the renderers must be prepared to GlyphList having 'normal' glyphs interspersed with
> color glyphs (they can appear due to font fallback). This isn't a problem for OpenGL renderer (used for on-screen painting),
> but GlyphListLoopPipe-based renderers (used for off-screen painting) needed an adjustment to be able to operate on
> specific segments of GlyphList.
> As an incidental optimization, calculation of GlyphList bounds ('getBounds' method) is performed now only when needed
> (most text renderers don't need this information).
> Speaking of the actual rendering of the glyph image, it's done by the straightforward glDrawPixels call in OpenGL renderer,
> and by re-using existing Blit primitive in off-screen renderers.
>
> **Testing**
>
> There's no good way to test the new functionality automatically, but I've added a test verifying that 'something' is
> rendered for the emoji character, when painting to BufferedImage.
>
> Existing tests pass after the change.

Dmitry Batrak has updated the pull request incrementally with two additional commits since the last revision:

 - 8263583: Emoji rendering on macOS
   
   extended test case to cover different types of target surfaces
 - 8263583: Emoji rendering on macOS
   
   implementation for Metal rendering pipeline

-------------

Changes:
  - all: https://git.openjdk.java.net/jdk/pull/3007/files
  - new: https://git.openjdk.java.net/jdk/pull/3007/files/3865b397..a9b2e9c3

Webrevs:
 - full: https://webrevs.openjdk.java.net/?repo=jdk&pr=3007&range=01
 - incr: https://webrevs.openjdk.java.net/?repo=jdk&pr=3007&range=00-01

  Stats: 132 lines in 2 files changed: 118 ins; 2 del; 12 mod
  Patch: https://git.openjdk.java.net/jdk/pull/3007.diff
  Fetch: git fetch https://git.openjdk.java.net/jdk pull/3007/head:pull/3007

PR: https://git.openjdk.java.net/jdk/pull/3007
Reply | Threaded
Open this post in threaded view
|

Re: RFR: 8263583: Emoji rendering on macOS

Dmitry Batrak-2
In reply to this post by Sergey Bylokhov-2
On Thu, 18 Mar 2021 10:02:11 GMT, Sergey Bylokhov <[hidden email]> wrote:

>> @mrserb I don't know how to check automatically that glyph painting works correctly. Could you please suggest a way to do it? In JetBrains Runtime we have a test that checks that rendered emoji glyph matches one of stored 'golden images', but I don't think that's suitable for OpenJDK, unless someone will volunteer to update that list when the need arises (e.g. when newer version of macOS changes the font, or rendering details). At the moment, we have already 7 golden images for a single glyph in our repository.
>
> You the current image from the MacEmoji test as a golden image for other formats/transforms/extraalpha/etc. For example, if DST type is changed then it is unlikely the shape of the emoji will be changed as well, or if it is too big/ too small.
>
> BTW It will be good if the test will fail on the current metal implementation.

Implementation for Metal pipeline has been added.

I've also updated the test case to check rendering into different types of images, including VolatileImage. The test will fail now, if corresponding OpenGL (or Metal, if it's explicitly enabled) implementation part would be rolled back.

-------------

PR: https://git.openjdk.java.net/jdk/pull/3007
Reply | Threaded
Open this post in threaded view
|

Re: RFR: 8263583: Emoji rendering on macOS

Alexey Ushakov-3
On Fri, 19 Mar 2021 13:54:34 GMT, Dmitry Batrak <[hidden email]> wrote:

>> You the current image from the MacEmoji test as a golden image for other formats/transforms/extraalpha/etc. For example, if DST type is changed then it is unlikely the shape of the emoji will be changed as well, or if it is too big/ too small.
>>
>> BTW It will be good if the test will fail on the current metal implementation.
>
> Implementation for Metal pipeline has been added.
>
> I've also updated the test case to check rendering into different types of images, including VolatileImage. The test will fail now, if corresponding OpenGL (or Metal, if it's explicitly enabled) implementation part would be rolled back.

Looks good!

-------------

PR: https://git.openjdk.java.net/jdk/pull/3007
Reply | Threaded
Open this post in threaded view
|

Re: RFR: 8263583: Emoji rendering on macOS

Sergey Bylokhov-2
On Fri, 19 Mar 2021 16:46:32 GMT, Alexey Ushakov <[hidden email]> wrote:

>> Implementation for Metal pipeline has been added.
>>
>> I've also updated the test case to check rendering into different types of images, including VolatileImage. The test will fail now, if corresponding OpenGL (or Metal, if it's explicitly enabled) implementation part would be rolled back.
>
> Looks good!

It looks fine to me. I will run the tests in CI just in case.

-------------

PR: https://git.openjdk.java.net/jdk/pull/3007