PR #3752 has the in-development spec proposal for the new texElementImage2D api, which is the WebGL endpoint for the upcoming HTML-in-Canvas web platform feature. This feature allows an HTML sub-tree to be renderered into a texture for use by WebGL programs.
HTML rendered into canvas in this way retains its interactivity (i.e., it can still be the target of input events), but this requires some strategy for doing accurate hit testing of canvas-rendered content for the purpose of determining the target of an input event. In simple terms, we need a way to accurately answer the question: what object is underneath the pointer?
Out of the box, HTML-in-Canvas will support defining a hit test region in terms of an HTML element's CSS box with a developer-provided affine transform applied. This will support many common scenarios where the element's rendered texture is drawn into a scene with a standard model-view-projection, but it does not support arbitrary non-linear projections (e.g. displacement maps).
This issue was previously discussed at a WebGL working group meeting (minutes), where @ladyhavoc suggested allowing the developer to specify a custom coordinate remapping function, which would take as input a point in the CSS box space of the canvas element, and return a point in the same coordinate system, which would then be used to hit test the elements in the canvas'es DOM subtree. The developer may implement the point remapping function to project the canvas point into their 3D scene, and if it overlaps where an HTML element texture was used, return the point projected into the abstract CSS box space of the hit element.
Subsequent to the WebGL WG discussion, this topic came up in discussion around the base HTML-in-canvas proposal. There appears to be general agreement around the idea of permitting the developer to specify a point remapping function, but there is not yet consensus on some of the details...
One important detail of DOM hit testing is that it's not sufficient to simply identify the HTML element targeted by an input event. We must also determine the location of the input event in the coordinate space of the element itself. This is used, for example, to determine the bounds of selected text when dragging the mouse pointer. As an example, if a GL program display HTML text wrapped around a sphere, then the mapping from the GL scene to an offset into the text-containing element would be non-linear.
One proposal discussed in an HTML-in-Canvas issue is to provide hooks to specify two levels of coordinate re-mapping: one at the canvas level to determine which element drawn into the canvas receive the input event; and once the event-receiving element is determined, a second per-element point remapping function could be used to compute the offset into the element's CSS box of the event.
My own opinion is that the two-level point remapping approach is not necessary; only remapping at the canvas level is needed. The developer is presumably already using the GL texture() function in their fragment shader to map GL clip space coordinates to texture coordinates (which in this case are essentially CSS box coordinates). Assuming their point remapping function mimics their shaders, this seems entirely sufficient to determine the offset of a point into an event target without needing a second per-element level of remapping.
I think the WebGL community will speak with the most authority on this subject, since point remapping is only relevant when using HTML-in-Canvas in conjunction with a 3D context. I'd like to get feedback and hopefully consensus on this topic, and bring the results back to the mainstream HTML-in-Canvas discussion.
PR #3752 has the in-development spec proposal for the new
texElementImage2Dapi, which is the WebGL endpoint for the upcoming HTML-in-Canvas web platform feature. This feature allows an HTML sub-tree to be renderered into a texture for use by WebGL programs.HTML rendered into canvas in this way retains its interactivity (i.e., it can still be the target of input events), but this requires some strategy for doing accurate hit testing of canvas-rendered content for the purpose of determining the target of an input event. In simple terms, we need a way to accurately answer the question: what object is underneath the pointer?
Out of the box, HTML-in-Canvas will support defining a hit test region in terms of an HTML element's CSS box with a developer-provided affine transform applied. This will support many common scenarios where the element's rendered texture is drawn into a scene with a standard model-view-projection, but it does not support arbitrary non-linear projections (e.g. displacement maps).
This issue was previously discussed at a WebGL working group meeting (minutes), where @ladyhavoc suggested allowing the developer to specify a custom coordinate remapping function, which would take as input a point in the CSS box space of the canvas element, and return a point in the same coordinate system, which would then be used to hit test the elements in the canvas'es DOM subtree. The developer may implement the point remapping function to project the canvas point into their 3D scene, and if it overlaps where an HTML element texture was used, return the point projected into the abstract CSS box space of the hit element.
Subsequent to the WebGL WG discussion, this topic came up in discussion around the base HTML-in-canvas proposal. There appears to be general agreement around the idea of permitting the developer to specify a point remapping function, but there is not yet consensus on some of the details...
One important detail of DOM hit testing is that it's not sufficient to simply identify the HTML element targeted by an input event. We must also determine the location of the input event in the coordinate space of the element itself. This is used, for example, to determine the bounds of selected text when dragging the mouse pointer. As an example, if a GL program display HTML text wrapped around a sphere, then the mapping from the GL scene to an offset into the text-containing element would be non-linear.
One proposal discussed in an HTML-in-Canvas issue is to provide hooks to specify two levels of coordinate re-mapping: one at the canvas level to determine which element drawn into the canvas receive the input event; and once the event-receiving element is determined, a second per-element point remapping function could be used to compute the offset into the element's CSS box of the event.
My own opinion is that the two-level point remapping approach is not necessary; only remapping at the canvas level is needed. The developer is presumably already using the GL texture() function in their fragment shader to map GL clip space coordinates to texture coordinates (which in this case are essentially CSS box coordinates). Assuming their point remapping function mimics their shaders, this seems entirely sufficient to determine the offset of a point into an event target without needing a second per-element level of remapping.
I think the WebGL community will speak with the most authority on this subject, since point remapping is only relevant when using HTML-in-Canvas in conjunction with a 3D context. I'd like to get feedback and hopefully consensus on this topic, and bring the results back to the mainstream HTML-in-Canvas discussion.