Viewport handling

Definitions

The list of viewport properties are as follows, along with the coordinate system they are represented in Java.

• offset - This is the x and y in device pixels of the pixel in the top-left corner of the user-visible display (in this case device pixels is the same as zoom-multiplied CSS pixels, and so the x and y will grow as the user zooms).
• size - This is the width and height in device pixels of the user-visible display area, and is unaffected by zooming.
• zoom - This is the zoom factor at which the page is shown to the user (> 1.0 means page elements are rendered large).
• page size - This is the width and height in device pixels of the content (as with offset, this will grow as the user zooms because it is CSS pixels multiplied by zoom).
• display port margins - This is the left, top, right, and bottom margins in device pixels around the user-visible display that we should paint as a buffer. These are in device pixels and are unaffected by zooming.
• display port - This is the area in device pixels of the page that we ask Gecko to draw.
• painted area - This is the area in device pixels of the page that Gecko has drawn for us, not including the display port margins. (Gecko will paint the area including display port margins, but for convenience we track these separately). The painted area size is unaffected by zooming, but the top and left coordinates of the painted area will grow as the zoom increases.

Documents are defined as follows:

• The "content document" of a tab refers to the document attached to the content window of that tab.
• The "displayed document" of a tab refers to the document to which user-visible content of a tab belongs. This may be different from the content document during page load, and we assume that there is no upper bound on how long they are different for.
• The "active tab" refers to the XUL browser element that is topmost in the browser.xul deck.
• The "browser content document" refers to the content document of the active tab.
• The "browser displayed document" refers to the document that is currently visible to the user. This may be different from the displayed document of the active tab during tab switch, and we assume that there is no upper bound on how long they are different for.

Threads

• The gecko thread. Gecko runs on this thread, as does anything in browser.js. "Messages" sent from browser.js to Java are synchronous JNI calls and also run on the gecko thread.
• The Java UI thread. Java receives UI events on this thread. When this results in messages to Gecko, those messages are queued on the Gecko event queue and handled asynchronously on the Gecko thread.
• The compositor thread. The compositor runs on this thread. This thread can make JNI calls to Java which will also run on the compositor thread.

Rules

• Java must maintain one set of properties (offset, size, zoom, page size, display port, and painted area).
• The compositor must query this Java for offset/zoom on every frame.
• Th painted area must be provided to Java synchronously every time Gecko draws (this is accomplished by JNI invocations in the widget code). This will be used for determining if we need to draw again, and by how much touch events need to be translated by.
• When the browser displayed document changes (e.g. tab switch, page load), java must be told of updated viewport properties (offset, zoom, page size) via the compositor when the first frame of the new content is composited.
• When the composition-related properties (offset, zoom, page size) of the selected tab change without the document changing (e.g. a meta viewport tag is added that changes the zoom, or javascript on the page calling ScrollTo), java must be told synchronously from browser.js. Java must return an updated display port when this happens, such that the updated display port is used for the draw that will happen.
• When the user performs pan/zoom actions, Java should send updated properties (offset, zoom, display port) to browser.js. (These may be throttled, such as when we know the newly visible area is still inside the display port, or in the middle of a pinch zoom.)
• browser.js must ignore viewport-dependent events (including clicks, double-taps, and viewport updates) from java during the period where the browser content document is different from the browser displayed document. Likewise, browser.js must not send any property updates to Java (e.g. resulting from user JS calling scrollTo) during this period. Java may still update its own viewport properties for the compositor's benefit.
• If the viewport size changes, Java must first send a resize notification. browser.js must, upon processing the resize, synchronously notify Java of receiving this event, along with any updated properties (offset, zoom, page size). While processing this event (still on the gecko thread), Java may recalculate properties (offset, zoom, display port margins) and may queue an event back on the gecko thread with the new properties. Drawing should be suppressed between the handling of the initial resize notification and handling of the final property update event, if there is one.
• A display port must always be set on the document element of the document being drawn. The display port is owned by Java, and must be updated whenever it becomes aware that the JS offset or zoom has changed. This includes scenarios such as: (1) when Java sends a viewport update to JS as a result of user performing pan/zoom actions; (2) in response to the synchronous event from browser.js informing Java of a offset/zoom/page size change; (3) when Java is notified by the compositor of a change in the displayed document.

Implementation Assertions

• The firstPaint flag on the presShell is set by browser.js if and only if the next draw will be of a different document than the previous draw was. It is cleared as close to the composition of that draw as possible.
• All messages sent to browser.js from the Java side must be sent on the Java UI thread (as opposed to the Java GL thread or the compositor thread invoking something in Java, or any background Java threads).

Gotchas

Paint suppression

Paint suppression in Gecko works sort of like this: when a new page is loaded, paints on it are suppressed for some time. The start and end of this suppression period are not well defined; the period starts around the time of the first reflow of the document, and ends in either 250ms (or some preffed value) or when some other conditions are hit (like the document reaches the ready state). Also complicating matters is that when paint suppression is enabled on a pres shell, the paint code will simply look for another pres shell to paint. During page load, when the old and new pres shells are still in existence, this results in the old document getting painted during this time.

To deal with this issue, an older version of the viewport code introduced additional paint suppression via a hook in browser.js. Although this worked, it was clunky and brittle with the off-main-thread compositor and refresh driver changes. The new approach we have taken is to allow the old document to continue being drawn as the new one loads; we detect the point at which the new document is about to be painted, and perform any necessary actions (such as meta viewport calculations and calling setResolution) at that point so that they take effect for the new document drawing and get carried through to Java via the compositor.

Display ports

If no display port is set, then the layer may end up in a non-scrollable state, which can potentially confuse the compositor and interfere with propagating the correct page sizes. Therefore a display port must always be set on the document element of the document being drawn. Note that at the point the onLocationChange event listener is triggered, the document element may not yet exist. Therefore setting the display port at that point and assuming it is set is a bad idea.

Rounding and conversions

Gecko does layout calculations in twips (1 twip == 1/60 of a CSS pixel). This is mostly irrelevant except for the fact that these are not floats and converting floats to twips is lossy. Drawing code in Gecko uses mostly floats; the zoom factor/resolution is a float. In order to make sure values line up and aren't off by a pixel here and there, all of these need to be taken into consideration when moving values around, rounding them as needed. See https://bugzilla.mozilla.org/show_bug.cgi?id=737510#c10 for an example of the hairiness this results in.

• Scroll offsets and page sizes in Gecko are always integer values in CSS pixels. When drawing, Gecko multiplies them by the resolution and rounds them *up* to the nearest device pixel. This is a lossy conversion; the compositor gets these integer device pixel values and passes them on to Java. Java keeps them as floats in device pixels. Therefore Java must be aware that page sizes it gets from the compositor have already undergone a lossy conversion and may not line up properly with internally-stored values.
• View size is always an integer in device pixels. All code that needs to keep this value should keep it as such.
• Zoom is always a float value. All code that needs to keep this value should keep it as such. browser.js passes this value to Gecko using a call to setResolution(float).
• Gecko itself tracks the display port in twips; the display port enters Gecko via the setDisplayPortForElement function which takes floating-point CSS pixel values. These values are converted directly to twips without intermediate rounding; therefore the display port in Gecko may include sub-pixel regions. Java keeps the display port in floats in device pixels, and must be aware that a slightly-lossy conversion is happening on those values in Gecko.
• When we calculate the painted area, we must take into account any lossy conversions that have happened on the coordinates that originated in Java. Specifically, say the Java code starts with (x,y)=(0.0,6.0) at a zoom level of 4.0; browser.js ends up calling window.scrollTo(0.0,1.5) which Gecko internally tracks as (0, 1). When Gecko paints, it will paint such that the CSS pixel (0,1) occupies the top-left 4x4 device pixels. When the widget code asks browser.js for the current Gecko viewport, browser.js will take (0,1) and multiply it by 4.0, giving (0.0, 4.0) and hand it to Java as the top-left corner of the painted area. This, while correct, is off by 2 pixels from what the original Java value was, and can be a source of error if not accounted for.

Deferred layout problems

Accessing layout properties like window.scrollX or body.scrollHeight might trigger some layout code to run in order to provide the correct value. While layout code is running, it could re-enter back into browser.js to run some event handlers. For example, I have observed this happening during device rotation in at least one version of the code - getViewport() is called which tries to access scrollX, which triggers the "resize" event handler in browser.js to run, which causes re-entrancy back into getViewport(). The outer call to getViewport() could end up returning inconsistent values as a result. (Note: bug 764467 should eliminate this problem, but I'm leaving this here as it is something that should be taken into consideration when touching this code).

Related bugs for more information

• bug 732091 - Make the compositor know when it's painting a new page
• bug 732564 - Get viewport handling good again
• bug 731829 - Race condition can leave checkerboarding on screen after rotation
• bug 731897 - make js set the java viewport synchronously
• bug 732089 - keep resolution on the document instead of the xul window
• bug 736729 - Display port update on page size change