Mobile/IME
Contents
Introduction
Here we will be collecting information about keyboard/IME handling and issues in Fennec.
IME architecture
Components
Android IME-handling code consists of three major parts: Java, widget, and content (XUL version also had IPC layer as the content process was separate).
Java part is mostly in GeckoInputConnection.java, which deals with Android API, and translates Java interface calls made by Android system to the corresponding Gecko events.
The IME events are the objects of GeckoEvent class with one of the IME_XXX types. They get sent using GeckoAppShell.sendEventToGecko() across JNI to nsWindow.cpp in widget, wrapped in AndroidGeckoEvent C++ class.
nsWindow does minimum packaging and dispatches the events further to content, where they go all over the place. Many of those events are handled in content/events/src/nsContentEventHandler.cpp, for example NS_QUERY_SELECTED_TEXT is dispatched to nsContentEventHandler::OnQuerySelectedText.
The IME events go only in one direction: from front-end to back-end, the reverse communication is done using notifications going from back-end to front-end. They pass through nsWindow::OnIMExxx in widget, across JNI to GeckoAppShell.NotifyIMEChange, and handled in GeckoInputConnection.notifyXXXChange.
After the major Java part redesign made in bug 595008 most of the input handling is done by the base Android class BaseInputConnection. We override only some of its methods to send the actual editable object changes to Gecko. The majority of the IME events from Java code are sent to Gecko from GeckoInputConnection.onTextChanged() method.
Ranges
Some IMEs are marking a part of the text that you're working on. For example, here a part of the text being entered is underlined:
Information about those parts is passed to Gecko as an array of ranges. Each range has several properties, including a type, which defines the nature of the range, styles like underlining, and text colors. The most noticeable and important range types are the composing text, and the text selection.
The ranges are stored as spans in the internal mEditable class member, and handled by the BaseInputConnection class.
Composition
A composing text is a range of text that you are currently working on. While it is in the composition state, it can be freely changed, these changes are considered as a batch operation, and are submitted as one update at the end of the composition, so the whole text modification is stored as one action in the edit history.
The actual composition state on Android side is handled by the BaseInputConnection class, with the composing range stored as a corresponding span in the mEditable member. GeckoInputConnection tracks the composing state using the mComposing flag. When it receives a text modification request from Android it sets this flag and sends IME_COMPOSITION_BEGIN event to Gecko. While in that state the composing text is sent to Gecko from sendTextToGecko() method. The composing may be ended either explicitly when finishComposingText() method is called by the system, or when the changes have to be committed for some reason. IME_COMPOSITION_END event is sent to Gecko to finish the composition.
As most of the external IME events handling is done by the BaseInputConnection class, Gecko composition state is a bit artificial, and sometimes may not exactly correspond to the state in BaseInputConnection class. There are areas for improvements here.
Debugging IME
Due to the asynchronous nature of IME interaction the best way to debug the issues is to read and try to understand the extended logs. There is a DEBUG flag in the GeckoInputConnection class, which enables debug log output. GeckoInputConnection.java also contains a special subclass DebugGeckoInputConnection, which is enabled by the same flag, and writes additional logs on each IME-related method call.
widget/android/nsWindow.cpp also writes additional IME-related logs when DEBUG_ANDROID_IME flag is defined.
