NPAPI:AdvancedKeyHandlingHistory

From MozillaWiki
Jump to: navigation, search

It took almost 10 years to finalize a spec for advanced key handling in NPAPI. This page provides information on past attempts.

Discussions and Documentation

Note: Some of the archive links here require authentication, however most relevant information behind those links has been copied to this document.

Mozilla bugs 78414 and 93149 have some information but are generally cluttered with useless comments. Deneb Meketa from Adobe posted a proposal on bug 93149, comments #34 and #36 discuss it.

There was a plugin-futures discussion started by Johnny Stenback in March of 2004 on the topic of handled status for key events. Johnny's discussion resulted in a plugin-futures wiki entry.

Aaron Leventhal of IBM started a relevant discussion on plugin-futures in April of 2004. The proposal discussed is an extension of Johnny's from March of 2004 and is actually fairly detailed.

API modification work is discussed in Mozilla bug 348279. The discussion there generally revolves around a proposal from Sun. Sun posted to plugin-futures about this work in March of 2007.

Karl Tomlinson wrote a message on Google Groups proposing an X11 windowless plugin API. There is some related info in that message.

Josh Aas wrote a blog post on the issue in September of 2008.

Past Proposals

Deneb Meketa, Adobe, April 2002

Original Proposal

First off, I think there are three things that need to be communicated:

1. Browser informs plugin that it is receiving focus, and indicates the reason (tab, shift-tab, or click)

2. Plugin informs browser that the user has tabbed out, and indicates the direction (tab or shift-tab)

3. Browser informs plugin that it has lost focus due to a click elsewhere, or the browser losing focus

Our idea is to handle all three of these using the existing NPP_HandleEvent mechanism.

There are already events for getFocusEvent and loseFocusEvent. The plugin returns 0 or 1 from getFocusEvent to indicate whether it can accept the focus.

Consider a new event called advanceFocusEvent. The browser would send this event whenever the tab key was pressed. It would include a parameter that indicates whether the tab direction is forwards or backwards. The plugin would return 0 to indicate that it does not understand advanceFocusEvent, 1 to indicate that it either cannot accept the focus or is relinquishing it (having run out of items in the specified direction), or 2 to indicate that it either has accepted the focus or is keeping it (because it has more items in the specified direction).

If the browser gets a 0 back from advanceFocusEvent, it could just do whatever it does today when the tab key is pressed. If it gets a 1, it would send a loseFocusEvent if the plugin had focus, then find the next focusable element in the page in the appropriate direction and attempt to set focus there (possibly sending advanceFocusEvent to another plugin). If it gets a 2, it would send a getFocusEvent if the plugin does not already have focus, and then stop.

The getFocusEvent and loseFocusEvent events would work as they do today for mouse events - no changes necessary.

I think this covers all three cases above, and allows existing plugins to work unaffected. But I could be wrong, and there might be something odious about implementing this particular idea on the browser side.

Comments

Deneb Meketa responding to Peter Lubczynski:

1. How is the plugin going to pass an event to the browser to notify that it should take focus?

Hmmm. I think my original intention was that the browser would always capture the tab key, and send advanceFocusEvent, and a return of 1 from the plugin would indicate "I have tabbed out, please take the focus back". However, I guess maybe the plugin has to actually *take* real input focus when it receives getFocusEvent, which would take the browser out of the event loop, unless we want to do some trick like spying on the child window messages - is that possible and reasonable? If not, we would need to add something like NPN_TakeFocus that the plugin could call when it reaches the end of its tab order.

2. How does the browser capture events sent to the plug-in window?

Yeah, I think this is the same question I just asked above. How does it work in NN 4.7? I can click inside a plugin, type into the plugin, and then press Ctrl+S and the NN Save dialog pops up. Isn't this exactly what we are asking about?: the plugin has focus, but the browser is somehow intercepting events? Or is it maybe that the browser has kept focus and is forwarding events to the plugin?

3. How does the browser know which keys to process and which to forward to the plug-in? (belongs with bug 78414)

Hmmm. Loretta makes a good point that plug-ins have more freedom if they receive all events, but that does make for an inconsistent browser user experience. It seems to me that the best solution is to go with a "most local" rule: if the plugin attaches a particular meaning to a key combination, then when the plugin has focus, it should get that key event. But otherwise the browser should get it. In theory, the plugin should return something from its event handler that indicates whether it consumed the event or not. But if we're dealing with keyboard events at the OS level, that may not be reliable; plugins might just always say yes, I handled it, which is no good. If we introduce an NPAPI-level way of sending keyboard events to plugins, then people could be encouraged to make a careful decision as to whether to say the event was consumed or not. But maybe that's too much hassle and somebody just needs to make an executive decision that There Are Certain Key Events That The Browser Always Gets. Hopefully this is a stable list of key combinations; it would be more annoying if user prefs in the browser could cause new key combinations to not reach the plugin.

Johnny Stenback, Mozilla, March 2004

Original Proposal

Adding the following NPN function would allow plugins to pass events to the browser. A handled status is included because it encourages plugins to expose all events to the browser, even if they handled it, which allows for any overrides that might be deemed necessary.

NPBool NPN_HandleEvent(NPP instance, NPEvent *event, NPBool handled);

Comments

One problem with this system is that it would require an accurate platform-specific specification of how the browser is supposed to interpret NPEvent data, which is often a pointer to a native structure, and often a synthesized one.

This also lacks an accurate description of how focus flow would work.

Aaron Leventhal, IBM, April 2004

Original Proposal

Adding the following NPN function would allow plugins to pass events to the browser. A handled status is included because it encourages plugins to expose all events to the browser, even if they handled it, which allows for any overrides that might be deemed necessary.

NPBool NPN_HandleEvent(NPP instance, NPEvent *event, NPBool handled);

  • If the user is not focused on the last focusable child element in the plugin, and they hit tab, the plugin should focus its next child element but should not send the tab key to the browser with NPN_HandleEvent()
  • If the user is focused on the last focusable child element and they hit tab, the plugin should reset its internal focus memory and send the tab key to the browser with NPN_HandleEvent()
  • Reverse that logic for backward tab navigation with shift+tab
  • Important keys to let the browser handle, if at all possible: All Alt+letter keys, Ctrl+tab/Ctrl+Shift+tab (tabbed browsing, in general a common highler level navigation shortcut), Ctrl+L (highlight location bar), F6/Shift+F6 (navigate by pane)
  • If the plugin wants to give focus to the browser, it can create a focus event and send that up with NPN_HandleEvent()

The next problem to be solved is letting the plugin know how it received focus: Since we apparently don't have the ability to use event parameters when sending events to the plugin, I suggest we add 2 new event types to npapi.h.

 enum NPEventType {
   NPEventType_GetFocusEvent = (osEvt + 16),
   NPEventType_LoseFocusEvent,
+  // Fired immediately before NPEeventType_GetFocusEvent
+  NPEventType_NavigateForwardIntoPlugin,
+  // if the focus was gained via forward or backward tab navigation
+  NPEventType_NavigateBackwardIntoPlugin,   
   NPEventType_AdjustCursorEvent,
   NPEventType_MenuCommandEvent,
   NPEventType_ClippingChangedEvent,
   NPEventType_ScrollingBeginsEvent = 1000,
   NPEventType_ScrollingEndsEvent
 };

By handling the two new events, the plugin can decide where it wants to start tab navigation from when it receives focus. If the user tabs forward into the plugin, the first focusable child would typically be focused. If the user tabs backward, the last focusable child would most likely be focused.

Comments

Rudi Sherry:

  • If the user is focused on the last focusable child element and they hit tab, the plugin should reset its internal focus memory and send the tab key to the browser with NPN_HandleEvent()

I would prefer that the browser handle the resetting of the focus in and out of the plugin; that is:

  • If the user is focused on the last focusable child element [of the plugin] and they hit tab, the plugin should send the tab key to the browser with NPN_HandleEvent()...

... which would be handled by the browser. The browser will, if it can, move its focus from the plugin area to its next focusable element/child/plugin. Part of doing this would be to call the original plugin with a reset-focus event.

This change handles the case where the plugin is the last focusable element of the browser, so hitting tab from the last focusable child of the plugin would not leave the browser with no focus.

  • Reverse that logic for backward tab navigation with shift+tab

which, in this case, is the same logic -- send the tab key to the browser.

Josh Aas:

This doesn't cover the case of windowed plugins on Linux and Windows. The focus event enum in the spec adds cases for Carbon Mac OS X events.

Oliver Yeoh and Danielle Pham, Sun, September 2007

Original Proposal

NPAPI Changes to Handle Keyboard Interaction with Plugin

1. Variables

  • NPPVpluginSupportsFocusBool
    • Plugin side variable which must be set to true if it can be focused by keyboard navigation.
  • NPNVSupportsFocusBool
    • Browser side variable that plugins can query if tab focus integration is available.

2. Events

  • NPEventType_GetFocusFirstEvent (for forward tab focus traversal)
    • Sent from the browser to plugin when focus should be given to the plugin's first child widget in the plugin's tab order. Or, Sent from plugin to the browser when focus should be given to the first widget in the browser's tab order that is right after the plugin.
  • NPEventType_GetFocusLastEvent (for reverse tab focus traversal)
    • Sent from the browser to plugin when focus should be given to the plugin's last child widget in the plugin's tab order. Or, Sent from the plugin to the browser when focus should be given to the last child widget in the browser's tab order that is right before the plugin.

Plugin side changes

1. Handshake Tab Navigation Support Between Browser and Plugin

1.1. The plugin should return true when the browser queries its NPPVpluginSupportsFocusBool value.

1.2. The plugin should query the browser's NPNVSupportsFocusBool to make sure tab focus traversal handling is available on the browser side.

2. Handle Tab Focus Events Into Plugin

In handling of tab focus events, the browser will not send any NPEventType_GetFocusEvent (existing event type defined in NPAPI) to plugin.

The browser sends the GetFocusFirstEvent to the plugin when tabbing forward and GetFocusLastEvent when tabbing backward into the plugin. This can be done via NPP_HandleEvent(). The plugin should implement NPP_HandleEvent() to handle these events according to the plugin's internal focus system (which is likely platform dependent.)

The underlying keyboard event (tab or modifier+tab) will be dropped, because:

  • That tab-key event took place outside of the plugin.
  • Its meaning is conveyed in GetFocusFirstEvent or GetFocusLastEvent
  • Sending that tab-key event easily confuses the plugin into advancing its internal focus.

Once received the GetFocusFirstEvent or the GetFocusLastEvent, plugin should take over input focus. That is, the plugin should listen to the event loop and forward any unconsumed key-strokes to the browser until it reaches the end of its tab order or until the user mouse-click to steal focus away from the plugin.

XEmbed plugins do not need to handle these events. XEmbed plugins will receive XEMBED messages which should be automatically handled by the plugin's toolkit.

3. Handle Tabbing Out of the Plugin

When user tab navigates out of the plugin, e.g, when the plugin reaches the end of its tab order, the plugin should send either the GetFocusFirstEvent to the browser when tabbing forward and GetFocusLastEvent when tabbing backward out of the plugin. This can be done via NPN_HandleEvent(). The browser should implement NPN_HandleEvent() to handle these event according to the browsers internal focus system.

The underlying keyboard event (tab or modifier+tab) will be dropped, because:

  • That tab-key event took place within the plugin, not the browser.
  • Its meaning is now conveyed in GetFocusFirstEvent or GetFocusLastEvent
  • Sending that tab-key event easily confuses the browser into advancing its internal focus.

Once received the GetFocusFirstEvent or the GetFocusLastEvent, the browser should take over input focus. That is, the browser should listen to the event loop.

XEmbed plugins need not to do this as it's toolkit should be sending the required XEmbed messages when tabbing out of the plugin. However, XEmbed plugins should still forward other unconsumed keystrokes to the browser.

4. Handle Mouse Click Events

No change to how mouse click events are handled.

5. Forward Unused Keystrokes to the Browser

5.1. Windows

For windowed plugins, the plugin forwards unconsumed WM_KEYDOWN messages to the subclassed window procedure.

NPBool nsPluginInstance::init(NPWindow* aWindow)
{
  ....
  
  // Save the subclassed WNDPROC. We need it to forward unused keystrokes
  // back to the browser.
  mOldWndProc = SubclassWindow(mhWnd, (WNDPROC)PluginWinProc);
}

LRESULT CALLBACK PluginWinProc(HWND hWnd, UINT msg, WPARAM wParam, LPARAM lParam)
{
  switch(msg) {
    case WM_KEYDOWN:
      if (!IsKeyUseful(wParam) {
        // We don't need this key. Forward to browser.
        return mOldWndProc(hWnd, msg, wParam, lParam);
      }
      
      // We need this key. Consume it.
      ...
      return 0;
  }
}

For windowless plugins, the plugin should simply return false when processing the WM_KEYDOWN in the NPP_HandleEvent() function.

5.2. Linux/Unix For XEmbed plugins, unused keystrokes should be forwarded to the socket window.

NPError nsPluginInstance::SetWindow(NPWindow* aWindow)
{
  GtkPlug *plug = gtk_plug_new((XID)aWindow->window);
  
  g_signal_connect_after(plug, "key_press_event", G_CALLBACK(plug_key_press_event_cb), NULL);
  ...
}

gboolean plug_key_press_event_cb (GtkWidget *widget, GdkEventKey *event, gpointer user_data)
{
  // forward unconsumed keystrokes to embedder
  XEvent xevent;
  GtkPlug *plug = GTK_PLUG(widget);
  GdkScreen *screen = gdk_drawable_get_screen (plug->socket_window);

  xevent.xkey.type = KeyPress;
  xevent.xkey.window = GDK_WINDOW_XWINDOW(plug->socket_window);
  xevent.xkey.root = GDK_WINDOW_XWINDOW(gdk_screen_get_root_window(screen));
  xevent.xkey.subwindow = None;
  xevent.xkey.time = event->time;
  xevent.xkey.x = 0;
  xevent.xkey.y = 0;
  xevent.xkey.x_root = 0;
  xevent.xkey.y_root = 0;
  xevent.xkey.state = event->state;
  xevent.xkey.keycode = event->hardware_keycode;
  xevent.xkey.same_screen = True;

  XSendEvent (GDK_WINDOW_XDISPLAY (plug->socket_window),
        GDK_WINDOW_XWINDOW (plug->socket_window),
        False,
        NoEventMask,
        &xevent);

  gdk_display_sync(gdk_screen_get_display (screen));
  
  return TRUE; 
}

For X11 plugins, the unused keystrokes should be forwarded to the plugin's parent window.

void xt_event_handler(Widget xtwidget, nsPluginInstance *plugin, XEvent *xevent)
{
  switch (xevent->type) {
    case KeyPress:
      if (!IsUsefulKey(xevent->xkey)) {
        /* we don't need this key. forward it to the embedder. */
        Window parentWindow = 0, rootWindow = 0, *childWindows;
        unsigned int childCount;
        
        XQueryTree(mDisplay, mWindow, &rootWindow, &parentWindow, &childWindows, &childCount);
        XFree(childWindows);
        
        XEvent event;

        event.xkey.type = xevent->type;
        event.xkey.window = parentWindow;
        event.xkey.root = rootWindow;
        event.xkey.subwindow = None;
        event.xkey.time = xevent->xkey.time;
        event.xkey.x = 0;
        event.xkey.y = 0;
        event.xkey.x_root = 0;
        event.xkey.y_root = 0;
        event.xkey.state = xevent->xkey.state;
        event.xkey.keycode = xevent->xkey.keycode;
        event.xkey.same_screen = True;

        XSendEvent(mDisplay, parentWindow, False, NoEventMask, &event);
        return;
      }
      ...
      break;
  }
}

5.3. Mac

The plugin should simply return false when processing the keyDown event in the NPP_HandleEvent() function.

Comments

How exactly is the browser supposed to send GetFocusFirstEvent/GetFocusLastEvent events to the plugin via existing NPP_HandleEvent. NPEvent data is different on each platform, it isn't clear how that would be conveyed when, for example, the type for NPEvent is "void*", a pointer to a native event structure.