UI Handler

Implementing UI Handlers

This chapter covers the UI Handler components of the universAAL platform. Each UI Handler controls one or more UI channels and thus enables explicit interaction between the system and its users. Consequently, UI Handlers have a huge impact on the way that users perceive the system and the design and implementation of new UI Handlers can be a rewarding task, as sophisticated variants have a good chance of becoming popular.

About UI Handlers

Chapter eight has already introduced the UI Bus, which is part of the platform's "UI Framework". The other two parts of the UI Framework are the Dialog Manager component and the UI Handlers. The purpose of the UI Framework is the realization of a "separation of the application layer and the presentation layer", which means that application developers will not implement actual user interfaces for their applications (like they would normally do when programming for a PC or a mobile device). They rather formulate an abstract representation of the "dialog" (the communication cycle) in question within their application, forward this information as a so-called "UI request" to the UI Bus, and then leave it to the platform's UI Framework to decide for a representational option among those that are available (and feasible). This allows the system to adapt much better to the specific user's preferences and the situation that he or she is currently in. A user lying on the bed, for example, may have his eyes closed and would thus not be able to realize the notifications that are shown on the TV screen and his smartphone's display. In this situation, it would be much better to make use of the some devices' loudspeakers for passing the information (if it is important enough for possibly waking the user up, of course). The smartphone's microphone could also be used to capture the user's reaction, if such is required.

To facilitate this kind of system behavior, applications leave it to the UI Bus to choose an appropriate UI Handler for the presentation of the dialog (after the Dialog Manager component has applied modifications to the UI request to reflect the user's personal preference settings, such as a language selection or the information that the user suffers from hearing impairment). The UI Bus is capable of doing this because all UI Handlers have to register themselves to the UI Bus with a specification of their capabilities. The chosen UI Handler receives the UI request from the UI Bus, presents the dialog to the user, captures the user's feedback (if any) and forwards this as an "UI response" back to the UI Bus. The UI response is marked with a specific DialogID to let the UI Bus know, who the recipient of the UI response is. Obviously, there must be at least one UI Handler available in the system as otherwise, the system will not be able to interact with the user at all (the UI Bus will have no options for choosing an interaction modality). Additional UI Handlers, however, improve the system's interaction capabilities by giving the UI Bus more options to chose from. This makes the implementation of new UI Handlers a core activity field of universAAL's developer community.

An exemplary UI Handler

The implementation of a new UI Handler component has two aspects. On the one hand, the UI Handler must be able to communicate with the rest of the platform, more specifically the UI Bus. On the other hand, the UI Handler needs to be able to convert the abstract dialog description that it receives from the sending application (by way of the UI Bus) into a data format that can be processed by the devices that the UI Handler is talking to (e.g., displays/keyboards and/or loudspeakers/microphones). In this section we will try to sketch out how you can register your component to the UI Bus, what to do with the UI requests that your component receives and how to send UI responses back to the bus. We will leave the problem of how to convert the information that is contained within the UI request (and also how to communicate with the technical devices that your UI Handler is utilizing) to your own programming expertise. This section is based on the predefined GUI UI Handler that is part of the platform's UI Framework (you need to join the UI Expert Group if you want access to the corresponding sources). We will use this as a basis to explain the required steps for the implementation of a custom UI Handler.

The GUI UI Handler consists of about a dozen classes. Two of these are of interest to us, these are the classes GUIIOHandler and MyOutputSubscriber (note that the classes' names still reflect the original separation of the UI Bus in two different buses as described in the first chapter). The GUIIOHandler class is the "main" class of the UI Handler. An instance of this class creates and manages an instance of MyOutputSubscriber, which handles the UI requests that are being forwarded to this UI Handler by the UI Bus. We will begin with an analysis of GUIIOHandler, and then proceed with a discussion of MyOutputSubscriber.

1.01: package org.universAAL.ui.handler.gui;
1.02: 
1.03: import org.slf4j.Logger;
1.04: import org.slf4j.LoggerFactory;
1.05: import org.universAAL.middleware.container.ModuleContext;
1.06: import org.universAAL.middleware.owl.Restriction;
1.07: import org.universAAL.middleware.ui.UIHandlerProfile;
1.08: import org.universAAL.middleware.ui.UIRequest;
1.09: import org.universAAL.middleware.ui.UIResponse;
1.10: import org.universAAL.middleware.ui.owl.Modality;
1.11: import org.universAAL.middleware.ui.rdf.Submit;
1.12: 
1.13: public class GUIIOHandler {
1.14:   
1.15:    private final static Logger log = LoggerFactory.getLogger(GUIIOHandler.class);
1.16:
1.17:    private static final String GUI_NAMESPACE = "http://gui.io.persona.ima.igd.fhg.de/GuiHandler.owl#";
1.18:  
1.19:    private static final String OUTPUT_LIST_OF_USERS = GUI_NAMESPACE + "listOfUsers";
1.20:
1.21:    private MyOutputSubscriber os = null;
1.22:
1.23:    public GUIIOHandler(ModuleContext context) {
1.24:     super();
1.25:     os = new MyOutputSubscriber(context, getOutputSubscriptionParams(), this);
1.26:     Login login = new Login(context, os);
1.27:    }
1.28:
1.29:    public void dialogFinished(Submit s) {
1.30:     Object o = s.getFormObject().getProperty(UIRequest.MY_URI);
1.31:     if (o instanceof UIRequest) {
1.32:       os.dialogFinished(new UIResponse(
1.33:           ((UIRequest) o).getAddressedUser(), ((UIRequest) o).getPresentationLocation(), s));
1.34:       } else {
1.35:        synchronized (os) {
1.36:       UIResponse ie = new UIResponse(os.currentUIRequest.getAddressedUser(), os.currentUIRequest.getPresentationLocation(), s);
1.37:       if (s.getDialogID().equals(os.dialogID))
1.38:           os.currentUIRequest = null;
1.39:       os.dialogFinished(ie);
1.40:       }
1.41:     }
1.42:   }
1.43:
1.44:    private UIHandlerProfile getOutputSubscriptionParams() {
1.45:     UIHandlerProfile oep = new UIHandlerProfile();
1.46:     oep.addRestriction(Restriction.getFixedValueRestriction(UIRequest.PROP_PRESENTATION_MODALITY, Modality.gui));
1.47:     return oep;
1.48:    }
1.49:
1.50:  }

In line 1.21, we create a new variable of our custom class MyOutputSubscriber, which is instantiated in the constructor in line 1.25. This object is used for collecting the UI requests that are send by applications and forwarded to this UI Handler by the UI Bus. The class has two methods, a public method called dialogFinished() (lines 1.29 to 1.42) and a private method called getOutputSubscriberParams() (lines 1.44 to 1.48).

The private method getOutputSubscriberParams() creates and returns an object of type UIHandlerProfile, which is needed for the construction of the MyOutputSubscriber object. As its name implies, the UIHandlerProfile is used for defining the handler's profile and describes its area of application. In line 1.46 we state that this handler is capable of processing requests that require a GUI modality (other supported modalities would be voice, gesture or SMS). Note that this is not the only type of property that can be used to describe the handler's capabilities. We have already predefined a wide variety of other properties like whether the handler is suited for interacting with users that have certain impairments or whether the handler will maintain privacy (think of a voice-based handler).

The public method dialogFinished(Submit s) is called when the user has finished his/her dialog related input to the system. In this specific case the method is called by a Swing interface controller object when the user clicks a specific button, thus confirming that the input is complete. The method's purpose is to create the UI response (an object of type UIResponse) that is send back to the calling application, containing the user's input.

2.01: package org.universAAL.ui.handler.gui;
2.02: 
2.03: import org.universAAL.middleware.container.ModuleContext;
2.04: import org.universAAL.middleware.rdf.Resource;
2.05: import org.universAAL.middleware.ui.UIHandler;
2.06: import org.universAAL.middleware.ui.UIHandlerProfile;
2.07: import org.universAAL.middleware.ui.UIRequest;
2.08: import org.universAAL.middleware.ui.rdf.Form;
2.09: 
2.10: public class MyOutputSubscriber extends UIHandler {
2.11: 
2.12:  protected UIRequest currentUIRequest = null;
2.13: 
2.14:  protected String dialogID = null;
2.15:
2.16:  protected SwingRenderer renderer = null;
2.17:
2.18:  protected MyOutputSubscriber(ModuleContext context, UIHandlerProfile initialSubscription, GUIIOHandler guiHandler) {
2.19:    super(context, initialSubscription);
2.20:    renderer = new SwingRenderer(guiHandler);
2.21:  }
2.22:
2.23:  public void adaptationParametersChanged(String dialogID, String changedProp, Object newVal) {
2.24:   synchronized (this) {
2.25:       if (dialogID.equals(this.dialogID)) {
2.26:       if (UIRequest.PROP_SCREEN_RESOLUTION_MAX_X.equals(changedProp)
2.27:           && newVal instanceof Integer
2.28:           && ((Integer) newVal).intValue() != currentUIRequest
2.29:               .getScreenResolutionMaxX()) {           
2.30:           renderer.updateScreenResolution(((Integer) newVal)
2.31:               .intValue(), -1, -1, -1);
2.32:           currentUIRequest.setScreenResolutionMaxX(((Integer) newVal)
2.33:               .intValue());
2.34:       } else if (UIRequest.PROP_SCREEN_RESOLUTION_MAX_Y
2.35:           .equals(changedProp)
2.36:           && newVal instanceof Integer
2.37:           && ((Integer) newVal).intValue() != currentUIRequest
2.38:               .getScreenResolutionMaxY()) {           
2.39:           renderer.updateScreenResolution(-1, ((Integer) newVal)
2.40:               .intValue(), -1, -1);
2.41:           currentUIRequest.setScreenResolutionMaxY(((Integer) newVal)
2.42:               .intValue());
2.43:       } else if (UIRequest.PROP_SCREEN_RESOLUTION_MIN_X
2.44:           .equals(changedProp)
2.45:           && newVal instanceof Integer
2.46:           && ((Integer) newVal).intValue() != currentUIRequest
2.47:               .getScreenResolutionMinX()) {           
2.48:           renderer.updateScreenResolution(-1, -1, ((Integer) newVal)
2.49:               .intValue(), -1);
2.50:           currentUIRequest.setScreenResolutionMinX(((Integer) newVal)
2.51:               .intValue());
2.52:       } else if (UIRequest.PROP_SCREEN_RESOLUTION_MIN_Y
2.53:           .equals(changedProp)
2.54:           && newVal instanceof Integer
2.55:           && ((Integer) newVal).intValue() != currentUIRequest
2.56:               .getScreenResolutionMinY()) {           
2.57:           renderer.updateScreenResolution(-1, -1, -1,
2.58:               ((Integer) newVal).intValue());
2.59:           currentUIRequest.setScreenResolutionMinX(((Integer) newVal)
2.60:               .intValue());
2.61:         }
2.62:       }
2.63:     }
2.64:   }
2.65:
2.66:  public void communicationChannelBroken() {
2.67:   // TODO Auto-generated method stub
2.68:  }
2.69:
2.70:  public Resource cutDialog(String dialogID) {
2.71:   synchronized (this) {
2.72:       if (dialogID.equals(this.dialogID)) {
2.73:       renderer.finish();
2.74:       Resource data = currentUIRequest.getDialogForm().getData();
2.75:       currentUIRequest = null;
2.76:       return data;
2.77:       } else
2.78:       return null;
2.79:   }
2.80:  }
2.81:
2.82:  public void handleUICall(UIRequest event) {
2.83:   Form f = event.getDialogForm();
2.84:   synchronized (this) {
2.85:       if (f.isMessage() && currentUIRequest != null) {
2.86:       f.setProperty(UIRequest.MY_URI, event);
2.87:       renderer.popMessage(f);
2.88:       } else {
2.89:       if (currentUIRequest != null)
2.90:           renderer.finish();
2.91:       currentUIRequest = event;
2.92:       dialogID = f.getDialogID();
2.93:       renderer.renderForm(f);
2.94:        }
2.95:      }
2.96:    }
2.97:
2.98: }

In line 2.10 you can see that the class MyOutputSubscriber extends the abstract class UIHandler. Only instances of a subclass of UIHandler can subscribe to the UI Bus for the receival of UI requests. MyOutputSubscriber implements the four abstract methods predefined by UIHandler, which are adaptionsParameterChanged() (lines 2.23 to 2.64), communicationChannelBroken() (lines 2.66 to 2.68, just a stub here), cutDialog() (lines 2.70 to 2.80) and handleUICall() (lines 2.82 to 2.96). In the constructor of MyOutputSubscriber, we call the constructor of UIHandler, passing it the ModuleContext and the UIHandlerProfile object that we created in the constructor of GUIIOHandler. The constructor method of UIHandler makes use of this profile to register our custom UI Handler to the UI Bus. Note that the constructor also creates a Swing controller object which will take care of handling the actual user interface.

The most important method in the class is handleUICall(). It gets called by the UI Bus when it forwards a new UI request to our UI Handler. We first need to extract the relevant dialog information from the received request (line 2.83) and then pass this Form object to our handler's renderer, in this case a Swing controller object (lines 2.87 and 2.93). Note that we made the actual procedure depend on whether the type of dialog is a Message. As you know from reading through chapter eight, this is the simplest of the four dialog types, because Messages do not require input from the user (besides his acknowledgement that he has received the Message, as can be achieved through a click on an OK-button). Because of this, Messages can be rendered even though the user is actually in the middle of another dialog (lines 2.85 to 2.87).

Moving on to the method cutDialog(), this method is required for aborting a dialog which is currently in progress (in other words a dialog that has been started but that has not yet received all the required input from the user). There is two things that need to be done here: we need to tell our renderer to stop its rendering work (line 2.73) and also need to preserve any data the user may have already provided (lines 2.74 to 2.76).

The method adaptationParametersChanged() is originally called by the Dialog Manager component when it notices that relevant parameters for a running dialog have changed. The Dialog Manager then informs the UI Bus about this, which has either the option of notifying the responsible UI Handler, calling this method, or of selecting another UI Handler instead, aborting the running dialog (using the cutDialog() method) and forwarding the acquired data from the aborted dialog to the new one.