Generating Interactive Prototypes from Query Annotated Discourse Models

1 Introduction

Model Based User Interface Development (MBUID) provides a method of developing user interfaces by working with higher-level interaction models. Such models are used to create interfaces that can be easily adapted to various context of use (i. e. different platforms, modalities and environments). Higher-level models can be transformed and adapted automatically to different contexts of use, requiring less time and ultimately driving down the cost of development process.

By using MBUID approaches the domain experts and other stakeholders not skilled at programming can be involved in the development process as there is an indication that they can understand higher-level models more easily than they would understand software code. For this reason MBUID approaches and tools are used in close cooperation with domain experts and are considered to be part of user-centered design process (Træ tteberg 2008).

Our aim is to support user-centered design processes. Therefore, allowing the domain experts to prototype by using MBUID is an important goal. By prototyping we mean rapid, iterative production of design alternatives with tools that allow the domain experts to focus on the model-generated interactive concepts rather than having to think about programming details (Beaudouin-Lafon and Mackay 2003).

In using currently existing MBUID approaches for prototyping, the domain experts could encounter a number of potential difficulties: Firstly even if they have composed a high-level interaction model and generated user interfaces from them, there may be other models (and software code) necessary before they can run and test the generated interfaces (García Frey et al. 2012). Secondly the domain experts may not understand the mechanisms of transformation from the high-level model they have made to the generated interfaces, and may thus experience frustration due to lack of control (Myers, Hudson and Pausch 2000). Thirdly even if the domain experts are allowed to modify the generated interfaces, such changes may be lost when the high-level model is modified, and the user interfaces are re-generated (the “round trip” problem) (Mann and Thompson 1988).

In this paper we address these challenges by introducing Query Annotations to a discourse-based MBUID approach. Query Annotations present familiar information to domain experts because they are formulated in SQL-like syntax that resembles natural language and that uses elements in the query definitions that originate from the domain. Furthermore, we will show how Query Annotations can be used in higher-level discourse models for data-binding (i. e. to connect interaction models to the domain models) and how they can be propagated all the way to the generated user interface. As such they can present to the domain experts a mapping between the higher level models and the generated interfaces, thereby facilitating an understanding of the generation mechanisms.

The most important purpose of applying Query Annotations to discourse-based modeling approach is to generate the basic CRUD^1[1] business logic code from such annotations. The result will be automatically generated, fully interactive prototypes that are ready to be tested by the domain experts.

Discourse-based modeling (Falb et al. 2006) has been chosen as the underlying MBUID approach over the more widespread task-based modeling (Paterno 2013) because it represents a more self-contained approach in regard to its models and tools. Task-based modeling consists of various models and appropriate tools that need to be orchestrated together to achieve the full path transformation from highest level models to final user interfaces. Having to go through a number of steps and modeling languages in order to complete an iteration cycle is counter-productive for a prototyping-oriented process. Our choice for the discourse-based MBUID is further supported by the recent arguments in regard to duality of task- and discourse-based approaches (Popp 2014).

The remainder of this paper is structured as follows: Section 2 describes the work related to our goals. The solution of Discourse Modeling combined with Query Annotations is introduced in Section 3. The use of the combination is illustrated by means of modeling an on-line shop application in Section 4. Section 5 demonstrates how a fully interactive prototype application is generated and how the domain experts can modify this application. Section 6 presents conclusion and future work.

2 Related Work

The authoritative framework for conceptualizing userinterface modeling research is the Cameleon Reference Framework (Calvary et al. 2003), which defines models at various levels of abstraction. Most Model Based User Interface Development approaches use the task-based interaction models with the primary example of approaches being MARIA (Paternò, Santoro and Spano 2009) and UsiXML (Limbourg et al. 2005). An alternative to task-based approaches are discourse-based approaches (Falb et al. 2006, Popp, Raneburger and Kaindl 2013), in which high-level concepts are modeled according to the principles of communication instead of tasks.

There is a duality of task-based and discourse-based models (Popp et al. 2014) in the sense that they can be used to specify the same type of interaction, but through different modeling concepts. Task-based models focus on the tasks that the user and the system need to accomplish, while the discourse-based models focus on the communication between the user and the system. Earlier work (Helmut Horacek et al. 2011) has also looked closely into the differences between the task- and discourse-based approaches. The more widespread task-based approaches have a large variety of tools available to work with models at various levels of abstraction. However, they require more orchestration of various tools and transformation models to go from higher-level models to the final user interface (García Fre et al. 2012, Limbourg and Vanderdonckt 2009). Discoursebased models, on the other hand, provide a fully automated process of generating final user interfaces (excluding business logic) from high-level interaction models.

Discourse-models have also been used to facilitate prototyping by supporting the exploration of design alternatives (Raneburger et al. 2014). In task-based modeling most attempts at prototyping have been done in the form of transforming early stage prototypes (sketches) into task-based models (Coyette et al. 2004). Both task-based and discourse-based prototyping approaches, however, require the existence of business logic (often in the form of Web services) before the full interaction of the interfaces with real data can be tested. This points to an overall tension between the exploratory nature of prototyping and the formal nature of model-based development (Træ tteberg 2008).

Data-binding (i. e. linking the user interfaces with the business logic) in discourse-based approaches has so far been achieved through Action-Notification Model (Popp, Kaindl and Raneburger 2013). Action-Notification Model is built on top of the Model-View-Controller (MVC) pattern and supports any business logic for which the appropriate adapter is created. Currently there is no automatic way of generating the business logic from Action-Notification Model although authors of (Popp, Kaindl and Raneburger 2013) indicate that it could in theory be possible. Task-based approaches mostly focus on connecting the task models to annotated web-services (Paternò, Santoro and Spano 2010). A notable exception is (Tran et al. 2009) where the authors use UsiXML to generate user interface and corresponding code for generic database functions. However, the code generation process is complicated, with many transformations, and includes several steps where user (developer) intervention is needed. Query Annotations (Kis and Bogdan 2013), that are used in this paper, also create the needed business logic code for database access. However, instead of mapping only to simple database functions, Query Annotations allow the domain experts to express complex queries.

Metawidgets (Kennard and Leaney 2010) allows retrofitting UI generation into existing applications by generating only database access widgets instead of whole applications. WebRatio (Brambilla 2008), a web modeling tool used in industry, provides support for modeling web interfaces and for generating the underlying functional core (or linking to the existing one) from the higher-level models. However, WebRatio and other modeling tools used in industry are targeting a single platform (e. g. web or mobile) and do not share the goals of addressing multiple contexts of use, that the MBUID approaches set.

Addressing the round-trip problem has been an ongoing challenge in the MBUID research. There is an extensive body of work summarized in Sanchez Ramon et al. (2013) on reverse engineering approaches that are capable to generate higher-level models from the final interfaces or lower level models and resources. However, these approaches work from a concrete level to an abstract level only, thus overriding the abstract level model based on a concrete model. The challenge of the full round-trip problem is to synchronize modifications at the lower-level models with the information present in the higher-level. The round-trip problem has been fully addressed by FLERP (Hennig et al. 2011), where the authors propose a semi-automatic solution for tackling various mapping problems when moving between abstract and concrete UI models. At the Concrete UI level FLERP uses the CAP3 (Bergh, Luyten and Coninx 2011) model to define the platform independent UI. Based on the original Canonical Abstract Prototypes (Constantine 2003), CAP3 has similarities to the approach proposed here since both give a graphical, although generic, representation of UI components, and both provide visual cues to the link between these components and the domain model elements. However, the generic representation has its limitations in giving the domain experts the understanding of the resulting UI look. As demonstrated in their research, the approach is more suitable for critical systems where proper functionality is much more important than the UI design.

3 Query Annotated Discourse Modeling

We will introduce Query Annotated Discourse Modeling by way of the diagrams in Figure 1. The diagrams represent previous Query Annotations (top) and Discourse Modeling (middle) approaches, as well as the solution, proposed in this paper, in which Query Annotations are applied to Discourse Modeling (bottom). The processes are explained in the context of the Cameleon Reference Framework (CRF) that defines four abstraction levels in MBUID: high-level Task & Concepts, modality independent Abstract User Interface (AUI), platform independent Concrete User Interface (CUI) and Final User Interface (FUI) at the lowest level.

Figure 1

The modeling process in the low-level Query Annotation approach (top), the Discourse Modeling approach (middle), and the Query Annotated Discourse Modeling (bottom).

3.1 Query Annotations

Currently Query Annotations are employed in a low-level modeling process, where they annotate concrete UI models (Kis and Bogdan 2013) and provide data-binding. The Template User Interface consists of concrete user interface components that are annotated with queries as shown in figure 2. Query Annotations automatically generate the program code for database access and modification (the CRUD operations), which results in an interactive prototype. Furthermore, they are also optionally shown in the generated user interface. The presence of Query Annotations both in the Template User Interface and in the generated one enables the domain experts to develop the mapping between the action of changing the queries and the resulted modification in the generated UIs.

Figure 2

Query Annotated Template UI, previously composed manually of concrete UI components and annotated with queries. Such interfaces will be automatically generated through the modeling process presented in this paper. Query annotations (in red) indicate the domain information that should be present in the final user interface, and serve for generating business logic that makes the final user interface functional.

Query Annotations are used in real world applications (Bogdan and Mayer 2009) where the use of queries allowed the non-programmers (domain experts) to continuously modify the user interfaces by changing the queries which are understandable to the domain experts and represent the domain knowledge. This enabled the domain experts to quickly see the result of their low-level modeling work and supported an iterative prototyping process, without the need for involving developers.

The main limitation of Query Annotations is that they have so far been present only at the low-level models. Therefore, if the user interfaces for different contexts of use (i. e. different platforms, modalities and environments) were to be prototyped, a different Template Interface for each context would need to be manually created and annotated with queries.

4 Modeling Example

In order to illustrate the Query Annotated Discourse Modeling approach introduced in this paper, we will use an example of a simple on-line shop interactive system. In the system the user should see existing product categories and, when a category is selected, the user should see the list of products in that category (provided that the category contains products). Next, the user should be able to add the selected products to a list (shopping cart). Finally, the user may decide to proceed to checkout, where the selected products are shown and the user needs to fill in the credit card information for the bill to be generated.

The domain experts start modeling the on-line shop system by defining the domain model for the three entities (ProductCategory, Product, CreditCard) shown in Figure 3. The domain model defines the Product entity that has a reference to ProductCategory. There is also a Bill entity that has a one-to-many reference to Products (i. e. the products that the user decides to buy) and a reference to the CreditCard entity with details (attributes) of the user’s credit card.

Figure 3

Domain model of the on-line shop example.

4.1 Discourse Modeling: Communicative Acts

The next step is to model the communication on the high level using the discourse-based modeling approach. We will illustrate this approach by examining the Discourse Model of the on-line shop example shown in Figure 4. All the constructs used in the model are listed in Table 1. Discourse Models define a high-level communication between a user and a software system by representing the communication as a discourse according to the principles of human communication. The basic elements of discourse are Communicative Acts (CAs), derived from the Speech-Act-Theory (Searle 1969), used to describe utterances (question, answer, informing, etc.) in the discourse. Such CAs are used to describe utterances either from the system (providing information, asking for input) or utterances made by the user.

Figure 4

Discourse Model for on-line shop example showing CommunicativeActs (leaves), AdjacencyPairs (diamond nodes), RSTRelations (square nodes), conditionalstatements (tree branch annotations), and QueryAnnotations (inside the tree nodes). The meaning of each construct is explained in Table 1.

Table 1

Modeling constructs of the Query Annotated Discourse Model.

Construct	Description	Representation
Communicative Acts (leaves)	Describe discourse utterances (e. g. question, answer, information). The green (darker) ones indicate discourse uttered by the system (i. e. prompting the user for input or displaying information) while yellow (lighter) indicate discourse uttered by the user (i. e. providing input).
Adjacency Pairs (diamond nodes)	Group together Communicative Acts indicating turn-taking in the discourse. For example, one such pair is asking a question and providing an answer.
RST Relations (square nodes)	Define the overall relations of elements of the discourse in a hierarchical tree structure. For example, Alternative relation indicates parts of discourse that are mutually exclusive. When two branches of a relation are of different importance, the more important one is called Nucleus and the less important one Satellite. For example, Elaborate indicates additional details that the Satellite branch provides for the Nucleus. In our example this is selecting a cat- egory (Nucleus) and a product from that category (Satellite).
Conditional statements (tree branch annotations)	Further extend RST Relations to enable or disable parts of the discourse tree depending on the condition that is typically related to the domain information.
Query Annotations (inside the tree nodes)	Represent propositional content of the discourse or, in other words, define which domain information is present in communicative acts ut- tered by the system or the user.

In Figure 4 CAs are represented as the leaf elements of the tree. The user action of selecting the product category is modeled by a SelectProductCategory ClosedQuestion CA, uttered by the system, and by an Answer to that question, uttered by the user. The leftmost Informing CA is uttered by the user and it serves to inform the system to proceed to checkout. The CAs that represent turn-taking in the discourse (e. g. question-answer) are paired according to the Conversation Analysis (Luff, Gilbert and Frohlich 1990) in Adjacency Pairs (diamonds in Figure 4).

5 Automatic Generation of Interactive Prototype

The first step in operationalizing the Discourse Model is to generate a Behavioral Model at the Abstract UI level. This step involves creation of the Partitioning State Machine (PSTM), shown in Figure 5. The PSTM consists of three possible states: selection of product category (S1), list of products in the selected category (S2), and checkout (S3). Each state corresponds to the Discourse Model subtree and represents one of the screens of the UI. The process of the PSTM generation is described in detail in Kis et al. (2014).

Figure 5

Partitioning State Machine generated according to Kis et al. (2014) for the on-line shop Discourse Model. S1 – selection of product category (no category is selected, or the selected one has no products); S2 – showing products from the selected category and adding them to the shopping cart; S3 – checkout, showing the products in the shopping cart and selecting to enter credit card information.

The Concrete User Interface (CUI) model is generated by applying transformation rules on the Discourse Model. As described in Kavaldjian, Falb and Kaindl (2009), the transformation rules match the communication elements of the Discourse Model (and their propositional content) with CUI templates. In the on-line shop example, the ClosedQuestion named SelectProduct and its Answer tagged AddToCart trigger the rule shown in Figure 6. When all the rule matching is done, the CUI Model is complete.

Figure 6

The Discourse Model to CUI transformation rule, part of work and toolbox by Kavaldjian, Falb and Kaindl (2009) (left), rule-triggering adjacency pair (top right), and the generated query-annotated Template UI fragment (bottom right).

During the CUI generation process, the Query Annotations from the Discourse Model are copied to the generated Graphical UI (GUI) widgets. Figure 6 shows, at the bottom right, the widgets generated from the associated CUI template and their Query Annotations. The container element in the generated widgets is annotated with the query (FROM Product p WHERE p.category = pc), while the child elements get the respective query projections (p.<attribute>. The widgets generated from the transformation rule template element and marked with IdOrFirst, are annotated with p.name), while the widgets generated from the transformation rule element and marked with RestAttributes are annotated with the query expressions p.price and p.description. Finally, the action ADD TO $productsInCart of the AddToCart Answer is assigned to the button underneath the Product Choice. The transformation rule keeps a connection to the original adjacency pair, thus our interpreter can detect that the ADD TO button action refers to the element selected in the Product Choice. This relation between the button and the chooser could also be made visible to the domain experts.

The top of Figure 7 shows the screen that corresponds to the state S2 rendered in Java Swing and that is obtained from the CUI model.^2[2] In the approach described in Kis and Bogdan (2013) this interface is called a Template Interface. The information populating the interface is only example data, as the queries have not yet been executed. The example data is obtained from the domain model definition. Such a Template Interface gives the domain experts a quick and intuitive preview of the UI, while still not being at the FUI level.

Figure 7

The Template User Interface for S1 (top left) and S2 (top right) with respective Query Annotations (bottom).

As part of the toolset presented to the domain experts, at the top of each screen there are two check boxes, which give the domain experts an option to further explore the generated interface. The annotate check box displays the Query Annotations as shown at the bottom of Figure 7. Displaying the annotations directly in the Template User Interface provides the domain experts with a better understanding of the connection between the high-level Discourse Model, the Domain Model, and the generated interface.

5.1 Query Execution and UI Population With Data. Testable UI

The second option in the toolset is the data option. This option will trigger the execution of the queries annotating the user interface and will populate the interface with data from the data-source. For each result of the executed query the child elements of the container that the query annotates are duplicated and, if needed, populated with data. The result for the S2 screen data population is shown in Figure 8, e. g. for each Product or ProductCategory the radio button and its respective labels are duplicated and populated with data.

Figure 8

The Template User Interface for S2 filled with actual data (top) and with Query Annotations (bottom).

Switching between the product categories shonw on the left of the Figure 8 (and pressing the Select button) triggers the query execution again and the interface is populated with new data. Finally, when the Proceed To Checkout button is pressed, the domain experts see the S3 screen shown at the right in Figure 9. Continuing in data mode, the domain experts can fill the credit card information and press the Submit button, which will create a new Bill and CreditCard objects in the data-source. The domain experts could also choose to switch to annotated, non-data mode (Figure 9 left) to examine only the query annotations at CUI level.

Figure 9

The Template Interface for S3 with Query Annotations (left) and data (right).

As our focus is on prototyping by domain experts, calling the application logic that actually does the purchase is beyond the scope of this paper, and is not strictly needed to test the generated UI. What is important for us is that the domain experts are able to navigate through the whole interface and test it without having to work at the application logic level where they might not have the necessary competence.

5.2 Domain Experts Work With Transformation Rules

We have shown how, in our implemented proof of concept, the domain expert can explore the generated user interface in several modes: the template (Figure 7 top), the query-annotated template (Figure 7 bottom), the data-populated interface (Figure 8 top, showing the FUI), and the query-annotated FUI (Figure 8 bottom). As shown in Kis and Bogdan (2013), the domain experts could use an interface builder to work with such query annotated UIs, but such work would be lost if the CUI is re-generated from the Discourse Model in the way we have illustrated here.

To address this, we elaborated conceptually on queryannotated transformation rules that can be worked upon by the domain experts, and which can package their work for future use. To illustrate that, the left part of Figure 10 shows a domain expert-oriented representation of a variant of the transformation rule in Figure 6 that is created specifically for adding to sets. The adjacency pair has a ClosedQuestion annotated with FROM <T> o WHERE … (meaning objects of a parametrized type T) and an Answer annotated with ADD TO <set of T>. The generated UI is represented in the query-annotated template form, showing the o.name expression for each result of the query FROM <T> o WHERE…. It should be noted that in this case the rule specifies that only the name attribute of the T objects will be shown.

Figure 10

Domain expert-oriented representation of an addition transformation rule.

The domain experts can work on this transformation rule to change the generated layout by, for example, using a list box instead of radio buttons, or by showing fields other than name for various possible values of T, etc. Furthermore, the domain experts can create a completely new CUI, matching the same discourse subtree with, for example, the interaction approach shown in Figure 11. Figure 11 displays the possible candidates at the left – using a list widget rather than using radio buttons – and the constructed set at the right, with an Add button in between. This new rule can also define that the button label is a left-to-right arrow. To show that the ADD TO action of the button refers to the choice made in the left list, a dotted line can show their connection, dictated by the adjacency pair.

Figure 11

CUI side of an alternative “Add-to-set” transformation rule.

In a modeling toolset that involves the domain experts at the transformation rule level, the transformation rules in Figures 10 and 11 will be offered as alternatives, since they are triggered by the same discourse model subtree (Figure 10 left). Furthermore, the rules can be shown both in the formal, type-parametric form (using <T> and <set of T>), and in the actual form (using Product and $productsInCart) to ease the domain experts’ work.

6 Conclusions and Future Work

The possibility to include designers, domain experts, and even end users in the development is an important factor in the user-centered design processes. Enabling early exploration of readily testable interfaces and iterating quickly over their variations helps bringing non-programmers early into the development process when prototyping is needed. However, most Model Based UI Development approaches require a significant understanding of various models and transformations and often even require writing software code before such testable interface can be reached. This increases the time between the iterations and hinders the non-programmers’ understanding of the transformation process thereby slowing down and discouraging prototyping.

This paper addresses these issues by modifying an existing MBUID approach. The inspiration for using Query Annotations on top of the existing approach comes from the previous results showing that queries specified for data-binding are easy to understand by the domain experts. These results described by Bogdan and Mayer (2009) show a decade of experience with non-programmers producing query annotated GUIs. The use of Query Annotations also motivated the choice of Discourse Modeling as the underlying MBUID approach for the proof of concept presented here.

By employing the queries to generate a simple databinding application logic we have addressed the issue of multiple models and implementation language code that is needed before the domain experts can get to test their generated UI. Even if this application logic does not make it to the final application, it is still very useful for the domain experts exploring and judging the qualities of the generated user interface. This enables the domain experts to focus on the results of the various model changes at various levels, rather than having to wait for a programmer to produce the necessary application logic.

The issue of domain expert understanding of the UI generation mechanisms has been addressed at two levels. Firstly, the Query Annotation present in the generated UI enables the domain experts to understand the internal mechanisms of the generated user interface, which helps them understand the transformation from a model to a UI. The query annotations also help to establish a connection between Task & Concepts and CUI levels. Secondly, the transformation rules can be better represented by using the Query Annotations, in a “language” that the domain experts already understand from their exploration of the Query-annotated models.

Finally, in our conceptual exploration we address the round trip issue by letting the domain experts change the Template User Interface and, at the same time, extract changes of the transformation rules involved. Furthermore, we allow the domain experts to propose alternative partial Template UIs for a certain rule matching the Discourse Model subtree, thereby providing for the creation of new transformation rules. By saving the domain experts’ work at the Template UI level as transformation rules, we ensure that the domain experts’ work will survive changes of the Discourse Model that do not break the matching Discourse Model subtree pattern. If a the pattern is broken, it may be natural that its generated partial Template UI is no longer needed. If it is still needed, the domain experts can retrieve it from the saved transformation rule. The domain experts can thus keep a repertoire of transformation rules and share transformation rules with other domain experts.

In future work we plan to apply the same principles of Query Annotation to other MBUID approaches, especially the commonly used ones based on Task Models. Our initial estimate is that the underlying Query Annotation principles can equally be applied to other MBUID approaches as long as their UI models have a hierarchical structure, which is the case in most approaches. Therefore the potential difficulties of applying Query Annotations to other approaches mostly depend on the adaptability of the implementation and the tools available for these approaches.

To expand Query Annotations beyond data-binding and CRUD operations, we are considering to explore how other types of application logic could be expressed in a declarative query-like form. Some examples of interest here are authentication and authorization, as common elements of software applications that also have impact on the generated UIs.