A case study of the MEUSec method to enhance user experience and information security of digital identity wallets

5.1 Step 1: definition of evaluation object

Input of Step 1: The Hidy wallet^[8] by the University of Applied Sciences Mittweida^[9] was used as evaluation object.

(1.1) Define wallet functions to be evaluated: Initially, the MU defined the wallet functions to be evaluated by selecting them from a suggested list of Krauß et al. ³⁰ The following wallet functions have been defined for evaluation by the MU:

1. Storing VC, in particular presentation and verification of the trustworthiness of the VC issuer.

2. Management of VC, in particular quick access to stored VC, deletion of VC and the display of VC.

3. Transfer of stored VC, in particular presentation and verification of the trustworthiness of the verifier, presentation and verification of the purpose of use of VC to be shared.

In addition to the wallet functions selected from the list of Krauß et al. ³⁰ the following wallet functions were defined by the MU:

1. Payment function, in particular requesting and making payments and

2. General operating functions, such as returning to previous screens.

(1.2) Identify security relevant components and potential attackers and (1.3) define scope of InfoSec evaluation: The ISE identified the security relevant components and potential attackers by viewing the wallet documentation. He also discussed these with the MU to define the scope of the InfoSec evaluation: (a) Malicious issuer who can create and sign VC, compromising integrity and confidentiality. (b) Issuer provides unwanted information in the VC (e.g., hidden health information) so that the issuer and verifier can co-operate, which compromises confidentiality and (c) Malicious verifier that uses VC for itself, which compromises confidentiality and integrity.

Output of Step 1: Defined evaluation object.

5.2 Step 2: preparation of user-based evaluation

Input of Step 2: Defined evaluation object from Step 1 (see Section 5.1).

(2.1) Determine requirements of the WU selection: The MU, UXE and ISE specified that 10 WU should be involved in Thinking aloud, differing in terms of gender, age, private operating system (e.g., iOS), previous experience with wallets and cryptocurrencies, IT affinity, German language skills and physical limitations (e.g., visual impairment). Nielsen and Landauer ³¹ recommend 5 WU for a user test. Due to the many different properties of the WU selection, it was decided to carry out the test with 10 WU.

(2.2) Define test cases for wallet functions: The MU, UXE and ISE planned the WU tasks for Thinking aloud. They created a paper guide with tasks for the WU to perform when conducting Thinking aloud:

1. Open the Hidy wallet on the home screen.

2. Request a payment of 5,000 Satoshi^[10] to their Hidy wallet.

3. Wait for the payment to arrive. At this point you would transfer Satoshis from an external application. In this case, the Hidy wallet is already topped up with Satoshis.

4. You can now close and reopen the Hidy wallet.

5. Add “Demo shop” to your wallet features.

6. Buy any ticket in the demo shop.

7. Display the ticket in the Hidy wallet.

8. You realise that you have bought the wrong ticket.

9. Delete the ticket from the Hidy wallet.

(2.3) Acquire WU : The WU were invited by the MU according to the defined characteristics (see Table 1) to a Thinking aloud slot of about 30 min. To collect the demographic data, the WU were given a questionnaire to complete. They also had to sign a declaration of consent for data processing. Table 1 shows the demographic data of the 10 WU.

(2.4) Set up test cases on end devices: The Hidy wallet was set up on an IPhone 14 Pro Max. In addition, the defined WU tasks were printed out so that the WU could use them for Thinking aloud. The devices for recording the smartphone screen and the WU (image and sound recording) were also set up.

Output of Step 2: Prepared user-based evaluation.

5.3 Step 3: execution of user-based evaluation

Input of Step 3: User-based evaluation prepared in Step 2 (see Section 5.2).

(3.1) Start lab recordings, conduct Thinking aloud (3.2) and (3.3) stop lab recordings and archive files: The MU first gave each WU (see Table 1) instructions (such as the paper guide from Section 5.2) and started the video and sound recordings. Then each WU performed the Thinking aloud ²⁵ with the paper guide under the supervision of the MU and UXE. Then, the MU stopped and archived the video and audio recordings. The WU received 20€for participating in the evaluation.

Output of Step 3: Thinking aloud recordings of each WU.

5.4 Step 4: evaluation of user-based evaluation results

Input of Step 4: Lab recordings archived in Step 3 (see Section 5.3).

(4.1) Collect strengths and weaknesses of UX and InfoSec: The MU, UXE and ISE reviewed the recordings made in the previous step. While doing so, they identified strengths and weaknesses by looking for actions that the WU either performed with ease or struggled to perform. Both strengths and weaknesses were recorded in a template proposed by the method, as seen in Table 2. The template contains a unique ID, a name with description of the issue and whether it is a strength or weakness regarding either InfoSec or UX. The previously in Section 5.1 defined wallet functions were assigned, as well as attributes regarding UX and InfoSec as defined by Morville ⁷ and DIN EN ISO/IEC 24760-1. ¹ The selection of suitable attributes is decided by a discussion between MU, UXE and ISE. Since during the review of the first three recordings only weaknesses were identified, because they were more obvious, the MU, UXE and ISE shifted their focus towards identifying strengths as well as weaknesses. Some UX and InfoSec weaknesses occured with multiple WU. Since the number of WU was chosen higher than the recommended 5 WU, this was to be expected. A weakness identified during evaluation was, that the VC are not displayed with names and can therefore only be identified by the associated image. Since this image needs to be loaded, network connectivity is required to distinguish the VC. Since this is both a UX and an InfoSec weakness, two separate entries were made. From an InfoSec point of view this affects the attributes “Confidentiality” and “Authenticity” since the WU might accidentally present the wrong VC to a verifier. It concerns the wallet functions “Transmission of saved VC”, “Saving VC” and “Display of VC”. The UX entry for the missing names can be seen in Table 2 while the entry for the InfoSec weakness can be accessed online (see footnote 7) with all other identified strengths (n = 7) and weaknesses (n = 41).

(4.2) Derive heuristics for UX and InfoSec: The MU, UXE and ISE derived heuristics based on the previously established strengths and weaknesses. A single strength or weakness might have multiple heuristics derived from it. A total of 27 heuristics were derived, which can be accessed online (see footnote 7). For example, both the heuristic “All VC should have unique names” (seen in Table 3) and “Images of VC should be displayed” are derived from the weakness “VC have no names”.

(4.3) Create collection and add derived heuristics: The MU used the previously defined heuristics to create a collection, which is available online (see footnote 7). The collection contains a consistent representation of all heuristics.

Output of Step 4: Heuristics for UX and InfoSec determined.

5.5 Step 5: preparation of expert-based evaluation

Input of Step 5: Heuristics for UX and InfoSec determined in Step 4 (see Section 5.4).

(5.1) Select heuristics from external collection and add them to own collection: The MU, ISE and UXE jointly went through the list of possible heuristics by Sauer et al. ²⁶ and added heuristics that were considered useful to extend their own collection. Therefore, the list of heuristics (see footnote 7) has been expanded from 27 to 32 heuristics. The added heuristics are:

1. The app should offer a search function for VC,

2. the app should be interoperable with other systems,

3. it should be possible to create profiles and assign VC to them,

4. data should be encrypted and transmission secure, and

5. the app is intended to notify the user of available updates.

(5.2) Literature research of heuristics, strengths and weaknesses of UX and InfoSec and (5.3) Derive heuristics for UX and InfoSec and add them to own collection: The MU, ISE and UXE decided not to conduct a literature search because the extended list of heuristics from Activity 5.1 covers the required functions. This decision is also included in the method and is therefore part of the process. This decision also eliminates Activity 5.3, which is intended to adapt the heuristics found during the literature search and add them to the collection.

(5.4) Update security relevant components and potential attackers and (5.5) update scope of InfoSec evaluation and heuristics: A new attacker has been identified who is gaining access to the storage of the device. If backups are unencrypted, they could be read directly by the attacker. Otherwise, standard attack vectors, such as weak passwords for the encryption of the backups, could be exploited. A second new attacker has been identified who is using known InfoSec vulnerabilities of the general operating functions.

(5.6) Assign weights to heuristics: The MU, ISE and UXE went through the heuristics together and assigned weights to them. The weights chosen were between 1 and 5, with 1 being barely important and 5 being essential. The defined weights can be seen in the list of heuristics (see footnote 7).

Output of Step 5: Heuristics of UX and InfoSec determined and weighted. The list of heuristics created is available online (see footnote 7).

5.6 Step 6: execution of expert-based evaluation

Input of Step 6: Heuristics of UX and InfoSec determined and weighted in Step 5 (see Section 5.5).

(6.1) Test defined wallet functions and (6.2) Rate fulfillment score for each heuristic: The ISE and UXE tested the defined wallet functions and rated each heuristic in the collection by its rate of fulfillment on a scale from “1: Not fulfilled” to “5: completely fulfilled”. Both the score and a reasoning for the score were added to the list of heuristics (see footnote 7).

(6.3) Create interaction matrix of defined heuristics: The MU created an interaction matrix to evaluate the interactions of different heuristics with each other. Since each heuristic interacts with every other heuristic and these interactions are not symmetrical (as in: heuristic A can affect heuristic B, but heuristic B might not affect heuristic A making their interactions asymmetrical) the number of fields in the matrix grows fast with an increasing number of heuristics. Since filling out the interactions for all 32 heuristics was deemed unfeasible in a reasonable amount of time, only heuristics with the weight of 5 were included in the interaction matrix. This reduced the number of heuristics from 32 to 12 and the number of fields in the matrix from 992 to 132. The resulting interaction matrix is available online (see footnote 7).

(6.4) Fill interaction matrix through discussion: The MU, ISE and UXE discussed the interactions of the selected heuristics. There are three possible interaction properties: “complementary”, “neutral” and “conflicting”. If the interaction between heuristic A and heuristic B is complementary, it means that fulfillment of heuristic A also helps fulfilling heuristic B. In a neutral interaction the two heuristics don’t affect each other and in case of a conflicting interaction, fulfillment of heuristic A will hinder the fulfillment of heuristic B. No conflicting heuristics were found in the discussion, which is why the interaction matrix only contains neutral and complementary entries. Table 4 shows some heuristics with their interacting properties. Each row represents the effect a heuristic has on other heuristics, while a column in the interaction matrix represents how a heuristic is affected by others. For example, heuristic 02 is complementary to heuristic 11, because unique names for VC improve the comprehensibility of InfoSec instructions. Conversely, heuristic 11 is neutral to heuristic 02, because comprehensible InfoSec instructions do not affect unique names for VC. The entire interaction matrix is available online (see footnote 7).

(6.5) Integrate heuristic interactions in own collection: The MU added the interactions to the list of heuristics (see footnote 7). Since most interactions were deemed to be “neutral”, only interactions that are “complementary” are explicitly noted in the list of heuristics.

Output of Step 6: (a) Interaction matrix of defined heuristics created and (b) heuristics scored.

5.7 Step 7: evaluation and validation of expert-based evaluation results

Input of Step 7: Heuristics scored in Step 6 (see Section 5.6).

(7.1) Aggregate scores for each attribute: First, for each heuristic, its weight and degree of fulfillment were multiplied to determine the score for that heuristic. These scores were then totaled for the attributes assigned to the heuristics. Thus, an absolute score was calculated for each attribute which can be seen in the first row in Table 5 for UX attributes and Table 6 for InfoSec attributes. The average score for the attributes was determined by dividing the absolute score by the number of referenced heuristics. These scores fill the second row of the Tables 5 and 6. With the maximum score per attribute, which results from the product of weight with maximum fulfillment score, the average maximum score can be calculated by dividing it by the number of heuristics, as shown in third and fourth row of the tables. Finally, the ratio of the average score to the average maximum score was calculated. This ratio was used to assess the degree of fulfillment for the respective attributes.

(7.2) Aggregate scores for UX and InfoSec: The scores from Activity 7.1 were used to determine the average score, average maximum score and the ratio of these scores at UX and InfoSec level. To do this, the mean value was determined using the mean values of the attributes that belong to UX or InfoSec. The results are shown in Tables 5 and 6.

The scores can be used in a further application of the MEUSec method to check whether the improvement suggestions from Step 8 have really led to an improvement.

(7.3) Feedback discussion and (7.4) Adjust heuristics: The feedback discussion was brief, as there were no problems with the heuristics and they did not need to be adjusted. As a result, Activity 7.4 was not required.

(7.5) Add own collection of heuristics to external collection of heuristics: This activity is part of the application when the software tool is fully developed and can be used to share the list of heuristics with other users. In this case, this activity can be seen as publishing the list of heuristics to Zenodo (see footnote 7) and making it available to interested parties.

Output of Step 7: Scores of UX and InfoSec aggregated.

5.8 Step 8: enhancement of UX and InfoSec

Input of Step 8: The interaction matrix defined in Step 6 (see Section 5.6) was used to collect improvement suggestions, as a different approach must be selected for each interaction property of the individual heuristics.

For conflicting heuristics, it must first be checked whether conflict solutions can be found. If not, either InfoSec or UX must be prioritised by using either the UX heuristic as a improvement suggestion or the InfoSec heuristic. The neutral and complementary heuristics serve directly as suggestions for improvement, as they do not influence each other negatively.

(8.1 and 8.2) Find solutions for conflicting heuristics and prioritize UX or InfoSec: As no conflicting heuristics have been identified in the interaction matrix, this activities could be skipped.

(8.3) Suggest improvements for the wallet based on complementary and neutral heuristics: Improvement suggestions could directly be derived for neutral and complementary heuristics, because they do not negatively affect each other. Therefore, the MU, ISE and UXE discussed the neutral and complementary heuristics to define improvement suggestions. A total of 26 improvement suggestions were collected and formulated in standardized form: ID, description and affected heuristics. Table 7 shows some improvement suggestions as examples. All improvement suggestions are available online (see footnote 7).

Output of Step 8: List of improvement suggestions.

5.9 Discussion of the wallet evaluation results

By conducting the evaluation, certain insights have been gained that further enhance the study of the implications between UX and InfoSec, especially usable security. These strengths, weaknesses, as well as improvement suggestions have been presented to the Hidy developers after conducting the evaluation. The resulting documents are available to the developers and can be used for further development. As of writing, the wallet is a prototype that is still a work in progress.

The strengths and weaknesses of the Hidy wallet identified using the MEUSec method are now discussed and compared with those from the literature.

As Sartor et al. ¹² show, the terminology used in their evaluated wallet is too technical and was not understood by test subjects. This weakness is linked to the design guideline “Use of understandable terms” by Sellung and Kubach ¹⁴ and was also identified by applying the MEUSec method to the Hidy wallet, e.g., as seen with terms like “credential” (see Section 5.4). This lack of understanding can lead to further problems, as users may struggle to perform basic tasks or misinterpret important features of the wallet. Furthermore, the technical terminology creates a barrier for less experienced users, reducing accessibility and potentially deterring adoption. It also raises security concerns, as users who do not fully understand key instructions are more likely to make mistakes. Ultimately, this undermines trust in the wallet and limits its target audience to highly technical users.

Sartor et al. ¹² also found that the initial tutorial of their evaluated wallet was not helpful enough for users. In addition, Khayretdinova et al. ¹⁰ identified that test subjects faced challenges in setting up their evaluated wallet and managing their VC due to the wallet’s complexity and lack of intuitive design. By applying the MEUSec method to the Hidy wallet, it was discovered that an introductory tutorial should be implemented in the Hidy wallet, related to the design guideline “User Onbording” by Sellung and Kubach. ¹⁴ The basic concept, functionalities and important tasks for users should be explained with the help of this tutorial at the first start of a wallet. In the past, basic functionality of wallets was not understood at all, like in the evaluation by Sauer et al. ²¹ and by Korir. ¹¹

In the evaluation of Sauer et al. ²¹ test subjects wanted more help hints in their evaluated wallet. This was also found in the evaluation of the Hidy wallet as there were no help hints and some elements like an “App Link” had no explanation at all, leaving test subjects wondering what its function was. This problem is tackled by the design guideline “Help and feedback” by Sellung and Kubach. ¹⁴ The guideline suggests using a clickable “i” or “?” that provides the user with additional information.

Sauer et al. ²¹ expressed that information can be hidden in a credential which is not shown to the user. This can happen in the Hidy wallet because the detail view does not show all stored information in the credential.

Like test subjects in the study of Satybaldy, ¹³ test subjects in the Hidy wallet evaluation experienced error messages that were not helpful. For example, an error message states “Creation failed” without telling the test subjects that the problem lies in the missing internet connection. Another error message stating “Sending failed” appeared after a successful purchase in the demo shop. The guideline “Error handling” by Sellung and Kubach ¹⁴ states that error messages should be explained to the user and give clear instructions on how to fix them.

More weaknesses found in the Hidy wallet are not represented in literature but conflict with other design guidelines from Sellung and Kubach. ¹⁴ The inconsistent naming of buttons and the use of different currencies in the demo shop and in the Hidy wallet conflict with the guideline “Use of consistent terms”. Overlapping success messages and screens that are hard to distinguish because they are too similar violate the guideline “Simplicity of use”. The guideline “Placement of information” requires that elements should be clearly recognized by the correct placement of text. This is not fulfilled, as added applications and credentials are only displayed using images and not their names. There are missing “back” buttons and keyboard fields that cannot be closed go against the “Operability” guideline, as these are not conform with user expectation. The “Home” screen does not meet users expectations and the demo shop interaction did not feel normal, as there was no receipt sent by e-mail and the resulting credential is not refundable and the deletion is irrevocable. This contradicts the guideline “Familiarity and Relatability” as this differs from experiences with other online shops and apps. In addition the detail view of a ticket credential did not offer additional meaningful information to the user.

While the Hidy wallet requires authentication every time the app is opened and the account balance is not directly revealed, the purchase history can be viewed directly, which should also be hidden by default. The guideline “Properly securing the wallet and functions” states that all sensitive data must be protected. The “Home” screen is perceived as clear, which is connected to the guideline “Minimalistic and simple design”, but there are loading screens, which are not recognizable as such, because the loading bar at the top of the screen is barely visible. This confuses the user and interrupts the users focus on the important functions.

In addition to the strengths and weaknesses of wallets found in the literature and in addition to the related design guidelines from Sellung and Kubach, ¹⁴ the following strengths and weaknesses of wallets were identified by applying the MEUSec method.

A weakness of the Hidy wallet is that it fails to clearly communicate to users that the app offers additional features beyond the basic wallet functionality. Despite these additional features, the app is still primarily a wallet, but this distinction is not reflected in its name. As a result, users often search for their purchased tickets in the demo shop rather than the wallet, and they mistakenly try to download the demo shop from the application store, assuming that it is an external app.

Another weakness is that the wallet currently only offers the German language. Therefore, some test subjects had problems using the Hidy wallet because they could not understand German well enough.

A strength of the Hidy wallet is that test subjects found the wallet application on the home screen simply because of its name.

6.1 Method evaluation results

The method evaluation results are described below on the basis of the defined evaluation criteria (see Section 4).

6.2 Discussion of the method evaluation results

Before developing the MEUSec method, Sauer et al. ³² conducted a systematic literature review to identify and compare existing procedures for evaluating the implications of UX and InfoSec. Some identified evaluation procedures (GOMS, ³³ SecureUse score, ³⁴ eye tracking ³⁵ and questionnaires such as SUS, ³⁶ UEQ, ³⁷ UEQS ³⁸ and AttrakDiff ³⁹ ) are pure UX evaluation procedures (without InfoSec scope) that can be applied to different software variants that differ in one security relevant parameter, ^{40 , 41 , 42} e.g., the authentication procedure. In this way, evaluation results about the implications of UX and InfoSec can be obtained. For example, a pure UX evaluation procedure is applied to a wallet with a 4-digit PIN and then to the (almost) same wallet with a 6-digit PIN. This makes it possible to evaluate the extent to which the UX changes when InfoSec is increased. However, this makes evaluating the implications of UX and InfoSec time-consuming, as only one security relevant parameter of the software system under evaluation can be changed per iteration. During the application of the MEUSec method, the implications of UX and InfoSec can be evaluated by creating the interaction matrix in one step, when assigning the interaction properties rather than evaluating only one security relevant parameter of the whole system.

With a heuristic evaluation, ⁴³ UX and InfoSec heuristics can be used to evaluate UX and InfoSec, but the implications between the heuristics and thus between UX and InfoSec are not evaluated. This is another advantage of the interaction matrix of the MEUSec method, as the implications between heuristics are evaluated.

Deriving heuristics and then formulating improvement suggestions based on the heuristics is more time and cost intensive than formulating improvement suggestions directly. However, this approach is more useful and effective as the heuristics can be used to determine a score for UX and InfoSec (and their attributes such as usability and integrity). The heuristics and the scores can then be reused in another application of the MEUSec method to check whether the improvement suggestions have really led to improvements (by comparing the new scores with the old scores).

Furthermore, the MEUSec method includes both an expert-based and user-based approach and not just one of the two, as the opinions of experts and users can differ. ⁴⁴ Of course, time and costs can be saved by using solely a user-based evaluation or only an expert-based evaluation. However, the combination of both approaches increases the quality of the results. A user-based evaluation alone is not sufficient for the InfoSec evaluation, as the users only evaluate the user interface and not any in-depth security components. Only one of the evaluation methods identified by Sauer et al. ³² includes both experts and end users. However, this in turn requires the evaluation of different software variants that differ in a security parameter. The remaining methods identified by Sauer et al. ³² are either expert- or end-user-based and should therefore not be used alone, but in combination, as the opinions of experts and users can differ. ⁴⁴ One evaluation procedure identified by Sauer et al. ³² that comes very close to a user- and expert-based approach is the heuristic walkthrough. ⁴⁵ Heuristic walkthrough ⁴⁵ first includes cognitive walkthrough ⁴⁶ and then heuristic evaluation. ⁴³ In cognitive walkthrough, ⁴⁶ evaluators put themselves in the role of end users and test the software system using predefined tasks. This is followed by a free-form evaluation using a heuristic evaluation. The MEUSec method is based on the heuristic walkthrough, but the cognitive walkthrough has been replaced by Thinking aloud. ²⁵ This has the advantage that actual end users are involved and not just evaluators who put themselves in the role of end users, as the opinions of experts and users can differ. ⁴⁴

The Thinking aloud method ²⁵ – as a user-based approach in the MEUSec method – is in itself a pure UX evaluation procedure (without InfoSec scope). However, a previous evaluation ²¹ has shown that the Thinking aloud method can also be used to evaluate UX aspects that have an impact on InfoSec. For example, users have expressed that they want to have a confirmation popup in the wallet when security-critical actions are performed, such as storing VC from unverified issuers in the wallet. This makes Thinking aloud suitable for the user-based evaluation of the MEUSec method.

Nevertheless, when using the MEUSec method, there is a risk of abstracting problems that could affect specific contexts and user groups. For example, heuristics could be used that are too abstract so that specific problems are not found or specific user properties are not addressed, e.g., when a heuristic related to accessibility does not consider individuals with specific accessibility difficulties.

In the MEUSec method, own heuristics are first formulated on the basis of the user-based evaluation and then heuristics from other heuristics are added to the own heuristics if necessary. This is more time-consuming than using heuristics from other heuristics directly. However, this minimizes the risk that user-based evaluation results are not sufficiently considered. This order was therefore deliberately chosen when applying the MEUSec method.

6.3 Suggestions for improving the MEUSec method

Based on the evaluation results of the MEUSec method (Section 6.1) and the discussion (Section 6.2), some improvement suggestions became apparent. These are summarized below.

Improvement suggestion (1) emerged in Step 1, while security relevant components and potential attackers should be identified by the ISE. While there are widely-known and accepted ways to define attacker capabilities in cryptography like Dolev and Yao, ⁴⁷ practical InfoSec is more fuzzy in this regard. The method could profit from a pre-selection of standardized attacker models with their respective capabilities, that can either be directly used by the ISE or adapted.

Improvement suggestion (2) came up when the MU was uncertain about what properties a WU constitutes. A template for these properties could help the MU to better understand these properties.

Improvement suggestion (3) concerns the paper guide of Thinking aloud: In its current form, the paper guide is static once the test cases have been set up. During our evaluation, it became clear that the paper guide must be adaptable, so it can be changed once again after the test cases have already been set up.

Improvement suggestion (4) came up during the creation of the heuristics. After the heuristics had been created, several of them were identified to share commonalities, so they could be easily summarized. This can be done in an additional reviewing step after the initial creation.

Improvement suggestion (5) is directly related to improvement suggestion 4, as the total number of heuristics greatly affects the time that is needed to create and fill out the interaction matrix. Besides summarizing, restrictions on the heuristics for the matrix could be made, for example by only taking heuristics with a poor degree of fulfillment into account.

Improvement suggestion (6) concerns the scoring in Step 7. After the initial scoring, the roles felt the need to once again talk about the scores, as heuristics that had been rated later on had an affect on heuristics rated earlier. This means that the scoring can not be seen as absolute, but rather as relative between the considered heuristics.

Improvement suggestion (7) is related to all steps of the method. Early on, the roles noticed that the process could greatly benefit from a template for documenting the method, so no information gets forgotten or is otherwise lost. This improvement could even be taken further, by having a software tool supporting the process.

Improvement suggestion (8) concerns the traceability of the decision making: While all roles could follow and agree on scores and the prioritisation, for archiving purposes or re-usage having a rationale on the taken decisions could help.

Improvement suggestion (9) is to move the feedback discussion (activity 7.3) and the adjustment of the heuristics (activity 7.4) so that they are performed after the activity of determining the fulfillment degree for each heuristic (activity 6.2). This means that any problems that occur with the heuristics can be rectified immediately. In addition, the feedback discussion and the adjustment of the heuristics should be combined, because during the feedback discussion only questions about the heuristics had to be clarified and the feedback discussion therefore did not require its own activity.

Improvement suggestion (10) is to combine the activities “test wallet functions” and “score heuristics” into one activity, i.e., to score the heuristics while testing the wallet functions. When evaluating the degree of fulfillment of the heuristics, UXE and ISE found that they had to test the wallet again and again because they could not remember exactly what happened when using the wallet in the previous activity.

Improvement suggestion (11) is to adjust the fulfillment score of the heuristics from 1–5 to 0–4. When calculating the scores in step 7, it became apparent that the calculated average score ratios were between 0.2 and 1, which does not match the scale normally expected of such scores.

Improvement suggestion (12) is to give the artifacts a changelog and allowing them to be adapted consistently with the other artifacts. Because during the implementation of the method, the already created artifacts had to be adjusted. This was not originally planned and should therefore be included.