Evaluation in Human-Computer Interaction (HCI) - Usability and Performance Metrics

3.1.1 - Goals of Evalution.

1. What is one of the primary objectives of evaluation?

(A) To increase the system's complexity
(B) To determine the scope and usability of the system's capability
(C) To eliminate user interaction
(D) To decrease system functionality

Correct Answer: (B) To determine the scope and usability of the system's capability

2. Which factor is crucial for the functionality of a system?

(A) It must be complex to use.
(B) It should not meet user needs.
(C) It should facilitate users in accomplishing their intended duties.
(D) It must have a high learning curve.

Correct Answer: (C) It should facilitate users in accomplishing their intended duties.

3. How should a system's design align with user expectations?

(A) It should require users to learn new tasks.
(B) It should provide capabilities that differ from user habits.
(C) It should allow users to complete tasks using familiar methods.
(D) It should complicate the user experience.

Correct Answer: (C) It should allow users to complete tasks using familiar methods.

4. What aspect of user interaction is important to evaluate?

(A) User satisfaction and ease of learning
(B) The complexity of the system
(C) The amount of data to be retained
(D) The time spent on the system

Correct Answer: (A) User satisfaction and ease of learning

5. In systems designed for leisure or entertainment, what additional factor should be evaluated?

(A) System speed
(B) User delight and emotional reaction
(C) Complexity of tasks
(D) Number of features

Correct Answer: (B) User delight and emotional reaction

6. What is the ultimate objective of the evaluation process?

(A) To increase system costs
(B) To identify specific design issues
(C) To limit user feedback
(D) To enhance system complexity

Correct Answer: (B) To identify specific design issues

7. Which of the following may indicate a design issue during evaluation?

(A) Meeting user expectations
(B) Users find the system too intuitive
(C) Unexpected outcomes or user confusion
(D) Streamlined user interactions

Correct Answer: (C) Unexpected outcomes or user confusion

8. What should evaluations track to determine how well a system supports tasks?

(A) The amount of money spent on the system
(B) User interaction and task performance
(C) The number of users
(D) System uptime

Correct Answer: (B) User interaction and task performance

9. Which of the following is a consideration in evaluating usability?

(A) User satisfaction and ease of learning curve
(B) Only technical performance
(C) User demographics
(D) The length of the system manual

Correct Answer: (A) User satisfaction and ease of learning curve

10. What could be a sign of putting users under too much stress?

(A) Requiring too little information
(B) Asking for too much information to be retained
(C) Providing clear instructions
(D) Offering multiple support options

Correct Answer: (B) Asking for too much information to be retained

3.1.2 - Evalution Through Expert.

11. What is the primary goal of conducting an initial evaluation of a system before implementation?

(A) To make the design more aesthetic
(B) To reduce the cost of development
(C) To avoid costly errors in the design
(D) To eliminate user feedback entirely

Correct Answer: (C) To avoid costly errors in the design

12. Which of the following is NOT a challenge when conducting user testing during the design phase?

(A) Cost of conducting regular user testing
(B) Gathering reliable feedback from unfinished ideas
(C) Changing the design before resource commitment
(D) Difficulty in evaluating interaction experiences with incomplete prototypes

Correct Answer: (C) Changing the design before resource commitment

13. Expert analysis relies on which of the following?

(A) Gathering extensive user input
(B) Evaluating systems based on recognized usability standards
(C) Testing only fully implemented systems
(D) Repeated user trials for every phase

Correct Answer: (B) Evaluating systems based on recognized usability standards

14. Which of the following is NOT a method of expert analysis mentioned in the text?

(A) Model-based evaluation
(B) Heuristic evaluation
(C) User-driven evaluation
(D) Cognitive walkthrough

Correct Answer: (C) User-driven evaluation

15. The cognitive walkthrough focuses primarily on evaluating what aspect of a system?

(A) Aesthetic appeal
(B) Ease of system learning through exploration
(C) Speed of system response
(D) Cost-effectiveness of the system

Correct Answer: (B) Ease of system learning through exploration

16. What is the key purpose of the "story" used in cognitive walkthroughs?

(A) To define the system's core functionality
(B) To explain why each step is appropriate or not for a new user
(C) To narrate the system’s back-end processes
(D) To make the walkthrough more engaging

Correct Answer: (B) To explain why each step is appropriate or not for a new user

17. During a cognitive walkthrough, evaluators answer all of the following questions EXCEPT:

(A) Will users understand the feedback they receive?
(B) Will users know what action they need to take?
(C) Will users pay for the system?
(D) Is the effect of the action the same as the user’s goal?

Correct Answer: (C) Will users pay for the system?

18. What is the main difference between heuristic evaluation and cognitive walkthrough?

(A) Heuristic evaluation involves user input, while cognitive walkthrough does not
(B) Heuristic evaluation uses expert judgment based on usability principles, while cognitive walkthrough evaluates learning through exploration
(C) Cognitive walkthrough is more suitable for finished systems than heuristic evaluation
(D) Heuristic evaluation only evaluates aesthetics

Correct Answer: (B) Heuristic evaluation uses expert judgment based on usability principles, while cognitive walkthrough evaluates learning through exploration

19. Which of the following is a heuristic used in heuristic evaluation?

(A) User guidance through long dialogues
(B) Detailed user manuals for every task
(C) User control and freedom
(D) Command-based input systems

Correct Answer: (C) User control and freedom

20. According to Nielsen, how many evaluators are typically required to identify 75% of usability issues in heuristic evaluation?

(A) 1–2 evaluators
(B) 3–5 evaluators
(C) 7–10 evaluators
(D) More than 10 evaluators

Correct Answer: (B) 3–5 evaluators

3.1.3 - Evalution Through Users: Styles of Evalution and Expremental Evalution Design.

21. What is the purpose of evaluation methods conducted by users?

(A) To assess usability with real users
(B) To analyze system complexity
(C) To improve hardware performance
(D) To bypass user testing

Correct Answer: (A) To assess usability with real users

22. Which of the following is a physiological monitoring technique used in evaluation?

(A) Eye-tracking
(B) Algorithm optimization
(C) Heart rate monitoring
(D) System benchmarking

Correct Answer: (C) Heart rate monitoring

23. At what stage of system development is user participation most common?

(A) During system installation
(B) During prototype development
(C) After full system launch
(D) Before requirements capture

Correct Answer: (B) During prototype development

24. Which evaluation technique can assist with early design phases, like requirements capture?

(A) Wizard of Oz technique
(B) User surveys and observation
(C) Advanced system testing
(D) Code optimization

Correct Answer: (B) User surveys and observation

25. Which of the following is a key advantage of laboratory-based studies?

(A) Access to advanced audio/visual recording equipment
(B) Testing the system in the user's natural environment
(C) High levels of disruption-free productivity
(D) Greater user collaboration

Correct Answer: (A) Access to advanced audio/visual recording equipment

26. Which of the following is a disadvantage of laboratory-based studies?

(A) Lack of control over external factors
(B) Difficulty simulating real-world contexts
(C) Absence of usable data
(D) Limited ability to test single-user tasks

Correct Answer: (B) Difficulty simulating real-world contexts

27. Which scenario is laboratory testing most suited for?

(A) Testing user interactions in hazardous locations
(B) Observing communication between multiple people
(C) Conducting field studies over a long period
(D) Measuring user reactions in social settings

Correct Answer: (A) Testing user interactions in hazardous locations

28. What is a major benefit of field studies in evaluation?

(A) Controlled testing environment
(B) Insight into natural user interactions
(C) Easier analysis through pre-set conditions
(D) Fewer disruptions from real-world tasks

Correct Answer: (B) Insight into natural user interactions

29. What is one challenge of field studies in user evaluation?

(A) Limited ability to record events
(B) High ambient noise and frequent disruptions
(C) Inability to observe user behavior
(D) Difficulty isolating dependent variables

Correct Answer: (B) High ambient noise and frequent disruptions

30. Which evaluation style allows for better observation of real-world user behavior?

(A) Field studies
(B) Laboratory studies
(C) Simulation techniques
(D) Theoretical evaluation

Correct Answer: (A) Field studies

31. What is one of the trade-offs of choosing laboratory studies over field studies?

(A) Reduced equipment costs
(B) Loss of contextual information
(C) Higher variability in data
(D) Fewer user interactions

Correct Answer: (B) Loss of contextual information

32. Which method involves controlled experiments to gather evidence for a hypothesis?

(A) Empirical methods
(B) User feedback analysis
(C) Observational techniques
(D) Physiological monitoring

Correct Answer: (A) Empirical methods

33. What is a critical aspect of participant selection for user evaluations?

(A) Selecting participants based on age only
(B) Ensuring participants are representative of the target user group
(C) Choosing users based on system familiarity
(D) Using undergraduate students for any system

Correct Answer: (B) Ensuring participants are representative of the target user group

34. What is an appropriate sample size for controlled experiments with statistical analysis?

(A) One to two users
(B) Five to seven users
(C) At least ten users
(D) Over 100 users

Correct Answer: (C) At least ten users

35. Which variable in experimental evaluation is manipulated by the experimenter?

(A) Dependent variable
(B) Independent variable
(C) Controlled variable
(D) External variable

Correct Answer: (B) Independent variable

36. What type of variable is the “menu selection speed” in an experiment?

(A) Independent variable
(B) Dependent variable
(C) Controlled variable
(D) External variable

Correct Answer: (B) Dependent variable

37. What does a hypothesis in experimental evaluation represent?

(A) A random prediction of user behavior
(B) A prediction about the relationship between variables
(C) A way to compare usability issues
(D) A survey of user preferences

Correct Answer: (B) A prediction about the relationship between variables

38. Which design minimizes learning effects by altering the order of conditions in an experiment?

(A) Between-subjects design
(B) Within-subjects design
(C) Randomized design
(D) Mixed design

Correct Answer: (B) Within-subjects design

39. Which experimental design involves different groups experiencing different conditions?

(A) Between-subjects design
(B) Within-subjects design
(C) Randomized design
(D) Mixed design

Correct Answer: (A) Between-subjects design

40. Which type of design reduces the effects of participant variation?

(A) Between-subjects design
(B) Within-subjects design
(C) Randomized design
(D) Mixed design

Correct Answer: (B) Within-subjects design

41. What is the primary consideration in selecting the experimental design?

(A) The representativeness of the participant group
(B) The equipment available
(C) The size of the testing space
(D) The number of possible variables

Correct Answer: (A) The representativeness of the participant group

42. Which experimental design uses both between-subjects and within-subjects variables?

(A) Mixed design
(B) Randomized design
(C) Controlled design
(D) Empirical design

Correct Answer: (A) Mixed design

43. Which guideline highlights the importance of examining data for outliers?

(A) Statistical tests
(B) Data examination
(C) Data storage
(D) Variable classification

Correct Answer: (B) Data examination

44. What type of variable can take any value within a range?

(A) Discrete variable
(B) Continuous variable
(C) Nominal variable
(D) Ordinal variable

Correct Answer: (B) Continuous variable

45. Which statistical tests are used when the data do not follow a normal distribution?

(A) Parametric tests
(B) Non-parametric tests
(C) Descriptive tests
(D) Inferential tests

Correct Answer: (B) Non-parametric tests

46. What does a log-transformation primarily help achieve?

(A) Increase data normality
(B) Simplify data interpretation
(C) Reduce outliers
(D) Aggregate data points

Correct Answer: (A) Increase data normality

47. Which statistical analysis focuses on rank rather than actual data values?

(A) Descriptive analysis
(B) Non-parametric analysis
(C) Parametric analysis
(D) Bivariate analysis

Correct Answer: (B) Non-parametric analysis

48. What is a key characteristic of mixed experimental designs?

(A) Use of only one type of variable
(B) Inclusion of both between-subjects and within-subjects variables
(C) Random assignment of subjects
(D) Focus on qualitative data only

Correct Answer: (B) Inclusion of both between-subjects and within-subjects variables

49. What is the main challenge in conducting group system evaluation experiments?

(A) Individual performance
(B) Equipment availability
(C) Group dynamics and interactions
(D) Data analysis

Correct Answer: (C) Group dynamics and interactions

50. Which type of analysis involves examining social interactions within a natural setting?

(A) Controlled experiment
(B) Ethnographic study
(C) Quantitative analysis
(D) Parametric testing

Correct Answer: (B) Ethnographic study

51. What does the term "confidence interval" refer to in statistical analysis?

(A) The range in which the data points lie
(B) The probability that the parameter falls within a specified range
(C) The average of all data points
(D) The maximum error of measurement

Correct Answer: (B) The probability that the parameter falls within a specified range

52. In experimental designs, what is the purpose of hypothesis testing?

(A) To determine the mean of a sample
(B) To make predictions about future experiments
(C) To evaluate if a difference exists between groups
(D) To collect qualitative data

Correct Answer: (C) To evaluate if a difference exists between groups

3.1.4 - Evalution Through Users: Observational Techniques and Query Techniques.

53. What is a primary advantage of using paper and pencil for data recording?

(A) Flexibility in writing styles
(B) High speed of recording
(C) Detailed information capture
(D) Easy to use digital tools

Correct Answer: (A) Flexibility in writing styles

54. Which method is recommended for minimizing the limitations of a single note-taker?

(A) Audio recording
(B) Video recording
(C) Designating a specific note-taker
(D) Using computer logging

Correct Answer: (C) Designating a specific note-taker

55. What is a major challenge when using audio recordings for data collection?

(A) Difficulty in analyzing data
(B) Inability to capture non-verbal cues
(C) High cost of equipment
(D) Lengthy transcription process

Correct Answer: (B) Inability to capture non-verbal cues

56. What is one of the key benefits of video recording in observational studies?

(A) Provides detailed written notes
(B) Captures audio cues effectively
(C) Allows observation of participant actions
(D) Easily transcribes spoken words

Correct Answer: (C) Allows observation of participant actions

57. What limitation does computer logging have regarding user activity?

(A) Inability to record keystrokes
(B) Does not provide rationale behind actions
(C) High cost of implementation
(D) Difficulty in data synchronization

Correct Answer: (B) Does not provide rationale behind actions

58. Why might user notebooks be less reliable for recording activities?

(A) They provide detailed information
(B) Records may be infrequent and basic
(C) They require advanced technical skills
(D) They are expensive to maintain

Correct Answer: (B) Records may be infrequent and basic

59. What is a significant drawback of transcribing audio recordings?

(A) Easy to interpret
(B) Time-consuming and may miss subtleties
(C) High accuracy in details
(D) Cost-effective compared to other methods

Correct Answer: (B) Time-consuming and may miss subtleties

60. Which system provides tools for automatic protocol analysis?

(A) EVA
(B) DRUM
(C) Noldus' Observer Pro
(D) Workplace project at Xerox PARC

Correct Answer: (A) EVA

61. What challenge is associated with synchronizing multiple data sources in observational studies?

(A) Easy to implement
(B) Reduces data collection time
(C) Requires advanced software tools
(D) Difficult to analyze data collectively

Correct Answer: (D) Difficult to analyze data collectively

62. What benefit does the post-task walkthrough provide?

(A) Eliminates the need for transcription
(B) Enhances understanding of participant behavior
(C) Minimizes the need for audio recordings
(D) Allows for real-time observation

Correct Answer: (B) Enhances understanding of participant behavior

63. What does the DRUM toolkit focus on?

(A) User experience design
(B) Video tagging and commentary
(C) Comprehensive evaluation methodology
(D) Cost-effective research solutions

Correct Answer: (C) Comprehensive evaluation methodology

64. Which recording method is best suited for capturing long-term user interactions?

(A) Paper and pencil
(B) Audio recording
(C) Computer logging
(D) Video recording

Correct Answer: (C) Computer logging

65. What is a primary goal of using protocol analysis tools?

(A) To reduce data collection time
(B) To automate the transcription process
(C) To assist with synchronizing and analyzing data
(D) To eliminate the need for manual note-taking

Correct Answer: (C) To assist with synchronizing and analyzing data

66. What is a common issue with using professional typists for video transcription?

(A) High cost of hiring typists
(B) Difficulty in creating accurate notations
(C) Typists miss subtleties in the data
(D) Typists are unable to handle technical jargon

Correct Answer: (C) Typists miss subtleties in the data

67. What is a key feature of the Noldus' Observer Pro system?

(A) It focuses solely on audio data
(B) It is designed for field research
(C) It offers transcription services
(D) It does not allow for video analysis

Correct Answer: (B) It is designed for field research

68. What is the primary purpose of query techniques in user evaluation?

(A) To analyze system performance
(B) To obtain user perspectives on a system
(C) To design the interface
(D) To collect demographic data

Correct Answer: (B) To obtain user perspectives on a system

69. What is one of the main benefits of conducting user interviews?

(A) They require less preparation
(B) They allow for deeper exploration of topics
(C) They provide quantitative data
(D) They are always controlled experiments

Correct Answer: (B) They allow for deeper exploration of topics

70. Which of the following is a primary category of query techniques?

(A) Data mining and analysis
(B) Interviews and questionnaires
(C) Observational studies and testing
(D) Statistical and non-statistical methods

Correct Answer: (B) Interviews and questionnaires

71. What is a key characteristic of questionnaires compared to interviews?

(A) They are more flexible
(B) They allow for deeper insights
(C) They require less time to administer
(D) They provide immediate feedback

Correct Answer: (C) They require less time to administer

72. What type of question in a questionnaire allows users to provide their own answers?

(A) Closed-ended
(B) Multi-choice
(C) Open-ended
(D) Scalar

Correct Answer: (C) Open-ended

73. Which question type measures the user's level of agreement with a statement?

(A) General
(B) Scalar
(C) Ranked
(D) Multi-choice

Correct Answer: (B) Scalar

74. What is an advantage of using scalar questions in a questionnaire?

(A) They are easy to interpret
(B) They provide detailed qualitative data
(C) They allow users to express varying degrees of opinion
(D) They require more time to analyze

Correct Answer: (C) They allow users to express varying degrees of opinion

75. What is a limitation of open-ended questions in questionnaires?

(A) They are difficult to analyze rigorously
(B) They provide too much quantitative data
(C) They can only be answered with a "yes" or "no"
(D) They are too structured

Correct Answer: (A) They are difficult to analyze rigorously

76. What is a common issue with the return rate of questionnaires?

(A) It is usually over 50%
(B) It is typically low, around 25% to 30%
(C) It always reflects the user base accurately
(D) It can be easily increased by sending more reminders

Correct Answer: (B) It is typically low, around 25% to 30%

77. What does conducting a pilot study for a questionnaire help to achieve?

(A) Increase the length of the questionnaire
(B) Identify design issues before wider distribution
(C) Guarantee a high response rate
(D) Ensure participants have prior experience

Correct Answer: (B) Identify design issues before wider distribution

78. Which of the following is an advantage of using video recordings in observational studies?

(A) Provides detailed written notes
(B) Allows observation of participant actions
(C) Easily transcribes spoken words
(D) Offers a controlled experimental setting

Correct Answer: (B) Allows observation of participant actions

79. What does the term "self-selecting" refer to in the context of questionnaire distribution?

(A) Only those chosen by the evaluator can respond
(B) Participants choose to respond based on their interest
(C) Everyone in the user base must complete the questionnaire
(D) Random selection of users for the study

Correct Answer: (B) Participants choose to respond based on their interest

80. What is the benefit of closed questions in questionnaires?

(A) They provide unlimited responses
(B) They require less effort from respondents
(C) They are always more expensive to administer
(D) They allow for detailed open-ended answers

Correct Answer: (B) They require less effort from respondents

81. What is the main limitation of using audio recordings for data collection?

(A) They are costly
(B) They cannot capture non-verbal cues
(C) They are difficult to analyze
(D) They require extensive transcription

Correct Answer: (B) They cannot capture non-verbal cues

82. What type of question allows respondents to rank their preferences?

(A) Open-ended
(B) Scalar
(C) Ranked
(D) Multi-choice

Correct Answer: (C) Ranked

3.1.5 - Choosing an Evalution Method

83. What is the first factor influencing the choice of evaluation method in the design process?

(A) The cost of the evaluation
(B) The stage in the design cycle
(C) The type of users involved
(D) The type of technology used

Correct Answer: (B) The stage in the design cycle

84. Which evaluation method is typically used during the design phase?

(A) User testing
(B) Cognitive walkthrough
(C) Field studies
(D) Full implementation evaluation

Correct Answer: (B) Cognitive walkthrough

85. What distinguishes laboratory studies from field studies?

(A) Control over user behavior
(B) Real-world context
(C) Involvement of more users
(D) Greater cost

Correct Answer: (A) Control over user behavior

86. Which evaluation technique is most likely to yield objective results?

(A) Think-aloud protocol
(B) Cognitive walkthrough
(C) Controlled experiment
(D) User interviews

Correct Answer: (C) Controlled experiment

87. What type of measures do subjective evaluation methods typically provide?

(A) Quantitative measures
(B) Qualitative measures
(C) Statistical analysis
(D) Numeric data

Correct Answer: (B) Qualitative measures

88. What is the main advantage of qualitative measures in evaluation?

(A) They are easy to analyze statistically.
(B) They provide detailed user insights.
(C) They are free from bias.
(D) They offer high-level data only.

Correct Answer: (B) They provide detailed user insights.

89. Which evaluation method is considered intrusive and may affect user behavior?

(A) System logging
(B) Post-task walkthrough
(C) Controlled experiment
(D) Think-aloud protocol

Correct Answer: (D) Think-aloud protocol

90. What factor should be considered regarding the resources needed for evaluation?

(A) The design's aesthetic appeal
(B) The evaluator's experience
(C) The stage of development
(D) The type of users involved

Correct Answer: (B) The evaluator's experience

91. What type of information can controlled experiments typically provide?

(A) Low-level details
(B) Subjective perceptions
(C) Detailed information on specific aspects
(D) High-level usability ratings

Correct Answer: (C) Detailed information on specific aspects

92. Why is it recommended to conduct early reviews in the design process?

(A) They are less costly.
(B) They allow for quicker fixes to issues.
(C) They require fewer resources.
(D) They involve only design professionals.

Correct Answer: (B) They allow for quicker fixes to issues.

93. Which evaluation technique is best suited for gathering high-level user perceptions?

(A) Controlled experiment
(B) Questionnaire
(C) Cognitive walkthrough
(D) Video recording

Correct Answer: (B) Questionnaire

94. What is a potential drawback of post-task walkthroughs?

(A) They provide immediate feedback.
(B) They rely on users' memory, which may be biased.
(C) They involve users throughout the process.
(D) They focus on the design stage only.

Correct Answer: (B) They rely on users' memory, which may be biased.

95. Which evaluation approach allows users to participate at every stage of the process?

(A) Participatory design
(B) Cognitive walkthrough
(C) Laboratory studies
(D) Controlled experiments

Correct Answer: (A) Participatory design

96. Which evaluation method is best for understanding user interactions in real-world settings?

(A) Laboratory studies
(B) Cognitive walkthrough
(C) Field studies
(D) Controlled experiments

Correct Answer: (C) Field studies

97. What is a common concern regarding the use of multiple evaluators in subjective methods?

(A) It can lead to unreliable results.
(B) It may introduce bias.
(C) It increases resource requirements.
(D) It complicates data collection.

Correct Answer: (B) It may introduce bias.

98. What is one main benefit of combining subjective and objective evaluation methods?

(A) It simplifies data analysis.
(B) It reduces resource requirements.
(C) It provides a more comprehensive understanding of user experience.
(D) It eliminates evaluator bias.

Correct Answer: (C) It provides a more comprehensive understanding of user experience.

99. In what situation might simulations be necessary for evaluation?

(A) When the system has been fully implemented.
(B) When access to the system's intended users is impractical.
(C) When the evaluator has extensive experience.
(D) When quick feedback is required.

Correct Answer: (B) When access to the system's intended users is impractical.

100. Which factor is least likely to influence the choice of evaluation method?

(A) The stage in the design process
(B) The evaluator's personal preferences
(C) The resources available
(D) The type of measures provided

Correct Answer: (B) The evaluator's personal preferences.