Cognitive Issues in Autonomous Spacecraft Control Operations: An Investigation of Software-Mediated Decision Making in a Scaled Environment
Chapter 1. Introduction
As advances in technology are applied in various complex, partially automated domains, the human controller becomes increasingly distant from the controlled process. This greater physical and psychological distance may have both helpful and harmful effects on human performance. Decision making, for example, may be helped by greater objectivity but hurt by a lack of involvement. The research described here investigated human decision making in a situational context modeled after advanced, unmanned spacecraft operations.
1.1 Background
For over 30 years, the National Aeronautics and Space Administration (NASA) has been placing unmanned spacecraft in low-Earth orbit for various scientific purposes, such as observations of weather systems and oceanographic features. Until very recently, ground monitoring and control of these spacecraft have required the presence of human operators on a 24-hour-a-day basis. As advances in technology have produced spacecraft that can operate more reliably than their predecessors, continuous human monitoring and control of on-board systems are no longer needed (e.g., Abedini, Moriarta, Biroscak, Losik, & Malina, 1995; Aked & Pylyser, 1996). However, the performance effects of removing humans from the continuous monitoring loop have not received any appreciable experimental attention.
Advanced capabilities take process-control engineers and analysts into a new on-call paradigm, where they are still part of the system but are needed only when the so-called ÒintelligentÓ automation calls for human help. In this model (Murphy & Norman, 1998; Patterson & Woods, 1997), human analysts do not need to be physically present at any particular support facility, and they are not tasked with monitoring mission operations. Their role is analogous to that of an on-call physician (Murphy & Norman, 1998).
Cognitive issues arise, however, when the on-call analyst must intervene because a problem exceeds the scope of the autonomous capabilities. The present research was designed to investigate a sub-set of the cognitive issues faced by decision makers in such environments.
1.2 Definition of Terms
á Autonomous system Ð a natural or artificial entity that exercises control over its own behavior and is able to make decisions without requiring specific outside intervention (cf. Takeuchi & Naito, 1995); an artificially autonomous system consists of hardware and software designed to support such functions as limited fault detection and resolution.
á Artificially intelligent software process (i.e., agent) Ð a programmed capability that exhibits the following characteristics: autonomy, persistence, migration, cloning/spawning, collaboration, and learning (Truszkowski, 1996a, b):
Autonomy Ð capability to perform functions in the background, without requiring human intervention for routine functioning
Persistence - capability to support inter-process activities that extend over significant time periods
Migration Ð capability to shift some of their task load across distributed nodes
Cloning/Spawning Ð capability to copy themselves in support of parallel processing
Collaboration Ð capability to interact with other software processes and with users
Learning Ð capability to add case-by-case experience and user feedback to its knowledge base, showing improved performance over time
Such a process is called an ÒagentÓ in the operational environment at NASA-Goddard (e.g., Truszkowski & Odubiyi, 1994).[1]
á Lights-out operations Ð a term from the field, denoting autonomous, unmanned ground control operations; with enough software autonomy built into the on-board and ground systems, control room personnel can turn out the lights and let the system run itself until such time as human intervention is needed; represents a level of automation beyond supervisory control; referred to here as the on-call model (Murphy & Norman, 1998; Peterson & Woods, 1997).
á Out-of-the-loop (OOL) -- describes the on-call analystÕs lack of moment-to-moment involvement with the controlled process; the on-call analyst is out of the traditional feedback loop that provides continuous updates on system status.
á Supervisory control Ð a level of automation that relieves the human operator of manual interaction with the monitored process, until such time as the automated monitoring capabilities signal that human intervention is required; present in the control room, the supervisory controller essentially monitors the system that monitors the controlled process (Moray, 1986; Sheridan, e.g., 1976, 1988a,b, 1997). The supervisory controller is present at the operational site.
In the remainder of this discussion, the term advanced software process (ASP) is used to refer to a software process with the characteristics listed above under Òartificially intelligent software process.Ó Although ÒagentÓ is the term used by designers and engineers in the field, it seems to go beyond metaphor in over-anthropomorphizing software processes (Shneiderman, 1997). Although some authors state that Òcomputerized procedures can be viewed as cognitive agents with their own monitoring strategies and intentionsÓ (Kontogiannis & Hollnagel, 1998), the term ASP is used here to avoid the pitfalls of anthropomorphizing, primarily the danger of attributing human-like judgment to an inanimate tool.
1.3 Literature Review
The cognitive issues in human interaction with autonomous systems have received little direct research attention. The key cognitive issues are reflected in the following topics for review:
á Effects of automation on human performance
á Trust versus over-reliance on automation
á Passive monitoring in supervisory control
á Cognitive demands in autonomous, ASP-based systems
á Limitations of decision making
á Information-display needs in on-call situations
á Performance effects of spatial visualization ability (SVA)
Previous research in these areas helped in formulating the hypotheses investigated in the present experimental context.
1.3.1 Effects of Automation on Human Performance
When a task that humans could perform or have performed is allocated to the computer, that task is said to have been automated (e.g., Parasuraman, 2000). Wickens (1992) sorts the various purposes of automation into the following categories (pp. 531-532):
1. Performing functions that the human operator cannot perform because of inherent limitations
2. Performing functions that the human operator can do but performs poorly or at the cost of high workload
3. Augmenting or assisting performance in areas in which humans show limitations
Automation that fulfills these purposes may be implemented at various levels, from intermediate gradations of semi-automation to full automation (cf. Endsley, 1997; Mitchell & Sundstrom, 1997; Parasuraman, 1997, 2000; Sheridan & Verplanck, 1978). In various domains, full automation of selected task groupings allows advanced software processes to function at some level of independence from direct human interaction.
Many authors have noted that, for all its promise, automation can negatively affect human performance (e.g., Bainbridge, 1987; Ballas, Heitmeyer, and Perez, 1991; Billings, 1990; Bowers, Deaton, Oser, Prince, & Kolb, 1995; Harris, Hancock, Arthur, & Caird, 1995; Hollnagel, 1992; Hollnagel & Woods, 1983; Hopkin, 1987, 1988, 1991; Kontogiannis & Hollnagel, 1998; Mitchell, 1983; Mosier, Skitka, Heers, & Burdick, 1998; Narborough-Hall, 1987; Norman, 1988; OÕHara, 1993; Parasuraman, 1997, 2000; Reason, 1990; Sarter & Woods, 1995a,b, 1997; Shaiken, 1986; Swain, 1987; Wei, Macwan, & Wieringa, 1998; Wiener, 1987; Wiener & Curry, 1980; Woods, 1994; Woods, Sarter, & Billings, 1997). Based on their findings, many of these authors have suggested that the uncritical acceptance of automation may make successful human intervention difficult, if not error prone, when such intervention inevitably becomes necessary. This possibility is of critical importance to human performance in an on-call environment.
The present research
was designed to investigate potential effects of ASP-based automation on human
decision making and task-completion time. The negative effects of automation
found in previous research suggest, for example, that automated identification
of problems and automated selection of visual displays may degrade human
decision-making performance.
1.3.2 Trust versus Over-reliance on Automation
The issue of too little or too much trust in the automation has sparked considerable discussion and research (e.g., de Keyser, 1986; Eidelkind, 1995; Lee & Moray, 1992; Muir, 1987, 1994; Muir & Moray, 1996; Zuboff, 1988). Noting that some human decision makers may not trust automated tools at all, while others may place too much trust in automated aids, Muir (1987) extends models of interpersonal trust to model human trust in automation.
Citing Sheridan and Hennessey (1984), Muir (1987) suggests that operators in supervisory-control environments, Òespecially novices, may be biased toward distrustÓ of the automation (1987, p. 534). Thus, experimental subjects, who do not have time to develop a professional level of expertise, may be biased toward distrust in the automation. Experts, however, may become accustomed to accepting machine diagnoses and problem resolutions if the automation has been highly reliable over a long period of time.
Indeed, research conducted by Mosier, Skitka, Heers, and Burdick (1998) found that Òincreased experience decreased the likelihood of catching the automation failuresÓ for current pilots of commercial glass-cockpit (highly automated) aircraft (p. 58). In related research, experienced flight dispatchers and pilots accepted flawed recommendations made by an apparently omniscient computer (Smith, McCoy, & Layton, 1997). Experts, then, may tend to be more trusting than novices are of highly familiar or supposedly competent automation.
In experiments on trust, Muir and other researchers have evaluated her 1987/1994 model of trust in the context of supervisory-control environments. In their analysis of operatorsÕ self-reported trust ratings, Lee and Moray (1992) found that level of trust was affected by overall system performance as well as by faults that disrupted system performance. Their experimental data indicate that an operatorÕs use of automatic controllers depends not on trust alone, but on a complex relationship between trust and self-confidence.
In research conducted at NASA-Goddard, Eidelkind (1995) investigated the role of trust in subjectsÕ willingness to delegate a detection task to a semi-autonomous software process. This study is one of the first to explore these issues in the context of ASP-based systems.
Eidelkind suggests that overly high trust in Òa supposedly reliable systemÓ leads to operators taking themselves out of the control loop (p. 47). This is the issue of complacency: operators can become too trusting and overly reliant on the automation (cf. Wiener, 1987). In a similar vein, Eidelkind discusses the potential costs of delegating tasks to advanced software processes (p. 8):
With the elimination of monitoring, delegation may lead to an actual and/or perceived loss of control over the task by the operator. In the case of perceived loss, even if the agent is highly reliable, feeling Ôout-of-the-loopÕ may cause unexpected problemsÉ If the operatorÕs mental model of the agent systemÕs state is actually hindered [by being out of the loop], a total loss of control can occur. When this occurs, the operator either fails to recognize agent breakdowns or, after positive recognition, lacks the ability to retake manual controlÉ
This is the classic view on the dangers of taking the operator out of the loop.
Because EidelkindÕs subjects were engaged in monitoring, they were operating more in a supervisory-control mode than in an on-call mode. In the on-call mode, the out-of-the-loop metaphor is no longer appropriate because the on-call analyst has never been completely in the loop (Murphy & Norman, 1998). The key issue becomes one of providing displays that support rapid, accurate situation assessment and effective intervention by an on-call analyst.
To investigate the demands on the human and the information/display requirements in the on-call model, the experimental design must specifically NOT allow subjects to monitor anything. The current research was designed primarily to compare performance under monitoring (i.e., supervisory control) and non-monitoring (i.e., lights-out, on-call) conditions. Specifically, the question is whether on-call subjects will respond quickly and effectively when they have not been monitoring system status.
Supervisory control has been implemented widely in the control of continuous processes (e.g., oil refining, nuclear power generation), control of vehicles (e.g., air-, sea-, and spacecraft), and robotic manufacturing systems. This paradigm is the norm in many of todayÕs complex, automated systems, both civilian and military. Although these systems have been criticized for taking the operator too far out of the control loop (e.g., by Mitchell, 1983), a key characteristic of supervisory control is that humans are continually present and routinely monitoring 24-hour-a-day system operations.
Cognitive issues arise in the supervisory-control paradigm because the operator serves as a passive monitor for long periods of time (e.g., Bushman & Mitchell, 1986; Lee & Moray, 1992; Mitchell, 1981, 1983; Mitchell & Saisi, 1987; Moray, 1986; Sheridan, 1976, 1988b; Wickens & Kessel, 1979). Vigilance and alertness are known to decline quickly under such conditions (e.g., Mackworth, 1948, 1950; Moray, 1986; Thackray, 1980; Thackray & Touchstone, 1989). When the signals to be detected (i.e., the targets) are infrequent, intermittent, and unpredictable, detection performance declines markedly during the first 30 minutes (Wickens, 1984). Given research findings on the vigilance decrement, researchers have been concerned that supervisory-control operators will not be ready to respond efficiently and effectively when called upon to deal with an anomaly (e.g., Mitchell, 1983; Wiener, 1987).
A major issue, the so-called out-of-the-loop performance problem, is described concisely by Endsley and Kiris (1995, p. 381):
System operators working with automation have been found to have a diminished ability both to detect system errors and subsequently to perform tasks manually in the face of automation failures, compared with operators who manually perform the same tasks.
These authors attribute observed decrements in out-of-the-loop performance to loss of situation awareness (SA) and loss of manual skills.
The between-subjects study conducted by Endsley and Kiris (1995) included five experimental conditions:
1) manual
2) decision support
3) consensual AI
4) monitored AI
5) full automation
The findings of this study provide strong supporting evidence for the classical view of the effects of the supervisory-control role on operator performance: Decisions took longer and understanding of system state was lower in the fully-automated condition. These findings support the earlier findings of Parasuraman and his colleagues, who furthered the empirical study of the cognitive effects of increased automation (Parasuraman, Molloy, & Singh, 1993).
Complacency is another cognitive issue associated with the supervisory-control paradigm (e.g., Molloy & Parasuraman, 1996; Parasuraman, Molloy, & Singh, 1993; Riley, 1994). The concern has been that operators will become overly trusting of the automation and will tend to think that any signal of a problem is really a false alarm (Wiener, 1987; Hopkin, 1988). Complacency has been cited as a contributing cause of the vigilance decrement in monitors of automated systems, i.e., the operator who trusts too much in the automation is less likely to think there is a need to check on what is happening (Bergeron, 1981; Endsley & Kiris, 1995; Parasuraman, Molloy, & Singh, 1993).
Complex interrelationships among factors such as trust, reliance on automation, and situation awareness underlie human performance in supervisory-control systems, including the more highly automated, transitional versions that incorporate some autonomous software processes. The present research was designed to investigate and explicate possible relationships between software-reported confidence in the automated problem diagnosis and two dependent variables: subjects' self-reported confidence[2] in their own decisions and subjects' self-reported reliance on the automated diagnosis.
1.3.4 Cognitive Demands in Autonomous, ASP-based
Systems
The literature on autonomous spacecraft systems tends to focus on the technical, engineering issues involved in achieving autonomy (e.g., Harvey, 1996; Lecouat & De Saint Vincent, 1996; Klein, Kulp, & Rashkin, 1996) and how to develop adaptive, ASP-based systems (e.g., Barber, Goel, Liu, Macfadzean, Martin, & Ramaswamy, 1997). Spacecraft engineers generally assume that mission analyst will be able to solve any problem requiring human expertise. Little or nothing has been said about demands of any kind that might be placed on human cognitive capabilities by this kind of situation. It is assumed that it will be possible to page the expert and that the expert will be able to deal with the problem, perhaps even from home at 2:00 a.m. (e.g., Abedini, Moriarta, Biroscak, Losik, & Malina, 1995; Aked & Pylyser, 1996; Hucteau, 1996). Issues of situation assessment and decision making typically go unmentioned in the engineering literature.
These issues have been articulated from a human-factors perspective (e.g., Truszkowski, 1996b). As described in his presentation, software agents occupy a virtual position between the human-computer interface and the command-and-control system. Instead of dealing directly with sensor information and historical trends in spacecraft health-and-safety data, the analyst will need information on how the ASPs came to the conclusion they did about the alleged anomaly.
A problem is that the ASPs will not be 100% reliable, and the analyst will need some way of validating the ASPsÕ conclusions. The cognitive demands associated with verifying automated decisions are described by Kontogiannis and Hollnagel (1998, p. 255, italics added):
Checking the reliability of automated decisions
is not an easy taskÉOperators may have to
undertake additional verification tasks, such
as deciding what data have been consulted by
the system, how the system handles unreliable
data, whether the system has perceived the
state of the process correctly, and so on.
In the present experimental task, subjects were required to decide whether the systemÕs perception of the onboard situation was correct, that is, the subjects were required to validate the simulated spacecraft's automated decisions.
As noted by Bainbridge (1997), it is sometimes ÒnecessaryÉto interpret the situation, rather than to assume that the situation is only and exactly what can currently be sensed in the environmentÓ (p. 352). The possibility that information provided by ASPS may be in error or may be incomplete imposes cognitive demands and raises performance issues for the analysts who are charged with fault resolution.
Publications based on a NASA-Goddard field study articulate the key cognitive issues associated with increased mission autonomy. For example, Murphy, Norman, Truszkowski, and Grubb (1997) note that empirical research is needed on the cognitive and human-performance effects of lights-out (on-call) automation. The present experimental environment was developed to begin investigating these effects.
1.3.5 Limitations in Decision Making
Human judgment and decision making have been widely studied from the rationalist, normative perspective, with the common finding that people are non-optimal decision makers (e.g., Kahneman & Tversky, 1972, 1973, 1982; Tversky & Kahneman, 1973, 1974). Human decision-makers do not consider all alternatives and their consequences but tend to rely on what has worked in the past.
As shown in the many studies summarized by Von Winterfeldt and Edwards (1986), Òhuman inference is routinely conservativeÓ (p. 533). This is thought to be the case because people do not fully extract the information available in cues that can reduce uncertainty, i.e., diagnostic cues, and because they consider all cues as Òequally informativeÓ (Wickens, 1992, p. 270). As suggested by Wickens (1992), this may occur because of a general need to reduce the load on short-term memory.
Given the limitations of human decision making, it would seem that automated aids might be called for. Designers cannot assume, however, that an automated decision aid will improve performance. For example, an evaluation of a fault-detection aid for a nuclear power plant simulation found that
aided operators performed better when
they had to diagnose malfuctions caused
by multiple failures, and when they had
not practice during their training
(Sassen, Buiel, & Hoegee, 1994).
In the cited study, having the aid did not improve subjects' diagnosis of malfunctions caused by single failures or their diagnosis of practiced problems; nor did having the automated aid increase the aided groupÕs confidence in their diagnoses.
A further limitation of decision making is people's tendency to fall back on general rules-of-thumb or heuristics when asked to make probability estimates or judgments. These educated guesses or short cuts will work sometimes, under some circumstances, but come with no guarantee of making a correct decision. It is thought that people tend to settle on heuristic solutions in part because more complex strategies impose greater demands on limited working memory (e.g., Reason, 1990; Wickens, 1992). Over 25 heuristics and biases have been identified (Kahneman, Slovic, & Tversky, 1982; Sage, 1981). A few examples of heuristics follow (Tversky & Kahneman, 1974):
á Anchoring Ð in revising a hypothesis or belief, the tendency to shift up or down only slightly from a mental setpoint established by the first item of evidence, the anchor
á Availability Ð judging the probability of A by the ease of bringing instances of A to mind
á Representativeness Ð judging the probability of A by the extent to which it resembles B
The
present research examines anchoring and adjustment from the displayed level of
agent confidence in the problem diagnosis.
Decision making is further constrained by a low correlation between confidence in one's decision and the accuracy of the decision (Plous, 1993). People may be extremely confident of a very wrong answer. As Loftus (1979) warns, ÒOne should not take high confidence as an absolute guarantee of anythingÓ (p. 101). In LoftusÕs research context, the eyewitness may be very sure in identifying the wrong person as a criminal. Although the results from eyewitness testimony bear on recall accuracy, they may be understood in terms of judgment and decision accuracy as well: The eyewitness must make a metacognitive judgment about his or her level of certainty and decide whether recall certainty is sufficient.
The present research investigated subjectsÕ confidence in their judgments of the accuracy of ASP-based anomaly reports. The expectation was that subjects' confidence in their decision would not be positively correlated with their accuracy, i.e., that previous findings would be replicated.
1.3.6 Information-Display Needs in On-Call Situations
An assumption made in planning for lights-out operations holds that fault-diagnosis-and-resolution aids should take the form of or incorporate capabilities for providing analysts with two-dimensional (2-D) and three-dimensional (3-D) displays of mission data (e.g., bar graphs, line graphs, pie charts). Such displays seem intuitively superior to tabular displays, e.g., the rows and columns of numbers traditionally presented to operators in mission control rooms.
Under some conditions, however, tables are superior to graphs in supporting performance (Meyer, Shinar, & Leiser, 1997). Thus, it is unclear whether graphical displays will improve usersÕ accuracy in evaluating problems reported by autonomous software processes. The need for graphics may depend on the nature of the problem (Boehm-Davis, Holt, Koll, Yastrop, & Peters, 1989).
The literature on textual versus graphical display design has generally been interpreted to support the conclusion that graphical displays are beneficial to tasks with spatial components, but not to purely symbolic tasks (e.g., Benbasat & Todd, 1993; Vessey, 1991). Research findings demonstrate that different graphical formats have different effects on judgment and decision making (e.g., MacGregor & Slovic, 1986; Meyer, Shinar, & Leiser, 1997). Tasks best supported by textual displays include determining specific numerical values (for example, 3752.0964). Tasks best supported by graphics include making comparisons that do not require highly specific values; determining the highest/lowest, biggest/smallest component; and extracting trends from data gathered over time (e.g., Boehm-Davis, Holt, Koll, Yastrop, & Peters, 1989; Dickinson, DeSanctis, & McBride, 1986).
As noted by Boehm-Davis and her colleagues (1989) and Shneiderman (1998), choosing an appropriate visual display depends on an understanding of the task to be performed and the kind of data available. In the domain of spacecraft control, where the primary tasks are monitoring for anomalous conditions and diagnosis and resolution of anomalies, it does not appear than anything more sophisticated than a 2-D trend display is considered necessary or operationally suitable by current operational personnel (Murphy, Norman, & Moshinsky, 1999). This seems to be the case even though the performance benefits of highly graphical and dynamic data displays have been demonstrated in NASA-like supervisory-control environments (e.g., Mitchell & Saisi, 1987).
In contrast to NASA's expectation that lights-out analysts will need graphical displays, experience on the Extreme Ultraviolet Explorer (EUVE) mission has been that textual problem files and optionally available trend displays are adequate to support the paged analystÕs process of anomaly isolation and resolution (Stroozas, personal communications, March 26, 1998; April 16, 1998). It is, thus, unclear whether visual formats other than 2-D trend displays will be needed in any lights-out environment.[3]
Whether implicit or explicit, an objective of all information-display efforts is to reduce requirements for users to remember commands, operations, and navigational paths (Norman, 1994). The most promising approaches are those that Òmake apparent hidden links and logical contingencies, andÉthat allow the user to perform spatial and intermediate operations on the interface rather than in the headÓ (Norman, 1994, p. 203). Effective visual displays help to overcome individual differences in spatial abilities that favor one group of users over another.
The current research compared various 2-D display techniques for their effectiveness in supporting subjects' performance on the experimental tasks. The expectation was that tables and bar charts would be superior to line graphs in supporting performance because the experimental task did not require trend detection.
1.3.7 Performance Effects of Spatial Visualization
Ability
Spatial visualization ability has been defined in similar ways by various researchers, e.g.,:
á the ability to deal with complex visual problems that require imagining the relative movements of internal parts of a visual image (Pellegrino & Hunt, 1991, p. 205)
á the ability to manipulate or transform the image of spatial patterns into other arrangements (Ekstrom, French, Harmon, & Dermen, 1976, p. 173)
á the mental manipulation of spatial information to determine how a given spatial configuration would appear if portions of that configuration were to be rotated, folded, repositioned, or otherwise transformed (Salthouse, Babcock, Skovronek, Mitchell, & Palmon, 1990, p. 128).
Mental manipulation and transformation are common elements of these overlapping definitions.
SVA is becoming widely recognized as Òthe primary cognitive factor driving differences in performance using computersÓ (Norman, 1994, p. 195): Those with high SVA perform well on computer tasks, but those with low SVA perform poorly. SVA was investigated as a mediator of performance in the present research because of its significant relationship to performance in other computer-based research (e.g., Alonso, 1998; Butler, 1990; Norman & Butler, 1989; Vincente, Hayes, & Williges, 1987).
Because low-SVA people are less able to rely on the mental imagery that seems to come naturally to high-SVA people, tasks that impose moderate processing demands on high-SVA subjects may impose high demands on low-SVA subjects (Alonso, 1998; Salthouse, Babcock, Mitchell, Palmon, & Skovronek, 1990). The strong implication drawn by these researchers is that low-SVA subjects will approach the limits of working memory sooner than will high-SVA subjects.
In a computer-based environment, where navigation of large, intricate data bases and complex menu structures is inescapable (cf. Norman, 1991b), low-SVA subjects may be at a distinct and widening disadvantage (Alonso, 1998; Norman, 1994). This disadvantage may be especially incapacitating when a low-SVA user is faced with a user interface low in apparency, i.e., one that hides the relationships among displayed and non-displayed objects and possible user operations (e.g., Alonso & Norman, 1996). In contrast, a highly apparent or intuitive user interface makes underlying contingencies transparent to the user.
The present research investigated the influence of SVA on performance that was supported by various 2-D approaches to information display. Based on previous findings in the literature, the expectation was that low-SVA subjects would be slower and less accurate compared to the high-SVA subjects.
1.4 Research Design
The overall research design is summarized in Figure 1.
1.4.1 Independent Variables
In this 2 X 3 X 3 X 2 factorial design, some variables were manipulated between-subjects and some within-subjects. The level-of-human-involvement variable was manipulated between subjects to reflect the real-world distinction between supervisory control and on-call environments: The key difference is that human operators are present and monitoring system operations under a supervisory-control paradigm, but they are not present in the control room and typically not monitoring system operations in an on-call context.
Both software-agent confidence and display-selection mode were
manipulated within subjects.
Monitoring | No
Monitoring
Display- Automated Manual | Automated Manual
Selection
Mode
Figure 1. Research design
These manipulations reflected real-world conditions under which
human operators are exposed to varying levels of software reliability and
varying levels of automation within the same system (cf. Parasuraman, 2000).
Software-based versus manual display selection reflected the likelihood that real-world automation of
display selection might be feasible in some cases but not in others. Treating display selection as
a within-subjects variable allowed observation of the effects of a varying
level of automation on subjectsÕ performance.
A fourth independent variable, type of display, was manipulated within subjects. The display types were table, bar chart, and line graph. For each practice and test problem, the data were available to the simulation in the form of a table, a bar chart, or a line graph. In the software-based mode of display selection, the choice among the three types was made at random. In the manual-selection mode, subjects were given the choice of displaying the situational data in tabular form, in a bar chart, or in a line graph.
1.4.2 Dependent Variables
SubjectsÕ performance was measured in terms
of speed (time to respond to an alert, time to complete a task) and the
accuracy of their situational assessments. Response time was measured from the onset of an alert to the
subjectÕs acknowledgment of the alert.
Task-completion time was defined as the time from acknowledgment of an
alert to submission of a decision, minus the time for entering a free-text
rationale for the decision. Accuracy was determined by comparison of the subjectÕs
problem diagnosis with an answer key.
Another dependent variable, type of display
selected in manual-selection mode, was recorded as (1) table, (2) bar chart, (3) or timeline. A confidence
rating was collected for each decision made by a subject. The confidence ratings were reported on
a scale from zero to 100 percent, at 10-unit intervals (e.g., 50, 60, 70, 80).
Prior to
the experimental session, subjects completed a short questionnaire on their
attitudes toward automation and their preferences for graphical versus tabular
displays. After the experimental
session, subjects completed a questionnaire designed to measure their attitudes
toward the simulated software and their reliance on agent confidence. Both
questionnaires presented statements that subjects rated on a nine-point scale
for their level of agreement or disagreement (1 = strongly disagree, 9 =
strongly agree). Responses to the pre- and post-test questionnaires provided
input to a repeated-measures analysis of changes in attitudes toward automation.
1.5 Hypotheses
1.5.1 Hypothesis 1 (H1)
Monitoring subjects will be quicker to submit their answers and more accurate in comparison to subjects in the on-call group.
Rationale: This hypothesis is based on similar predictions in the literature on monitoring behavior and supervisory control. This result is expected because the on-call subjects will have been engaged in other, unrelated activities in between problems. It should take them longer to get back into the problem-solving process and to submit their answers to problem alerts. According to this line of thinking, accuracy is likely to suffer from forgetting due to interference by the distractor task, which the monitoring subjects will not have been experienced. The on-call model predicts, however, that performance of the on-call subjects will be at least as good as performance of the monitoring subjects.
1.5.2 Hypothesis 2 (H2)
Subjects who are given the option of selecting the display type for more than half of the test tasks will choose tables and bar charts more often than they choose timelines.
Rationale: Subjects will find it easier to extract the values that they need from tables and bar charts as compared to line graphs.
1.5.3 Hypothesis 3 (H3)
Decision accuracy will be better for tables and bar charts than it will be for line graphs.
Rationale: Tables and bar charts are better suited for supporting the comparison of actual values with normal ranges. Tables give exact values, which the subject can then compare with the values given in the agent rationale. The height of the bars in a bar chart gives a visual cue to relative value and directly supports comparison with other bars. Line graphs, however, are better suited to detection of trends over time because each line connects the points at which observations were taken for specific components.
1.5.4 Hypothesis 4 (H4)
The displayed level of agent confidence will serve as an anchor from which subjects will adjust their own level of confidence in their answer to a specific problem.
Rationale: When given a value in a problem statement, subjects use that value as an ÒanchorÓ and adjust their estimate in either direction from the anchor (Kahneman, Slovic, & Tversky, 1982). The assumption here is that the value given for agent confidence will serve as an anchor from which subjects will adjust their own confidence level.
1.5.5 Hypothesis 5 (H5)
a. There will be no significant relationship between subjective confidence ratings and accuracy.
b. Low subjective confidence will be negatively (inversely) correlated with task-completion times.
Rationale: a. Published research often reports a lack of correlation between subjective confidence and decision accuracy (e.g., Fischhoff, Slovic, & Lichtenstein, 1977; Loftus, 1979; Plous, 1993 ). In some situations, people tend to be overconfident, that is, certain of wrong answers. Highly confident eye witnesses can identify the wrong suspect. Although domain experts are typically well calibrated, non-experts who are high in confidence will not necessarily be high in accuracy or have low response times. b. Low confidence logically implies the need to spend more time solving a problem or making a decision. Thus, low confidence should be associated with high task-completion times.
1.5.6 Hypothesis 6 (H6)
Self-reported reliance on the automation will be higher in the on-call condition than in the monitoring condition.
Rationale: Low human involvement makes the subject more dependent on the software. The on-call subject will be forced to think that the automation is reliable. The monitoring subject may be less reliant on the software because of greater familiarity with the normal operations of the simulated MOCHA system.
1.5.7 Hypothesis 7 (H7)
On-call subjects will report that automated systems need less monitoring than will subjects in the monitoring condition.
Rationale: Based on their experience with MOCHA, the on-call subjects can be expected to realize that it is not necessary for them to be paying full attention to MOCHA in order to solve the problems. However, the monitoring subjects can be expected to develop the attitude that monitoring is necessary because that is what they were required to do. To some extent, this would justify the severe boredom that they experienced, i.e., if I had to go through that, it must be necessary. People tend to base their assessment of what is needed on their own experiences in similar situations.
1.5.8 Hypothesis 8 (H8)
High-SVA subjects will perform better in both the monitoring and on-call conditions (i.e., their response times will be shorter and their accuracy better than for low SVA subjects).
Rationale: High SVA gives an advantage over low SVA for faster completion of computer-based tasks (Vincente, Hayes, & Williges, 1987). Since this effect has been found for various computerÐbased tasks (e.g., Alonso, 1998) and has been described as pervasive (Norman, 1994), there is every reason to expect to find similar effects in the MOCHA environment.
Chapter 2. Method
2.1 Participants
Undergraduate
psychology students at the University of Maryland were randomly assigned to the
monitoring or on-call conditions and received class credit for their
participation. Fifteen subjects
participated in pilot studies and 83 in the experiment. Data were analyzed for 42 men and 41
women. Ages ranged from 18 to 28
years old (mean = 19.9, standard deviation = 2.2).
Class
status ranged from freshman to senior (28 freshmen, 29 sophomores, 15 juniors,
and 11 seniors). Levels of other demographic variables are summarized in Table
1:
Table 1
Self-reported Experience on a Nine-point Scale
(1 = Novice, 9 = Expert) (N = 83)
|
Experience withÉ |
Minimum |
Maximum |
Mean |
Std.
Deviation |
| Computers |
1.00 |
9.00 |
5.74 |
1.71 |
| Tables |
3.00 |
9.00 |
5.87 |
1.58 |
| Line graphs |
3.00 |
9.00 |
6.23 |
1.52 |
| Bar graphs |
3.00 |
9.00 |
6.51 |
1.60 |
Subjects rated the
usefulness of graphics at a mean of 6.81 on a nine-point scale (1 = Useless, 9
= Very useful), with a standard deviation of 1.63 (min = 1, max = 9).
2.2 Materials
A
consent form approved by the University of Maryland's Institutional Review
Board (IRB) was used to document subjects' informed willingness to participate
in the MOCHA experiment (Appendix A).
A paper-and-pencil, pre-test survey of attitudes toward automation
(Appendix A) included 12 statements, which subjects rated on a nine-point scale
(1 = disagree, 9 = agree). A standard test of spatial-visualization ability
(SVA)[4] was programmed for on-line
presentation to subjects. (See Figure 8 in Appendix B for a sample problem.)
Background training materials (Appendix C) were developed by the experimenter
to give subjects a basic overview of spacecraft-control operations and to
define terms that they would encounter in the experiment.
An
on-line survey of consumer attitudes and expenditures (Appendix D) was used for
the distractor task in the on-call condition. Items in the distractor survey
were adapted from such sources as USA Today, Information Week, Time
Digital,
and questionnaires developed by the U. S. Census Bureau. A paper-and-pencil
survey of post-test attitudes toward automation included 11 items (Appendix A),
which subjects rated on a nine-point scale (1 = disagree, 9 = agree). The
post-test items generally substituted "MOCHA" for the term
"automation" in some pre-test items, but items were added asking
about the extent to which subjects relied on the agent's confidence in making
their decisions and in setting their own confidence.
2.3 Simulation Environment
With
support from the NASA grant, a simulation for use in experimental research was
developed in the Laboratory for Automation Psychology. The experimenter dubbed
the simulation "MOCHA", an acronym for Mars Observer Calls
Home Again.[5] The design of MOCHA was
informed by interviews with personnel at NASA-Goddard Space Flight Center and
by materials gathered from the World Wide Web about the Mars Observer and other
unmanned spacecraft (e.g., NASA, 1993, 1997).
Both the
design of MOCHA and the experimental conditions were influenced by the
Lights-Out Ground Operations System (LOGOS) Program at NASA-Goddard and by
discussions with LOGOS personnel.
Other sources were consulted for technical information on spacecraft
components and operations (e.g., Carraway, 1996; Fleeter, 1996; Lord, 1996;
Marshall, Landshof, & van der Ha, 1996; Morrison, 1996; Neal, Lewis, &
Winter, 1995). Personnel at the Johns Hopkins Applied Physics Laboratory
described their personal experiences with day-to-day operations and spacecraft
anomalies on the NEAR and MSX missions. The experimenter also drew on over 10
years of experience conducting human factors analyses, including cognitive task
analyses, in NASA-Goddard's mission control centers, as documented in, e.g.,
Fischer and Murphy, 1983; Murphy and Mitchell, 1986; Sheppard, Murphy, and
Stewart, 1985, 1991; Stewart and Murphy, 1984.
The
resulting MOCHA simulation roughly mimics some of the characteristics of actual
autonomous spacecraft operations. As such, it represents a "scaled"
environment, which retains key features of the real environment but reduces
real-world complexity (Ehret, Gray, & Kirschenbaum, 2000). Reflecting some
of the capabilities envisioned for the LOGOS prototype, the MOCHA concept
includes a simulated fault-detection-and resolution ASP that is, by definition,
capable of detecting and resolving simple anomalies (i.e., system faults). The
problem statements imply that software of this kind is onboard MOCHA, but there
is no actual fault-detection software. Again, by definition, this ASP operates
autonomously until unable to resolve an on-board problem. As do its real-world
counterparts, the fault-detection ASP alerts the analyst/subject when there is
a situation that it cannot handle. Examples of problem alerts and other MOCHA
screens are provided in Appendix B (Figures 7 through 16).
As
illustrated in Appendix B, MOCHAÕs user interface (UI) provides graphical views
of the system data associated with problem reports. One view of the data for
each problem is either chosen for presentation by another simulated ASP or
requested by the subject. The UI provides a means for the subject to view
parameter values and trends in spacecraft health-and-safety data. The UI permits the subject to
select a response to each alert, to enter free-text explanations of responses,
and to enter subjective confidence in each response. Subjects are permitted to change their response an unlimited
number of times before submitting it.
The MOCHA simulation was developed on a
Windows NT platform in VisualCafŽ, a development tool for building JavaTM
applets and applications (Symantec, 1995). MOCHA displays were presented to
subjects on a 20-inch monitor. On-call subjects accessed displays for the
distractor task via the World Wide Web using the Netscape browser on a 17-inch,
color-synchronized Macintosh monitor. The MOCHA simulation and the distractor
survey are available from the Laboratory for Automation Psychology.
2.4 Procedure
2.4.1 Pilot Studies
To
evaluate whether subjects were able to perform the experimental tasks and to
fine-tune the design, several pilot studies were conducted. Fifteen students
participated in these iterative studies. Because it was found that the training
portion of the session was taking over one hour, the training materials were
condensed, and the number of practice problems was reduced from ten to
six. For the same reason, the
number of test problems was reduced from 18 to 10. These changes made it
possible to complete an entire session in 90 to 120 minutes. The pilot studies
indicated that subjects were able to develop a strategy for solving the
problems and that they were able to navigate through the experimental task.
2.4.2 Pre-Experimental Procedure
Participants
read and signed the consent form and provided various demographic data, e.g.,
age, sex, level of general computer experience (Appendix A). They completed a pre-experimental
questionnaire on their attitudes toward automation, similar to those used by
Muir (1987), Lee and Moray (1992), and Eidelkind (1995). This questionnaire is
provided in Appendix A. As an
introduction to MOCHA and spacecraft terminology, the test administrator
described the Mars Observer graphic (Figure 7 in Appendix B).
Next,
subjects took the on-line version of the VZ-2 test of spatial-visualization
ability (Ekstrom, French, Harman, & Dermen, 1976). Figure 8 (Appendix B)
provides a sample problem. Subjects were given six minutes to complete the SVA
test. At the end of that time or
after they had completed all of the problems, subjects received on-line feedback
on the number right out of 20 problems. VZ-2 scores were saved to a database
for later analysis.
In the
training portion of the session, all subjects read basic background information
on spacecraft control and completed six practice tasks before starting the
experimental session. Training
occurred before subjects were told either to monitor status messages
(monitoring condition) or to work on the survey (on-call condition). Training included background reading on
the basics of NASA ground control of planetary missions (Appendix C).
The MOCHA environment provided
additional training by displaying information associated with each element in
the hierarchy of spacecraft components (Figure 9, Appendix B). By clicking on a component, subjects in
training received a text description of that componentÕs function in the
overall system and information on the normal range of parameters associated
with that component (e.g., temperature, pressure). The purpose of the spacecraft simulation and the training
materials was to put the relatively easy problems into a fairly complex,
real-world context. When taken out of context and stated in generic terms, the
problems involve simple comparisons.
Following
their background reading and traversal of the component hierarchy, subjects
were given a set of six practice tasks.
The test administrator explained the steps involved in navigating
MOCHAÕs user interface, but did not explain how to solve the problems.[6] The subject had to develop his or her own strategy for
deciding whether a problem was as reported, a false alarm, or a different
problem in the same sub-system.
The correct strategy for making a decision
was to compare the normal range of values given in the problem statement with
the values represented in the display.
If the component reported as out-of-range in the problem statement was,
in fact, out-of-range, the answer was Òproblem as reported.Ó If a different component in the same
sub-system was out-of-range, it was a different problem; and, if no components
were out-of-range, it was a false alarm. Training and practice typically took
between 30 and 45 minutes.
2.4.3 Experimental Procedure
Subjects in both conditions completed 10
test tasks, which were presented in a randomly chosen order for each subject.
2.4.3.1 Monitoring Condition
After completing the pre-experimental phase,
each monitoring subject was asked to focus attention on the main MOCHA
screen. Until an alert occurred, a
monitoring subject viewed status messages that appeared in sequence in the
lower left portion of the screen. Figure 10 in Appendix B illustrates the
status messages, which report on the normal, autonomously controlled activities
of spacecraft systems and instruments.
When an alert occurred, the subject received
an alert message, which was displayed in the problem description area in the
lower left portion of the screen (e.g., Figures 11, 12, and 13 in Appendix
B). An alert message specified the
source of the problem and the software's (i.e., agent's) confidence in the diagnosis.
The subject was to acknowledge the alert by clicking on a button labeled
ÒAcknowledge Alert.Ó This action
stopped the Alert heading from blinking in red.
In the automated display-selection mode, the
monitoring subject then viewed a table, bar chart, or timeline that provided
information on the problem. One of these three display formats was
automatically presented at random. In the manual display-selection mode,
subjects received a dialog box asking them to choose the display format (table,
bar chart, or line graph) for viewing diagnostic information (Figure 14,
Appendix B). Based on the strategy
used to make a decision, the subject determined whether the problem was as
reported in the alert; whether there was a different problem; or whether the
alert was a false alarm. The
subject selected a radio button corresponding to one of these decisions and was
then asked to provide a free-text rationale for the decision (Figure 15,
Appendix B). Finally, the subject
was prompted to enter a level of confidence in his or her decision (by setting
the indicator on the slider bar displayed in the response area). The subject then submitted his or her
response, received feedback on the accuracy of the decision, and returned to
the monitoring task.
2.4.3.2 On-call Condition
After completing the pre-experimental phase,
each on-call subject was asked to focus attention on a distractor task. This group of subjects did not monitor
status messages while waiting for an alert. In this condition, the subject
worked on a survey of opinions and consumer activity (Appendix D) at a
Macintosh terminal located adjacent to the MOCHA workstation.
When an alert occurred, the MOCHA user
interface emitted a tone to attract the on-call subjectÕs attention. This subject then followed the same
procedure described above for the monitoring subject:
1. Acknowledge the problem
alert.
2. Read the problem statement
and the range of normal values.
3. View a display of subsystem
status, as provided by the automation or as manually selected (table, bar
chart, or timeline).
4. Compare the information presented in the problem description
to the information given in the table, bar chart, or timeline.
5. Decide on the nature of the
problem and select the corresponding descriptor in the response area (Problem
as reported, False alarm, or Different problem.)
6. Enter a free-text rationale
for the decision.
7. Enter a level of confidence
in the decision.
8. Submit the response and
receive feedback.
In both conditions, feedback
had three components:
1) a statement of the
subjectÕs answer; 2) output of the subjectÕs confidence level; and 3) the
correct answer. Feedback was
displayed for three seconds. The on-call subject then returned to the
distractor task, which was intended to mimic some unrelated activity that an
actual on-call analyst might be engaged in when not monitoring spacecraft
operations.
2.4.4 Data Capture and Analysis
The simulation software recorded the
following data elements for each subject: control number, grouping condition,
VZ-2 score, time to acknowledge alerts, display-selection mode for each
problem, display type chosen or displayed automatically, order of problems,
agent confidence for each problem, answers to problems, free-text explanations
for decisions, time to enter free-text explanations, confidence levels for
decisions, and task-completion times. Output files were imported into the
Statistical Package for the Social Sciences for analysis (SPSS, 1997).
Chapter 3. Results
The alpha level for significant findings was
generally set at 0.05. In some
cases, results are reported if p < .10, since this was largely an exploratory
study.
3.1 Effects of Practice
As shown
in Figure 2, practice was effective in bringing mean submission time to an
asymptote of 35.13 seconds for the final practice task (N = 83, s.d. = 14.27).
All subjects received the same practice tasks in the same order. Although the
experimental tasks were given in random order, none of their mean submission
times exceeded the asymptote.
3.2 Monitoring versus On-call Group Differences
Subjects
were assigned at random to experimental conditions and had not been exposed to
the experimental treatment when they took the test of spatial-visualization
ability (SVA). However, analysis of variance showed that the groups differed in
SVA (F(1,80) = 4.97, p < .05).
The monitoring group had a higher mean SVA (mean = 13.42, s.d. = 3.78)
than did the on-call group (mean = 11.54, s.d. = 3.85).
|
|
Figure
2. Mean task-completion time (in
seconds)
reaches asymptote over six
practice tasks
On the
question of whether the monitoring and on-call groups differed in performance
time or accuracy (H1), tests of the between-groups effects showed no
significant difference on either decision accuracy (score on the test problems)
or speed (task-completion time).
Similarly,
there was no significant difference between groups in the time to acknowledge
problem alerts for the test tasks, although it was reasonable to expect that
the on-call group would be slower than the monitoring group. A multivariate
analysis of variance data, with SVA entered as a co-variate, yielded findings
of no differences between groups on the accuracy and speed measures.
An
unexpected finding was that the groups differed in acknowledgment time for the
practice tasks (F(1,81)= 10.908, p < .01), even though
practice occurred before the monitoring and on-call conditions were
implemented. Mean acknowledgement time on the practice tasks was 34.7 seconds
(s.d. = 22.9) for the subjects who were to act as monitors during the test
session (N = 41) and 64.2 seconds (s.d. = 52.6) for those who were to act as
on-call analysts during the test session (N = 42).
Another
unexpected finding was an interaction between grouping condition and sex for
decision accuracy (F(1,79) = 3.932, p = .05). As shown in Table 4, women in the
on-call condition differed from men in both conditions and from female monitors
in the accuracy of their responses to the MOCHA problems:
Table 4
Interaction of MOCHA
Grouping Condition and Sex on Decision Accuracy (N = 83)
Human
Involvement Prior to Alert
Male
7.79 (2.11) 8.10
(2.25)
Female 8.24 (1.64) 6.63
(2.53)_________
This interaction is
illustrated in Figure 3.
|
|
MSC Time
1.02 5.78 .02_ns_
Hypothesis
|
Result
|
|
H1: Monitoring subjects will be quicker
to submit their answers and more accurate in comparison to subjects in the
on-call group. |
No
significant difference for either accuracy or speed; on-call women less
accurate as compared to men and women in other groups. |
|
H2:
Subjects who are given the option of selecting the display type for more than
half of the test tasks will choose tables and bar charts more often than they
choose line graphs. |
High-manual
selection group showed a preference for bar charts and an aversion to line
graphs as compared to group given only 1-5 opportunities to select display
format. |
|
H3:
Decision accuracy will be better for tables and bar charts than it will be
for line graphs. |
No
significant differences among the display formats for decision accuracy;
performance with tables was significantly faster than with line graphs. |
|
H4: The displayed level of agent
confidence will serve as an anchor from which subjects will adjust their own
level of confidence. |
Subjects
adjusted their confidence up for agent confidence of .5 or .7 but down for
agent confidence of .9, suggesting a non-linear relationship or possible
ceiling effect. |
|
H5a: There will be no significant
relationship between subjective confidence ratings and accuracy. |
Subjective
confidence and decision accuracy were significantly correlated, irrespective
of grouping condition. |
|
H5b: Low subjective confidence will be
negatively correlated with task-completion times. |
No
significant relationship found between subjective confidence and
task-completion time. |
|
H6: Self-reported reliance on the
automation will be higher in the on-call condition than in the monitoring
condition. |
No
significant difference between monitoring and on-call conditions for the
self-reported effect of agent confidence on problem solutions or own
confidence; low-SVA group reported relying more on agent confidence than did
the medium- or high-SVA groups. |
|
H7: On-call subjects will report that automated
systems need less monitoring than will subjects in the automated condition. |
No
significant difference between monitoring and on-call groups; low-SVA
subjects increased their rating of the need to monitor automated systems from
pre- to post-test. |
|
H8: High-SVA subjects will perform better
in both the monitoring and on-call conditions (i.e., their response times
will be shorter and their accuracy better than for low SVA subjects). |
SVA
was significantly correlated with decision accuracy but not with time to
acknowledge problem alerts or task-completion time; SVA was found to be a
stronger predictor of accuracy for men than it was for women. |
The
difference between the monitoring and on-call groups in SVA could be seen as a
confounding factor (K. J. Klein, personal communication, September 3, 2000). It
might be expected, for example, that the monitoring group would show greater
accuracy on the test problems because of their higher mean SVA (13.45 (3.78)
versus 11.5 (3.85) for the on-call group). However, since the two groups did
not differ on any of the performance measures, the difference in SVA apparently
did not have this effect. This may
be the case because both SVA means fall within the medium range (10-14 SVA
problems correct) of the SVA groups and because the variability on SVA within
the monitoring and on-call groups was essentially identical.
Alternatively, the difference in SVA may have affected the performance data in
such a way that the confounding effects of SVA masked a true performance
difference between the groups (N. S. Anderson, personal communication,
September 5, 2000).
If SVA is more predictive of performance for males than it is for females, jobs that require mental rotation and projection of displayed objects may be more difficult for low-SVA males than they are for low-SVA females. Since low-SVA appears to be a cognitive disability, there is a need for user-interface design that is sensitive to users with this inherent problem.
4.9 General Discussion
To what extent can the findings of this research be generalized beyond the laboratory, beyond MOCHA, and beyond the population of college students? Although the results reported here may not generalize in the strict sense beyond the laboratory or beyond MOCHA, issues have been considered that are crucial to the success of human analysts in out-of-the-loop operations, not only in aerospace domains, but also in other process-control industries. The MOCHA research provides a baseline for much-needed further work, both in the laboratory and in the field, using more complex decision-making tasks.
A strong case has been made for the validity and usefulness of research using scaled worlds (Ehret, Gray, & Kirschenbaum, 2000), with operational personnel as research participants. Even though scaled worlds present low-fidelity simulations to participants, the relatively simple tasks presented in MOCHA only faintly represent tasks that actual operators might perform in autonomous spacecraft control. MOCHA probably does not qualify as a low-fidelity simulation of autonomous spacecraft operations for any but the most novice research participants. To use a MOCHA-like system with actual spacecraft analysts, the tasks would need to be made far more realistic and complex, possibly involving simultaneous anomalies in different components (K. Moe, personal communication, HCIL Open House, June, 1999). In an actual autonomous-control environment, the software would be handling all but the most knotty problems (i.e., those that cannot be reduced to rules or algorithms). Analysts might also be dealing with more than one spacecraft. It would also be informative to study participants' performance over time (K. J. Klein and B. Shneiderman, personal communications, September 20, 2000).
Adding an "apparency condition" to a future experiment is recommended, since the current MOCHA displays were probably relatively low in apparency. Subjects had to go back and forth from the problem description to the display in order to make a decision. If limited short-term memory (STM) is a problem for low-SVA people (Salthouse, Babcock, Mitchell, Palmon, & Skovronek, 1990), the low-SVA subjects in the present experiment may have been experiencing STM overload. Placing the problem description closer to the display or allowing subjects to drag the problem description over to the display might have been helpful. A display that mapped the normal parameters to the displayed values would have greatly increased apparency, thereby helping to overcome the disadvantage of low SVA.
Although this research used college students, the results for SVA generalize to current and potential users of computer systems. A standard, accurate measure of spatial ability was used (Dupree & Wickens, 1982; Vincente, Hayes,& Williges, 1987). Significant results were found when subjects were grouped according to low, medium, and high SVA scores. There is no reason to think that these results do not generalize beyond college students. Since they do generalize, there is cause for concern. Research and development are needed to provide low-SVA users with the same level of access to information and on-line tools as enjoyed by high-SVA users.
Project Title: Beyond Supervisory Control: Investigating Cognitive Issues
in Human Interaction with Autonomous Satellites
I state that I am over 18 years of age, in good
physical health, and wish to participate in a program of research being
conducted by Dr. Kent L. Norman at the Graduate School, University of Maryland,
College Park, Department of Psychology.
I understand that the purpose of this research is
to explore the effects on human performance of various user-interface design
concepts and data representations.
Prior to participating in brief training on the experimental tasks, I will be asked to complete a standard test of spatial ability and to rate my agreement/disagreement with a set of statements about computers. During an hour-long experimental session, I will be asked to monitor visual displays and perform decision-making tasks.
All information in the study is confidential, and my name will not be
identified at any time.
I understand that, in general, there are minimal risks associated with
the use of video-display terminals (VDTs). Every standard precaution will be taken to ensure that these
risks are minimized through proper installation and maintenance of the
equipment. There are no physical
or psychological risks associated with the experiment itself.
I understand that the experiment is not designed to help me personally
but that the investigators hope to learn more about human performance with
various user interfaces and data representations for the ultimate improvement
of human interaction with highly automated systems. I understand that I am free to ask questions or to withdraw
from participation at any time without penalty.
Faculty Advisor: Kent L.
Norman
Department
of Psychology
301/405-5924
Ph.D. Candidate: Betty Murphy
Department
of Psychology
301/457-4988
Signed:
_______________________________________ Date:
Demographics Survey
Last Four Digits of Your Social Security Number _______________
TodayÕs Date _______________________
We are collecting the following information to help in the data analysis. All personal information will be kept confidential.
1. Your age ______
2. Your sex ______ Male or ______ Female
3. Your class status:
_____ Freshman _____ Sophomore _____ Junior
_____ Senior _____ Grad student _____ Other
4. Please circle one number to indicate your experience with computers:
Novice Expert
1 2 3 4 5 6 7 8 9
5. Please circle one number to indicate your experience reading and interpreting tables of numbers:
Novice Expert
1 2 3 4 5 6 7 8 9
6. Please circle one number to indicate your experience reading and interpreting line graphs:
Novice Expert
1 2 3 4 5 6 7 8 9
7. Please circle one number to indicate your experience reading and interpreting bar graphs:
Novice Expert
1 2 3 4 5 6 7 8 9
8. Please circle one number to indicate how useful,
overall, you have found graphs to be in your experience with them:
Useless Very useful
1 2 3 4 5 6 7 8 9
9. Place an X by any of the following statements that
describes your typical reaction to graphs. You may mark more than one.
_____ When I encounter a graph in a text, newspaper,
or magazine, I tend to ignore it (skip it
completely).
_____ I prefer reading graphs to reading tables of
numbers.
_____ I would prefer seeing a table of numbers rather
than seeing a graph of the numbers.
_____ I find graphs useful for remembering
information.
_____ When encountering a graph, I usually ÒskimÓ the
graph for an overall idea of what it represents, but I do not study it in detail.
_____ Graphs are generally a waste of space.
_____ I almost always look at graphs when I encounter
them in texts, newspapers, and magazines.
Pre-Experimental Automation Survey
Automation Survey
Instructions: Please rate
your level of agreement or disagreement with the statements below. You may check in between numbers
if necessary to reflect your level of agreement/disagreement.
Disagree Agree
give faulty
information.
2. Computers should be 1 2
3 4 5
6 7 8
9
designed to support
the tasks of human users.
3. I have found computers to 1 2 3 4
5 6 7
8 9
be more reliable than
people are.
4. Software designers do not 1 2 3 4
5 6 7
8 9
know what users
need.
5. In my experience, 1 2
3 4 5
6 7 8
9
computers are
dependable
across a range of
operations.
6. I trust a human assistant 1 2 3 4
5 6 7
8 9
more than I trust
software
processes.
7. Based on their past 1 2
3 4 5
6 7 8
9
performance,software
processes are
predictable.
8. Intelligent software 1 2
3 4 5
6 7 8
9
agents detract from the
userÕs sense of
controlling
the computer.
9. Highly automated 1 2 3 4
5 6 7
8 9
systems need constant
monitoring.
10. Artificial intelligence 1 2 3 4
5 6 7
8 9
has the
potential to endow
computer
systems with human-
like
capabilities, such
as
judgment and planning.
Disagree Agree
11. Users care more about 1 2
3 4 5
6 7 8
9
getting their work done
than they care about
feeling a sense of mastery
over the computer.
12. Artificial intelligence 1 2
3 4 5
6 7 8
9
offers more
promises
than
practical solutions.
Post-Experimental Automation Survey
Instructions: Please rate your level of agreement or disagreement
with the statements below. You may
check in between numbers if necessary to reflect your level of agreement/disagreement.
Disagree Agree
1.
The MOCHA system 1 2
3 4 5
6 7 8
9
often gave faulty
information.
2. The MOCHA software 1 2 3
4 5 6
7 8 9
was designed to support
the tasks of human users.
3. MOCHA is more reliable 1
2 3 4
5 6 7
8 9
than a human assistant
would be.
4. When solving the MOCHA 1
2 3 4
5 6 7
8 9
problems,I relied on the
agentÕs confidence level
for guidance.
5. MOCHAÕs designer did 1 2 3
4 5 6
7 8 9
not know what its users
would need.
6. MOCHA behaves 1 2 3
4 5 6
7 8 9
dependably across
a range of operations.
7. I would trust a human 1 2
3 4 5
6 7 8
9
assistant more than I
trust MOCHA.
8. MOCHAÕs software 1 2 3
4 5 6
7 8 9
processes perform in
predictable ways.
9. MOCHA operations need 1 2
3 4 5
6 7 8
9
constant monitoring.
10. The software agentÕs 1 2
3 4 5
6 7 8
9
confidence level influenced
my confidence level.
11. Increasing automation 1
2 3 4
5 6 7
8 9
is something to be
concerned about.
Appendix B. MOCHA Screen Shots
Blank page for figure 7
Blank page for figure 8
Blank page for figure 9
Blank page for figure 10
Blank page for Figure 11
Blank page for figure 12
Blank page for figure 13
Blank page for figure 14
Blank page for Figure 15
Blank page for Figure 16
Appendix C. Training and Test Materials
MOCHA Problem Descriptions with Agent Reasoning
Following is the full set of problems developed by the experimenter. In the final version of the simulation, six of these problems were used for the practice tasks and 10 for the test tasks.
121
PCS Sensor 1 temperature low
Agent Reasoning:
The acceptable temperature range for sensors is 5 to 50 degrees C.
122
PCS Sensor 2 temperature low
Agent Reasoning:
The acceptable temperature range for sensors is 5 to 50 degrees C.
123
PCS Sensor 3 temperature low
Agent Reasoning:
The acceptable temperature range for sensors is 5 to 50 degrees C.
124
EPS Battery 1 temperature low
Agent Reasoning:
Battery temperature ranges between -10 and 25 degrees C.
125
EPS Battery 2 temperature low
Agent Reasoning:
Battery temperature ranges between -10 and 25 degrees C.
126
EPS Solar array panel 1 output low
Agent Reasoning:
Each panel in the solar array produces 190 Watts per orbit.
127
EPS Solar array panel 2 output low
Agent Reasoning:
Each panel in the solar array produces 190 Watts per orbit.
128
EPS Solar array panel 3 output low
Agent Reasoning:
Each panel in the solar array produces 190 Watts per orbit.
129
EPS Solar array panel 4 output low
Agent Reasoning:
Each panel in the solar array produces 190 Watts per orbit.
130
EPS Solar array panel 5 output low
Agent Reasoning:
Each panel in the solar array produces 190 Watts per orbit.
131
EPS Solar array panel 6 output low
Agent Reasoning:
Each panel in the solar array produces 190 Watts per orbit.
132
EPS Battery 1 charge low
Agent Reasoning:
Battery charge ranges between 20 and 42 amps.
133
EPS Battery 2 charge low
Agent Reasoning:
Battery charge ranges between 20 and 42 amps.
134
EPS Battery 1 discharge low
Agent Reasoning:
Discharge rates must be kept within 20 and 42 amps.
135
EPS Battery 2 discharge low
Agent Reasoning:
Discharge rates must be kept within 20 and 42 amps.
136
C&DHS Transmitter power low
Agent Reasoning:
Forty-four watts (44 W) are needed to support downlink.
137
C&DHS Transmitter temperature low
Agent Reasoning:
Transmitter temperatures are maintained between -25 and 55 degrees C.
138
C&DHS Receiver temperature low
Agent Reasoning:
Receiver temperatures are maintained between -25 and 55 degrees C.
139
C&DHS Transmitter temperature high
Agent Reasoning:
Transmitter temperatures are maintained between -25 and 55 degrees C.
140
C&DHS Receiver temperature high
Agent Reasoning:
Receiver temperatures are maintained between -25 and 55 degrees C.
141
PCS Sensor 1 temperature high
Agent Reasoning:
Sensor temperature ranges between 5 and 50 degrees C.
142
PCS Sensor 2 temperature high
Agent Reasoning:
Sensor temperature ranges between 5 and 50 degrees C.
143
PCS Sensor 3 temperature high
Agent Reasoning:
Sensor temperature ranges between 5 and 50 degrees C.
144
EPS Thermal: Heater 1 not responding to off command
Agent Reasoning:
Heater on/off status corresponds to the on/off commanding schedule.
145
EPS Thermal: Heater 2 not responding to off command
Agent Reasoning:
Heater on/off status corresponds to the on/off commanding schedule.
146
EPS Thermal: Heater 3 not responding to off command
Agent Reasoning:
Heater on/off status corresponds to the on/off commanding schedule.
147
C&DHS Orbit Status: Deviation in x-axis
Agent Reasoning:
There can be no more than plus or minus 50 meters deviation per axis per orbit.
148
C&DHS Orbit Status: Deviation in y-axis
Agent Reasoning:
There can be no more than plus or minus 50 meters deviation per axis per orbit.
149
C&DHS Orbit Status: Deviation in z-axis
Agent Reasoning:
There can be no more than plus or minus 50 meters deviation per axis per orbit.
150
C&DHS Tape recorder 1 pressure low
Agent Reasoning:
Pressure must be maintained between 400 and 800 pounds per square inch (psi).
151
C&DHS Tape recorder 2 pressure low
Agent Reasoning:
Pressure must be maintained between 400 and 800 pounds per square inch (psi).
152
C&DHS Tape recorder 1 pressure high
Agent Reasoning:
Pressure must be maintained between 400 and 800 pounds per square inch (psi).
153
C&DHS Tape recorder 2 pressure high
Agent Reasoning:
Pressure must be maintained between 400 and 800 pounds per square inch (psi).
154
EPS Battery 1 voltage low
Agent Reasoning:
The range of acceptable voltage is 22 to 34 volts (V).
155
EPS Battery 2 voltage low
Agent Reasoning:
The range of acceptable voltage is 22 to 34 volts (V).
156
EPS Battery 1 voltage high
Agent Reasoning:
The range of acceptable voltage is 22 to 34 volts (V).
157
EPS Battery 2 voltage high
Agent Reasoning:
The range of acceptable voltage is 22 to 34 volts (V).
158
C&DHS Antenna pointing accuracy less than 95%
Agent Reasoning:
Pointing accuracy is maintained between 95 and 100 percent.
159
C&DHS Antenna pointing stability less than 95%
Agent Reasoning:
Pointing stability is maintained between 95 and 100 percent.
160
PCS Orientation Status: Deviation in x-axis
Agent Reasoning:
Acceptable deviation in spacecraft orientation is plus or minus 10 degrees.
161
PCS Orientation Status: Deviation in y-axis
Agent Reasoning:
Acceptable deviation in spacecraft orientation is plus or minus 10 degrees.
162
PCS Orientation Status: Deviation in z-axis
Agent Reasoning:
Acceptable deviation in spacecraft orientation is plus or minus 10 degrees.
163
PCS Actuator 1 temperature low
Agent Reasoning:
Functional temperature range for actuators is -25 to 50 degrees C.
164
PCS Actuator 2 temperature low
Agent Reasoning:
Functional temperature range for actuators is -25 to 50 degrees C.
165
PCS Actuator 3 temperature low
Agent Reasoning:
Functional temperature range for actuators is -25 to 50 degrees C.
166
PCS Acuator 1 temperature high
Agent Reasoning:
Functional temperature range for actuators is -25 to 50 degrees C.
167
PCS Actuator 2 temperature high
Agent Reasoning:
Functional temperature range for actuators is -25 to 50 degrees C.
168
PCS Actuator 3 temperature high
Agent Reasoning:
Functional temperature range for actuators is -25 to 50 degrees C.
169
PCS Sun sensor 1 no lock on sun
Agent Reasoning:
Sun sensors must be locked on the sun.
170
PCS Sun sensor 2 no lock on sun
Agent Reasoning:
Sun sensors must be locked on the sun.
171
PCS Star tracker 1 temperature low
Agent Reasoning:
Functional temperature range for star trackers is 5 to 50 degrees C.
172
PCS Star tracker 2 temperature low
Agent Reasoning:
Functional temperature range for star trackers is 5 to 50 degrees C.
173
PCS Star tracker 1 temperature high
Agent Reasoning:
Functional temperature range for star trackers is 5 to 50 degrees C.
174
PCS Star tracker 2 temperature high
Agent Reasoning:
Functional temperature range for star trackers is 5 to 50 degrees C.
Sample Status Messages for the Monitoring Condition
The following status messages were developed by the experimenter and displayed to monitoring subjects in between problem alerts.
Solar array tilted 30 degrees toward Sun.
Solar array actuator response time: 1.0 second
Sun sensor locked on sun.
Horizon sensor locked on Martian horizon.
Star trackers locked on guide stars.
Instrument booms 1 and 2 checkout: normal
Gyroscopes: nominal
orientation.
MOCHA spectrometer completing scheduled observation of upper
atmosphere..
Small thrusters fired for 2.0 seconds to adjust orbit.
Thruster response time:
0.82 second
Propellant reserves: 26.5
kilograms
Propellant tanks repressurized.
MOCHA mapping camera beginning daily low-resolution imaging.
Low-resolution images recording to Tape recorder 1.
Data continuity, Tape recorder 1:
99.90%
Recording rate, Tape recorder 1:
18.5 Mb per second
Available data capacity, Tape recorder 1: 1.0 Gb
Heaters 1 and 2 turned off
per schedule.
Heater actuator response time:
0.65 second
MOCHA radiometer beginning scheduled atmospheric dust measurement.
Radiometer data recording to Tape recorder 2.
Data continuity, Tape recorder 2:
99.90%
Recording rate, Tape recorder 2:
15 Mb per second
Available data capacity, Tape recorder 2: 0.8 Gb
Tape recorder 2 sending data from MOCHA spectrometer to pre-processor.
Pre-processing completed.
Pre-processor sending data to transmitter.
Power available to support downlink: 44 W
Antenna locked on Deep Space Network receiver.
Downlink data rate: 85 kb
per second
Spectrometer data transmitted successfully.
Spectrometer data transmission acknowledged.
Round-trip communication time: 19.5 min
Heater 3 turned on per
schedule.
Heater actuator response time:
0.5 second
Temperature check of high-gain antenna: normal
Changes to observation schedule received for MOCHA magnetometer.
Uplink data rate: 455 bits
per second
Battery pressure check:
normal
Propellant temperature check:
normal
Propellant pressure check: normal
Altitude check: 250
miles (402.5 kilometers)
Current speed: 7, 500 mph
(12,000 kilometers per hour)
Total power usage last orbit:
1100 W
Time on last orbit: 4.2 hr
Minutes on battery power (last orbit): 39.5
Number of rotations on last orbit: 1.0
Local time at crossing Martian equator on last orbit: 1:58 p. m.
Solar array tilted 45 degrees toward Sun.
Solar array actuator response time: 1.25 second
Sun sensor locked on sun.
Horizon sensor locked on Martian horizon.
Star trackers locked on guide stars.
Routine command updates received from Deep Space Network.
Current rate of command reception: 11.5 commands per second.
Uplink data rate: 450 bits
per second (bps)
Current inclination to Martian equator: 91 degrees
MOCHA radiometer completing atmospheric dust measurement.
MOCHA magnetometer beginning daily measurement of magnetic field.
Magnetometer data recording to Tape recorder 1.
Data continuity, Tape recorder 1:
99.95%
Recording rate, Tape recorder 1:
18 Mb per second
Available data capacity, Tape recorder 1: 0.5 Gb
Narrow-angle and wide-angle imaging tests completed on MOCHA camera.
Gravitational wave experiment completed.
Altimeter checkout: normal
Tape recorder 2 sending radiometer data to pre-processor.
Pre-processing completed.
Pre-processor sending radiometer data to transmitter.
Power available to support downlink: 44 W
Antenna locked on Deep Space Network.
Downlink data rate: 85 kb
per second
Radiometer data transmitted successfully.
Radiometer data acknowledged.
Round-trip communications time:
19.0 min
MOCHA camera completing low-resolution imaging.
Tape recorder 1 sending imaging data to pre-processor.
Pre-processing completed.
Preprocessor sending imaging data to transmitter.
Power available to support downlink: 44 W
Antenna locked on Deep Space Network receiver.
Downlink data rate: 80
kbits per second
Imaging data transmitted successfully.
Imaging data transmission acknowledged.
Round-trip communication time:
20.2 minutes
MOCHA camera focusing adjustment in progress.
MOCHA camera focusing adjustment completed.
Routine test of ground-to-spacecraft communications completed.
Tape recorder 2 sending gravitational wave data to pre-processor.
Instructions for Research Participants and Training in the Experimental Task
Beyond
Supervisory Control HRS 198-13
Instructions
for Research Participants
First, please complete the consent form and the short survey on your background.
[Participant completes forms.]
Now you will spend about an hour responding to some pre-experimental questionnaires and being trained in the experimental task. You will then spend about an hour participating in the experiment.
Pre-testing:
You will be asked to complete two pre-test surveys:
1. Spatial Visualization Ability (SVA)
2. Questionnaire on Automation
We will now proceed with this part of todayÕs activities.
[Research assistant administers the VZ2 and the automation survey.]
Training
in the Experimental Task:
[Participant reads the following:]
Background: The sponsor of this research is the NASA-Goddard Space Flight Center in Greenbelt, Maryland. Control room operators at NASA-Goddard spend a lot of time monitoring the performance of various unmanned satellites. In the future, they will become involved with satellite operations only when there is a problem that cannot be handled by on-board software.
MOCHA: For purposes of this research, we have developed a simulated satellite environment named MOCHA, for Mars Observer Calls Home Again. The actual Mars Observer was an unmanned spacecraft designed to conduct mapping and weather studies in orbit around Mars. During the experiment, you will be the ground controller for MOCHA. You will be given all the information you need to perform your tasks.
Instruments on-board MOCHA:
á Magnetometer -- measures the planetÕs magnetic field
á Radiometer Ð measures radiation coming from the planetÕs surface
á Spectrometer Ð measures wavelengths of light reflected from the planetÕs surface
á Altimeter Ð measures the spacecraftÕs altitude above the planet
á Mapping Camera Ð used for imaging and mapping features on the planetÕs surface
Basic Concepts:
Ground controllers focus on maintaining the Òhealth and safetyÓ of the spacecraft and its ability to maintain commanded positions so that the location and orientation of the spacecraft can be documented. If position is unknown, it is impossible to aim the instruments in the right direction for observing. Various indicators of operational status must be kept within acceptable ranges to assure the that the spacecraft remains functional. Temperature and pressure ranges, for example, are monitored by on-board functions, which report out-of-range conditions to the ground controller.
Occasionally, during an unmanned spacecraftÕs life cycle, some abnormal events may occur. These ÒanomaliesÓ or ÒfaultsÓ are unexpected and not initially understood by ground controllers. A process of determining the origin of the abnormal activity proceeds until ground controllers are satisfied that they know why the anomaly occurred and have developed a plan to resolve it, if possible. This process of Òfault isolation and resolutionÓ begins with the verification that something unusual has actually occurred.
Your Primary Task: The experiment is set up to represent the kind situation that will exist as spacecraft ground control centers become more automated. Your primary task will be to respond to problem alerts. In a few minutes, you will work through a set of practice alerts that resemble the actual problems you will respond to in the experiment. By doing the practice problems, you will learn how to make the required decisions.
Familiarization with MOCHA: At this point, you may go ahead and cycle through the various ÒboxesÓ that represent components of MOCHA. When you click on a box, corresponding text will appear in the window on the right-hand side of the screen. Please read these descriptions, which are intended to give you some background on the various components.
[Time for familiarization]
Practice Session: Now we will begin the practice problems. When an alert occurs, your objective is to decide whether there is a problem as reported, a false alarm, or a different problem. The alerting device is an autonomous software process, called an Òagent,Ó that reports each problem with some degree of ÒAgent ConfidenceÓ (50%, 70%, or 90%). This confidence level means, for example, that the agent is 70% sure that there is a problem as described in the Problem Description. Because the agent may be mistaken, you should take the Agent Confidence level as an estimate. You will use the practice time to develop a method of confirming or disconfirming the problem reported by the agent.
When you have
completed all the practice problems, you will begin the experimental
session. When you have finished
the experimental session, you will be asked to complete a brief questionnaire. Be sure that the research assistant has
your full name and Social Security Number for reporting your participation.
THANKS VERY MUCH FOR
YOUR PARTICIPATION !!
Experimenter: Betty Murphy Faculty advisor:
Dr. Kent L. Norman
bmurphy@psyc.umd.edu Kent_Norman@lap.umd.edu
Training in System Components
When a subject clicked on a button in the MOCHA hierarchy of components, the corresponding text was displayed in the System Data portion of the screen. So, for example, when the subject clicked on the button labeled Electrical Power Subsystem, the second paragraph below was displayed. These readings constituted the MOCHA-specific portion of the training session. For display purposes, the text was reformatted to break up long textual passages. An outline format and white space were used to provide visual relief.
About MOCHA Reports
The available reports provide all the information needed to evaluate a reported anomaly. Reports contain current readings of values on the various characteristics of components, for example, temperature, position, and on/off status. They also contain historical, trend data for some components.
About the Electrical Power Subsystem (EPS)
The Electrical Power Subsystem provides the electricity needed to operate spacecraft components. Power is stored in two 42-ampere batteries. When the spacecraft is in sunlight, solar power is converted to electricity by six solar panels on the spacecraftÕs one solar array. When the spacecraft is in shadow, stored power is available from the batteries.
About the Rocket Propulsion Subsystem (RPS)
The Rocket Propulsion Subsystem consists of four large thruster rockets on-board the spacecraft. Controlled thrust from these rocket engines is used to adjust the position of the spacecraft, for example, to adjust the height and shape of the orbit around Mars.
About the Command and Data Handling Subsystem (C&DHS)
The Command and Data Handling Subsystem includes the onboard data-processing capabilities as well as the capabilities for transmitting data to the ground and receiving commands from the ground. The C&DHS includes computer components, such as the central-processing unit (CPU) and mass memory, as well as communication resources, such as tape recorders. Both scientific and engineering data are stored on the tape recorders for later transmission to the ground. Following a pre-determined schedule, the on-board computer controls the position of the spacecraft, manages power usage, initiates the collection of scientific data by the observing instruments, and transmits the data to the ground.
About the Position Control Subsystem (PCS)
The Position Control Subsystem includes sensors that enable a determination of the spacecraftÕs position in three dimensions and actuators that enable a change of position in the spacecraft or any of its moveable components (e.g., solar array, scientific instruments). The PCS is concerned with maintaining the stability of the spacecraftÕs position in support of instrument pointing accuracy. The function of one actuator is to flip the spacecraft over if it stabilizes in an upside-down position.
About Battery Health & Safety
Battery health and safety depends on maintaining a minimum charge, while not exceeding a maximum charge, and on staying within a range of acceptable temperatures, specified as follows:
Min (value) Min (value)
Charge Temp
Max (value) Max (value)
Appendix D: Distractor Survey for the On-Call Condition
The following items were selected from the full distractor survey for presentation in this appendix.
Sample Survey Questionnaire
Instructions: Please complete the following questions to reflect your opinions as accurately as possible and to answer factual questions to the best of your knowledge. Your information will be kept strictly confidential.
1. Should divorce in this country be more difficult to obtain, easier to obtain, or stay as it is now?
o More difficult to obtain
o Easier to obtain
o Stay as it is now
2. Please look at the following list of qualities for a child and then answer the questions below the list.
1. has good manners
2. tries hard to succeed
3. is honest
4. is neat and clean
5. has good sense and sound judgment
6. has self control
7. he acts like a boy or she acts like a girl
8. gets along well with other children
9. obeys his or her parents well
10. is responsible
11. is considerate of others
12. is interested in how and why things happen
13. is a good student
a. The qualities listed above
may all be important, but which three
would you say are the most
desirable for a child to have?
__________
__________
__________
b. Which of these three is the most desirable of all?
__________
c. All of the qualities listed above may be desirable, but which three do you consider least important?
__________
__________
__________
d. Which of these three is least important of all? ______________
3. What do you think the chances are these days that a man wonÕt get a job or promotion while an equally or less qualified woman gets one instead?
o Very likely
o Somewhat likely
o Somewhat unlikely
o Very unlikely
4. a. On average, women who are employed full time earn less than men earn. Which of the following reasons do you think is the most important reason for this:
o WomenÕs family responsibilities keep them from putting as much time and effort into their jobs as men do.
o Men work harder on the job than women do.
o Employers tend to give men better paying jobs than they give women.
4. b. How confident are you of your answer to question number 4 a? (Click on one number in the following scale.)
Not confident at all Very confident
1 2 3 4 5 6 7 8 9
5. We are faced with many problems in this country, none of which can be solved easily or inexpensively. Below is a list of some problems, and for each please indicate your opinion about the money weÕre spending.
a. Space exploration program
o Spending too much
o Spending about the right amount
o Spending too little
b. Improving and protecting the environment
o Spending too much
o Spending about the right amount
o Spending too little
c. Improving and protecting the nationÕs health
o Spending too much
o Spending about the right amount
o Spending too little
d. Solving the problems of the big cities
o Spending too much
o Spending about the right amount
o Spending too little
e. Halting the rising crime rate
o Spending too much
o Spending about the right amount
o Spending too little
f. Dealing with drug addiction
o Spending too much
o Spending about the right amount
o Spending too little
g. Improving the nationÕs education system
o Spending too much
o Spending about the right amount
o Spending too little
h. Foreign aid
o Spending too much
o Spending about the right amount
o Spending too little
i. Social Security
o Spending too much
o Spending about the right amount
o Spending too little
j. Parks and recreation
o Spending too much
o Spending about the right amount
o Spending too little
6. If you had to make the following choice, which option would you choose?
o a 100 % chance of losing $50
o a 25% chance of losing $200, and a 75% chance
of losing nothing
7. a. Which is a more likely cause of death in the United States Ð being killed by falling airplane parts or by a shark?
o Falling airplane parts
o Shark
7. b. How confident are you of your answer to question number 7 a? (Click on one number in the following scale.)
Not confident at all Very confident
1 2 3 4 5 6 7 8 9
8. If you had to make the following choice, which option would you choose?
o a sure gain of $240
o a 25% chance to gain $1000, and 75% chance to gain nothing
9. If you had to make the following choice, which option would you choose?
o a sure loss of $750
o a 75% chance to lose $1000, and 25% chance to
lose nothing
10. If you were given a choice, which of the following options would you prefer?
o $1,000,000 for sure
o a 10% chance of getting $2,500,000,
an 89% chance of getting $1,000,000,
and a 1% chance of getting $0
11. If you were given a choice, which of the following options would you prefer?
o an 11% chance of getting $1,000,000,
and an 89% chance of getting $0
o a 10% chance of getting $2,5000,000,
and a 90% chance of getting $0
12. Suppose an unbiased coin is flipped three times, and each time the coin lands Heads up. If you were going to win $100 for guessing right, which side would you choose for the next toss?
o Heads
o Tails
13. Does the act of voting for a candidate change
your opinion about whether that candidate will
win the election?
o Yes
o No
o Not sure
14. Which of the following sequences of XÕs and OÕs seems more like it was generated by a random process (e.g., by flipping a coin)?
o XOXXXOOOOXOXXOOOXXXOX
o XOXOXOOOXXOXOXOOXXXOX
15. a. Which is the more common cause of death in the United States, diabetes or homicide?
o Diabetes
o Homicide
15. b. How confident are you of your answer to question number 15a? (Click on one number in the following scale.)
Not confident at all Very confident
1 2 3 4 5 6 7 8 9
16. Is the percentage of African countries in the United Nations greater or less than 65%?
o Greater than 65%
o Less than 65%
17. a. What is the exact percentage of African countries in the United Nations?
o 75%
o 70%
o 65%
o 50%
o 45%
o 40%
o 35%
o 30%
o 25%
o 20%
o 15%
o 10%
o Other Please specify: _____%
17. b How confident are you of your answer to question number 17 a? (Click on one number in the following scale.)
Not
confident at all Very confident
1 2 3 4 5 6 7 8 9
18. Are the chances of nuclear war between the United States and China greater or less than 1%?
o Greater than 1%
o Less than 1%
19. How confident are you of your answer to question number 18? (Click on one number in the following scale.)
Not confident at all Very
confident
1 2 3 4 5 6 7 8 9
20. Listed below are various aspects of jobs. Which one of the following job characteristics do you think is the most important to you personally?
__________
a. A job that allows one to work independently
b. A job that is useful to society
c. Good opportunities for advancement
d. An occupation that is recognized and respected
e. An occupation that leaves one a lot of leisure time
f. An interesting job Job security
g. Responsible job tasks
h. High income
i. An occupation in which one can help others
j. A lot of contact with other people
k. Gives a feeling of doing something meaningful
l. Safe and healthful working conditions
21. Which of these statements comes closest to your own idea of how gasoline and oil prices are decided? Each company sets its own prices to meet the competition, or the oil companies get together and set prices for their products?
o Each company sets its own prices
o Oil companies set prices together
22. Some people say that we will have plenty of oil 25 years from now. Others say that at the rate we are using our oil, it will be all used up in about 15 years. Which of these ideas would you guess is most nearly right?
o Plenty of oil in 25 years
o Oil used up in 15 years
23. a. Which one of the following five problems do you think is the most important problem facing this country at present:
o Crime and violence
o Poor quality of government leaders
o Racial tensions
o Drug use
o Breakdown of morality among people in general
23. b. How confident are you of your answer to question number 23 a? (Click on one number in the following scale.)
Not confident at all Very
confident
1 2 3 4 5 6 7 8 9
24. Some people feel that the federal government should see to it that all people have adequate housing, while others feel each person should provide his/her own housing. Which comes closest to how you feel about this?
o Government should provide housing
o Each person should provide own housing
25. Do you think that the government in Washington ought to reduce the income differences between the rich and the poor, perhaps by raising the taxes of wealthy families or do you think that the government should not concern itself with reducing this income difference between the rich and the poor?
o Reduce income differences
o DonÕt reduce income differences
26. In the past three months, have you had any expenses for any of the following, either for your family or someone else?
Computer information services?
Yes o
No o
TV computer games and computer games software?
Yes o
No o
Hand held computer games and computer board games?
Yes o
No o
Toys and games?
Yes o
No o
Hobbies?
Yes o
No o
Moving, storage, and freight express?
Yes o
No o
Purchase of pets, pet supplies, and medicine for pets?
Yes o
No o
Pet services?
Yes o
No o
Veterinarian expenses for pets?
Yes o
No o
28. In the past 3 months, have you purchased any alcoholic beverages in restaurants, taverns or cocktail lounges?
Yes o
No o
29. Should women be given preference in hiring to correct past discrimination, or should they be treated the same as all other candidates?
o Favor giving preference
o Oppose giving preference
30. In general, how do you feel about your time - would you say that you always feel you have enough leisure time, only sometimes feel you have enough leisure time, or almost never feel you have enough leisure time?
o Always have enough leisure time
o Sometimes have enough leisure time
o Never have enough leisure time
31. In the past three months, have you purchased dinners, other meals or snacks in restaurants, cafeterias, cafes, drive-ins, other such places?
Yes o
No o
32. In the past three months, have you used public pay phone service?
Yes o
No o
33. a.In the past three months, have you used coin-operated laundry or dry cleaning machines?
Yes o
No o ¨ Go to Item 34
33. b. What was the total cost for these machines?
Amount ¨ $_______.00
33. c. Was any of this amount for items other than clothes?
Yes o ¨ How much? $______.00
No o
34. a. In the past three months, have you sent clothes or other items to the dry cleaners or laundry?
Yes o
No o ¨ Go to Item 35
34. b. What was the total cost for dry cleaning and laundry services?
Amount ¨ $_______.00
34. c. Was any of this amount for items other than clothes?
Yes o ¨ How much? $______.00
No o
35. (omitted from appendix)
36. What is your usual MONTHLY expense for haircutting, styling, and other related services that you receive?
Amount ¨ $_______.00
37. How many hours per day do you watch TV?
o Up to 2 ½ hours
o 2½ hours to 3 hours
o 3 hours to 3½ hours
o 3½ hours to 4 hours
o 4 hours to 4½ hours
o More than 4½ hours
38. In the past three months, did you owe any money to any of the following?
Revolving credit accounts including store, gasoline, and general purpose credit cards, such as Sears, Amoco, Visa, MasterCard, etc.
Yes o ¨ How much did you owe? $______.00
No o
Stores for installment credit accounts?
Yes o ¨ How much did you owe? $______.00
No o
Banks and savings and loan companies?
Yes o ¨ How much did you owe? $______.00
No o
Credit unions?
Yes o ¨ How much did you owe? $______.00
No o
Finance companies?
Yes o ¨ How much did you owe? $______.00
No o
Insurance companies?
Yes o ¨ How much did you owe? $______.00
No o
Doctors, dentists, hospitals, or other medical practitioners for expenses not covered by insurance?
Yes o ¨ How much did you owe? $______.00
No o
Other credit sources?
Yes o ¨ How much did you owe? $______.00
No o
39. a. Which one of the following five problems do you think is the most important problem facing this country at present?
o Breakdown of morality among people in general
o Drug use
o Racial tensions
o Poor quality of government leaders
o Crime and violence
39. b. How confident are you of your answer to question number 35 a? (Click on one number in the following scale.)
Not confident at all Very confident
1 2 3 4 5 6 7 8 9
40. Please indicate whether you agree or disagree with this statement: Social conditions are more to blame than individuals for crime and lawlessness in this country.
o Agree
o Disagree
41. Do you think most companies that lay off workers during slack periods could arrange things to avoid layoffs and provide steady work right through the year?
o Yes
o No
o No opinion
42. Do you think anything should be done to make it easier for people to pay doctor or hospital bills?
o Yes o No
43. When you speak of profits, are you thinking of profit on the amount of sales, on the amount of money invested in the business, on year-end inventory, or what?
o Profit on sales
o Profit on investment
o Profit on inventory
o Profit on other bases.
44. Should our country be more active in world affairs?
o Yes
o No
45. Would you say that the average family gets more electricity for its money today than it did 10 years ago?
o Yes
o No
46. Electricity in the Washington Metropolitan area comes primarily from the following source:
o Steam generators
o Water power
o Nuclear power
o None of the above
47. The following color is the complement of blue:
o Red
o Yellow
o Green
48. If you had a choice between two identical products, one a brand name and the other an off-brand, but the brand name cost $3.50 more than the off-brand, which would you be likely to buy?
o the off-brand product
o the brand-name product
49. a. On a U. S. penny, the head of Lincoln looks in the following direction:
o To the right
o To the left
49. b. How confident are you of your answer to question number 49 a? (Click on one number in the following scale.)
Not
confident at all
Very confident
1 2 3 4 5 6 7 8 9
50. If private companies were permitted to compete with the U. S. Postal Service, do you think service would be better, worse, or about the same?
o Better
o Worse
o About the same
51. a. Please rate your level of familiarity with the Internet/World Wide Web:
o Novice
o Intermediate
o Expert
51. b. About how much time do you spend daily using the Internet/World Wide Web:
o less than 30 minutes
o 30 minutes to one hour
o more than one hour
52. Over the last year, approximately how much have you spent on gifts to family members and friends?
o $1 to $50.99
o $51 to $100.99
o $101 to 150 or more
53. If you were stranded on a desert island, which one of the following technologies would you like to have available in working order?
o Television
o Telephone
o Computer with Internet hookup
54. During the 1998 holiday season, consumer spending online amounted to a total of
o $1.2 billion
o $2.4 billion
o $3.6 billion
55. a. Most users of the Internet report the following effect, or lack of effect, on their television viewing time:
o No effect
o Watching less TV
o Watching more TV
55. b. How confident are you of your answer to question number 55a? (Click on one number in the following scale.)
Not
confident at all
Very confident
1 2 3 4 5 6 7 8 9
56. a. On the average, approximately how much time do you spend weekly using the Internet/World Wide Web?:
o 16 to 20 hours or more
o 11 to 15 hours
o 6 to 10 hours
o 0 to 5 hours
56.b. Please indicate the number of items you have purchased over the Internet:
o 5 or more
o 3-4
o 1-2
o none
56.c. Please indicate the level of concern you have about the security/confidentiality of information you provide about yourself over the Internet:
o high concern
o moderate concern
o little concern
57. During the 1998 holiday season, U. S. retail sales amounted to a total of
o $648 billion
o $670 billion
o $689 billion
58. On the average, how much time do you spend weekly doing volunteer work in your community:
o 0 to 3 hours
o 4 to 6 hours
o 7 to 9 hours or more
59. If private companies could compete with the Postal Service to deliver first-class mail, the cost of postage stamps would:
o Increase
o Stay the same
o Decrease
60. Do you think the United States should forbid public speeches against democracy?
o Yes
o No
o No opinion
61. Do you or any member of your family pay rent for your living quarters?
o Yes
o No
62. a. What is the rental charge to you or your family for this unit?
Amount $____.00
o DonÕt Know
62. b. What period of time does this cover?
o Month
o Other Specify ________________________
62. c. In the past 3 months, how many payments have been made?
Number ____
63. Does the rental payment include the cost of...
... Electricity?
o Yes
o No
... Gas?
o Yes
o No
... Piped-in water?
o Yes
o No
... Heating?
o Yes
o No
... Trash/Garbage collection?
o Yes
o No
64. Do you think that advertising is less truthful today than it was a year or two ago?
o Yes
o No
65. What is the name of the company which provides telephone service for your residence?
Name of telephone company ¨ ________________________
66. How many telephone bills have you received in the past three months?
Number ¨ ____
67. a. What was the total amount of these telephone bill(s)?
Amount ¨ $____.00
DonÕt Know o
67. b. In what month was the latest bill received?
Month ¨ ________
67. c. Does the total amount of the bill include...
... A basic service charge?
Yes o
No o
... Long distance call charges?
Yes o
No o
68. a. Do you share the cost of your telephone service with at least one other member of your household?
No o
Yes o
68. b. If yes, how many other members of your household share the cost of telephone service:
4 or more o
3 o
2 o
1 o
69. (omitted from appendix)
70. In the past three months, have you or your family rented any vehicles?
Yes o
No o ¨ Go to item 72.
71. Did you rent any ...
... automobiles?
Yes o
No o
... trucks, including vans?
Yes o
No o
... motorized camper-coaches?
Yes o
No o
... other attachable-type campers?
Yes o
No o
... motorcycles, motor scooters, or mopeds (motorized bicycle)?
Yes o
No o
... boats, with a motor?
Yes o
No o
... boats, without a motor?
Yes o
No o
... trailers, other than camper type, such as for a boat or cycle?
Yes o
No o
... any other vehicles?
Yes o
No o
72. Do you own any of the following vehicles ...
... automobiles?
Yes o
No o
... trucks, including vans?
Yes o
No o
... motorized camper-coaches?
Yes o
No o
... trailer type campers?
Yes o
No o
... other attachable-type campers?
Yes o
No o
... motorcycles, motor scooters, or mopeds (motorized bicycle)?
Yes o
No o
... boats, purchased with a motor?
Yes o
No o
... boats, purchased without a motor?
Yes o
No o
73. Answer the following questions only if you own an automobile, truck or van. If you donÕt own any of these, go to question 74.
73. a. What is the year, make, and model?
First vehicle Second vehicle
Year ¨ 19__ Year ¨ 19__
Make ¨_____________ Make ¨______
Model ¨____________ Model ¨_____
73. b. How many cylinders does it have?
First vehicle Second vehicle
Cylinders ¨ ___ Cylinders ¨ ___
73. c. Does it have ...
... automatic transmission?
First vehicle Second vehicle
Yes o Yes o
No o No o
... power steering?
First vehicle Second vehicle
Yes o Yes o
No o No o
... power brakes?
First vehicle Second vehicle
Yes o Yes o
No o No o
... air conditioning?
First vehicle Second vehicle
Yes o Yes o
No o No o
... sun roof?
First vehicle Second vehicle
Yes o Yes o
No o No o
... turbo charged engine?
First vehicle Second vehicle
Yes o Yes o
No o No o
... diesel engine?
First vehicle Second vehicle
Yes o Yes o
No o No o
... four wheel drive?
First vehicle Second vehicle
Yes o Yes o
No o No o
73. d. How many doors does it have?
First vehicle Second vehicle
Doors ¨ ____ Doors ¨ ____
73. e. Was it new or used when acquired?
First vehicle Second vehicle
New o New o
Used o Used o
73. h. How many miles are currently on the vehicle?
First vehicle Second vehicle
Miles ¨ _______ Miles ¨ _______
73. i. In what month and year was it purchased?
First vehicle Second vehicle
Month ¨ _______ Month ¨ _______
Year ¨ ______ Year ¨ ______
73. j. Was a trade-in allowance received?
First vehicle Second vehicle
Yes o¨ How much?_____ Yes o¨ How much?_____
No o No o
73. k. What was the amount paid for it after trade-in allowance and discount?
First vehicle Second vehicle
Paid ¨ $________ Paid ¨ $________
73. l. Did this price include sales tax?
First vehicle Second vehicle
Yes o Yes o
No o No o
73. m. Was any portion of the purchase price financed?
First vehicle Second vehicle
Yes o Yes o
No o ¨ Go to item 74 No o ¨ Go
to item 74
73.n. What was the amount of the cash down payment?
First vehicle Second vehicle
Amount ¨ $_______.00 Amount ¨ $_____.00
73.p. How much was borrowed, excluding any interest?
First vehicle Second vehicle
Amount ¨ $_______.00 Amount ¨ $_____.00
73.q. What was the number of payments contracted for?
First vehicle Second vehicle
Number ¨ ___ Number ¨ ___
73.r. In what month and year was the first payment made?
First vehicle Second vehicle
Month ¨ _______ Month ¨ _______
Year ¨ ______ Year ¨ ______
73.s. What is the amount of each payment?
First vehicle Second vehicle
Amount ¨ $_______.00 Amount ¨ $_____.00
74. People should not be allowed to express opinions that are harmful or offensive to members of other religious or racial groups.
o Strongly agree
o Agree
o Neither agree nor disagree
o Disagree
o Strongly disagree
75. If given a choice, would you prefer to submit information about your household to the Census Bureau on an electronic form over the Internet or on a paper form by mail?
o By Internet
o By mail
76. Are you confident that the confidentiality of your information is well protected by government agencies?
o No
o Yes
o Not sure
77. Are you currently employed?
o Yes
o No ¨ Go to item 81.
78. When it comes to Social Security, if you could choose, would you opt to get out of the system and pay no Social Security taxes or be in the system and pay the Social Security taxes?
o Get out of the system and pay no taxes
o Be in the system and pay taxes
79. Do you think any part of the Social Security trust fund should be invested in the stock market?
o Yes
o No
80. Do you favor privatizing the Social Security system?
o Yes o No
81. Do you think about retirement at all?
o Yes
o No ¨ Go to item 90.
82. At what age do you plan to retire?
o 60 or younger
o 61-64
o 65
o 66 or older
83. Do you expect Social Security to provide more or less than 50% of your retirement income?
o Less than 50%
o 50% or more
84. What worries you most about retirement?
o Poor health
o Financial problems
o Boredom
o Alienation
85. When you reach retirement age, will Social Security benefits pay the same as now, less, or nothing at all?
o Same as now
o Less than now
o Nothing at all
86. Mark all major sources of your expected retirement income:
o IRA
o Pension(s)
o Social Security
o Assistance from family members
o Private savings
o 401(k) or 403(b)
87. In your retirement, do you expect to be better or worse off overall than your parents, or about the same?
o Better off
o Worse off
o About the same
o Not sure
88. Compared to your parentsÕ generation, do you expect to be worse off or better off in the following areas during your retirement?
Health care : o Worse o Better
Being independent: o Worse o Better
Having enough money: o Worse o Better
Personal health: o Worse o Better
Ability to stay involved: o Worse o Better
89. (omitted from appendix)
90. (omitted from appendix)
Thank
you. This is the end of the
survey.
References
Abedini, A., Moriarta, J., Biroscak, D., Losik, L., and Malina, R. F. (1995). A low-cost, automonous ground station operations concept and network design for EUVE and other earth-orbiting satellites. Berkeley, CA: Center for EUV Astrophysics, Technology Innovation Series (Publication Number 667).
Aked, R., and Pylyser, E. (1996). An operational concept for a small automonous satellite. Fourth International Symposium on Space Mission Operations & Ground Data Systems ÒSpaceOps 96Ó (SP-394, Vol. 2, pp. 905-912). Noordwijk, The Netherlands: ESA Publications.
Alonso, D. L. (1998). The effects of individual differences in spatial visualization ability on dual task performance. (Unpublished Doctoral Dissertation). College Park, MD: University of Maryland, Department of Psychology.
Alonso, D. L., & Norman, K. L. (1996). Apparency of contingencies in single panel menus (CAR-TR-824; CS-TR-3644). College Park, MD: University of Maryland, Center for Automation Research, Department of Psychology.
Arkes, H. R., Christensen, C., Lai, C., & Blumer C. (1987). Two methods of reducing overconfidence. Organizational Behavior and Human Decision Processes, 39, 133-144.
Baecker, R. M., Grudin, J., Buxton, W. A. S., & Greenberg, S. (1995). From customizable systems to intelligent agents. Readings in human-computer interaction: Toward the year 2000 (2nd ed., pp. 783-792). San Francisco, CA: Morgan Kaufmann.
Bainbridge, L. (1997). The change in concepts needed to account for human behavior in complex dynamic tasks. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 27, 351-359.
Bainbridge, L. (1987). Ironies of automation. In J. Rasmussen, K. Duncan, and J. Leplat (Eds.), New technology and human error (pp. 271-283). New York: Wiley.
Ballas, J. A., Heitmeyer, C. L., & Perez, M. A. (1991). Interface styles for the intelligent cockpit: Factors incluencing automation deficit. Proceedings of AIAA Computing in Aerospace 8 (pp. 657-667). Washington, DC: American Institute of Aeronautics and Astronautics.
Barber, K. S., Goel, A., Liu, T. H., Macfadzean, R. H., Martin, C. E., & Ramaswamy, S. (1997). Sensible agents. SMC Õ97 Conference Proceedings: 1997 IEEE International Conference on Systems, Man, and Cybernetics (Vol. 5, pp. 4146-4151). Piscataway, NY: Institute of Electrical and Electronics Engineers (IEEE).
Benbasat, I., & Todd, P. (1993). An experimental investigation of interface design alternatives: Icon vs. text and direct manipulation vs. menus. International Journal of Man-Machine Studies, 38, 369-402.
Bergeron, H. P. (1981). Single pilot IFR autopilot complexity benefit trade-off study. Journal of Aircraft, 18, 705-706.
Billings, C. E. (1990). A concept of human-centered automation. Abstracts of AIAA/NASA/FAA/HFS Symposium, Challenges in Aviation Human Factors: The National Plan (pp. 3-5). Washington, DC: American Institute of Aeronautics and Astronautics.
Boehm-Davis, D. A., Holt, R. W., Koll, M., Yastrop, G., & Peters, R. (1989). Effects of different data base formats on information retrieval. Human Factors, 31, 579-592.
Bowers, C., Deaton, J., Oser, R., Prince, C., & Kolb, M. (1995). Impact of automation on aircrew communication and decision-making performance. International Journal of Aviation Psychology, 5, 145-167.
Bushman, J. B., & Mitchell, C. M. (1986). Modeling the supervisory control hierarchy in a command and control environment. Proceedings of the 1986 IEEE Conference on Systems, Man, and Cybernetics (pp. 14-17). New York: Association for Computing Machinery.
Butler, S. A. (1990). The effect of method of instruction and spatial visualization ability on the subsequent navigation of a hierarchical data base (CAR-TR-488; CS-TR-2398). College Park, MD: Department of Psychology and the Human-Computer Interaction Laboratory, University of Maryland.
Carraway, J. (1996). Assessing operations complexity. In D. G. Boden and W. J. Larsen (Eds.), Cost-effective space mission operations (pp. 113-133). New York: McGraw-Hill.
de Keyser, V. (1986). Technical assistance to the operator in case of incident: Some lines of thought. In E. Hollnagel, G. Mancini, & D. D. Woods (Eds.), Intelligent decision support in process environments (pp. 229-253). Berlin: Springer-Verlag.
Dickinson, G. W., DeSanctis, G., & McBride, D. J. (1986). Understanding the effectiveness of computer graphics for decision support: A cumulative experimental approach. Communications of the ACM, 29(1), 40-47.
Dupree, D. A., & Wickens, C. D. (1982). Individual differences and stimulus discriminability in visual comparison reaction time. Proceedings of the Human Factors Society 26th Annual Meeting(pp. 809-811). Santa Monica, CA: Human Factors Society.
Edwards, W., & Von Winterfeldt, D. (1986). On cognitive illusions and their implications. In H. R. Arkes and K. R. Hammond (Eds.), Judgment and decision making: An interdisciplinary reader (pp. 642-679). New York: Cambridge University Press.
Ehret, B. D., Gray, W. D., & Kirschenbaum, S. S. (2000). Contending with complexity: Developing and using a scaled world in applied cognitive research. Human Factors, 42, 8-23.
Eidelkind, M. A. (1995). Operator trust and task delegation strategies in semi-autonomous agent systems: Experimental agent integration in missions operations room (Unpublished MasterÕs thesis). Washington, DC: George Washington University.
Ekstrom, R. B., French, J. W., Harman, H. H., & Dermen, D. (1976). Manual for kit of factor-referenced cognitive tests. Princeton, NJ: Educational Testing Service.
Endsley, M. (1997). Level of automation: Integrating humans and automated systems. Proceedings of the Human Factors and Ergonomics Society 41st Annual Meeting (Vol. 1, pp. 200-209). Santa Monica, CA: Human Factors and Ergonomics Society.
Endsley, M., & Kiris, E. O. (1995). The out-of-the-loop performance problem and level of control in automation. Human Factors, 37, 381-394.
Fischer, S. G., & Murphy, E. D. (1983). Network Control Center full operational capability (FOC) human engineering support (CSC/TM-83/6143). Report prepared for Goddard Space Flight Center by Computer Sciences Corporation under Contract NAS5-22900, Task Assignment 561.
Fischhoff, B., Slovic, P., & Lichtenstein, S. (1977). Knowing with certainty: The appropriateness of extreme confidence. Journal of Experimental Psychology: Human Perception and Performance, 3, 552-564.
Fleeter, R. (1996). Reducing spacecraft cost. In J. R. Wertz & W. J. Larson (Eds.), Reducing space mission cost (pp. 161-191). Boston: Klewer Academic.
Gist, M. E., & Mitchell, T. R. (1992). Self-efficacy: A theoretical analysis of its determinants and malleability. Academy of Management Review, 17, 183-211.
Harris, W. C., Hancock, P. A., Arthur, E. J., & Caird, J. K. (1995). Performance, workload, and fatigue changes associated with automation. International Journal of Aviation Psychology, 5, 169-185.
Harvey, R. (1996). Applications of spacecraft autonomy and their influence over mission operations. Fourth International Symposium on Space Mission Operations & Ground Data Systems ÒSpaceOps 96Ó (SP-394, Vol. 2, pp. 913-919). Noordwijk, The Netherlands: ESA Publications.
Hollnagel, E. (1992). The intelligent use of intelligent systems. In O. Kaynak, G. Honderd, & E. Grant (Eds.), Intelligent systems: Safety, reliability and maintainability issues (pp. 42-59). New York: Springer-Verlag.
Hollnagel, E., & Woods, D. D. (1983). Cognitive systems engineering: New wine in new bottles. International Journal of Man-Machine Studies, 18, 583-600.
Hopkin, V. D. (1988). Human factors aspects of the AERA 2 program. Farnborough, UK: Royal Air Force Institute of Aviation Medicine.
Hopkin, V. D. (1987). Human factors implications of progressive air traffic control automation. In R. S. Jensen (Ed.), Proceedings of the Fourth International Symposium on Aviation Psychology (pp. 179-185). Columbus, OH: Ohio State University.
Hopkin, V. D. (1991). The impact of automation on air traffic control systems. In J. A. Wise, V. D. Hopkin, and M. L. Smith (Eds.), Automation and systems issues in air traffic control (NATO Advanced Science Institutes Series F, Vol. 73, pp. 3-19). New York: Springer-Verlag.
Hucteau, M. (1996). COPS: Supervision and performance of COSPAS-SARSAT system. Fourth International Symposium on Space Mission Operations & Ground Data Systems ÒSpaceOps 96Ó (SP-394, Vol. 2, pp. 1080-1087). Noordwijk, The Netherlands: ESA Publications.
Kahneman, D., Slovic, P., & Tversky, A. (Eds.) (1982). Judgment under uncertainty: Heuristics and biases. Cambridge, UK: Cambridge University Press.
Kahneman, D., & Tversky, A. (1973). On the psychology of prediction. Psychological Review, 80, 251-273.
Kahneman, D., & Tversky, A. (1972). Subjective probability: A judgment of representativeness. Cognitive Psychology, 3, 430-454.
Kahneman, D., & Tversky, A. (1982). The simulation heuristic. In D. Kahneman, P. Slovic, & A. Tversky (Eds.), Judgment under uncertainty: Heuristics and biases. Cambridge, UK: Cambridge University Press.
Klein, J., Kulp, D., & Rashkin, R. (1996). State modeling and pass automation in spacecraft control. Fourth International Symposium on Space Mission Operations & Ground Data Systems ÒSpaceOps 96Ó (SP-394, Vol. 2, pp. 1023-1029). Noordwijk, The Netherlands: ESA Publications.
Kontogiannis, T., & Hollnagel, E. (1998). Application of cognitive ergonomics to control room design of advanced technologies. International Journal of Cognitive Ergonomics, 2, 243-268.
Lecouat, F., & De Saint Vincent, A. (1996). System autonomy through ground operations automation. Fourth International Symposium on Space Mission Operations & Ground Data Systems ÒSpaceOps 96Ó (SP-394, Vol. 2, pp. 928-937). Noordwijk, The Netherlands: ESA Publications.
Lee, J., & Moray, N. (1992). Trust, control strategies and allocation of functions in human-machine systems. Ergonomics, 35, 1243-1270.
Loftus, E. F. (1979). Eyewitness testimony. Cambridge, MA: Harvard University Press.
Lord, M. J. (1996). Ground systems. In D. G. Boden & W. J. Larson (Eds.), Cost-effective space mission operations (pp. 369-408). New York: McGraw-Hill.
MacGregor, D., & Slovic, P. (1986). Graphic representation of judgmental information. Human-Computer Interaction, 2, 179-200.
Mackworth, N. H. (1948). The breakdown of vigilance during prolonged visual search. Quarterly Journal of Experimental Psychology, 1, 5-61.
Mackworth, N. H. (1950). Researches on the measurement of human performance. (Medical Research Council Special Report Series No. 268. London: HM Stationery Office.) (Reprinted in H. W. Sinaiko (Ed.), (1961), Selected papers on human factors in the design and use of control systems (pp. 174-331). New York: Dover.
Maes, P. (1994). Agents that reduce work and information overload. Communications of the ACM, 37(7), 30-40, 146. (Reprinted in Baecker, Grudin, Buxton, & Greenberg, 1995, pp. 811-821).
Maes, P. (1996). Interface agents (Tutorial presented at User Interface Õ96). North Andover, MA: User Interface Engineering.
Marshall, M. H., Landshof, J. A., & van der Ha, J. C. (1996). Reducing mission operations cost. In J. R. Wertz & W. J. Larson (Eds.), Reducing space mission cost (pp. 193-227). Boston: Klewer Academic.
McGee, V. E. (1971). Principles of statistics: Traditional and Bayesian. New York: Appleton-Century-Crofts.
Meyer, J., Shinar, D., & Leiser, D. (1997). Multiple factors that determine performance with tables and graphs. Human Factors, 19, 268-286.
Mitchell, C. M. (1981). Human-machine interface issues in the Multisatellite Operations Control Center-1 (MSOCC-1)(NASA Technical Memorandum 83826). Greenbelt, MD: NASA-Goddard Space Flight Center.
Mitchell, C. M. (1983). The human as supervisor in automated systems. In C. M. Mitchell, P. M. Van Balen, and K. L. Moe (Eds.), Human factors considerations in system design (NASA Conference Publication 2246, pp. 257-287). Greenbelt, MD: NASA-Goddard Space Flight Center.
Mitchell, C. M., & Saisi, D. L. (1987). Use of model-based qualitative icons and adaptive windows in workstations for supervisory control systems. IEEE Transactions on Systems, Man, and Cybernetics, SMC-17, 573-593.
Mitchell, C. M., & Sundstrom, G. A. (1997). Human interaction with complex systems: Design issues and research approaches. IEEE Transactions on Systems, Man, and CyberneticsÑPart A: Systems and Humans, 27, 265-273.
Molloy, R., & Parasuraman, R. (1996). Monitoring an automated system for a single failure: Vigilance and task complexity effects. Human Factors, 38, 311-322.
Moray, N.(1986). Monitoring behavior and supervisory control In K. R. Boff, L. Kaufman, and J. P. Thomas (Eds.), Handbook of perception and human performance(Vol. II: Cognitive processes and performance, pp. 40-1 - 40-51). New York: Wiley.
Morrison, M. (1996). Spacecraft performance and analysis. In D. G. Boden and W. J. Larson (Eds.), Cost-effective space mission operations (pp. 465-484). New York: McGraw-Hill.
Mosier, K. L., Skitka, L. J., Heers, S., & Burdick, M. (1998). Automation bias: Decision making and performance in high-tech cockpits. International Journal of Aviation Psychology, 8, 47-63.
Muir, B. M. (1987). Trust between humans and machines, and the design of decision aids. International Journal of Man-Machine Studies, 27, 527-539.
Muir, B. M. (1994). Trust in automation: Part I. Theoretical issues in the study of trust and human intervention in automated systems. Ergonomics, 37, 1905-1922.
Muir, B. M., & Moray, N. (1996). Trust in automation, Part II. Experimental studies of trust and human intervention in a process control simulation. Ergonomics, 39, 429-460.
Murphy, E. D., & Mitchell, C. M. (1986). Cognitive attributes: Implications for display design in supervisory control systems. International Journal of Man-Machine Studies, 25, 411-438.
Murphy, E. D., & Norman, K. L. (1998). Beyond supervisory control: Human performance in the age of autonomy. Presented at the Third Automation Technology and Human Performance Conference, Norfolk, VA, March 26. To appear in Automation technology and human performance: Current research and trends. Hillsdale, NJ: Erlbaum.
Murphy, E. D., Norman, K. L., & Moshinsky, D. Y. (1999). VisAGE usability study (Prepared for NASA-Goddard Space Flight Center under Grant No. NAG5-3425; HCIL-TR-99-04;LAP-TR-1999-01). College Park, MD: Laboratory for Automation Psychology, University of Maryland.
Murphy, E. D., Norman, K. L., Truszkowski, W., & Grubb, T. (1997). Investigating cognitive issues in lights-out, spacecraft-ground mission operations. Poster sessions: Abridged proceedings of HCI International Õ97 (p. 95). New York: Elsevier.
Narborough-Hall, C. S. (1987). Automation implications for knowledge retention as a function of operator control responsibility. In D. Diaper and R. Winder (Eds.), People and computers II (pp. 269-282). New York: Cambridge University Press.
NASA. (1993). Mars Observer mission status. Last accessed at http://ftp.seds.org/puÉOBSERVER/mars03.18.93 on 3/18/93.
NASA. (1997). COBE: COsmic Background Explorer. Last accessed at http://spectrum.lbl.gov/www/cobe/cobe.html on 10/30/97.
Neal, V., Lewis, C. S., & Winter, F. H. (1995). Spaceflight. New York: Macmillan.
Norman, D. A. (1988). The design of everyday things. New York: Doubleday/Currency.
Norman, K. L. (1991a). Models of the mind and machine: Information flow and control between human and computers. In M. C. Yovits (Eds.), Advances in computers (Vol. 32, pp. 201-253). New York: Academic Press.
Norman, K. L. (1991b). The psychology of menu selection: Designing cognitive control at the human/computer interface. Norwood, NJ: Ablex.
Norman, K. L. (1994). Spatial visualization -- A gateway to computer-based technology. Journal of Special Education Technology, XII(3), 195-206.
Norman, K. L., & Butler, S. (1989). Search by uncertainty: Menu selection by target probability (CAR-TR-432; CS-TR-2230). College Park, MD: University of Maryland, Center for Automation Research and the Department of Computer Science.
OÕHara, J. M. (1993). The effects of advanced technology systems on human performance and reliability. Proceedings of the Topical Meeting on Nuclear Plant Instrumentation, Control, and Man-Machine Interface Technologies (pp. 253-259). La Grange Park, IL: American Nuclear Society.
Parasuraman, R. (2000). Designing automation for human use: Empirical studies and quantitative models. Ergonomics, 43, 931-951.
Parasuraman, R. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39, 230-253.
Parasuraman, R., Molloy, R., & Singh, I. L. (1993). Performance consequences of automation-induced Òcomplacency.Ó International Journal of Aviation Psychology, 3, 1-23.
Patterson, E. S., & Woods, D. D. (1997). Shift changes, updates, and the on-call model in space shuttle mission control. Columbus, OH: Ohio State University. Accessible at http//csel/eng.ohio~state.edu:8080/~csel/UPDATES97.htm.
Pelligrino, J. W., & Hunt, E. B. (1991). Cognitive models for understanding and assessing spatial abilities. In H. A. H. Rowe (Ed.), Intelligence: Reconceptualization and measurement (pp. 203-225). Hillsdale, NJ: Erlbaum.
Plous, S. (1993). The psychology of judgment and decision making. New York: McGraw-Hill.
Reason, J. (1990). Human error. New York: Cambridge University Press.
Riley, V. (1994). A theory of operator reliance on automation. In M. Mouloua & R. Parasuraman (Eds.), Human performance in automated systems: Current research and trends (pp. 8-14). Hillsdale, NJ: Erlbaum.
Sage, A. P. (1981). Behavioral and organizational considerations in the design of information systems and processes for planning and decision support. IEEE Transactions on Systems, Man and Cybernetics, SMC-11, 640-678.
Salthouse, T. A., Babcock, R. L., Mitchell, D. R. D., & Palmon, R., & Skovronek, E. (1990). Sources of individual differences in spatial visualization ability. Intelligence, 14, 187-230.
Salthouse, T. A., Babcock, R. L., Skovronek, E., Mitchell, D. R. D., & Palmon, R. (1990). Age and experience effects in spatial visualization. Developmental Psychology, 26, 128-136.
Sarter, N. B., & Woods, D. D. (1995a). ÒFrom tool to agentÓ: The evolution of (cockpit) automation and its impact on human-machine coordination. Proceedings of the Human Factors and Ergonomics Society 39th Annual Meetings (pp. 79-83). Santa Monica, CA: Human Factors and Ergonomics Society.
Sarter, N. B., & Woods, D. D. (1995b). ÒHow in the world did we ever get into that mode?Ó Mode error and awareness in supervisory control. Human Factors, 37, 5-19.
Sarter, N. B., & Woods, D. D. (1997). Team play with a powerful and independent agent: Operational experiences and automation surprises on the Airbus A-320. Human Factors, 39, 553-569.
Sassen, J. M. A., Buiel, E. F. T., & Hoegee, J. H. (1994). A laboratory evaluation of a human operator support system. International Journal of Human-Computer Studies, 40, 895-931.
Shaiken, H. (1986). The human impact of automation. IEEE Control Systems Magazine, 6 (Dec.), 3-6.
Sheppard, S. B., Murphy, E. D., & Stewart, L. J. (1985). Ground control human factors study: Theoretical model development (Contract NAS5-27684, Task 500-21). McLean, VA: Computer Technology Associates, Inc.
Sheppard, S. B., Murphy, E. D., & Stewart, L. J. (1991). A methodology for assessing the human factors impacts of increased automation. Proceedings of the Human Factors Society 29th Annual Meeting (pp. 556-560). Santa Monica, CA: Human Factors Society.
Sheridan, T. B. (1976). Preview of models of the human monitor/supervisor. In T. B. Sheridan & G. Johannsen (Eds.), Monitoring behavior and supervisory control (pp. 175-180). New York: Plenum.
Sheridan, T. B. (1997). Supervisory control. In G. Salvendy (Ed.), Handbook of human factors and ergonomics (2nd ed., pp. 1295-1327). New York: Wiley.
Sheridan, T. B. (1988a). Human and computer roles in supervisory control and telerobotics: Musings about function, language and hierarchy. In L. P. Goodstein, H. B. Anderson, & S. E. Olsen (Eds.), Tasks, errors and mental models (pp. 149-160). New York: Taylor & Francis.
Sheridan, T. B. (1988b). Task allocation and supervisory control. In M. Helander (Ed.), Handbook of human-computer interaction (pp. 159-173). New York: Elsevier.
Sheridan, T. B., & Hennessy, R. T. (Eds.). (1984). Research and modeling of supervisory control behavior: Report of a workshop. Washington, DC: National Academy Press.
Sheridan, T. B., & Verplanck, W. L. (1978). Human and computer control of undersea teleoperators (Tech. Rep.). Cambridge, MA: M.I.T., Man-Machine Laboratory.
Shneiderman, B. (1998). Designing the user interface: Strategies for effective human-computer interaction (3rd ed.). Reading, MA: Addison-Wesley.
Shneiderman, B. (1997). Direct manipulation versus agents: Paths to predictable, controllable, and comprehensible interfaces. In J. M. Bradshaw (Ed.), Software agents (pp. 97-106). Cambridge, MA: AAAI Press/The MIT Press.
Shneiderman, B. (1995). Looking for the bright side of user interface agents. interactions, II.I, 13-15.
Shneiderman, B. (2000). Universal usability: Pushing human-computer interaction research to empower every citizen. Communications of the ACM, 43 (May), 85-91.
Smith, P., McCoy, C. E., & Layton, C. (1997). Brittleness in the design of cooperative problem-solving systems: The effects on user performance. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 27, 360-371.
SPSS, Inc. (1997). SPSS, Release 8.0. Chicago, IL: Author.
Stewart, L. J., & Murphy, E. D. (1984). Ground control human factors study: Approach for monitoring/analyzing the effects of automation on control system personnel (Contract NAS5-27684, Task 500-11). McLean, VA: Computer Technology Associates, Inc.
Swain, A. L. (1987). Four human factors problems for system operators as dependence on automation increases. In G. Salvendy (Ed.), Cognitive engineering in the design of human-computer interaction and expert systems (Advances in Human Factors/Ergonomics, Vol. 10B, pp. 105-112). Amsterdam: Elsevier.
Symantec. (1995). VisualCafŽ. Cupertino, CA: Symantec Corporation.
Takeuchi, A., & Naito, T. (1995). Situated facial displays: Towards social interaction. Proceedings of CHI Õ95 Conference on Human Factors in Computing Systems (pp. 450-455). New York: Association for Computing Machinery.
Thackray, R. I. (1980).
Boredom and monotony as a consequence of automation: A consideration
of the evidence relating boredom and monotony to stress (FAA-AM-80-1). Washington, DC: Department of
Transportation/Federal Aviation Administration, Office of Aviation Medicine.
Thackray, R.I., & Touchstone, R. M. (1989). A comparison of detection efficiency on an air traffic control monitoring task with and without computer aiding (DOT/FAA/AM-89/1). Washington, DC: Department of Transportation/Federal Aviation Administration, Office of Aviation Medicine.
Truszkowski, W. (1996a). An introduction to agent technology. (Presentation viewgraphs). Greenbelt, MD: NASA-Goddard Space Flight Center).
Truszkowski, W. (1996b). ÒLights outÓ operations: Human/computer interfaces/interactions (HCI). 1996 Technology workshop Ð Autonomous Òlights outÓ operations workshop: Operational challenges and promising technologies (Presentation viewgraphs, pp. 355-367). Greenbelt, MD: Mission Operations and Data Systems Directorate, NASA-Goddard Space Flight Center.
Truszkowski, W., & Odubiyi, J. (1994). An agent-oriented approach to automated mission operations. Proceedings of the Third International Conference on Space and Ground Systems, NASA-Goddard Space Flight Center, Nov. 1994.
Tversky, A., & Kahneman, D. (1973). Availability: A heuristic for judging frequency and probability. Cognitive Psychology, 5, 207-232.
Tversky, A., & Kahneman, D. (1974). Judgments under uncertainty: Heuristics and biases. Science, 185, 1124-1131.
Vessey, I. (1991). Cognitive fit: A theory-based analysis of the graphs versus tables literature. Decision Sciences, 22, 219-240.
Vincente, K., Hayes, B. C., & Williges, R. C. (1987). Assaying and isolating individual differences in searching a hierarchical file system. Human Factors, 29, 349-359.
Von Winterfeldt, D., & Edwards, W. (1986). Decision analysis and behavioral research. New York: Cambridge University Press.
Wei, Zhi-Gang, Macwan, A. P., & Wieringa, P. A. (1998). A quantitative measure for degree of automation and its relation to system performance and mental load. Human Factors, 40, 277-295.
Wickens, C. D. (1984). Engineering psychology and human performance. Columbus, OH: Charles E. Merrill.
Wickens, C. D. (1992). Engineering psychology and human performance (2nd ed.). New York: HarperCollins.
Wickens, C. D., & Kessel, C. (1979). The effects of participatory mode and task workload on the detection of dynamic system failures. IEEE Transactions on Systems, Man, and Cybernetics, SMC-9, 24-34.
Wiener, E. L. (1987). Fallible humans and vulnerable systems: Lessons learned from aviation. In J. A. Wise and A. Debons (Eds.), Information systems: Failure analysis (pp. 163-181). Berlin: Springer-Verlag.
Wiener, E. L., & Curry, R. E. (1980). Flight-deck automation: Promises and problems. Ergonomics, 23, 995-1011.
Woods, D. D. (1994). Automation: Apparent simplicity, real complexity. In R. Parasuraman & M. Mouloua (Eds.), Automation technology and human performance (pp. 3-17). Hillsdale, NJ: Erlbaum.
Woods, D. D., Sarter, N., & Billings, C. (1997). Automation surprises. In G. Salvendy (Ed.), Handbook of human factors and ergonomics (2nd ed., pp. 1926-1943). New York: Wiley.
Zuboff, S. (1988). In the age of the smart machine: The future of work and power. New York: Basic Books.