By: Michael Kramer
A recent Aerospace study sought to determine whether graphical representations of engineering thought processes could help capture the knowledge of senior engineers.
Two generations of space systems engineers have designed, developed, and refined the processes, technologies, and techniques for building satellites. Many of these pioneers have left the aerospace industry, while others are approaching retirement. Much of their knowledge will depart with them unless it is captured and organized in some useful way. As one manager said, “More than 30 percent of my people will retire in the next three years, while almost 30 percent of my people have less than three years of experience. How can we bridge the knowledge from one generation to the next?”
A research project conducted at Aerospace from March through August 2005 attempted to collect, organize, and efficiently reuse the knowledge of domain experts. The focus was on identifying the processes that experienced engineers use to drive initial design decisions when they are translating mission requirements into design requirements. The project was designed to apply “concept maps,” which graphically depict the relationships among interdependent ideas. Three goals were identified: to demonstrate that concept maps can be used for knowledge acquisition among multiple domain experts; to develop a prototype knowledge representation model from the concept maps; and to assess the utility of that model by examining a limited problem to see if the maps help solve it.
Concept mapping was originally developed as a tool for education. It’s based on the theory that people learn by associating new concepts with concepts they already know. New concepts are tied to existing concepts by the perceived relationships among them, and prior knowledge is used as a framework for understanding and acquiring new knowledge. A tripartite association of concept-relationship-concept evolves from this learning; this relationship triplet is called a precept. Concept maps can be created to visually describe these relationships with numerous precepts. They are used to stimulate ideas and are believed to help foster creativity.
Knowledge Harvesting and Evocative Objects
This research project attempted to harvest the knowledge of Aerospace experts by eliciting their memories of a project and tying them to other engineers’ memories of the same project. One way to elicit and harvest knowledge is through the use of evocative objects. For example, a former Xerox Palo Alto Research Center director tells the story of how Xerox repairmen and design engineers were brought together in the hope that they could collaborate to produce better designs. The meeting was going nowhere until the director rolled in one of the machines, letting the repair people take it apart while sharing stories about the parts. The stories centered on what worked, what failed, and what drove the design decisions that led to the development of the parts. Taking the machine apart evoked the men’s memories and helped them share what they knew.
Getting satellites back from space is difficult, at best, so taking one apart to get the design stories was unlikely for the Aerospace project. Some other type of evocative object was therefore needed. Interviewing subject matter experts and developing concept maps seemed a fruitful option. The initial concept maps would serve as evocative images, drawing out the memories of other engineers, enabling the researchers to develop even more detailed maps.
A basic technique for knowledge harvesting is to choose a known, solved problem and review it with experts in that field. This is what was done with the Aerospace engineers. The statement of requirements for a satellite development program was reviewed as a representative tough problem. Two sections of those requirements were chosen for study—the attitude determination and control system, and the power system. These were chosen in part because they have very little overlap and because the people who work on them are in separate facilities and have very little interaction. The directors of the Control Analysis Department within the Vehicle Systems Division and the Electrical and Power Systems Department within the Electronics and Sensors Division were asked to suggest leaders in design for their respective domains, and those leaders were asked to recommend domain experts. At the conclusion of the search, two domain experts from the Control Analysis Department and three from the Electrical and Power Systems Department agreed to help develop concept maps covering their respective domains. The five participants averaged more than 22 years of experience.
The engineers’ stories and responses to solving the requirements-translation problem were elicited and transcribed and then reviewed with each engineer. In most cases, a few additional concepts were expressed during this review. The stories and additions were then converted into concept maps. The process began with collecting every unique concept name and its relational connectors to other concept names. These precepts were sorted from most abstract to most concrete and mapped to a single diagram. The experts were then given their concept maps for review. As expected, these maps functioned as evocative objects. As each expert reviewed them, new concepts, relationships, and connections were defined and added. This second round of concept maps depicted a vastly increased amount of information.
Each of these experts brought a unique background in terms of education, experience, and temperament, all of which influenced their approaches to design. One of the goals of this project was to gather the varied knowledge of numerous experts, but it was also important to preserve the valuable differences among experts’ knowledge. A comparison of the ordered lists of concepts showed multiple instances of similar or identical concepts among different experts. On the other hand, there were differences among the experts’ lists of concepts, and many more differences in the relationships between them. The experts in each domain used a core taxonomy; these common terms were the language of that domain, and the relationships among the concepts reflected each expert’s approach to that domain. The maps from the engineers in the Electrical and Power Systems Department shared a common vocabulary, but they also reflected individual observations. Similarly, the maps from the domain experts in the Control Analysis Department reflected the language those experts use and their approach to a design problem. The next step was to combine the concept maps from all the experts in a domain. First, their common concepts and relationships were mapped out. This produced a core map that reflected their agreement. Then, the remaining relationships and concepts were added, with color coding to preserve and reflect each expert’s approach to translating the requirements.
The essential questions at this point were, “Do the concept maps accurately depict the experts’ approaches to their domains, and do they provide a useful tool for knowledge capture and sharing?” To assess this, a Solomon four-group test was run. The Solomon test involves gathering four groups of similar people who are randomly assembled out of a test population. Two tests are administered: The first group takes only the first test; the second group takes the first test and then, after some intervention or learning process, takes the second test; the third group skips the first test, but takes the second test after completing the learning process; the fourth group takes only the second test, without any preliminary learning process. In this case, the first and second tests were the same: Engineers were given one hour to review a statement of requirements on orbit for the satellite program in development and to list design requirements.
The test was administered first with one group, then with the second group. As expected, both groups fared similarly because they were randomly assembled from a group with similar backgrounds and similar translation capabilities. The second group was then given a concept map and the same problem to review again. The second pass through the problem generated different, additional, and more diverse design requirements. This may have been because the engineers had the concept map to work with, or it might have been because they had seen the problem once already. The third group was then given the problem and the concept map at the same time. This group used the concept map as a guide for translating the statement of mission requirements to design requirements. The third group translated significantly more requirements than the second group did after two passes at the problem.
Because the first and second tests were the same in this project (listing design requirements), the first group functioned as the fourth by repeating the test. An analysis of the results indicates that for both the Electrical and Power Systems group and the Control Analysis Department group, the use of the concept maps led to the translation of significantly more design requirements.
The results of this research project show that concept maps can serve as evocative objects in the recall of concepts and precepts used by domain experts. As the experts reviewed and corrected their maps, they recalled more concepts and more relationships among them. The structural, graphic, and visual nature of the concept map fostered memory recall. In addition, concept maps from individual domain experts can be combined to enable the joint representation of different approaches to translating mission requirements into design requirements.
The Solomon test also revealed the usefulness of concept maps as guides in the translation of design requirements. The use of the concept maps produced larger numbers of requirements and significantly more diverse requirements than a single review of the given problem, or even a second review without the concept maps as a guide. The test results are at best suggestive, not determinative, of the utility of this model, and further testing with larger populations is required to establish significant utility. Nonetheless, it would be of obvious benefit to the national security space community if knowledgeable people with a map but without the direct intervention of domain experts could translate a wide and diverse set of design requirements. The concept mapping model could be designed so that domain experts would help generate domain maps and review the requirements translated by less experienced people. This would make efficient use of a dwindling senior workforce. Aerospace is planning a follow-up project to explore the possibility of an interactive domain map with some form of semantic update based on later stages of design and development.
- J. D. Novak, Learning, Creating and Using Knowledge: Concept Maps as Facilitative Tools in Schools and Corporations (Lawrence Erlbaum and Associates, Mawah, NJ, 1998).
- J. D. Novak and D. B. Gowin, Learning How to Learn (Cambridge University Press, New York, 1984).
- S. Turkle, The Second Self: Computers and the Human Spirit (Simon and Schuster, New York, 1985).
- U. Sekaran, Research Methods for Business, Fourth ed. (John Wiley and Sons, New York, 2003).