Home Page

The proposed concept is designed to aid the Air Mobility Command (AMC) to bring their "dysfunctional systems" into an operational functional state by applying an integrated Systems Engineering (SE) approach with an Instructional System Design (ISD) approach based on the principles and philosophies of Total Quality Management (TQM). Additionally, this concept identifies some required information and training system management tools to be integrated with the functional training system that will allow AMC and AMC's Aircrew Training System (ATS) prime contractors to take a proactive approach in enhancing training effectiveness, continually improving training output product quality, and increasing cost efficiencies by focusing on the overall mission objective of AMC's training system.

Informational Note: The "Total Training Concept" is not the single contractor approach for developing and operating AMC's training system as espoused in previous training system training approaches. The major short coming of the single contractor concept is that it is in direct conflict with the Federal Acquisition Regulations (FARs). The presented concept promotes the fundamental foundation of the FARs by providing a vehicle through which AMC can: increase their training system's ability to provide more effective training; provide a consistently higher quality student output from the various aircrew training systems; decrease the overall life cycle operating cost of the total training system; promote equitable cost competitiveness among Aircrew Training System prime contractors; provide AMC with the tools and information to adequately evaluate true "Best Value" among ATS prime contractor bids; enhance AMC's student information management capability; and promote more effective and efficient aircrew/aircraft cross training.

Ms. Joan E. Hendrix is employed by Simms Industries, Inc. as the C-17 ATS Courseware Product Assurance Manager. In this capacity, Ms. Hendrix performs Independent Verification and Validation Services for the C-17 Aircrew Training System. Ms. Hendrix has been instrumental in the design and implementation of all C-17 ATS Formative, Summative, and (currently) Operational Evaluation activities. Ms. Hendrix is a Ph.D. candidate in the Instructional Psychology and Technology program at the University of Oklahoma.

Mr. Raymond G. Moore is a Systems Engineering and Life Cycle Cost Analyst contracted with General Technology Corporation, under contracted with McDonnell Douglas Training Systems to develop an automated life cycle cost modeling tool for the C-17 Aircrew Training System. Mr. Moore has also developed Reliability and Maintainability Models and predictions, program plans, and integrated logistic support analysis reports for numerous defense related programs. As a member of the Society of Automotive Engineers (SAE) G11 review committee, he provided subject matter expert review for sections of the SAE Reliability, Maintainability and Supportability (RMS) Handbook. Mr. Moore also publishes "The 8A Connection", a quarterly listing of Small Disadvantaged Businesses qualified to meet Government Prime Contractor's SDB set aside subcontracting requirements. Mr. Moore has a degree in Business Administration from Sam Houston State University and additional graduate training in Electrical/Electronics Engineering from the University of Texas at Arlington.

The training needs and operational requirements of the Airlift Mobility Command (AMC) currently exist in a fragmented and dysfunctional training system. Little design interface nor evaluative interchange exists between and among the current Aircrew Training Systems (ATS) within AMC. Although this is partially caused by employing different training contractors, the lack of a systematic interface among contractors does not have to be the standard. The entire AMC training system could be significantly improved by implementing a "Total Training Concept" which uses an integrated Systems Engineering (SE) and Instructional Systems Design (ISD) approach to develop systems specifications for all AMC training, each specific ATS, and the infrastructure of the major components and processes within an ATS. The two disciplines of Systems Engineering and Instructional System Design are uniquely compatible when jointly applied to the specification, design, and development of Aircrew Training Systems. Specialists in both disciplines are trained to approach analysis, design, and implementation issues using a top-down structure to identify the most effective and efficient design required to meet critical objectives and operational needs.

Training systems that cannot be changed cannot be improved. Planning for change is very different than planning for continued, routine operation of a training system. Many more systems variables need to be taken into account, and systems thinking is critical to the success of these change efforts. The art of systems thinking includes learning to recognize the ramifications and tradeoffs of the selected action. One of the values of a systems-analytical approach is the identification of data gaps where more research and evaluation information is required for informed decision making.

With the current structure of "stand-alone" Formal Schools, the outputs from basic training schools are ultimately the input to follow-on Formal School training. For maximum effectiveness and efficiency, there should be a match between basic school output performance criteria and follow-on Formal School entry level proficiency criteria; however, due to the stand-alone design of the Formal School systems, this is not always the case. There may be training content redundancies and/or training gaps between the basic and the follow-on schools.

Currently, two anomalies exist within AMC's training system approach that are the primary cause of the detriments to AMC's training effectiveness and desired training cost efficiencies. These two anomalies are:

AMC's total training system has become a "dysfunctional system" comprised of functional systems operating as independent entities that have not focused on the "total" end product.

The independent functional systems of AMC's current training system have taken a reactive approach to identifying and resolving training deficiencies and budget/cost overrun impacts due to a lack of sufficient viable historical ATS data.

The current AMC Aircrew Training Systems have been designed using a traditional "piece-meal" approach. Each ATS has been built as a "stand-alone" training system, with its own unique requirements, structure, criteria, and standards. This can result in problems when one ATS's product, or output, is the follow-on ATS's entry level student, or system input. At a higher level, such training systems do not enable adequate comparisons of training effectiveness and cost efficiencies. Because each system has its own standards and type of data collected and reported, uniform data is not available for cross systems comparisons. The lack of uniform criteria, standards, data collection and reporting processes, and the tools to perform these functions does not allow the Air Force the ability to systematically evaluate the results of their total training systems.

The criteria for ATS design should be standardized to determine program output and follow-on program entry level skill requirements. The "Total Training Concept" proposes a systematically designed, implemented, and evaluated training system approach which would specify standard learning and evaluation criteria for all AMC Aircrew Training Systems. The criteria should specify such items as student entry level requirements, student evaluation procedures, graduate performance standards, uniform data collection and analysis processes, and implementation of standardized data-collection, analysis and evaluation tools. The benefits of designing and implementing such a system would include reduced training remediation requirements, lowered total life cycle costs, enhanced cross training of aircrews, and would allow for the overall comparison of training effectiveness and cost efficiency across Aircrew Training Systems.

In addition, the "Total Training Concept" proposes the development of a centralized data collection and reporting system and the development and implementation of a set of data analysis tools. The centralized data collection and reporting system could provide "real time" information in a paperless environment that would provide AMC with accurate information ranging from an individual student's training history to data on the current output (graduating classes) from an ATS. Combining a set of analysis tools with the centralized data collection and reporting system, AMC could continually assess the on-going cost and performance of each ATS within the AMC training system, project future budget needs and perform "what-if" funding profile assessments, and plan for accomplishment of AMC's objectives (set by national defense strategies) using up-to-date, real time data regarding the current operational performance and cost of AMC's training capabilities.

The current "information age" technology provides the ability to collect large volumes of data. When multiple types of data are collected, the ability to identify and extract the "right" data and then process this data into useable information, is required. Having the right information in a timely manner, with the tools to re-process this information as may be needed for use by multiple disciplines, provides management with the ability to proactively control decisions in order to produce an effective and efficient operational organization.

We propose an approach to training systems specification and design that can be viewed as model building. Models serve as visions that can, and should, inspire the next generation of engineers. In our vision, we believe that a model can be built for use in restructuring AMC's total training system so that it can function as a unified set of systems performing the intricate task of integrated training, rather than function as a disconnected grouping of independent training systems.

Macro-engineers, or model builders, operate in a complex environment. As macro-engineers, we must work with the decision-makers in the target organization to define and build a shared vision of an optimal system of training systems. A shared vision shows where the organization wants to go and what it will be like when they get there. The power to determine training policy and to determine engineering and design approaches is usually shared by numerous individuals within organizations. The sharing of this power means that many individuals with widely divergent perspectives must develop a shared vision before the macro-engineer's concepts can be considered for implementation.

Three primary customers within the Air Force would need to develop a shared vision in order to implement the proposed training system engineering approach. These include AMC, responsible for generating and defining the training needs; the Air Force Education and Training Command (AETC), responsible for all Air Force training systems; and the Air Force Systems Command (ASC), responsible for program and contract administration. Members of these three groups, with appropriate support from knowledgeable Systems Engineers and Instructional Systems Designers, would provide the foundation for shaping and developing a vision of the future which would result in the systematic design and structure of their aircrew training systems.

The approach needed to bring the "Total Training Concept" into reality is rooted in the philosophies of Concurrent Engineering. By integrating the engineering processes of Systems Engineering with Instructional Systems Design to formulate a single engineering effort focused on a common objective, a set of comprehensive design criteria would be established that provide a macro to micro view which we refer to as the "Total Training Concept". This concept begins with a macro view and then breaks down the elements of a training system (the micro view) in order to develop a complete understanding of the "training system".

The processes of Systems Engineering and Instructional Systems Design fit together in ways which enhance the implementation of a Concurrent Engineering philosophy. Systems Engineering consists primarily of functional analysis, synthesis, evaluation and decisions. Instructional Systems Design consists primarily of analysis, design, development, implementation, and evaluation. In SE, the analysis activities consist of the system needs analysis, system requirements analysis, functional analysis, hardware requirements analysis, and software requirements analysis. ISD refers to this phase as Front-end Analysis, which includes task analysis, target population analysis, needs analysis, learning analysis, and media analysis. Following the analysis phases, SE performs the synthesis and integration/tradeoff evaluations in preparation for specifications development. In ISD, the design phase follows the analysis activities, and the output, or product of this phase is a complete set of systems design specifications, or training specifications. In SE, the result of this phase is a complete set of "A", "B", and "C" specifications. Both the SE and ISD engineering processes begin by identifying and analyzing a need, converting this need into a set of requirements, and then processing these requirements into a design solution which, when implemented, satisfies the identified need. Both of these engineering processes employ the application of scientific and engineering efforts to:

Transform an identified need into a description of system operational performance parameters and objectives through the use of an iterative process of analysis, definition, synthesis, and evaluation.

Integrate related technical parameters and ensure compatibility of all physical, functional, and program interfaces in a manner that optimizes the total system definition and design.

The output product of the Systems Engineering and Instructional Systems Design analysis and design processes is a descriptive set of system design documents. Following the design specifications phase, the ISD process continues with the development process. In this phase, the systems specifications and design specifications are translated into actual products to be used within the training system. Upon full development of the system, implementation of the training occurs. Both ISD and SE incorporate systematic evaluations at all phases of the analysis, design, and development processes. Figure 1 portrays the combined processes of Systems Engineering and Instructional Systems Design.

The problem of the current dysfunctional training systems could be alleviated by the use of a Concurrent Engineering process which would produce a complete set of integrated system specifications that combine the fundamental concepts of Instructional Systems Design, Systems Engineering, and the principles of Total Quality Management.

This set of integrated specifications would form the skeleton around which the current training system dysfunctionalities could be identified, evaluated, and corrected in order to produce the optimal functional training system. The comprehensive and systematic design of this Concurrent Engineering process would provide the "A", "B", and "C" level specifications for AMC's Aircrew Training Systems that would result in a "Total Training System" with increased training effectiveness, improved cost efficiencies, and higher quality output products.

In the ISD/SE Concurrent Engineering process, the activities of requirements analysis and system design would be accomplished by an integrated team of ISD and SE professionals. The first objective of this Concurrent Engineering team would be the application of Total Quality Management principles to identify the "user" and "supplier" interfaces and the overall integration requirements of the operational entities that comprise AMC's training system. The second objective of this Concurrent Engineering team would be to integrate the ISD processes with the SE processes in order to formulate a comprehensive set of design documents which sufficiently combine training requirements with system design requirements. While the idea of an integrated ISD and SE engineering process is not totally new, we find from previous experience

that most training system designs occur as a result of parallel ISD and SE efforts which are then merged at the end of the individual processes. With the ISD/SE Concurrent Engineering process, all of the ISD and SE engineering activities would be integrated into a single, cohesive engineering process resulting in a set of design documents guaranteed to create the "optimal system" which would produce the best quality product for the least cost. The following provides an generalized overview of the set of design documents resulting from the ISD/SE Concurrent Engineering approach that would be used to re-engineer AMC's dysfunctional training system (see Figure 2).

The "A" specification, or System specification, would be centered around the TQM principles of identifying the "user" and "supplier" relationships, interfaces, and requirements in order to promote an understanding the total training system's operational needs and the operational needs of the functional areas (i.e. aircrew training systems) that comprise AMC's training system. These high level needs would then be analyzed using the standard ISD and SE analysis processes to convert total training systems needs into a set of operational and functional requirements. This set of operational and functional requirements would consider the operational elements that comprise AMC's training system and define how these operational areas should function together in the total training system. One of the intents of this "A" specification will be to promote interchangeability among aircrew training systems, identify commonalities among aircrew training systems so that "optimal utilization" concepts could be employed to reduce replications (i.e. create once, use several times), and enhance the identification and development of tools to collect and process common information from the aircrew training systems that AMC could use to increase training quality while minimizing total cost of ownership.

The "B" specification, or Development specification, would be a set of general "design-to" requirements identifying the infrastructure and the necessary functional areas that should comprise an individual aircrew training system. This specification would establish the performance characteristics for the aircrew training system, as well as, the specific interface requirements for the aircrew training system in order for it to function as an efficient and effective part of AMC's training system. In addition to operational training system interface requirements, this specification would identify the specific interface requirements for the training system to AMC's data collection and reporting system. This interface requirement would not only identify the physical characteristics for interfacing to AMC's data collection and reporting system, but also identify the types of data that must be collected/reported, and the format that must be used by the training system operator in order to provide adequate traceability of the data. Standardization of format conventions will enhance the process of root cause analysis of training system problems. In addition to the B specification identifying the required functional areas of an aircrew training system, it would also identify the general performance characteristics and attributes required of those functional areas (e.g. the aircrew training system must have a training management system that ....this training management system must be capable of collecting individual student performance data and transferring this data to AMC's student records data collection system ...., etc.). The B specification would be constructed to enhance AMC's objective of the optimal training system, yet still allow for design flexibility in order to promote cost competitiveness among aircrew training system prime contractor developers. This specification would also provide AETC, AMC, and ASC a foundation for developing a standardized system of metrics for evaluating the operational performance of the aircrew training systems and comparing procurement values among competitive aircrew training system developers. Lower level B specifications would be developed as a joint effort by AMC/ASC/AETC and the aircrew training system developer to identify specific design and performance characteristics for a specific aircrew training system within AMC's total training system.

The "C" specification, or Product specification, would be a set of "build-to" requirements for the component parts that comprise an aircrew training system. This specification would identify the primary operational performance and design characteristics that each of the component parts (e.g. training management system, training support system center, etc.) of a aircrew training system must possess. In essence, this specification would identify the form, fit, and function of the aircrew training system's component parts. The "C" specification would be the outline used to standardize the structure of aircrew training systems in order to promote commonalties among aircrew training systems. These common elements would give the Air Force the ability to fully develop and utilize a data collection and reporting system for the total training system.

Additionally, this standardized structure for aircrew training systems would provide an avenue for equality assessments and fair competitive assessments among aircrew training system developers. The lower level "C" specifications would be developed by the aircrew training system developer (i.e., contractor), thus allowing competitiveness among aircrew training system developers to be embedded within their ability to competitively develop and/or operate the aircrew training system.

One of the advantages of a comprehensive set of integrated specifications would be its usability in either re-engineering an existing dysfunctional system, or ensuring the continuing integrity of a functional system when developing and integrating a new component part of the existing training system. Because the ISD/SE Concurrent Engineering approach provides a global view of what constitutes the "training system", the results of the process will be highly useable and applicable to most training systems' change and development efforts.

Although we cannot present a full set of specifications in this forum, we have identified some key features which we believe are critical to the optimal design of the "Total Training Concept" training systems.

The ATSs should have compatible training management systems. The training management systems provide student, training, and curriculum management functions which are key to the generation, collection, and reporting of systems performance data. The design of the training management systems should be such that the key performance data can be electronically transported to the follow-on training management system. Such transferable data should include top level performance metrics of the courseware, student throughput, student graduation performance, as well as, student biographical data and training histories. The system should report and status such systems data for individual students, the entire ATS, or comparatively across systems when the data is provided. This type of system would provide for the required curriculum, scheduling, and student management functions; it would also gather and report student records, courseware performance records, system performance records, and top level metrics required for informed management decisions.

The use of standardized data collected across all ATSs, combined with a set of uniform evaluation criteria and a common tool used to perform training system life cycle cost analysis and other cost/performance evaluations, will reduce the overall cost associated with managing AMC's training system operations and evaluating operational performance of the ATSs that comprise AMC's training system. Likewise, the standardized data, uniform evaluation criteria, and common analysis tool could be used by the operators of the ATSs to analyze continuing ATSs' operations, identify potential problems, and initiate corrective actions before problems become program detriments.

This approach would also give the government and prime contractors a standardized approach and a set of common metrics in order to establish an ATS bench mark for both performance and cost. This ATS bench mark could be used by both the government and ATS developers/operators to determine "what is" and "what is not" an efficient and effective training operation in terms of training and cost. This "bench marking" for ATS operations would also provide the government with baseline data needed for evaluating competitive bids between ATS developers and operators, while at the same time providing the ATS developers and operators a realistic "target" for compiling ATS bids. In addition, this approach could prevent the government from procuring systems that could not deliver as expected, and could assist the ATS developers/operators with establishing competitive bids embedded with realistic and achievable operational and cost goals.

By creating a more functional system that takes advantage of several cohesive and integrated parts, each contributing to increases in training effectiveness and cost efficiencies, the total life cycle cost of AMC's training system could be minimized and more easily managed.

The evaluation processes across ATSs should be standardized. This includes both the evaluation of the ATSs' performance, and the evaluation of trainees' performance. Student performance should be evaluated according to uniform standards in order to provide a common scale of interpretation across the range of student performance. Uniform evaluation also implies uniform data collection and reporting. Statistical data should be categorized, organized, and calculated in standard sets, so that each data set can be accurately interpreted within other aircrew training systems. The consistent design of the lower level data allows for the generation of higher level reports, or systems metrics. Such uniform metrics across training systems would allow adequate comparisons for basing government program decisions.

The "Total Training Concept" supports the execution of the traditional phases of Formative, Summative, and Operational Program Evaluations, as well as, the incorporation of systematic evaluations in the early stages of the engineering specification generation process. These evaluation activities are intended to be iterative in nature for the facilitation of continuous process improvements within the aircrew training systems program. Furthermore, systematic evaluations will aid in the identification of cost drivers. More specific or detailed information from the training systems results in more accurate information to minimize cost drivers. The standardized evaluation processes will function to identify variables, such as negative training and training deficiencies, that can influence or drive life cycle costs.

The training systems which result from the evolution and development of the shared vision will provide benefits to individual students, the involved training programs, the AMC customer, and to the supporting Air Force agencies.

Training systems must adhere to a standardized set of student learning criteria. In aircrew training, this is especially critical. Training materials and methodological procedures should be carefully configured and controlled so that students are afforded uniform learning opportunities for optimizing learning results. Levels of required student proficiencies must be clearly defined, documented, and communicated in order to ensure adequate (and consistent) student learning.

Standardized learning must translate into consistent student performance. Measurable student performance indicators must be defined, documented, and communicated. The standards for student performance should be clearly related to that performance which would be expected of fully trained aircrews. This is not to imply that student performance does not build and improve with exposure, practice, and subsequent skill building, but it does mean that the required performance must be representative of skill sets required for aircrew duties.

Like the requirement for consistent student performance standards, uniform graduate performance standards will ensure equitable training results. In the case of graduate performance, this requirement is even more critical. Training system graduates must be able to participate in required operational missions without detriment to the functioning of the aircrew duties. The "Total Training System" concept supports the requirement of mastery learning and performance. Due to the critical nature of AMC aircrew mission duties, any standard less than performance mastery should not be acceptable. Substandard graduate performance may place aircrew missions at serious risk.

The standardization of graduate performance requirements and standards ensures that training system graduates will possess the requisite knowledge, skills, and abilities when entering the follow-on training system. The correct match between student entry level skills and the training curriculum requirements facilitates program optimization by reducing excessive training time, excessive remediation, and excessive training systems utilization above that accommodated within the planned throughput.

Standardization of learning criteria, student performance standards, graduate performance standards, and student entry level knowledge and skills will ultimately function to reduce overall training system remediation requirements for systems within the "Total Training System" construct. Any reduction of the training remediation requirements will impact the demands placed upon the training system resources, resulting in reduced overall life cycle costs.

Because training systems will be designed and constructed with standard interface and other systems requirements, problem identification and resolution will be facilitated. Individual ATSs will be better able to utilize lessons learned from other ATSs within the system. Vigilant and systematic evaluations will indicate areas to monitor for possible problems. Root cause analysis procedures will be employed to identify the prime cause of the problem and to plan the most appropriate (effective and efficient) corrective action to be implemented. Routine re-evaluation will provide status feedback on the effectiveness of the corrective action. Because of the linked design and interface of the individual ATSs within the "Total Training System", the sharing of information, data, and problem solutions will improve. Individual training systems would be considered as "cooperative" units rather than as "competitors". Uniformity of basic design specifications will render more meaningful and applicable uses of lessons learned from training systems problem identification and resolution.

The centralized collection of standardized data measuring the attributes of each ATS, its output product (i.e., trained student), and how that output product was derived will allow AMC to compare the similarities and compatibilities within the output products of the ATSs. Likewise, this would enhance AMC's ability to perform comparisons of a trained student's academic knowledge and skills attributes to the academic knowledge and skills requirements for other aircrew system positions. These comparisons would enable AMC to effectively utilize previously obtained academic knowledge and skills to minimize the total cost of training for AMC aircrews. Enhancement of cross training, or multiple aircrew position capability of individual AMC personnel, would minimize the effects of defense down-sizing.

The standardization of data collection and reporting allows for valid cross systems comparisons of indicators concerning program effectiveness. The system data should require minimal manipulation for the generation of these specified program metrics. Training program effectiveness metrics should include student performance metrics, courseware performance metric, and overall systems performance metrics.

The standardization of data collection and reporting, combined with a set of analytical tools, will allow for the identification of component replications in the ATSs. With this data, the functionalities of one efficiently operating ATS could be utilized to meet the operational needs of multiple ATSs, thus reducing the total operational cost of AMC's training system. Additionally, the capability of cross system comparisons would allow the life cycle cost of one ATS to be accurately compared to the life cycle cost of all other ATSs. Variances, and the cause of the variances, in the life cycle cost of an ATS could be easily assessed by using data that is not only standardized, but is collected and reported within a standard set of criteria.

Anything that impacts the training program is a major impact to the training system's Life Cycle Cost (LCC). Because the training program is a sub-component of a training system, it is a major driver in the LCC of a training system. The use of standardized data, combined with a centralized data collection and reporting system, could provide the major source of data for LCC analysis. This data could then be used by a training system's LCC model to provide LCC assessments not only of the ATSs that comprise AMC's training system, but also for the overall LCC of AMC's total training system. Because the major focus of a training system's LCC analysis would be the training program, the same data needed for training program performance evaluations could be used to perform the LCC analysis efforts.

Accurate and specific data, collected by a centralized and standardized data collection system for the various ATS training programs, would help identify the specific impacts/influences that changes in ATS inputs and/or operations have on the training systems LCC. This detailed information would enable the identification of the real cost drivers in an ATS and to AMC's training system. For example, more student remediation results in more training device time requirements, and more maintenance and logistics requirements, etc.. Early identification of increases in remediation requirements, and the subsequent reduction of these remediation requirements, could be accomplished prior to a negative impact on the system's life cycle costs.

The training system's Summative and Operational Evaluation will aid in the identification of cost drivers. The use of standardized data for Summative and Operational Evaluations should be structured to gather data in a form which will be conducive to feeding the LCC model. Therefore, the Summative and Operational Evaluations would identify variables, such as negative training and training deficiencies, that can influence or drive life cycle costs.

The data fed into the LCC model must be as accurate as possible to avoid the "garbage in/garbage out" syndrome. Inputing inaccurate data would result in erroneous LCC predictions. The use of the "Total Training Concept's" standardized data collection and reporting system would provide the accurate data need to perform a valid LCC analysis. Furthermore, the LCC model/analysis must have accurate data that is not too labor intensive to gather or pre-process prior to use in the LCC analysis. The proposed "Total Training Concept's" structure of standardized data, with a centralized collection and reporting system, would provide the needed data in a cost efficient manner, using the "collect once, use multiple times" approach. In this approach, LCC data would be maintained in a single database; the same database would be used for other program assessments. This would avoid the current problem of using incompatible assessment systems or methodologies and, more importantly, minimize duplication of data, data collection, and input efforts (as well as, data storage), which also saves costs.

Blanchard, B.S., and Fabryck W. J. (1990). Systems engineering and analysis. Englewood Cliffs, New Jersey: Prentice Hall.

Burkman, E. (1987). Factors affecting utilization. In Instructional technology: Foundations. Ed. R.M. Gagne. Hillsdale, New Jersey: Lawrence Erlbaum Associates, Inc..

Defense Systems Management College. (1990). Systems engineering management guide. Washington, D.C.: U.S. Government Printing Office.

Dick, W., and Carey, L. (1985). The systematic design of instruction. Glenwood, Illinois: Scott, Foresman and Company.

Hammer, M., and Champy, J. (1993). Reengineering the corporation. New York: Harper-Collins Publishers.

Lacy, J.A. (1992). Systems engineering management: Achieving total quality. New York; McGraw-Hill, Inc..

Morgan, R.M. (1987). Planning for instructional systems. In Instructional technology: Foundations. Ed. R.M. Gagne. Hillsdale, New Jersey: Lawrence Erlbaum Associates, Inc..

Rogers, E.M. (1983). Diffusions of innovations. New York: The Free Press.

Rubin, S.E. (1994). Public schools should learn to ski: A systems approach to education. Milwaukee, Wisconsin; ASQC Quality Press.

Senge, P.M., Robert, C., Ross, R.B., and Smith, B.J. (1994). The fifth discipline fieldbook: Strategies and tools for building a learning organization. New York; Doubleday.

Siegel, P., and Byrne, S. (1994). Using quality to redesign school systems: The cutting edge of common sense. San Francisco: Jossey-Bass Publishers.

Smith, P.L., and Ragan, T.J. (1993). Instructional design. New York: Macmillan Publishing Company.