Terms of Reference (TOR)
for MORS Workshop:

Test & Evaluation, Modeling & Simulation, and VV&A: Quantifying the Relationship Between Testing and Simulation

Background

As the use of models continues to become more prevalent in system design, expectations are that they also will become a more central tool supporting system development and test and evaluation.  In many cases, models or simulations could be expected to become a primary evaluation tool for addressing Operational Requirement Documents (ORDs) and Critical Technical Parameters (CTPs), and for resolving Critical Operational Issues (COIs).

With this increased reliance on M&S in systems acquisition, VV&A of M&S becomes even more critical.  While there have been considerable advances in M&S technology, there has been limited comprehensive research and review of VV&A methodology.  Most VV&A is accomplished with Subject Matter Experts (SMEs) who review simulation predictions and test results, and provide a subjective opinion on the adequacy of the model.  There are few examples of an objective process or quantitative criterion that allows the decision maker to fully understand why the model is or isn’t good enough for the particular application.  Furthermore, in many instances, there is inadequate definition of the intended application of the model – a critical element in any accreditation.

While SMEs play an important role in the accreditation process, test and evaluation data resulting from the use of real equipment and systems should provide a major source of data for V&V of M&S.  Statistical tests comparing test results and model results, and perhaps even some overall measure of “goodness of fit”, should be able to provide the analyst with a means of quantifying the validity of model results and assessing the degree to which models can support weapon system evaluation.

An issue of concern to the analyst and decision-maker in using models to supplement field-testing is the lack of credibility of the model in predicting outcomes outside its validated domain space.  While testing all aspects of a model is not typically economically feasible, validating a model in the center of the envelope and using it to predict performance at the edge of the envelope is not practical or valid.

  1.  Objectives and Goals

This workshop has three interrelated, though distinct, objectives.  These are to explore guidance and develop recommendations related to the:

1. Experience of the M&S community in the use of test and evaluation (T&E) data to support the VV&A of M&S.

2. Experience of the T&E community in properly using verified, validated, and accredited M&S to support both Developmental and Operational T&E processes.

3. Experience of the T&E and M&S communities in Accreditation thought processes and implementation.

The overall goals of this workshop are to:

1.      Existing policies and procedures,

2.      Previous or known barriers to implementation

3.      Lessons learned in these two areas.

The meeting will highlight potential ways of:

  2.  Approach

The proposed MORS Workshop, “Test & Evaluation, Modeling & Simulation, and VV&A: Quantifying the Relationship between Testing and Simulation,” will be organized by three general topic areas: T&E for M&S V&V, M&S Support to T&E, and Accreditation Processes.

Workshop Co-Chairs:  Dr. Marion Williams, Ms. Annie Patenaude

Opening remarks:
M&S expectations and the challenge of VV&A:  Mr. Walt Hollis, accepted

Perspectives:  System Acquisition:  TBD
Army OT&E:  Mr. Brian Barr, accepted
Navy OT&E:  Mr. Steve Whitehead, accepted
Air Force OT&E:  Dr. Marion Williams, accepted
DOT&E:  Dr. Ernest Seglie, accepted
MDA : Dr. Patricia Sanders, accepted
JTE/DT: Mr. Richard Lockhart

Working Groups: To support the workshop goals and objectives, case studies will be sought in model applications at all levels.  The working groups will attend briefings of the case studies and focus in three areas:

The case studies will be solicited from those using various types of models and analysis to support the Test and Evaluation.

Synthesis Group Co-Chairs: Priscilla Glasow, Chris Fossett

Publications Bulldog:  Lana McGlynn

3.  Working Group Chairs and Co-Chairs.  The chairs and co-chairs of the working groups, along with their contact information, are identified in Table 1 below.

4.  Meeting Location. 

The workshop will be held in Albuquerque, NM, 15-17 October 2002 at Kirtland AFB.  The MORS Hotel will be the Radisson Hotel & Conference Center, 505-888-3311. 

5.  Agenda 

Table 2 highlights the top-level schedule for the three-day workshop.

6.  Attendees

The working group chairs will be key OTA representatives from each of the Services and Agencies.  Co-chairs will be MORS members from T&E and analysis communities.  Invitees will be from DoD, Industry, and Academia with interest in T&E M&S use.

7.      Products

In addition to the workshop products of PHALANX article and briefs to MORS Sponsors, the goals of this workshop support bringing a credible synthesis of recommendations to the practicing community.  The products from the working groups will be synthesized into out-briefs to the T&E Leads and Commanders in the Services and Agencies, with guidance and recommendations for T&E VV&A methodologies. These products will provide a draft set of Best Practices and identify existing gaps in VV&A and T&E policies, procedures and practices that would be fruitful areas for future investment.

Additionally, the Working Group Co-Chairs will present a synopsis of the workshop, its recommendations, and resulting actions during a special session of the MORS Symposium in June 2003.

Organizing Committee