Working Group 33 Charter
Analytical Rigor in Experimentation

(Last Updated 20 August 2007)

PURPOSE: The purpose of the Analytical Rigor in Experimentation Working Group is to provide an opportunity for military, government, and civilian operations research analysts to examine topics, methodologies, analyses, and innovations pertinent to all aspects of designing, executing, and reporting of Experimentation in the Department of Defense (DoD) and other government departments and agencies such as the Department of Homeland Security (DHS).

DISCUSSION: Within the last eight years, each of the Services along with the US Joint Forces Command in conjunction with Government and civilian agencies have chartered organizations within their respective domains to plan, execute, and report experiments to effect changes in doctrine, training, and technologies. During this time, each of these organizations has been struggling with experimentation implementation issues and, as a result, has been building their own repertoire and standards in experimentation improvements and innovations based on trial and error.

The Analytical Rigor in Experimentation Working Group will bring together Service, Government, academic, and industry experiment practitioners to share lessons learned in order to improve experimentation strategies and methods. Some recurring experimentation issues to be addressed are the following: designing credible experiments within limited resources, developing coherent strategies for multiple successive experiments, developing meaningful Measures of Merit (MOMs), use of small sample sizes, use of inexact technology surrogates, overlaying experiments on training exercises or field tests, difficulties in training participants, problems with integrating simulation into experiments, impact of data collection on player participation, timeliness of analysis, impact of continually evolving experimental objectives, and implementation and sharing of results/lessons learned.

The Analytical Rigor in Experimentation Working Group will solicit input and discussion from the analysts who are involved in the design, execution, and analyses of experiments. Additionally, in order to understand and improve the entire process, participants throughout the entire experimentation community will be included. This community includes the organization leaders who develop and approve experiment programs and budgets, Service and Government staff agencies who are customers of experimentation results, members of the acquisition community who participate with experimental hardware or software, combat developers who provide experimental concepts and scenarios, Service operational testers and modelers who use experiment data, Service and Combatant Commander trainers who provide field exercises and train the participants for field experiments, and service operational forces who support experimentation as participants.

The Analytical Rigor in Experimentation Working Group will consider all venues for experimentation: field experiments, constructive modeling experiments, human-in-the-loop virtual simulation experiments, analytically assisted table-top experiments, and wargaming.

COMMUNITY OF PRACTICE : The Analytical Rigor in Experimentation Working Group will be the sponsor for the MORS Experimentation Community of Practice (COP), a group that continues to discuss experimentation issues and looks for synergy and consistency across government, industry, and academia. The Experimentation COP members meet face-to-face annually during the MORS Symposium.

OTHER WORKING GROUPS: The Analytical Rigor in Experimentation Working Group is a member of Composite Group F (Advances in Military Operations Research). Joint sessions are encouraged with working groups within CG F: WG-29 – M&S and Wargaming WG-31 – Computing Advances in Military OR WG-32 – Warfighter Performance & Social Science Methods. Additionally, Joint sessions are possible with working groups in other Composite Groups: CG A, Strategic and Defense – WG 5 (Homeland Defense & Civil Support); CG C, Joint Warfare – all Working Groups; CG D, Resources/Readiness/Training – WG 22 (Analytic Support to Training); and CG E, Acquisition – WG 24 (MOMs) and WG 25 (T&E).