Working Group 22: Experimentation


(Updated 21 October 14)

Purpose:  The Experimentation Working Group (WG) 22 provides an opportunity for military, government and civilian operations research analysts to examine topics, methodologies, analyses, and innovations pertinent to all aspects of designing, planning, executing, analyzing and reporting the results of experimentation in the Department of Defense (DOD) and other government departments and agencies such as the Department of Homeland Security (DHS) as it applies to military operations research issues and techniques.

Discussion:  Within the last ten years, each of the military Services, several combatant commands, and other government agencies like DHS have chartered organizations within their respective domains to design, plan, execute, analyze and report the results of experiments that effect changes across the spectrum of Doctrine, Organization, Training, Materiel, Leadership, Personnel and Facilities (DOTMLPF).  Each of these organizations has been faced with challenges to effectively implement an experimentation program that is responsive to the needs of decision makers.  As a result, these groups have been independently developing their own unique approaches to experimentation.

The Experimentation Working Group brings together Service, government, academic, and industry experiment practitioners to share tools, methodologies, and lessons learned that can improve experimentation strategies and methods.  The following are example issues:

·      Designing credible experiments with limited resources

·      Developing coherent strategies for campaigns of experimentation

·      Developing meaningful Measures of Merit (MOMs)/Measures of Effectiveness

·      Accounting for small sample sizes

·      Conducting experiments in training exercises or field tests

·      Addressing challenges with participants

·      Integrating modeling into experiments

·      Reconciling data collection and player participation

·      Analyzing results in a timely manner

·      Addressing continually evolving experimental objectives

·      Effectively sharing results and lessons learned

·      Transitioning results into tangible action

·      Assessing ROI

·      Analyzing qualitative data with rigor


The Experimentation Working Group will solicit input and discussion from the analysts who are involved in designing, planning, executing, analyzing, and reporting the results of experiments.  Additionally, in order to understand and improve the entire process, participants throughout the experimentation community will be included when appropriate.  This broader community includes the decision makers who develop and approve experiment programs and budgets and who use experimentation results to improve their decisions; military Services and government staff agencies who are customers of experimentation results; members of the acquisition community who participate with experimental hardware or software; combat doctrine developers who provide experimental concepts and scenarios; military Service operational testers and modelers who use experiment data; military service and combatant command trainers who provide field exercises and train the participants for field experiments; and military Service operational forces (i.e., “warfighters”) who support experimentation as participants and benefit from the results of effective experimentation.

The Experimentation Working Group will consider all venues for experimentation: field experiments, constructive simulation experiments, human-in-the-loop virtual simulation experiments, analytically assisted tabletop experiments, and wargaming.

Other Working Groups:  Candidates for joint discussions with WG 22 include, but are not limited to:

·      Homeland Security, Homeland Defense and Civil Support (WG 4)

·      Air and Missile Defense (WG 9)

·      Joint Campaign Analysis (WG 10)

·      Land and Expeditionary Warfare (WG 11)

·      Maritime Operations (WG 12)

·      Strike Warfare and Power Projection (WG 13)

·      Air Warfare (WG 14)

·      Analytic Support to Training (WG 20)

·      Modeling and Simulation (WG 28)

·      Computational Advances in OR (WG 29)

·      Special Operations and Irregular Warfare (WG 32)

·      Social Science Methods and Applications (WG 33)