|
| |
Improving Defense Analysis Through Better Data Practices - A
Workshop
Terms of Reference (Draft 12 March 2003)
1. Background
Effective quantitative analysis depends on the
availability of credible data. Each
of the Services, the Joint Staff, and the many organizations within the Office
of the Secretary of Defense have established a variety of procedures to gather,
generate, maintain, transform and use military data to support an array of
analytic activity, from budget programming, to strategic planning, to wargaming.
In fact, the collection and application of credible data is a major part
of every analytic effort and a key contributor to the value of every final
product. In order to better support strategic analyses conducted by
the Department, Defense Planning Guidance,
Fiscal Years 2004-2009, May 2002, tasked the Director of Program Analysis
and Evaluation to establish policy and assign responsibility for generating,
collecting, developing, maintaining, and disseminating data associated with
current and future U.S. and non-U.S. forces.
Existing directives and pending instructions provide general policy for
accomplishing this task. To manage,
gather, create and apply analytic data, the Services and defense organizations
currently employ a wide variety of methodologies, many resulting from several
years of evolution. At the same
time, commercial enterprises and the academic community have generated many new
alternative methodologies for handling data within their disciplines to include
new technologies that could be applicable to the military. This
paper describes a Military Operations Research Society special meeting to review
current data processes used by DoD to support analysis, identify best data
practices throughout the DoD and broader community, and recommend, where
appropriate, best practices that could by implemented to improve the analytic
data processes used by the Department.
2. Discussion
Recent MORS special meetings have addressed a myriad of analytic
focus areas such as weapons of mass destruction and urban operations; new
analytic tools, to include the new sciences; and, in the near future, the
shaping of analytic personnel to support the needs of the combatant commanders.
Key to each of these areas and noted in most special meeting final
reports is the importance of good data to the usefulness of the supporting
analysis and the credibility of the final products.
Directives within the Department provide general guidelines for data
verification and validation (data V&V) to support the range of analyses
conducted. These directives do not
include guidance on specific methodologies, techniques, tools, procedures or the
use of metadata that would help ensure data is of the appropriate quality and
used in the appropriate ways to best contribute to the credibility of the final
product. In fact, some study
results are undermined by the lack of credible data to support the analysis,
while others are marred by the misapplication of credible data to models,
scenarios, and methodologies that were outside the scope the data was designed
to support. In fact, concerns
regarding the collection and application of warfighting databases have often
served to impede the production of credible joint warfighting analyses. For example, program offices carefully develop system
lethality and vulnerability results to address specific threats in predetermined
contexts in order to produce estimates of military utility associated with the
fielded system. These data may or
may not be appropriate for use in joint warfighting analyses addressing the
effectiveness of an array of systems operating jointly against an array of
threats that were outside the scope of the original data development process.
While the problems may not be due to any intent by the analysts involved
to skew the estimated effectiveness results, misapplications and
misrepresentations still occur. As a result, concerns about the use of the data have led to
barriers within the Department to developing balanced, objective and credible
databases and analyses.
These issues are not unique to DoD or to the various Service
elements. Non-DoD government
agencies (such as the Bureau of Labor Standards), commercial business
organizations, other governmental and inter-governmental organizations (to
include the World Bank) and the academic community in general deal with similar
issues across all elements of enterprise activity.
These groups, as well as many agencies within the military, have
developed specific methodologies to assist in the production of credible
databases in order to limit unintended biases and misapplication of data within
general warfare analysis areas or across other applications.
MORS is in a unique position to help address these data production,
management, and application issues by inviting analysts, policy makers, database
managers and experts in the field to review existing procedures and then to
identify and recommend best data practices for military and joint warfighting
analysis to the Department. These
best practices can then be used as a standard to guide more specific policies
within the Department of Defense, particularly with respect to supporting
strategic analyses, as well as to help individual units gather and maintain
credible data for specific purposes. Such a special meeting will support current plans for the
Under Secretary of Defense for Policy to coordinate the development and/or
updating of DoD scenarios, as well as for other activities within the Department
to ensure the availability of credible data for a range of military activities.
3. Goals and Objectives
The purpose of this workshop would be to identify best practices for
generating, collecting, developing, maintaining, disseminating and applying data
and metadata to help support better analyses as well as to help establish
improved data policy and procedures within the Department of Defense.
In particular, the workshop will examine current practices by
organizations both inside the Department and throughout the inter-agency,
international, commercial and academic worlds to determine those that
incorporate best data practices. In addition, the workshop would assess, to the extent
practical, new technologies to assist in the management and application of data.
As a part of this activity, the workshop would address potential tools,
techniques, procedures and technologies for identifying, collecting, verifying,
managing, and disseminating data and analytical baselines within the Department.
Specific best practices will be identified regarding:
 |
What data content would best support the development of
analytic baselines (defined by DODD 8260.1 as a package comprising a
scenario, concept of operations, and integrated data used by DoD Components
as a base case for strategic analyses) to include methods of collecting and
storing data, metadata, assumptions, scenarios contexts and other pertinent
information?
|
|
What cost effective methods and approaches best support
verification and validation checks on data received, to include associated
metadata in data repositories?
|
|
What techniques are available to ensure data is appropriate
for its intended uses be they extensions of existing applications or new
applications?
|
|
What effective and efficient methods are available for
preparing data sets and associated metadata for use in specific studies and
analyses to conclude the configuration of related data sets, delivering data
according to criteria established by oversight bodies and archiving data and
other study products for later reporting and application?
|
| What guidelines and methodologies are available for
prioritizing data development, storage, management, and dissemination
activities?
|
In addition, to the extent possible, the workshop will
identify organizations and activities within the Department that are perceived
to be authoritative data sources as well as identify characteristics that help
define an authoritative data source, including the internal practices processes
and procedures that help ensure data credibility.
3.1 Goals
This workshop will afford the military OR community an opportunity to
achieve the following goals:
(1) Establish a vision for improving the generation and
management of credible data to support the analytic activities of the
Department;
(2) Identify specific best practices that could be adopted by
the Department, the Joint Staff, the Services, and their supporting agencies to
realize this vision;
(3) Define how MORS can work with the sponsors
over time to implement improvements to data production and management.
3.2 Objectives
The objectives of the workshop are to:
| Review current DoD processes for developing, managing, using,
and sharing data in support of military analyses to include defining common
terms
|
| Identify best practices and standards within DoD and the
broader analytic community for generating, collecting, developing,
maintaining, and disseminating data; suggest how applicable best practices
could be implemented in DoD
 | Review expected impact of technology on practices and
processes
|
| Note issues and problems that serve as barriers to
implementing best practices
|
|
| Identify and assess potential sources of data needed for a
wide variety of analyses
| Provide a current listing of authoritative data sources
within the Department
|
|
| Educate the community on the new Data Directive, to include
plans for implementation (data repository, access to the data, etc.)
|
4. Approach
In order to achieve these goals and objectives, the subjects of data
management, data generation, data creation, and data transformation and
application will be examined thoroughly. General
presentations by Department and Service experts regarding current practices and
desired best practices will be followed by descriptions of methodologies that
have been developed and implemented by other government agencies, international
and/or allied organizations, industry, and academia.
Working groups will address data practices within specific activity and
contextual frameworks. Working groups will be asked to review existing practices
in their area within DoD as a starting point and then will discuss relevant
practices by non-DoD organizations as well as techniques and approaches
addressed in the literature. By
developing a vision of the desired outcome in each area, the working groups will
then identify and recommend efficient and effective practices in their areas for
consideration by the Department. The
working group focus areas, selected issues, and desired results are detailed
below.
4.1 Working Groups
Following general presentations, the workshop attendees will be
organized into five subject area working groups plus a synthesis working group.
Depending on the numbers of the participants, additional groups can be
generated to address issues in more depth with different individuals in order to
provide a broader range of best practice options for the Department.
The focus will be on identifying best data practices as well as barriers
and procedures that inhibit the ability to manage, gather, create, share,
transform and apply credible data as well as on technologies developed to
support these activities. Specific
working groups will be as follows:
| Data Management
|
| Obtaining Data From Outside Sources
|
| Creating Data Internally
|
| Transforming and Employing Data
|
| Data Technology Support
|
| Synthesis
|
4.1.1 Data Management Working Group
This working group will review current DoD data management practices
and investigate data management approaches by other pertinent government
organizations, such as the Bureau of Labor Statistics, as well as industry and
international organizations to determine their applicability to managing data
for military analysis. The group
will identify best community practices of data management, to include enterprise
business rules, models, and structures to facilitate effective data management
and data sharing. In addition, the
group will review storage; access; pedigree; validation and verification
(V&V) requirements; currency; tools; preservation of data context, and the
use of meta data. The group
will suggest ways to implement best practices if appropriate.
In addition, the group will address the handling of critical technical
and compartmentalized data, and suggest generic ways for addressing
classification issues in support of joint analytic processes.
The group will produce a summary briefing report identifying specific
best practices for data management by the Department and its organizations and
agencies.
4.1.2 Obtaining Data from Outside
Sources Working Group
This “Data Demand” group will review current data gathering and
data sharing processes within DoD as well as “best practice” data gathering
and data sharing approaches in the larger community. Such practices could include composable frameworks and
supporting procedures; tools, utilities and supporting standards; critical mass
of end-use content and supporting access; and life-cycle sustainment and
supporting elements. The group will
address red and blue data issues such as release, responsibilities and
liabilities and suggest ways to improve the process.
To the extent possible, the group will identify currently perceived
authoritative sources for general data types. The group will also identify
databases that are vital to support the current OSD Analytic Agenda, but that
are either of poor quality or not available at all.
To the extent possible, the group will provide possible solutions that
could remedy this situation in a timely fashion. The group will produce a
summary briefing report identifying best data gathering practices, along with
current impediments to effective data collection and recommended approaches to
facilitate the adoption of best practices by Department organizations and
agencies.
4.1.3 Creating Data Internally
Working Group
This “Data Provider” working group will review current Department
and community practices for data creation, to include the expected attributes of
an authoritative data source. It
will identify the data content associated with various types of data produced to
support joint studies, experiments and wargames, list potential sources for this
data, understand and evaluate how data is developed by various sources, and
identify best practices in this arena. The group will also address the implications of the
Intelligence community’s transition to Matlab/Simulink representations for
threats. Finally, the group will
suggest ways to incorporate best practices in the data creation process, as
appropriate. Particular attention
will be paid to specific techniques that could facilitate sharing of effective
approaches to data creation across defense organizations.
These will include cost- and analytic resource-effective methods and
procedures to ensure the appropriate application of one Service’s or
Organization’s or Program Office’s data to issues and studies conducted by
other defense organization. The
group will produce a summary briefing report identifying data content, potential
sources, evaluation of data creation, and the identified best practices, along
with current impediments to credible data creation and recommended approaches to
facilitate the adoption of best practices by Department organizations and
agencies.
4.1.4 Transforming and Employing
Data Working Group
This “Data Application” working group will review current DoD and
best practices of data application, analysis and sharing to include
appropriate use of data and maintaining data integrity when aggregating,
disaggregating and transforming data for different applications.
The group will suggest ways to implement best practices if appropriate.
In its investigation, the group will review best practices from other
government agencies, pertinent international or allied organizations, industry
and DoD business applications. The
examination will include discussion of how data is currently applied to
department strategic analyses and what methods are available to assure the best
application of that data to both traditional and new analytic problem domains.
In particular, this group will identify specific tools and procedures to
help ensure the appropriate application of existing data or the appropriate
transformation of data from existing databases to best support issues
confronting the joint community. The group will produce a summary briefing
report describing the data transformation and application methods and approaches
explored, the best practice applications that may be addressed with known
techniques, and those applications that would be better served by the
development of new techniques. The
briefing report will also list current impediments to credible data application
and suggest recommended approaches to facilitate the adoption of best practices
within the Department.
4.1.5 Data Technology Support
Working Group
This working group will address state of the art technology to
support the needs of the other data enterprise areas of management, collection,
creation, transformation and employment. The group will identify and assess the impact and appropriate
use of new technologies in areas such as data composition and data mining.
The group should pay particular attention to methodologies used to track
databases and database changes to ensure traceability throughout the analytic
process as well as cost-effective methods to ensure reasonable levels of data
verification and validation are maintained. Discussions will also include data
engineering and data interchange standards.
In addition to addressing current DoD practices, the group will identify
best practices used by industry, other government agencies, and international
organizations. The group will produce a summary report identifying best
practice data support technologies, along with current impediments to
implementation of these technologies and recommended approaches to facilitate
their adoption by Department
organizations and agencies.
4.1.6 Synthesis Working Group
This working group will take a broad view,
identifying the high level issues across the domains of data generation,
management, and application that enable or impede the successful evolution of
joint and warfighting and strategic analysis.
The group will provide the integrating perspective to help highlight best
data practices, particularly in those areas where techniques that reach across
all areas will best meet the long-range requirements of the Department.
This working group will provide the integrating perspective.
4.2 Working Group Tasking
Working groups will be directed to address the following questions
and concerns:
|
How do the Department and its Services and organizations
currently perform the functions of the specific area that it is addressing
(data management, collection, creation, or application)?
|
|
How are these functions performed by other enterprises in
other government departments, commercial enterprises, international
organizations or by the scientific and academic communities?
|
|
What features of existing or proposed alternative methods,
approaches and techniques appear to be best suited for adoption and
implementation within the Department?
|
Working groups will identify practices that help or hinder
operations within their focus area and then will define the key best practices
that they feel would have the greatest impact on improving data management, data
gathering, data creation, data transformation, data application and sharing
within the Department. Each “best
practice” will be defined in sufficient detail to help potential DoD users to
understand how the method could be implemented and what metrics could be used to
measure progress toward full implementation at both the organizational and
department level. Time permitting,
a proposed plan for implementing the most important practices will also be
developed along with the likely impact of current or emerging technologies on
best data practices in the previously described areas.
5. Sequence of Events
A warm-up session for working group co-chairs will be held the
evening before the first day of the workshop. This session will be held at IDA. The purpose of this warm-up session is to review and discuss
tasking for the working groups.
The first day will be devoted to a plenary session.
The Workshop Co-Chairs will present the goals, objectives, and
organization for the workshop. A MORS Sponsor will present a detailed charge to the workshop
participants. Session speakers will
provide alternative perspectives on the current state of data generation,
management, application and sharing for joint warfighting analysis within the
Department; as well as best data practices from other government and commercial
enterprises and from the scientific/academic world.
There will be a social event (mixer) at the end of the first
day.
All of the second day will be devoted to working group
sessions and discussions. The
morning of the third day will be devoted to the preparation of working group
presentations. The afternoon of the
third day will be devoted to working group presentations in plenary session.
The morning of the fourth day will be made available to
working group co-chairs for final editing of the working group annotated
briefings.
6.
Agenda
|
Day/Time
|
Activity
|
POC
|
|
Monday
|
March 24, 2003
|
|
|
1700
|
Working Group Co-Chair Warm-Up Session
|
Workshop Co-Chairs
|
|
|
|
|
|
Tuesday
|
March 25, 2003
|
|
|
0700
|
Registration and Continental Breakfast
|
MORS Office
|
|
0800
|
MORS President's Welcome
|
Ted Smyth
|
|
0805
|
Facility Host Welcome
|
Phil Major
|
|
0810
|
Sponsor’s Welcome
|
Eric Coulter
|
|
0815
|
Chair's Welcome, Workshop Overview
|
Tom Allen
|
|
0830
|
Keynote, DoD Data Enterprise
|
John Osterholz
|
|
0910
|
Keynote, DoD Analytic Agenda
|
Chris Lamb
|
|
0950
|
Break
|
|
|
1000
|
OSD Enterprise
|
Tony Simon
|
|
1030
|
DoD Data Directive & Analytic Agenda Implementation
|
Jim Stevens
|
|
1100
|
Service/JS Perspectives
|
Senior Analyst Panel Discussion
|
|
1230
|
Buffet Lunch
|
|
|
1330
|
Industry
Standards for Data Generation
|
Invited Speaker, Industry
|
|
1415
|
Non-DoD Data Management
|
Kurt
Schnebele, NOAA
|
|
1500
|
Break
|
|
|
1515
|
Best Practices in Data Applications/Sharing
|
Invited Speaker
|
|
1600
|
Best Practices in Data
|
Invited Speaker
|
1645
|
Working Group Kick-Off Meetings
|
Working Group Chairs
|
|
1730
|
Mixer
|
MORS Office
|
|
|
|
|
Wednesday
|
March 26, 2003
|
|
|
0715
|
Continental Breakfast
|
MORS Office
|
|
0800
|
Working Group Session #1
|
Working Group Co-Chairs
|
|
0945
|
Break
|
|
|
1000
|
Working Group Session #2
|
Working Group Co-Chairs
|
|
|
|
|
|
1130-1300
|
Staggered Lunch, IDA Cafeteria:
|
MORS Office
|
|
1300
|
Working Group Session #3
|
Working Group Co-Chairs
|
|
1500
|
Break
|
MORS Office
|
|
1515
|
Working Group Session #4
|
Working Group Co-Chairs
|
|
1700
|
Working Group Chair & Co-Chairs Hotwash
|
Workshop Chair
|
|
|
|
|
|
Thursday
|
March 27, 2003
|
|
|
0715
|
Continental Breakfast
|
MORS Office
|
|
0800
|
Working Groups Session #5
|
Working Group Co-Chairs
|
|
0945
|
Break
|
MORS Office
|
|
1000
|
Working Groups Session #6 (Prepare Briefing)
|
Working Group Co-Chairs
|
|
1200-1330
|
Staggered Lunch, IDA Cafeteria
|
MORS Office
|
|
1330
|
Working Groups: Present Briefings, WG 1, 2, 3
|
Working Group Co-Chairs
|
|
1500
|
Break
|
MORS Office
|
|
1515
|
Working Groups: Present Briefings, WG 4, 5 and Synthesis
Group
|
Working Group Co-Chairs
|
|
1645
|
Workshop Wrap-Up
|
Workshop Chair
|
|
1700
|
Adjourn
|
Workshop Chair
|
|
|
|
|
|
Friday
|
March 28, 2003
|
|
|
0800
|
Working Group Co-Chairs complete Working Group Annotated
Briefing
|
Working Group Co-Chairs
|
|
1200
|
Adjourn Workshop
|
Workshop Chair
|
7. Attendees
Attendance will be controlled via invitation. Attendees will include
invited experts from OSD, all Services, the Joint Staff, Federally Funded
Research and Development Centers, operational commanders, DoD contractors, and
data base developers and managers from other government departments, commercial
firms and academia. Workshop chairs will control membership of their sessions in
conjunction with the Organizing Committee. Attendance will be limited to 150
people.
8. Products
There will be six specific products generated as a result of this
workshop:
|
A collection of common terms and definitions associated with
data management, application and generation available for use by the
military analysis community. An
attempt will be made to generate a draft of this document prior to the
workshop, but it will be updated and finalized as a part of the workshop
process.
|
|
An integrated list of data best practices in the areas
addressed.
|
|
An Executive Summary of the workshop in the form of a text
document and a scripted briefing for the MORS Sponsors addressing the
workshop objectives, findings, conclusions, and recommendations.
|
|
A proceedings document containing summaries of all sessions
and copies of appropriate briefing slides and presentations.
|
|
A general session presentation for the 71st MORSS.
|
|
A PHALANX Article.
|
9. Planning and Organizing Chairs and Committees
|
General Chair
|
Tom Allen
|
|
Co-Chair
|
Jim Bexfield, FS
|
|
Plenary Facilitator
|
Scott Simpkins
|
|
Data Management WG
|
Simone Youngblood, Jim Stevens
|
|
Data Gathering WG
|
Jack Sheehan, Phillipe Loustaunau
|
|
Data Creation WG
|
Sam Fragapane, Bryan Paris, Dennis Leedom
|
|
Data Employment WG
|
Clay Bowen, Ernie Boehner, Rudy Pabon
|
|
Data Technology WG
|
Bob Might, Ron Smits, Jim Richardson
|
|
Synthesis WG
|
Stu Starr, FS, Roy Reiss, Kirk Yost
|
|
Admin Coordination
|
Brian Engler, Natalie Kelly
|
|
MORS Bulldog
|
Bill Dunn
|
|
OSD Rep
|
Jim Bexfield, FS
|
|
Joint Staff Rep
|
Bob Orlov
|
|
Army Rep
|
Paul Deitz
|
|
Navy Rep
|
Herb Cupo
|
|
Air Force Rep
|
Roy Reiss
|
|
Marine Corps Rep
|
Phil Exner
|
|
DMSO
|
Simone Youngblood
|
|
MORS
|
Lee Dick
|
|
Other
|
|
10. Administration
Name:
Ms. Natalie Kelly, MORS, 1703 N. Beauregard St, Suite 450, Alexandria, VA
22311
Dates: 25-27 March 2003
Location: Institute
for Defense Analyses, Alexandria, VA
Fee: $210 Federal government; $420
all others
Attendance: 120 - 150
Dear
Colleague Letter
Read
Aheads
Directions to Institute for Defense Analyses
|
