https://ntrs.nasa.gov/search.jsp?

R=20140003093 2020-08-03T17:03:18+00:00Z

NASA/CR–2014-218178

CFD Vision 2030 Study: A Path to

Revolutionary Computational Aerosciences
Jeffrey Slotnick and Abdollah Khodadoust
Boeing Research & Technology, Huntington Beach, California

Juan Alonso
Stanford University, Stanford, California

David Darmofal
Massachusetts Institute of Technology, Cambridge, Massachusetts

William Gropp
National Center for Supercomputing Applications, Urbana, Illinois

Elizabeth Lurie
Pratt & Whitney, United Technologies Corporation, East Hartford, Connecticut

Dimitri Mavriplis
University of Wyoming, Laramie, Wyoming

March 2014
 NASA STI Program . . . in Profile

Since its founding, NASA has been dedicated to the  CONFERENCE PUBLICATION.
advancement of aeronautics and space science. The Collected papers from scientific and
NASA scientific and technical information (STI) technical conferences, symposia, seminars,
program plays a key part in helping NASA maintain or other meetings sponsored or co-
this important role. sponsored by NASA.

The NASA STI program operates under the  SPECIAL PUBLICATION. Scientific,
auspices of the Agency Chief Information Officer. technical, or historical information from
It collects, organizes, provides for archiving, and NASA programs, projects, and missions,
disseminates NASA’s STI. The NASA STI often concerned with subjects having
program provides access to the NASA Aeronautics substantial public interest.
and Space Database and its public interface, the
NASA Technical Report Server, thus providing one  TECHNICAL TRANSLATION.
of the largest collections of aeronautical and space English-language translations of foreign
science STI in the world. Results are published in scientific and technical material pertinent to
both non-NASA channels and by NASA in the NASA’s mission.
NASA STI Report Series, which includes the
following report types: Specialized services also include organizing
and publishing research results, distributing
specialized research announcements and feeds,
 TECHNICAL PUBLICATION. Reports of providing information desk and personal search
completed research or a major significant phase support, and enabling data exchange services.
of research that present the results of NASA
Programs and include extensive data or For more information about the NASA STI
theoretical analysis. Includes compilations of program, see the following:
significant scientific and technical data and
information deemed to be of continuing  Access the NASA STI program home page
reference value. NASA counterpart of peer- at http://www.sti.nasa.gov
reviewed formal professional papers, but
having less stringent limitations on manuscript  E-mail your question to [email protected]
length and extent of graphic presentations.
 Fax your question to the NASA STI
 TECHNICAL MEMORANDUM. Scientific Information Desk at 443-757-5803
and technical findings that are preliminary or of
specialized interest, e.g., quick release reports,  Phone the NASA STI Information Desk at
working papers, and bibliographies that contain 443-757-5802
minimal annotation. Does not contain extensive
analysis.  Write to:
STI Information Desk
 CONTRACTOR REPORT. Scientific and NASA Center for AeroSpace Information
technical findings by NASA-sponsored 7115 Standard Drive
contractors and grantees. Hanover, MD 21076-1320
NASA/CR–2014-218178

CFD Vision 2030 Study: A Path to

Revolutionary Computational Aerosciences

Jeffrey Slotnick and Abdollah Khodadoust

Boeing Research & Technology, Huntington Beach, California

Juan Alonso
Stanford University, Stanford, California

David Darmofal
Massachusetts Institute of Technology, Cambridge, Massachusetts

William Gropp
National Center for Supercomputing Applications, Urbana, Illinois

Elizabeth Lurie
Pratt & Whitney, United Technologies Corporation, East Hartford, Connecticut

Dimitri Mavriplis
University of Wyoming, Laramie, Wyoming

National Aeronautics and

Space Administration

Langley Research Center Prepared for Langley Research Center

Hampton, Virginia 23681-2199 under Contract NNL08AA16B

March 2014
The use of trademarks or names of manufacturers in this report is for accurate reporting and does not constitute an
official endorsement, either expressed or implied, of such products or manufacturers by the National Aeronautics
and Space Administration.

Available from:

NASA Center for AeroSpace Information

7115 Standard Drive
Hanover, MD 21076-1320
443-757-5802
 Table of Contents

1  EXECUTIVE SUMMARY .............................................................................................................................. 1 

2  INTRODUCTION ............................................................................................................................................ 4 
3  VISION OF CFD IN 2030 ................................................................................................................................ 5 
4  CURRENT STATE .......................................................................................................................................... 6 
5  CFD TECHNOLOGY GAPS AND IMPEDIMENTS................................................................................... 9 
5.1  Effective Utilization of High-Performance Computing (HPC) ..................................................................9 
CASE STUDY 1: Current Utilization of HPC at NASA .........................................................................10 
5.2  Unsteady Turbulent Flow Simulations Including Transition and Separation...........................................12 
CASE STUDY 2: LES Cost Estimates and 2030 Outlook ......................................................................13 
5.3  Autonomous and Reliable CFD Simulation .............................................................................................15 
CASE STUDY 3: Scalable Solver Development.....................................................................................16 
5.4  Knowledge Extraction and Visualization .................................................................................................18 
5.5  Multidisciplinary/Multiphysics Simulations and Frameworks .................................................................18 
6  TECHNOLOGY DEVELOPMENT PLAN ................................................................................................. 20 
6.1  Grand Challenge Problems .......................................................................................................................20 
CASE STUDY 4: Impact of CFD Tool Development on NASA Science and Space Exploration
Missions ...................................................................................................................................................21 
6.2  Technology Roadmap ...............................................................................................................................22 
7  RECOMMENDATIONS ............................................................................................................................... 27 
7.1  Development of a Comprehensive Revolutionary Computational Aerosciences Program ......................28 
7.2  Programmatic Considerations...................................................................................................................30 
CASE STUDY 5: Community Verification and Validation Resources .................................................32 
7.3  Strategic Considerations ...........................................................................................................................33 
CASE STUDY 6: Sponsored Research Institutes ....................................................................................34 
8  CONCLUSIONS ............................................................................................................................................. 37 
9  ACKNOWLEDGMENTS .............................................................................................................................. 37 
10  REFERENCES ............................................................................................................................................... 37 
APPENDIX A. HIGH PERFORMANCE COMPUTING TRENDS AND FORECAST FOR 2030 ............. 45 

i
 List of Figures

Figure 1. Technology Development Roadmap ………………………………………………………………… .. 23

Figure 2. Proposed Enhanced Revolutionary Computational Sciences Program ....................................................29 
Figure 3. Proposed New Revolutionary Computational Sciences (RCA) Program Structure .................................29 
Figure 4. Changing Predictions About Semiconductor Sizes ..................................................................................45 

List of Tables

Table 1. Estimated Performance for Leadership-class Systems ................................................................................49 

ii
 follow-up, a workshop was held with academic, industrial,
1 Executive Summary and government participants from the general aerospace en-
gineering community with a stake in simulation-based engi-
The ability to simulate aerodynamic flows using computa-
neering. The results from the survey and workshop were syn-
tional fluid dynamics (CFD) has progressed rapidly during
thesized and refined by the team, with considerable additions
the last several decades and has fundamentally changed the
through internal discussions and feedback from sponsoring
aerospace design process. Advanced simulation capabilities
NASA officials. The overall project spanned a period of 12
not only enable reductions in ground-based and in-flight
months and resulted in a series of findings, a vision for the
testing requirements, but also provide added physical in-
capabilities required in the year 2030, and a set of recom-
sight, enable superior designs at reduced cost and risk, and
mendations for achieving these capabilities.
open new frontiers in aerospace vehicle design and perfor-
mance. Throughout the evolution of physics-based simula- Findings
tion technologies in general, and computational fluid dy-
namics methods in particular, NASA’s Aeronautics Re- 1. NASA investment in basic research and technology
search Mission Directorate has played a leading role in the development for simulation-based analysis and de-
development and deployment of these technologies. How- sign has declined significantly in the last decade and
ever, today the aerospace CFD community finds itself at a must be reinvigorated if substantial advances in
crossroads due to the convergence of several factors. In simulation capability are to be achieved. Advancing
spite of considerable successes, reliable use of CFD has simulation capabilities will be important for both na-
remained confined to a small but important region of the tional aeronautics and space goals, and has broad im-
operating design space due to the inability of current meth- plications for national competitiveness. This will re-
ods to reliably predict turbulent-separated flows. At the quire advances in foundational technologies, as well as
same time, High Performance Computing (HPC) hardware increased investment in software development, since
is progressing rapidly and is on the cusp of a paradigm shift problem and software complexity continue to increase
in technology that may require a rethinking of current CFD exponentially.
algorithms and software. Finally, during the last decade, 2. HPC hardware is progressing rapidly and technolo-
government investment in simulation-based technology for gies that will prevail are difficult to predict. However,
aerospace applications has been significantly reduced and there is a general consensus that HPC hardware is on the
access to leading-edge HPC hardware has been constrained cusp of a paradigm shift that will require significantly
both in government and industry. Sustaining future advanc- new algorithms and software in order to exploit emerg-
es in CFD and related multidisciplinary analysis and opti- ing hardware capabilities. While the dominant trend is
mization tools will be critical for achieving NASA’s aero- toward increased parallelism and heterogeneous archi-
nautics goals, invigorating NASA’s space program, keeping tectures, alternative new technologies offer the potential
industry competitive, and advancing aerospace engineering for radical advances in computational capabilities, alt-
in general. The improvement of a simulation-based engi- hough these are still in their infancy.
neering design process in which CFD plays a critical role is 3. The use of CFD in the aerospace design process is
a multifaceted problem that requires a comprehensive long- severely limited by the inability to accurately and re-
term, goal-oriented research strategy. The objective of this liably predict turbulent flows with significant regions
report is to develop such a plan, based on factual infor- of separation. Advances in Reynolds-averaged Navier-
mation, expert knowledge, community input, and in-depth Stokes (RANS) modeling alone are unlikely to over-
experience. come this deficiency, while the use of Large-eddy simu-
This report represents the findings and recommendations of a lation (LES) methods will remain impractical for various
multidisciplinary team that was assembled in response to a important applications for the foreseeable future, barring
NASA Research Announcement (NRA) with the goal of any radical advances in algorithmic technology. Hybrid
formulating a knowledge-based forecast and research strate- RANS-LES and wall-modeled LES offer the best pro-
gy for developing a visionary CFD capability in the notional spects for overcoming this obstacle although significant
year 2030. The diverse team members bring together deep modeling issues remain to be addressed here as well.
expertise in the areas of aerodynamics, aerospace engineer- Furthermore, other physical models such as transition
ing, applied mathematics, and computer science, and the and combustion will remain as pacing items.
team includes members with extensive experience from in- 4. Mesh generation and adaptivity continue to be signif-
dustry, academia, and government. A multipronged strategy icant bottlenecks in the CFD workflow, and very lit-
was adopted for gathering information and formulating a tle government investment has been targeted in these
comprehensive research plan. Input from the broader interna- areas. As more capable HPC hardware enables higher
tional technical community was sought, and this was ob- resolution simulations, fast, reliable mesh generation and
tained initially through the development and compilation of adaptivity will become more problematic. Additionally,
an online survey that garnered more than 150 responses. As a adaptive mesh techniques offer great potential, but have

1
 not seen widespread use due to issues related to software  Seamlessly integrates with other disciplinary codes for
complexity, inadequate error estimation capabilities, and enabling complex multidisciplinary analyses and opti-
complex geometries. mizations.
5. Revolutionary algorithmic improvements will be re- The Vision includes a much higher level of integration be-
quired to enable future advances in simulation capa- tween advanced computational methods and improved
bility. Traditionally, developments in improved discreti- ground-based and flight test techniques and facilities in order
zations, solvers, and other techniques have been as im- to best advance aerospace product development efforts and
portant as advances in computer hardware in the devel- reduce technical risk in the future.
opment of more capable CFD simulation tools. Howev-
er, a lack of investment in these areas and the supporting A number of Grand Challenge (GC) problems are used that
disciplines of applied mathematics and computer science constitute the embodiment of this vision of the required CFD
have resulted in stagnant simulation capabilities. Future capabilities in 2030, and cover all important application areas
algorithmic developments will be essential for enabling of relevance to NASA’s aeronautics mission as well as im-
much higher resolution simulations through improved portant aspects of NASA’s space exploration mission. Four
accuracy and efficiency, for exploiting rapidly evolving GC problems have been identified:
HPC hardware, and for enabling necessary future error 1. Wall resolved LES simulation of a full powered air-
estimation, sensitivity analysis, and uncertainty quantifi- craft configuration in the full flight envelope
cation techniques. 2. Off-design turbofan engine transient simulation
6. Managing the vast amounts of data generated by 3. Multidisciplinary Analysis and Optimization (MDAO)
current and future large-scale simulations will con- of a highly flexible advanced aircraft configuration
tinue to be problematic and will become increasingly 4. Probabilistic analysis of a powered space access con-
complex due to changing HPC hardware. These in- figuration
clude effective, intuitive, and interactive visualization of These Grand Challenge problems are chosen because they
high-resolution simulations, real-time analysis, man- are bold and will require significant advances in HPC us-
agement of large databases generated by simulation en- age, physical modeling, algorithmic developments, mesh
sembles, and merging of variable fidelity simulation data generation and adaptivity, data management, and multidis-
from various sources, including experimental data. ciplinary analysis and optimization to become feasible. In
fact, they may not be achievable in the 2030 time frame, but
7. In order to enable increasingly multidisciplinary are used as drivers to identify critical technologies in need
simulations, for both analysis and design optimiza- of investment and to provide benchmarks for continually
tion purposes, advances in individual component measuring progress toward the long-term goals of the re-
CFD solver robustness and automation will be re- search program.
quired. The development of improved coupling at high
fidelity for a variety of interacting disciplines will also Recommendations
be needed, as well as techniques for computing and cou- In order to achieve the Vision 2030 CFD capabilities, a
pling sensitivity information and propagating uncertain- comprehensive research strategy is developed. This is for-
ties. Standardization of disciplinary interfaces and the mulated as a set of recommendations which, when consid-
development of coupling frameworks will increase in ered together, result in a strategy that targets critical disci-
importance with added simulation complexity. plines for investment, while monitoring progress toward the
Vision vision. Two types of recommendations are made: a set of
specific programmatic recommendations and a series of
A knowledge-based vision of the required capabilities of more general strategic recommendations. The programmat-
state-of-the-art CFD in the notional year 2030 is developed ic recommendations avoid the identification of specific
in this report. The Vision 2030 CFD capability is one that: technologies and the prescription of funding levels, since
 Is centered on physics-based predictive modeling. these decisions are difficult at best given the long-range
nature of this planning exercise. Rather, long-range objec-
 Includes automated management of errors and uncer-
tives are identified through the vision and GC problems,
tainties.
and a set of six general technology areas that require sus-
 Provides a much higher degree of automation in all steps
tained investment is described. A mechanism for prioritiz-
of the analysis process.
ing current and future investments is suggested, based on
 Is able to effectively leverage the most capable HPC the periodic evaluation of progress toward the GC prob-
hardware of the day. lems.
 Has the flexibility to tackle large-scale capability tasks
in a research environment but can also manage large
numbers of production jobs for database applications.

2
Programmatic Recommendation 1: NASA should devel- reallocation of resources, sharing with other NASA mission
op, fund, and sustain a base research and technology directorates, leveraging other government agency HPC as-
(R&T) development program for simulation-based analy- sets, or through any combination of these approaches.
sis and design technologies. The presence of a focused base
Programmatic Recommendation 4: NASA should lead
R&T program for simulation technologies is an essential
efforts to develop and execute integrated experimental
component of the strategy for advancing CFD simulation
testing and computational validation campaigns. System-
capabilities. This recommendation consists of expanding the
atic numerical validation test datasets and effective mecha-
current Revolutionary Computational Aerosciences (RCA)
nisms to disseminate validation results are becoming more
program and organizing it around six technology areas identi-
important as CFD complexity increases. NASA is ideally
fied in the findings:
positioned to lead such efforts by leveraging its unique exper-
1. High Performance Computing (HPC) imental facilities in combination with its extensive in-house
2. Physical Modeling CFD expertise, thus contributing valuable community re-
3. Numerical Algorithms sources that will be critical for advancing CFD technology.
4. Geometry and Grid Generation The development of new experimental testing technologies
5. Knowledge Extraction and facilities is expected to play a continuing role not just in
6. MDAO aerospace product development, but increasingly in computa-
tional method validation.
The physical modeling area represents an expansion of the
current turbulence modeling area under the RCA program Strategic Recommendation 5: NASA should develop, fos-
to encompass other areas such as transition and combustion, ter, and leverage improved collaborations with key re-
while the numerical algorithms area corresponds to a cur- search partners and industrial stakeholders across disci-
rent emphasis in the RCA program that must be broadened plines within the broader scientific and engineering
substantially. The other areas constitute new, recommended communities. In an environment of limited resources,
thrust areas within the RCA program. achieving sustained critical mass in the necessary simulation
technology areas will require increased collaborations with
Programmatic Recommendation 2: NASA should devel-
other stakeholders. Mutually beneficial collaborations are
op and maintain an integrated simulation and software
possible between NASA mission directorates, as well as with
development infrastructure to enable rapid CFD technol-
other US government agencies with significant ongoing in-
ogy maturation. A leading-edge in-house simulation capa-
vestments in computational science. Tighter collaboration
bility is imperative to support the necessary advances in CFD
with industry, specifically in simulation technology areas,
required for meeting the 2030 vision. Maintaining such a
would also be beneficial to both parties and a joint Computa-
capability will be crucial for understanding the principal
tional Science Leadership team is proposed to coordinate
technical issues and overcoming the impediments, for inves-
such collaborations. At the same time, investments must look
tigating new techniques in a realistic setting, and for engag-
beyond the traditional aerospace engineering disciplines to
ing with other stakeholders. In order to be sustainable, dedi-
drive substantial advances in simulation technology, and
cated resources must be allocated toward the formation of a
mechanisms for engaging the wider scientific community,
streamlined and improved software development process that
such as focused research institutes that engage the broader
can be leveraged across various projects, lowering software
academic community, should be explored.
development costs, and releasing researchers and developers
to focus on scientific or algorithmic implementation aspects. Strategic Recommendation 6: NASA should attract
At the same time, software standards and interfaces must be world-class engineers and scientists. The ability to achieve
emphasized and supported whenever possible, and open the long-term goals for CFD in 2030 is greatly dependent on
source models for noncritical technology components should having a team of highly educated and effective engineers and
be adopted. scientists devoted to the advancement of computational sci-
ences. Mechanisms for engaging graduate and undergraduate
Programmatic Recommendation 3: NASA should utilize
students in computational science with particular exposure to
and optimize HPC systems for large-scale CFD develop-
NASA aeronautics problems must be devised. These include
ment and testing. Access to large-scale HPC hardware is
student fellowships, as well as visiting programs and intern-
critical for devising and testing the improvements and novel
ships, which may be facilitated through external institutes
algorithms that will be required for radically advancing CFD
and centers.
simulation capabilities. Although the current NASA para-
digm favors computing for many small, production jobs
(“capacity”) over larger, proof-of-concept jobs (“capability”),
a mechanism must be found to make large-scale HPC hard-
ware available on a regular basis for CFD and multidiscipli-
nary simulation software development at petascale to ex-
ascale levels and beyond. This may be done through internal

3
2 Introduction through the development of advanced algorithms within the
same time frame.1,16-18
The rapid advance of computational fluid dynamics (CFD)
However, the last decade has seen stagnation in the capabili-
technology during the last several decades has fundamentally
ties used in aerodynamic simulation within the aerospace
changed the aerospace design process. Aggressive use of
industry, with RANS methods having become the high-
CFD is credited with drastic reductions in wind tunnel time
fidelity method of choice and advances due mostly to the use
for aircraft development programs1-4, as well as lower num-
of larger meshes, more complex geometries, and more nu-
bers of experimental rig tests in gas turbine engine develop-
merous runs afforded by continually decreasing hardware
ment programs.5-6 CFD has also enabled the design of high-
costs.19-23 At the same time, the well-known limitations of
speed, access-to-space, and re-entry vehicles in the absence
RANS methods for separated flows have confined reliable
of suitable ground-based testing facilities.7-9 In addition to
use of CFD to a small region of the flight envelope or operat-
reducing testing requirements, physics-based simulation
ing design space.24 Simultaneously, algorithmic development
technologies such as CFD offer the added potential of deliv-
has been substantially scaled back within NASA and access
ering superior understanding and insight into the critical
to leading-edge HPC hardware has been constrained, both at
physical phenomena limiting component performance, thus
NASA and within industry.25 In some sense, current CFD has
opening new frontiers in aerospace vehicle design.10-11
become a commodity based on mature technology, suitable
Physics-based simulations in general, and CFD in particular, only for commodity hardware and reliable only for problems
are front and center in any aerospace research program since for which an extensive experience base exists.
these are crosscutting technologies that impact all speed re-
Continued advances in physics-based simulation technolo-
gimes and all vehicle classes. This is evidenced in the Na-
gies in general, and CFD in particular, are essential if NASA
tional Research Council (NRC) commissioned decadal sur-
is to meet its aeronautics research goals, as well as for suc-
vey on aeronautics12 which identifies five common themes
cessfully advancing the outcomes in the National Aero-
across the entire aeronautics research enterprise, the first two
nautics R&D plan.13 The required advances in fuel burn,
being physics-based simulation and physics-based design
noise, emissions, and climate impact will only be realized
tools. Similarly, these technologies affect all of the outcomes
with vastly more sophisticated analysis of future configura-
in the current National Aeronautics R&D Plan,13 and contin-
tions. Beyond Aeronautics, NASA’s space missions rely
ued advances in these technologies will be critical for meet-
heavily on computational tools developed within ARMD7-9,
ing the stated outcomes. 26-30
and superior designs at lower cost and risk will require
Since the advent of scientific computing, NASA’s Aero- radical advances in new CFD tools.31 Additionally, the loss
nautics Research Mission Directorate (ARMD) has played a of the leadership role NASA ARMD once played in the de-
leading role in the development and deployment of CFD velopment of simulation-based engineering technology has
technologies.14 Successive external reviews of NASA Aero- larger implications for the aerospace industry in particular,
nautics programs during the last two decades by organiza- and national competitiveness in general.17,32,33 Due to the
tions such as the National Academy of Engineering (NAE) long lead times and high risk involved, industry must rely on
and others12 have repeatedly praised the world-class status government agencies to develop and demonstrate new simu-
and leading-edge technical contributions of the simulation- lation technologies at large scale, after some investment in
based engineering tools developed under these programs. In proof-of-concept at universities. In recent years, the National
fact, many algorithms, techniques, and software tools in use Science Foundation (NSF) and Department of Energy (DOE)
today within and beyond the aerospace industry can trace have led investing in computational science-based research
their roots back to NASA development or funding. and in deploying leading-edge HPC facilities, although with
a different focus based more on scientific discovery rather
The development of computational aerodynamics has been
than engineering product design.16, 18, 34-36 As noted by a blue-
characterized by a continual drive to higher fidelity and more
ribbon panel report convened by the NSF, simulation-based
accurate methods from the 1970s to the 1990s, beginning
engineering is fundamentally different from science-based
with panel methods, proceeding to linearized and nonlinear
simulation and is in danger of being neglected under the cur-
potential flow methods, inviscid flow (Euler) methods, and
rent scenario, with important implications for national com-
culminating with Reynolds-averaged Navier-Stokes (RANS)
petitiveness.37, 38
methods. These advances were arrived at through sustained
investment in methodology development coupled with acqui- Thus, there is a national imperative to reinvigorate the in-
sition and deployment of leading edge High Performance vestment in physics-based engineering simulation tools in
Computing (HPC) hardware made available to researchers.15 general, and in CFD in particular, and NASA is uniquely
While Moore’s law has held up remarkably well, delivering a positioned to fill this role. In addition to enhancement of
million-fold increase in computational power during the last CFD technology, effective use of existing, and potentially
20 years, there is also ample evidence that equivalent or new, ground-based testing facilities will be required to pro-
greater increases in simulation capabilities were achieved vide a complementary set of tools to best advance aerospace

4
product development efforts and reduce technical risk in the 3 Vision of CFD in 2030
future. Sustaining future advances in CFD and related multi-
disciplinary analysis and optimization tools along with strong Given the inherent difficulties of long-term predictions, our
support of ground-based and flight testing technologies and vision for CFD in 2030 is grounded on a desired set of capa-
facilities will be key for achieving NASA next generation bilities that must be present for a radical improvement in
aeronautics goals, keeping industry competitive, invigorating CFD predictions of critical flow phenomena associated with
NASA’s space program, and advancing aerospace engineer- the key aerospace product/application categories, including
ing. With investment, the resulting engineering design pro- commercial and military aircraft, engine propulsion, ro-
cess would decrease risk, reduce time-to-market, improve torcraft, space exploration systems, launch vehicle programs,
products, and facilitate truly revolutionary aerospace vehicles air-breathing space-access configurations, and spacecraft
through the ability to consider novel designs. Without such entry, descent, and landing (EDL).
an investment, the engineering design process will look much
the same in 2030 as it does today and act as a barrier to revo-
lutionary advances in aerospace and other industries of na-
tional importance.
The improvement of a simulation-based engineering design
process in which CFD plays a critical role is a multifaceted
problem. Having relied on mature algorithms and ridden the
wave of ever-decreasing commodity computer hardware
costs, the CFD development community now finds itself
poorly positioned to capitalize on the rapidly changing HPC
This set of capabilities includes not only the accurate and
architectures, which include massive parallelism and hetero-
efficient prediction of fluid flows of interest, but also the
geneous architectures.38-40 New paradigms will be required to
usability of CFD in broader contexts (including uncertainty
harness the rapidly advancing capabilities of new HPC hard-
quantification, optimization, and multidisciplinary applica-
ware.41, 42 At the same time, the scale and diversity of issues
tions) and in streamlined/automated industrial analysis and
in aerospace engineering are such that increases in computa-
design processes. To complicate things further, CFD in 2030
tional power alone will not be enough to reach the required
must effectively leverage the uncertain and evolving envi-
goals, and new algorithms, solvers, physical models, and
ronment of HPC platforms that, together with algorithmic
techniques with better mathematical and numerical properties
improvements, will be responsible for a large portion of the
must be developed.1,16-18 Finally, software complexity is in-
realized improvements.
creasing exponentially, slowing adoption of novel techniques
into production codes and shutting out production of new The basic set of capabilities for Vision 2030 CFD must in-
software development efforts, while simultaneously compli- clude, at a minimum:
cating the coupling of various disciplinary codes for multi-
1. Emphasis on physics-based, predictive modeling. In
disciplinary analysis and design.43 The development of a
particular, transition, turbulence, separation, chemically
long-range research plan for advancing CFD capabilities
reacting flows, radiation, heat transfer, and constitutive
must necessarily include all these considerations, along with
models must reflect the underlying physics more closely
the larger goal of comprehensive advances in multidiscipli-
than ever before.
nary analysis and optimization capabilities.
2. Management of errors and uncertainties resulting from
The objective of this study is to develop such a plan, based all possible sources: (a) physical modeling errors and
on factual information, expert knowledge, and the in-depth uncertainties addressed in item #1, (b) numerical errors
experience of the team and the broader community. The arising from mesh and discretization inadequacies, and
strategy begins by defining the required capabilities for CFD (c) aleatory uncertainties derived from natural variabil-
in the notional year 2030. By contrasting this vision with the ity, as well as epistemic uncertainties due to lack of
current state, we identify technical impediments and formu- knowledge in the parameters of a particular fluid flow
late a technology development plan. This in turn is used to problem.
develop a research strategy for achieving the goals of the 3. A much higher degree of automation in all steps of the
Vision 2030 CFD capability. The outcome of the research analysis process is needed including geometry creation,
plan is a set of recommendations formulated to enable the mesh generation and adaptation, the creation of large da-
successful execution of the proposed strategy. tabases of simulation results, the extraction and under-
standing of the vast amounts of information generated,
and the ability to computationally steer the process. In-
herent to all these improvements is the requirement that
every step of the solution chain executes high levels of
reliability/robustness to minimize user intervention.

5
 Critical Flow Phenomena Addressed in This Study

• Flow separation: e.g., smooth-body, shock-induced, blunt/bluff body • Wake hazard reduction and avoidance
• Laminar to turbulent boundary layer flow transition/reattachment • Wind tunnel to flight scaling
• Viscous wake interactions and boundary layer confluence • Rotor aero/structural/controls, wake and multirotor interactions, acoustic
• Corner/junction flows loading, ground effects
• Icing and frost • Shock/boundary layer, shock/jet interactions
• Circulation and flow separation control • Sonic boom
• Turbomachinery flows • Store/booster separation
• Aerothermal cooling/mixing flows • Planetary retro-propulsion
• Reactive flows, including gas chemistry and combustion • Aerodynamic/radiative heating
• Jet exhaust • Plasma flows
• Airframe noise • Ablator aerothermodynamics
• Vortical flows: wing/blade tip, rotorcraft • Plasma flows

4. Ability to effectively utilize massively parallel, hetero- sufficiently predictive and automated to be used in criti-
geneous, and fault-tolerant HPC architectures that will cal/relevant engineering decisions by the non-expert user,
be available in the 2030 time frame. For complex physi- particularly in situations where separated flows are present.
cal models with nonlocal interactions, the challenges of
Additionally, the Vision includes a much higher level of in-
mapping the underlying algorithms onto computers with
tegration between advanced computational methods and im-
multiple memory hierarchies, latencies, and bandwidths
proved ground-based and flight test techniques and facilities
must be overcome.
in order to best advance aerospace product development ef-
5. Flexibility to tackle capability- and capacity-computing
forts and reduce technical risk in the future.
tasks in both industrial and research environments so
that both very large ensembles of reasonably-sized solu- Finally, as part of our vision, we define a set of Grand Chal-
tions (such as those required to populate full-flight enve- lenge (GC) problems (as shown in the graphic on the next
lopes, operating maps, or for parameter studies and de- page) that are bold and in fact may not be solvable in the
sign optimization) and small numbers of very-large- 2030 timeframe, but are used as drivers to identify critical
scale solutions (such as those needed for experiments of technologies in need of investment, and to serve as bench-
discovery and understanding of flow physics) can be marks for continually measuring progress toward the long-
readily accomplished. term development goals. These GC problems were chosen
to embody the requirements for CFD in 2030, and cover all
6. Seamless integration with multidisciplinary analyses that
important application areas of relevance to NASA’s aero-
will be the norm in 2030 without sacrificing accuracy or
nautics mission, as well as important aspects of NASA’s
numerical stability of the resulting coupled simulation,
space exploration mission. Details on the GC problems are
and without requiring a large amount of effort such that
presented in Section 6.
only a handful of coupled simulations are possible.

Included in this desired set of capabilities is a vision for how

CFD in 2030 will be used: a vision of the interaction between
4 Current State
the engineer/scientist, the CFD software itself, its framework At present, CFD is used extensively in the aerospace industry
and all the ancillary software dependencies (databases, mod- for the design and analysis of air and space vehicles and
ules, visualization, etc.), and the associated HPC environ- components. However, the penetration of CFD into aero-
ment. A single engineer/scientist must be able to conceive, space design processes is not uniform across vehicle types,
create, analyze, and interpret a large ensemble of related sim- flight conditions, or across components. CFD often plays a
ulations in a time-critical period (e.g., 24 hours), without complementary role to wind tunnel and rig tests, engine certi-
individually managing each simulation, to a pre-specified fication tests, and flight tests by reducing the number of test
level of accuracy. There should be less emphasis on the me- entries and/or reducing testing hours.3-5 But, in many circum-
chanics of running and collecting the information, and more stances, CFD provides the only affordable or available source
emphasis on interpreting and understanding the results of the of engineering data to use in product design due to limita-
work. Like the predictive nature of large-scale, finite- tions either with model complexity and/or wind tunnel capa-
element-based, linear structural analyses that are assumed in bility, or due to design requirements that cannot be addressed
the aerospace industry, information derived from computa- with ground-based testing of any kind.8, 10, 31 As a result, CFD
tions of fluid flow must carry the same stamp of approval or, technology development has been critical in not only mini-
at least, a reasonable estimate of possible errors contained in mizing product design costs, but also in enabling the design
the information provided. At the moment, CFD is not yet of truly novel platforms and systems.

6
 Grand Challenge Problems

GC1 LES of powered aircraft configuration

across the full
flight envelope

GC2 Off-design turbofan engine transient

simulation

GC3 MDAO of a highly-flexible advanced

aircraft configuration

255606.002.ppt
GC4 Probabilistic analysis of a powered
space access configuration

Generally, the design process is composed of three key phas- CFD is often used to gain important insight into flow phys-
es: conceptual design, preliminary and detailed design, and ics (e.g., multiple plume interactions) used to properly lo-
product validation. The current usage of CFD tools and pro- cate external components on the surface of launch vehicles
cesses in all three of these design phases is summarized be- or spacecraft.9,27,29 CFD is also increasingly providing sub-
low. stantial portions of the aero and propulsion performance
database.7,9,30 In many cases, wind tunnel data is used only
Conceptual Design. CFD is often used in the early, con-
to anchor the CFD data at a few test points to provide con-
ceptual design of products where it has been both previous-
fidence in the CFD database. CFD is the primary source of
ly calibrated for similar applications using data-morphing
data for the hypersonic flight regime when ground testing
techniques, as well as for new configurations where little
is limited or does not exist.8, 44
or no engineering data is available to guide design deci-
sions. Simplified models are typically used during the con- Product Validation and Certification. As the product
ceptual optimization phase to allow reasonably accurate development process moves into the validation phase and
trades to be made between drag, fuel consumption, weight, certification testing, CFD is often used to confirm perfor-
payload/range, thrust, or other performance measures. Use mance test results, assess the redesign of components that
of simplified models is necessary to allow often time- show potential for improved performance, and to answer
consuming optimization processes to be used in the overall any other questions that arise during product testing. Typi-
design effort, but inherently places conservatism into the cally, product configurations evolve over the testing period
final design. This conservatism derives from the use of based on a combination of measured results and engineer-
models that are too similar within the existing product de- ing judgment bolstered by the best simulation capability
sign space, other geometric simplifications, or the use of available. In general, CFD modeling capability grows to
low-fidelity CFD tools that trade off flow physics model- capture the required scope and physics to answer the ques-
ing accuracy for execution speed. tions raised during testing. The expense of responding to
often unplanned technical surprises—which results in more
Preliminary/Detailed Design. Once a product develop-
time on the test stand or in flight test, or changes in hard-
ment program is launched, CFD is a necessary and uni-
ware—drives conservatism into aerospace designs and is a
formly present tool in the detailed configuration design
significant motivation for improving the accuracy and
process. For example, CFD is indispensable in the design
speed of CFD. If CFD is sufficiently accurate and fast, en-
of cruise wings in the presence of nacelles for commercial
gineers can move from their traditional design space with
airplanes, and for inlet and nozzle designs; wind tunnels
greater confidence and less potential risk during testing.
are used to confirm the final designs.1,4 In both military and
commercial aircraft design processes, CFD is the primary For each of these design phases, the performance (in terms
source of data for aircraft load distributions and ground ef- of numerical efficiency and solution accuracy) of CFD is of
fect estimations.10 Similarly, gas turbine engine manufac- critical importance. A high-level view of the current state of
turers rely on CFD to predict component design perfor- CFD in several key technical areas is given below.
mance, having reduced the number of single-component
High Performance Computing (HPC). The effectiveness
rigs substantially as CFD capability has become more ma-
and impact of CFD on the design and analysis of aerospace
ture.5,6 Increasingly, multicomponent and multiphysics
products and systems is largely driven by the power and
simulations are performed during the design cycle, but the
availability of modern HPC systems. During the last dec-
long clock times often associated with these processes re-
ades, CFD codes were formulated using message passing
stricts their widespread adoption. For space exploration,

7
(e.g., MPI) and thread (e.g., OpenMP) software models for curs in areas where the design work is routine and the in-
expressing parallelism to run as efficiently as possible on vestment in automation makes business sense; single-
current generation systems. However, with the emergence prototype designs and novel configurations continue to
of truly hierarchical memory architectures having numer- suffer the pacing limits of human-in-the-loop workflows
ous graphical processing units (GPUs) and coprocessors, because the payoff for automating is not evident. This is-
new CFD algorithms may need to be developed to realize sue is not unique to the aerospace industry.
the potential performance offered by such systems. Gov-
Solution Uncertainty and Robustness. In practice, CFD
ernment labs, such as Oak Ridge National Lab (ORNL),
workflows contain considerable uncertainty that is often
Argonne National Lab (ANL), and the NASA Advanced
not quantified. Numerical uncertainties in the results come
Supercomputing (NAS) facility at NASA Ames Research
from many sources, including approximations to geometry,
Center, have often led the way with the acquisition and
grid resolution, problem setup including flow modeling
testing of new hardware. Much research on testing and tai-
and boundary conditions, and residual convergence. Alt-
loring of CFD algorithms takes place on these platforms
hough NASA and professional organizations such as
with heavy participation from academia, national labs and
ASME and AIAA have created standards for the verifica-
to some extent industry as well. Government computing
tion and validation of CFD and heat transfer analyses52-54,
resources are also used to tackle large-scale calculations of
such techniques are not widely used in the aerospace in-
challenge problems, such as the detailed direct numerical
dustry. With a few notable exceptions, CFD is carried out
simulation (DNS) of spray injector atomization or high fi-
on fixed grids that are generated using the best available
delity simulations of transition and turbulent separation in
practices to capture expected flow features, such as at-
turbomachinery. However, because of the high cost of
tached boundary layers.55 Such approaches cannot reliably
these leadership-class systems, industry and academia of-
provide adequate resolution for flow features when loca-
ten purchase smaller commodity clusters utilizing similar
tions are not known a priori, such as shocks, shear layers,
types of processors when the latest hardware technology is
and wakes. Although grid refinement is often seen as a
fully demonstrated on CFD problems and other important
panacea to addressing grid resolution issues, it is seldom
applications.
done in practice (with the exception of a few workshop test
Turbulence Modeling. Current practices for CFD-based cases) because uniform refinement is impractical in 3D.
workflows utilize steady Reynolds-average Navier-Stokes Adaptive mesh refinement strategies offer the potential for
(RANS) with 1- or 2-equation turbulence models,46-48 alt- superior accuracy at reduced cost, but have not seen wide-
hough hybrid unsteady RANS/LES methods are increas- spread use due to robustness, error estimation, and soft-
ingly common for certain classes of simulations in which ware complexity issues. Achieving consistent and reliable
swirling and intentionally separated flows are dominant, flow solver or residual convergence remains problematic in
such as combustors. Techniques to combine near-wall many industrial cases. Although many CFD codes are able
RANS regions and outer flow field, large-eddy simulation to demonstrate convergence for a few simple problems, for
(LES) regions in these hybrid methods are immature. many flows involving difficult flow physics and/or com-
Many CFD design processes include an estimation of plex geometries such as an aircraft in high-lift configura-
boundary layer transition, using a range of models, from tion, many of the current solver techniques employed are
purely empirical to coupled partial-differential equation not strong enough to ensure robust convergence.56 Engi-
(PDE) solutions of stability equations.49, 50 Both approach- neering judgment is required to interpret results that are not
es involve much empiricism, may be missing some well converged, which introduces conservatism into deci-
modes of transition, and are evolving. As a result, no sion-making. Furthermore, the use of steady-state flow
generalized transition prediction capability is in wide- solvers itself is in question for many flows of engineering
spread use in Navier-Stokes CFD, and the default prac- interest.
tice is to run the codes “fully turbulent”. Steady-state
Multidisciplinary Analysis and Optimization (MDAO).
CFD accounts for a vast majority of simulations; unsteady
Although the basic concepts of MDAO are fairly well ac-
flow predictions are inherently more expensive and not yet
cepted in the community, the routine use of MDAO meth-
uniformly routine in the design process, with some excep-
ods is not, by any means, pervasive. At moderate levels of
tions.51
fidelity (commensurate with analyses conducted during the
Process Automation. Current CFD workflows are often conceptual design phase), it is common industrial practice
paced by the geometry preprocessing and grid generation to perform coupled multidisciplinary analyses (MDA) of
phases, which are significant bottlenecks. In some cases, the most tightly integrated disciplines in a design. Aero-
where the design effort involves components of similar structural analyses, conjugate heat transfer calculations,
configurations, specialized, automated processes are built and aero-acoustic simulations all tend to take place in air-
that considerably reduce set-up time, execution of the craft, spacecraft, jet engine, and rotorcraft analysis and de-
CFD solver, and post-processing of results. This process sign processes. High fidelity CFD is not routinely used in
to production capability of the CFD workflow only oc- such MDAs, although recent years have witnessed a signif-

8
 icant rise in the coupling of state-of-the-art CFD with addi- 5.1 Effective Utilization of High-Performance
tional disciplines. While frameworks for the coupling of Computing (HPC)
disciplinary analyses are widely available,57-60 the ability to
couple CFD with other high fidelity descriptions of partic- CFD in 2030 will be intimately related to the evolution of the
ipating disciplines is limited by the availability of coupling computer platforms that will enable revolutionary advances
software and, more fundamentally, by a lack of general in simulation capabilities. The basic framework for Vision
methodologies for accurate, stable, and conservative 2030 CFD must map well to the relevant future programming
MDAs. The application of optimization techniques in in- paradigms, which will enable portability to changing HPC
dustry is mostly limited to single-discipline simulations. environments and performance without major code rework.41
Although conceptual design tools have long benefited from However, the specific architecture of the computing plat-
multidisciplinary optimization (MDO) approaches (with forms that will be available is far from obvious. We can,
various modules at the lowest fidelity levels), high fidelity however, speculate about the key attributes of such machines
CFD-based optimizations are still rare. During the past and identify key technology gaps and shortcomings so that,
decade, the development of advanced surrogate modeling with appropriate development, CFD can perform at future
techniques and the introduction of adjoint-based optimal exascale levels on the HPC environments in 2030.
shape design techniques have enabled the use of CFD in Hybrid computers with multiple processors and accelerators
aerodynamic optimization of aircraft and gas turbine com- are becoming widely available in scientific computing and,
ponents. However, the use of optimization with multiple although the specific composition of a future exascale com-
disciplines treated using high-fidelity methods is still with- puter is not yet clear, it is certain that heterogeneity in the
in the realm of advanced research and is by no means a computing hardware, the memory architecture, and even the
routine practice. network interconnect will be prevalent. Future machines
will be hierarchical, consisting of large clusters of shared-
5 CFD Technology Gaps and Impedi- memory multiprocessors, themselves including hybrid-chip
ments multiprocessors combining low-latency sequential cores
with high-throughput data-parallel cores. Even the memory
Given the current state of CFD technology, tools, and pro- chips are expected to contain computational elements,
cesses described above, necessary research and develop- which could provide significant speedups for irregular
ment to address gaps and overcome impediments in CFD memory access algorithms, such as sparse matrix operations
technology are fundamental to attaining the vision for CFD arising from unstructured datasets. With such a running
in 2030 outlined earlier. Five key technical areas were iden- target on the horizon, the description of 2030 CFD is
tified during this Vision 2030 study and rose to the highest grounded on a target supercomputer that incorporates all of
level of importance as identified from a user survey and the representative challenges that we envision in an ex-
workshop of a large international community of CFD re- ascale supercomputer. These challenges are certainly relat-
searchers and practitioners. In the subsections below, the ed to heterogeneity, but more concretely, may include mul-
effective utilization of HPC is first considered. This in- ticore CPU/GPU interactions, hierarchical and specialized
cludes both the implications of future computing platforms networks, longer/variable vector lengths in the CPUs,
and the requirements imposed by potential emerging future shared memory between CPU and GPUs, and even a higher
programming paradigms to deal with exascale challenges. utilization of vector units in the CPUs. Research must be
Next, we describe the desired level of capability (in 2030) conducted so that we are ready to tackle the specific chal-
for the prediction of unsteady, turbulent flows including lenges presented.
transition and separation. We continue with a discussion of
research topics in autonomous and reliable CFD simulation The wildcard in predicting what a leading edge HPC system
techniques that aim at providing both a high level of auto- will look like is whether one or more of several current nas-
mation in the analysis process and the required algorithms cent HPC technologies will come to fruition. Radical new
(both for meshing and the solution process) to ensure confi- technologies such as quantum computing, superconducting
dence in the outcomes. Then, in order to derive useful in- logic, low-power memory, massively parallel molecular
formation from the simulations, the discussion on smart computing, next generation “traditional” processor technol-
knowledge extraction from large-scale databases and simu- ogies, on-chip optics, advanced memory technologies (e.g.,
lations considers the research required to automate the pro- 3D memory) have been proposed but are currently at very
cess of sifting through large amounts of information, often low technology readiness levels (TRL). Many of these
at a number of different geographic locations, and extract- revolutionary technologies will require different algorithms,
ing patterns and actionable design decisions. Finally, we software infrastructures, as well as different ways of using
end with a discussion on multidisciplinary and multiphysics results from CFD simulations.61
simulations that describes the research work required to
perform seamless, accurate, and robust simulations of mul-
tiphysics problems in which CFD would be an integral part.

9
CASE STUDY 1: Current Utilization of HPC at NASA
HPC utilization at NASA is almost entirely focused on capacity computing (running many, relatively small jobs) with little capability com-
puting (running jobs utilizing a significant amount of a leadership class high performance computer). The largest NASA HPC system is
Pleiades at the NASA Advanced Supercomputing (NAS) division. As of June 2013, this system is currently ranked 19th in the world in
terms of its performance on the linear algebra benchmark LINPACK.1 As of October 2013, Pleiades consists of:
 11,136 nodes with Intel Xeon processors for a total of 162,496 cores
 64 nodes with NVIDIA graphics processing units for a total of 32,768 cores
 417 TB of total memory
From the NAS website, the theoretical peak performance of this configuration is quoted as 2.88 pFLOP/s and the demonstrated
LINPACK rating is 1.24 pFLOP/s. By comparison, the current fastest system on the Top 500 list is Tianhe-2 at the National University of
Defense Technology in China that has a theoretical peak performance of 54.9 pFLOP/s and a demonstrated LINPACK rating 33.9
pFLOP/s. The Top 10 HPC systems are provided in the embedded table, including Pleiades for comparison, and shows that Pleiades is
a factor of 2 to 30 times slower than these Top 10 systems (in terms of the LINPACK performance). While Pleiades is within a factor of
about 10 of the world’s fastest HPC systems, it is rarely used at anywhere near its full capability. For example, a snapshot of the Pleia-
des job queue2 (taken on October 24, 2013 at 2:00PM Eastern) shows the following utilization:
 469 jobs running
 Average cores used per job: 457
 Maximum cores used per job: 5,000 (the only job running more than 1000 cores)
Thus, although the Pleiades system has approximately 160K CPU cores (and another 32K GPU cores), the average job size is less than
1K cores and Pleiades is therefore acting as a capacity facility. Further, the usage of Pleiades is over-subscribed with job queues often
having delays numbering in the days, so that even in its role as a capacity facility, Pleiades is insufficient to meet NASA’s needs.
Top 500 System LINPACK Theoretical Peak
Cores
Ranking Site (pFLOP/s) pFLOP/s)
1 Tianhe-2 (China) 33.9 54.9 3,120,000
2 Titan (USA: DOE) 17.6 27.1 560,640
3 Sequoia (USA: DOE) 17.2 20.1 1,572,864
4 K computer (Japan) 10.5 11.3 705,024
5 Mira (USA: DOE) 8.6 10.1 786,432
6 Stampede (USA: University of Texas) 5.2 8.5 462,462
7 JUQUEEN (Germany) 5.0 5.9 458,752
8 Vulcan (USA: DOE) 4.3 5.0 393,216
9 SuperMUC (Germany) 2.9 3.2 147,456

255606.003.ppt
10 Tianhe-1A (China) 2.6 4.7 186,368
19 Pleiades (USA: NASA) 1.2 2.9 195,264

By comparison, the DOE has an HPC strategy that encompasses both capacity and capability computing. A key enabler of this strategy
is the significant HPC resources at the DOE (for example, the DOE has 4 of the Top 10 supercomputer sites shown in the table: Titan,
Sequoia, Mira, and Vulcan). This wealth of HPC resources allows the DOE to dedicate systems to both capacity and capability compu-
ting. For example, DOE leadership systems have job queue policies that (1) strongly favor large jobs that will use a significant fraction of
a leadership system and (2) limit the potential that these systems are flooded by capacity computations. The DOE also has programs
such as Innovative and Novel Computational Impact on Theory and Experiment (INCITE)3 specifically designed to encourage capability
computing. INCITE allocates up to 60% of the Leadership Computing Facilities at Argonne and Oak Ridge National Laboratories to na-
tional and international research teams pursuing high-impact research that can demonstrate the ability to effectively utilize a major frac-
tion of these machines in a single job. Utilization data for DOE’s Leadership Facilities bears out the impact these policies have had on
the pursuit of capability computing. For example, on DOE’s Mira system, which is about four times larger than Pleiades, the average job
size was 35K cores during the period from April through October 2013. The smallest job size during that time was 8K cores while the
largest job size used essentially the entire system at nearly 800K cores. Comparisons can also be made between Pleiades and DOE’s
“Intrepid” system. Intrepid is the 58th fastest supercomputing site with 164K cores, a LINPACK performance of 0.46 pFLOP/s, and a

10
 peak performance of 0.56 pFLOP/s. During the period from October 2011 through October 2013, Intrepid’s average job size was 9K
cores, with its smallest job being 256 cores, and largest job size used all 164K cores. Although Intrepid is a somewhat less capable sys-
tem than Pleiades, the utilization patterns are in stark contrast. Further, for both Mira and Intrepid,
the overall utilization is still high, with Mira’s scheduled availability being utilized at 76% and Intrep-
id’s at 87%.4 A recent extreme example of capability computing using CFD is the 1.97M core simu-
lation of shock interaction with isotropic turbulence (image shown) that was performed by combin-
ing the DOE’s Sequoia and Vulcan systems.5,6
Looking toward CFD in the 2030s and beyond, the need for improved physics-based modeling in
CFD is driving increasingly expensive simulations that will only be possible by leveraging leader-
ship class HPC systems. Without NASA’s leadership in the application of capability computing to
CFD, the adoption of these technologies in the United States aerospace engineering industry will
be hampered.
1 http://www.top500.org/
2 http://www.nas.nasa.gov/hecc/resources/pleiades.html
3 http://www.doeleadershipcomputing.org/incite-program/
4 Data courtesy of the Argonne Leadership Computing Facility at Argonne National Laboratory, which is supported by the Office of Science of the U.S. Department of
Energy under contract DE-AC02-06CH11357.
5 Bermejo-Moreno, I., Bodart, J., Larsson, J., Barney, B., Nichols, J., and Jones, S. "Solving the Compressible Navier-Stokes Equations on up to 1.97 Million Cores
and 4.1 Trillion Grid Points." SC13, November 17-21 2013, Denver, CO, USA.
6 Larsson, J., Bermejo-Moreno, I., and S. K. Lele. "Reynolds- and Mach-Number Effects in Canonical Shock-Turbulence Interaction." Journal of Fluid Mechanics,
717:293–321, 2013.

We envision the leading edge HPC system in 2030 will 4. Robust CFD code scalability. As described earlier, an
have a peak capability of about 30 exaFLOPS when based HPC system in 2030 will require tremendous levels of
on the evolution of current technologies. To achieve this parallelization. Unfortunately, robust CFD flow solver
anticipated hardware performance, and the required flow scalability even on current multicore platforms is sore-
solver software enhancements to enable effective CFD on ly lacking. Few applications can make efficient use of
2030 computing systems, a number of technology gaps and more than O(1,000) cores, although the largest ma-
impediments must be overcome. chines today are available with O(1,000,000) cores.62
In contrast, 20 years ago, production CFD codes ran
1. Hardware system power consumption. Current state-
routinely on the largest available shared-memory vec-
of-the-art computing systems consume too much power
tor machines. To address these challenges new, ex-
to be scaled up substantially, utilize structural compo-
tremely parallel CFD algorithms that balance compu-
nents that are too large, and do not provide the level of
ting and communication need to be developed. Fur-
computational and communication speed necessary.
thermore, there needs to be investment in the develop-
Development of advanced HPC hardware technologies
ment of CFD codes built on highly optimized libraries
with a special focus on power consumption and error
and middleware. In contrast, current CFD codes and re-
protections and recovery is needed.
lated processes are rather monolithic today, making it
2. Higher levels of software extraction. The increased
difficult to change algorithms or implementations. A
complexity of HPC exascale systems in 2030 will re-
future CFD code and surrounding processes should be
quire higher levels of automation and the ability to hide
modular, allowing replacement of components with
this complexity from the subject matter experts. The
new components developed in academia or from com-
whole software infrastructure stack does not scale to
mercial vendors easily and transparently. Such a modu-
the level of complexity of future HPC systems and
lar approach would also enable coupling of MDA/O
needs to be more resistant to errors. To overcome this
processes.
gap, research into industrial strength implementations
5. Lack of scalable CFD pre- and post-processing
of the necessary middleware, especially operating sys-
methods. Despite the deficiencies in current CFD
tems, compilers, communication, and I/O libraries, as
solver scalability, the situation for the surrounding in-
well as deployment and monitoring systems needs to
frastructure of pre- and post-processing software is
continue.
even worse. In order to streamline and accelerate the
3. Advanced programming environments. Another
entire CFD workflow and design process, the devel-
critical component in the development of the full future
opment of basic scalable pre- and post-processing
HPC ecosystem is the development of basic, highly
methods must be addressed. This includes geometry
scalable and error resistant algorithms, decomposable
representation and mesh generation on the front end as
software architectures, and programming environments
well as visualization, database generation, and general
that allow scientific subject matter experts to express
knowledge extraction from large datasets on the back
algorithms at the appropriate level of abstraction.
end.

11
6. Lack of access to HPC resources for code develop- invited speakers in the session on turbulence predicted the
ment. Another key issue is the lack of access to large- continued use of RANS, including one and two-equation
scale HPC resources as an integral part of software de- models, as opposed to the more complex Reynolds-Stress
velopment. Consistent and reliable access to leading- Transport models. They also predicted the extensive use of
edge HPC hardware is critical for devising and testing hybrid methods. However, LES-dominant methods for the
new techniques that enable more advanced simulations, range of engineering problems of interest (specifically for
as well as for demonstrating the impact that CFD tech- higher Reynolds numbers) will likely not be feasible based
nology enhancements can have on aerospace product on current estimates of HPC computing performance in
development programs.11, 39 Algorithmic choices and 2030 using standard CFD approaches (see “Case Study 2:
software implementation strategies are directly affected LES Cost Estimates and 2030 Outlook” below).
by the type of hardware made available during the soft-
In the area of viscous turbulent flows with transition and
ware development process, and the stagnating scalability
separation, a number of technology gaps and impediments
of current production CFD codes is at least partly at-
must be overcome to accurately model these flows in the
tributable to the inability to test these codes consistently
2030 timeframe. Specifically, they are:
on large-scale HPC hardware. The resulting situation of
scalability-limited simulation tools reduces demand for 1. Lack of a theoretically based, hybrid RANS-LES
large-scale capability computing since few codes can turbulence simulation capability. Ideally, unsteady
take advantage of HPC hardware, while driving demand flow simulations using advanced turbulence models
for throughput or capacity computing. Allocating a por- (e.g., DES, full LES) should be used to resolve the key
tion of HPC computing resources for highly scalable turbulent length scales that drive development and prop-
software development programs will be essential for agation of flow separation. There has been progress in
pushing the boundaries of CFD simulation capabilities.45 the representation of post-separation physics with the
use of hybrid RANS-LES, or in general, turbulence-
5.2 Unsteady Turbulent Flow Simulations In- resolving methods (i.e., at least the part of the domain
cluding Transition and Separation that is solved in LES mode).67, 68 In contrast, however,
the prediction of pre-separation physics is still provided
Perhaps the single, most critical area in CFD simulation by RANS models, which have seen nearly stagnant de-
capability that will remain a pacing item by 2030 in the velopment for 20 years.48 Unfortunately, hybrid methods
analysis and design of aerospace systems is the ability to are currently cost-prohibitive for routine use on realistic
adequately predict viscous turbulent flows with possible configurations at Reynolds numbers of interest in aero-
boundary layer transition and flow separation present.23, 31, space, at least in the thinner regions of the boundary lay-
63, 64
While steady, fully turbulent attached flows can be er such as near the wing attachment line. Another key
predicted reasonably well with current RANS methods at impediment in fielding a robust hybrid RANS-LES ca-
all speed regimes,23, 65, 66 all types of separated flows are pability is the changing nature of the interface between
still difficult to predict. In particular, smooth body separa- RANS and LES regions. For hybrid methods to be rou-
tion remains very hard to simulate accurately and efficiently tinely used, a seamless, automatic RANS-to-LES transi-
for high-speed (buffet-limited) stall, low-speed high-lift, tion in the boundary layer is urgently required.69
inlets at crosswind conditions, engine simulations and com- 2. Availability and convergence of complex turbulence
pressor stall, flows at the edges of the design envelope, and models in practical codes. A recurring issue in using
for maneuvering flight with moving control surfaces. In elaborate RANS models with second moment closures
general, two critical components of flow physics need to be (e.g., Reynolds Stress Transport methods) for practical
modeled accurately: the exact location of separation as con- applications is both their availability in widely used flow
trolled by boundary-layer physics and the feedback from solvers (e.g., FUN3D, OVERFLOW) and their notori-
the separated region to the boundary layer. ously poor convergence characteristics for flow simula-
Based on feedback from the CFD survey and the follow-up tion involving complex geometries and/or complex flow
workshop held as part of this study, it is clear that the ma- physics.70 The key impediments are the complexity of
jority of the engineering and scientific community believes the models themselves manifesting in myriad variations,
that RANS-based turbulence models, in conjunction with inadequate attention to numerics during design of the
the expanded use of hybrid RANS-Large Eddy Simulation models, and the lack of powerful solution techniques in
(LES) methods, will be the norm in 2030. This sentiment these codes that may be needed to solve the flow and
was confirmed from discussions at the workshop: all of the turbulence model equations.71

12
CASE STUDY 2: LES Cost Estimates and 2030 Outlook
The predictions of when LES will be available in a reasonable turnaround time for engineering use have been performed by numerous
researchers. Here, we focus on wall-modeled LES (WMLES) in which the anisotropic near-wall region is modeled in some manner such
that the LES is responsible only for the larger, more isotropic outer flow. In 1979, Chapman estimated that such wall-modeled LES
would be possible in the 1990s for practical aerodynamic applications.1 This clearly has not been realized in practice and one key factor
in the optimistic predictions of Chapman was an underestimation of the computational work required for LES. Since that time, Spalart, et
al. in 19972 and 20003 revised the computational cost estimates and predicted that full-wing LES would not be available for engineering
use until 2045. Most recently, Choi and Moin4 revisited Chapman’s estimate applying the analysis of Spalart to show that the required
resolution for wall-modeled LES in the turbulent portion of a boundary layer flow (i.e., after transition) scales asymptotically with Reyn-
olds number, that is the number of grid points N ~ ReL (Chapman had estimated N ~ ReL2/5).
A potential concern is that these estimates ignore the cost of the laminar and transitional region of the boundary layer. In fact, because
this region is significantly thinner than the turbulent boundary layer (even though it is generally a much smaller fraction of the chord), the
computational cost may be non-negligible. To be precise, we follow Spalart, et .al. 1997 and count the number of cubes of volume  ,
3

where  is the boundary layer thickness. We consider both the laminar (including transition) and turbulent region of a boundary layer
on a unit aspect ratio NACA 0012 wing. The flow is modeled using the 2D coupled integral boundary layer method of Drela5 with transi-
tion estimated using an eN method (Ncrit = 9). The table below shows that the number of cubes in the laminar region is 10 to 100 times
larger than in the turbulent region. Thus, we conclude that a key issue in the application of WMLES will be the modeling of the laminar
and transitional region.
Rec Nlamcubes Nturbcubes Ncubes (Total)

255606.004.ppt
1e6 1.1e6 1.3e4 1.1e6
1e7 1.1e7 1.5e5 1.1e7
1e8 9.1e7 3.1e6 9.4e7

We can estimate the performance of WMLES on HPC in 2030. We base this estimate on existing second-order accurate finite volume
and finite difference discretizations with explicit time integration. While clearly other options exist, in particular higher-order methods, this
combination is representative of the class of algorithms currently being applied throughout aerospace CFD on LES and DES simula-
tions. Thus, we are making estimates based solely on how increased computational power will impact the ability to perform WMLES
simulations. Specifically, we make the following assumptions:

 The mesh is an isotropic refinement of the boundary layer cubes with n points in each direction (and thus n3 unknowns in a single
cube). In this example, we choose n=20.
 The timestep of the explicit method is equal to hmin / a  where hmin   min / n and a  is the freestream speed of sound.
 The number of floating point operations per timestep per point is Citer. In this example, we choose Citer=1250.
 The time integration is performed over CT convective timescales. In this example, we choose CT =100.

The table below shows the petaFLOP/s required to achieve a 24-hour turnaround for Mach 0.2 flow around a unit aspect-ratio geometry
(estimates for high aspect ratio wings can be obtained by scaling by the desired aspect ratio). We note that the FLOP cost scales with
approximately ReL1.3, which is due to ~ReL for gridding requirements and ReL1/3 for timestep requirements. Estimates for wall-resolved
LES4 show gridding requirements that scale with ReL13/7 which gives FLOP costs scaling with ReL2.5.
Rec Ndof Niter FLOP PFLOP/s
255606.005.ppt

1e6 9.0e9 4.6e7 5.2e20 6

1e7 8.5e10 1.5e8 1.6e22 180
1e8 7.5e11 4.6e8 4.3e23 5,000

We can then compare these estimates to existing HPC capability as well as estimated capability in 2030. At present, the world’s top
HPC machine is Tianhe-2, a supercomputer developed by China’s National University of Defense Technology, with a theoretical peak
performance of 55 PFLOP/s (and an actual achieved performance of 34 PFLOP/s on the Linpack benchmark). Thus, by today’s capabil-
ity, wall-modeled LES is feasible in a 24-hour turnaround time at Reynolds number of about 1 million on unit-aspect ratio geometries us-
ing existing algorithms. Looking ahead to 2030, the leadership class HPC machine is estimated to have a theoretical peak performance
of about 30 exaFLOP/s (see Appendix A). Thus, by 2030, we could expect to perform these types of calculations on the leadership HPC
machine.

13
 Additional conclusions based on these results are:
 For the higher aspect ratios that are more relevant to external flows, the costs will be an order of magnitude larger and thus out of the
reach of even 2030 leadership HPC machines at the high Reynolds numbers of interest.
 At lower Reynolds numbers, the cost differences between wall-modeled and wall-resolved LES disappear. Thus, for lower Reynolds
number applications, e.g., in some components of turbomachinery, wall-resolved LES is feasible on leadership class machines today.6
 As additional complexities are introduced, e.g., the full geometric complexity of a turbomachinery or a high-lift configuration, the cost will
increase further.
 The cost estimate assumes that the laminar and transition regions are simulated using the same resolution (per cube) as in the turbulent
region, i.e., the transition process is simulated. If the transition process could instead be modeled so that the grid resolution was essen-
tially that required for steady laminar boundaries, then the cost of the laminar and transition region would become negligible compared to
the turbulent region and reduces the above cost estimates by a factor of 10 to 100.
 The wall modeling for this type of LES is a current weakness that could limit the reliability of using this approach for separation predic-
tion.
 While these types of LES calculations may be feasible on leadership class HPC machines, engineering CFD calculations are not often
pursued at this level of parallelism. Rather, engineering CFD calculations tend to be performed with 1,000s and rarely 10,000s of com-
pute nodes. Thus, to realize these capabilities will require effort to exploit existing and future HPC performance.
 The potential impact of algorithmic work could be significant. For example, a 10-fold improvement due to algorithmic performance (e.g.,
through adaptivity, or higher-order discretizations, or improved solution algorithms) could bring these 24-hour calculations down to a few
hours. Further, this could relieve some of the pressure on wall modeling and transition modeling by facilitating increased grid resolution
(or improved accuracy at less cost), and head toward wall-resolved LES.
1. Chapman, D. R., “Computational Aerodynamics Development and Outlook,” AIAA J. 17, 1293 (1979)
2. Spalart, P. R., Jou, W.-H., Strelets, M., & Allmaras, S. R., “Comments on the feasibility of LES for wings, and on a hybrid RANS/LES approach” (invited). First
AFOSR Int. Conference on DNS/LES, Aug. 4-8, 1997, Ruston, Louisiana. (In “Advances in DNS/LES,” C. Liu & Z. Liu Eds., Greyden Press, Columbus, OH).
3. Spalart, P. R., “Strategies for turbulence modeling and simulations,” Int. J. Heat Fluid Flow 21, 252-263., 2000.
4. Choi, H., and Moin, P., “Grid-point requirements for large eddy simulation: Chapman’s estimates revisited,” Phys. Fluids 24, 011702 (2012)
5. Mueller, T. J. (ed.), Low Reynolds Number Aerodynamics, Lecture Notes in Engineering, Volume 54, 1989, pp 1-12.
6. Tucker, P. G., “Computation of unsteady turbomachinery flows: Part 2-LES and hybrids”, Progress in Aerospace Sciences. ISSN 0376-0421, 2011

3. Effects of grid resolution and solution scheme in as- iments, more specific test data from complex, integrated
sessing turbulence models. A key gap in the effective- flow fields using geometries that are more representative
ness of current and future turbulence models is the effect of complex aerospace systems is needed. Impediments
of grid resolution and solution scheme on both the accu- include balancing test cost and model complexity, diffi-
racy and convergence properties of the models. Studies culty in designing experiments, geometry deviations,
show that adequate grid resolution is required to capture measurement detail, and accuracy of CFD.
the full range of turbulence structures in models ranging 6. Robust transition prediction capability. Boundary
from simple eddy-viscosity formulations to full LES and layer transition is not well predicted (if at all) in CFD
DNS simulations.55 Additionally, choice of solution practice, impacting wind tunnel to flight scaling, laminar
scheme may be important when using marginal grid res- flow prediction and control, turbomachinery design, and
olution for complex geometries. Much work has been hypersonic transition/heating analysis, among others.
performed on building-block geometries,23,65 but real- Transition modeling for lower Reynolds number appli-
world cases are now too complex to assess full grid con- cations is particularly lacking, with specific impact on
vergence.56 high bypass ratio turbomachinery and for the lower
4. Insufficient use of foundational validation/ calibra- Reynolds number vehicles being designed today. Cur-
tion datasets to drive physics-based improvements to rently, en methods are difficult to use. However, there
turbulence prediction. Key experimental datasets are have been some novel and promising developments in
critically important in the ongoing development and re- transition prediction methods (e.g., the Langtry-Menter
finement of the full range of turbulence models from correlation-based model),50 but these partial-differential
RANS to LES. Typical impediments include test cost, equation (PDE) based methods (as opposed to en tech-
large number of cases needed, and instrumentation limi- niques) must be calibrated for a wide range of flow re-
tations. Moreover, many existing datasets are often not gimes and problems of interest, and should be viewed as
effectively exploited to use all available data in assessing in-development and somewhat risky. Still, these meth-
and improving models. ods are being improved and are propagating into both
5. Insufficient use of real-world experiments to validate government and commercial CFD codes72 even as they
turbulence models. In addition to building-block exper- (currently) do not account for the cross-flow mode of

14
 transition. Solid research is needed both on the PDE and (CAD) packages. However, there is currently no single
en tracks, with emphasis on both accuracy and ease of standard for representing surface or solid geometries
coupling with RANS codes. within CAD tools, complicating efforts to fully auto-
7. Lack of explicit collaboration among turbulence mate the link between mesh generation and geometry
researchers. There is a general lack of close coordina- definition. Furthermore, many existing CAD geometry
tion between turbulence modelers and researchers both definitions are poorly suited for CFD analyses, either
in the aerospace field itself (scattered amongst academ- due to insufficient accuracy (non water-tight geome-
ia, industry, and government), and between researchers tries often adequate for manufacturing purposes), or
in aerospace and related fields. In order to generate the due to excessive detail not essential for the CFD analy-
new ideas necessary to address the key issues of flow sis. This results in the need to incorporate specialized
separation and transition, it is imperative that a more post-processing tools such as shrink-wrapping in the
concerted effort be undertaken to connect members of former case, and/or de-featuring techniques in the latter
the aerospace turbulence community to others in case. At the same time, additional information such as
weather prediction, bio-fluids, and hydrodynamic slope and curvature or even higher surface derivatives
fields. may be required for the generation of curved mesh el-
ements suitable for use with higher-order accurate CFD
5.3 Autonomous and Reliable CFD Simulation discretizations.73,74 Finally, for adaptive meshing pur-
Today, most standard CFD analysis processes for the simu- poses, tight coupling between the CFD software and
lation of geometrically complex configurations are onerous, geometry definition is required to enable low-overhead,
both in terms of cycle time and process robustness. Even on-demand, geometry surface information queries
for simpler configurations that are typically analyzed during within the context of a massively parallel computing
the conceptual design phase, full automation is essential in framework.75
order for a conceptual designer to effectively exploit the 2. Poor mesh generation performance and robustness.
capacity of high performance computers and physics-based Significant human intervention is often required in the
simulation tools. Based on feedback from the engineering mesh generation process due to lack of robustness. This
and scientific communities as determined through our CFD is evidenced by the inability of current mesh generation
Vision 2030 survey and workshop, the key issues related to software to consistently produce valid high-quality
CFD automation and reliability can be categorized into the meshes of the desired resolution about complex configu-
broad areas of mesh generation and adaptivity, discretiza- rations on the first attempt. Additionally, many current
tions, solvers and numerics, and error control and uncertain- mesh generation algorithms (e.g., advancing-front meth-
ty quantification. ods) do not scale appropriately on parallel computer ar-
Mesh generation and adaptivity chitectures, and most mesh generation software is either
run sequentially, or using a small number of computer
Today, the generation of suitable meshes for CFD simula- cores or processors. On the other hand, CFD solver
tions about complex configurations constitutes a principal technology has demonstrated very good scaling on mas-
bottleneck in the simulation workflow process. Often the sively parallel machines,76-78 and is demanding ever
mesh generation phase constitutes the dominant cost in larger meshes, which the mesh generation community is
terms of human intervention, and concerns about the cost finding increasingly difficult to deliver due both to
and reliability of mesh generation were raised repeatedly in memory and time constraints using desktop commodity
the survey and workshop. However, since a computational hardware. During the last decade or more, developments
mesh is merely a means to enable the CFD simulation, ul- in mesh generation software came from third-party
timately the mesh generation process should be invisible to commercial vendors and NASA investment in this area
the CFD user or engineer. Given a suitable geometry repre- has essentially evaporated. However, fundamental ad-
sentation and a desired level of solution accuracy, a fully vances in computational geometry and other areas will
automated meshing capability would construct a suitable be key to improving the reliability, robustness, and par-
mesh and adaptively refine this mesh throughout the solu- allel scalability of mesh generation capabilities, particu-
tion process with minimal user intervention until the final larly as larger simulations using finer meshes about more
accuracy levels are met. This process allows the user to complex geometries are sought. Additionally, paradigm
focus on the final solution without concern for the construc- shifts in meshing technology (i.e., cut cell methods,
tion and maintenance of the underlying mesh. Achieving strand grids, meshless methods) may lead to revolution-
this vision of fully automated meshing requires overcoming ary advances in simulation capabilities.79-82
various important current impediments.
Discretizations, Solvers, and Numerics
1. Inadequate linkage with CAD. Configuration geome-
try definitions required by mesh generation software The core of an autonomous and reliable CFD capability
are generally provided by computer-aided design must rely on efficient and robust discretization and solution

15
strategies. Discretizations must be tolerant of localized poor 1. Incomplete or inconsistent convergence behavior.
mesh quality while at the same time be capable of deliver- Most current CFD codes are capable of producing fully
ing high accuracy at low cost. Solution techniques must be converged solutions in a timely manner for a variety of
scalable, efficient, and robust enough to deliver converged simple flow problems. However, most often these same
solutions under all reasonable conditions with minimal user tools are less reliable when applied to more complex
intervention. One of the principal concerns raised through- flow fields and geometries,56,85 and may fail, or require
out our survey and workshop was the high level of expertise significant user intervention to obtain adequate results.
and human intervention often required for performing and There are many possible reasons for failure, ranging
understanding CFD analyses, with a consensus that a prin- from poor grid quality to the inability of a single algo-
cipal requirement for relieving this dependency will require rithm to handle singularities such as strong shocks, un-
added investment in basic numerical methods research. der-resolved features, or stiff chemically reacting terms.
Current gaps and impediments with numerical methods What is required is an automated capability that delivers
include:

CASE STUDY 3: Scalable Solver Development

The development of optimal solvers has been central to the success of CFD methods since the early days of numerical simulation, for
both steady-state and time-implicit problems. Optimal solvers are defined as methods that are capable of computing the solution to a
problem with N unknowns in O(N) operations. Because the number of unknowns in industrial CFD problems is most often very large
(106 < N < 109), optimal solvers offer the potential for orders of magnitude increase in solution efficiency compared to simple iterative
solvers, which most often scale as O(N2) or higher.
Multigrid methods constitute the most successful and widely used optimal solvers for CFD problems. These methods were developed
for CFD applications at an early stage, with considerable NASA investment. In the late 1970s, joint NASA collab-
orative work with academic leaders in multigrid solver technology produced some of the first successful multigrid
solvers for potential flow methods1, followed by efficient multigrid solvers for the Euler equations2, and the Na-
vier-Stokes equations. The success was such that multigrid methods were implemented and used in virtually all
important NASA CFD codes, including TLNS3D, CFL3D, OVERFLOW, and more recently FUN3D. Multigrid
methods have become essential solver components in commercial production codes such as Fluent and
STARCCM+, and have received particular attention within the DOE where they are used in various large-scale
production codes. Despite their early success, many impediments remain for successfully extending these solv-
ers to larger and more complex problems. While most early NASA investment focused on geometric multigrid for structured meshes, ex-
tending these solvers to complex geometry CFD problems or even abstract matrix inversion problems requires the development of alge-
braic multigrid methods (AMG). At the same time, improvements to current multigrid strategies are required if these methods are to
scale effectively on emerging massively parallel HPC hardware. Although NASA investment in further research on multigrid methods
has stalled since the early 1990s, considerable research has been directed toward developing more optimal AMG solvers designed for
use on petascale and exascale hardware within the DOE. For example, the Scalable Linear Solver group at Lawrence Livermore Na-
tional Laboratory has developed parallel AMG technology and related methods such as Smoothed Aggregation techniques that maintain
optimal solver qualities while delivering vastly improved scalability on massively parallel machines. Current capabilities include the
demonstration of the solution of a problem involving over 1012 degrees of freedom with good scalability on over 100,000 cores.3 Alt-
hough these solvers are publicly available, they have not drawn the interest of the aerospace CFD community, and will likely require
considerable investment to modify and extend to production aerospace CFD problems.
Multigrid method developments are often reported at dedicated multigrid specialist conferences. For example, the first successful multi-
grid solution of the Euler equations was reported at the 1983 Copper Mountain Multigrid Methods conference.2 This conference series
was traditionally well attended by NASA participants and, as recently as 1996, the conference proceedings were edited and published
by NASA Langley4. However, during the last decade there has been virtually no NASA presence at these conferences. This has been
accompanied by a significant decline of scalable solver papers published in AIAA venues, while NASA CFD codes have remained con-
fined to the same multigrid technology that was developed in those early years.
1. South, J. C. and Brandt, A., “Application of a Multi-level Grid Method to Transonic Flow Calculations,” in Transonic Flow Problems in Turbomachinery, (Adam, T.C.
and Platzer, M.F. eds.) Hemisphere, Washington, pp 180-207, 1977
2. Jameson, A., “Solution of the Euler Equations for Two Dimensional Transonic Flow by a Multigrid Method,” Proceedings of International Multigrid Conference, Cop-
per Mountain, April 1983, Applied Mathematics and Computation, Vol. 13, 1983, pp. 327-356.
3. Baker, A. H., Falgout, R. D., Kolev, Tz. V., and Yang, U. M., “Scaling Hypre’s Multigrid Solvers to 100,000 Cores,” in High Performance Scientific Computing: Algo-
rithms and Applications, M. Berry, et al., eds., Springer (2012). LLNL-JRNL-479591.
4. Melson, N. D., Manteuffel, T. A., McCormick, S. F., Douglas, C. C. (eds.), “Seventh Copper Mountain Conference on Multigrid Methods,” NASA CP 3339, Septem-
ber 1996.

16
 hands-off solid convergence under all reasonable antici- quantification raises the risk that engineering decisions are
pated flow conditions with a high tolerance to mesh ir- based on inaccurate and/or uncertain results. The Vision
regularities and small-scale unsteadiness. Reaching this 2030 survey highlighted the need for improvements in error
goal necessarily will require improvements in both dis- quantification. Furthermore, in terms of reliable and auto-
cretizations and solver technology,86 since inadequate mated CFD simulations, discretization error estimation is a
discretizations can permit unrealizable solutions, while key ingredient for the realization of a solution adaptive pro-
temperamental solvers may be unable to reach existing cess. Current error control and uncertainty quantification
valid solutions. Although incremental improvements to gaps and impediments include:
existing algorithms will continue to improve overall ca-
1. Limited use of existing error estimation and control
pabilities, the development of novel robust numerical
methods. Significant progress has been made in the es-
techniques such as monotone, positivity-preserving,
timation and control of discretization errors, in particu-
and/or entropy-preserving schemes87,88 and their exten-
lar in terms of output-based techniques.90-93 However,
sion to complex problems of industrial relevance offers
while these techniques were demonstrated by multiple
the possibility of radical advances in this area.
groups for steady 2D RANS94 and 3D inviscid flows,83
2. Algorithm efficiency and suitability for emerging the applications to 3D RANS and unsteady flows were
HPC. In previous decades, NASA has invested heavily limited, in particular for complex geometries. These
in numerics and solver technology, and it is well doc- more complex applications have been severely imped-
umented that equivalent advances in numerical simula- ed by the inadequacies of 3D anisotropic and time-
tion capability have been enabled through the devel- dependent adaptive meshing, as well as by poor ro-
opment of more efficient algorithms compared to ad- bustness of current discretization and solution algo-
vances in HPC hardware.1,16-18 However, during the last rithms (i.e., to be able to solve flow and adjoint equa-
decade, algorithmic investment has been dramatically tions on potentially poor quality meshes during the
curtailed, with the result that the many flow solvers in adaptive process).
use today were developed more than 20 years ago and
2. Inadequacy of current error estimation techniques.
are well known to be suboptimal. Because solver opti-
While discretization error estimation techniques for
mality is an asymptotic property, as larger simulations
outputs have improved during the past 10 years, these
are attempted, the potential benefits of better solvers
techniques do have fundamental limitations that could
grow exponentially, possibly delivering orders of mag-
influence their application to increasingly complex
nitude improvement by the exascale computing
problems. In particular, output-based error estimation
timeframe. At the same time, the drive to more com-
techniques are based on linearizations about existing
plex flows (including more complex turbulence mod-
(approximate) solutions and as a result can have signif-
els, stiff chemically reacting terms, or other effects)
icant error when the flows are under resolved (even in
and tightly coupled multidisciplinary problems will re-
the case of a linear problem, the techniques generally
quire the development of novel techniques that remain
only provide error estimates and are not bounds on the
stable and efficient under all conditions. Finally, exist-
error). Furthermore, for unsteady, chaotic flows (which
ing numerical techniques were never conceived with
will be a key phenomenon of interest as turbulent DES
massive parallelism in mind, and are currently unable
and LES simulations increase in use moving forward)
to capitalize on the emerging massively parallel and
linearized analysis will produce error estimates that
heterogeneous architectures that are becoming the
grow unbounded with time (due to the positive Lya-
mainstay of current and future HPC. In order to im-
punov exponent for chaotic flows).95,96 In these situa-
prove simulation capability and to effectively leverage
tions, existing output-based methods will be swamped
new HPC hardware, foundational mathematical re-
by numerical error, rendering the sensitivity infor-
search will be required in highly scalable linear and
mation meaningless. This issue will affect not only er-
nonlinear solvers not only for commonly used discreti-
ror estimation but also design optimization moving
zations but also for alternative discretizations, such as
forward.
higher-order techniques.89 Beyond potential advantages
3. Limited use of uncertainty quantification. The con-
in improved accuracy per degree of freedom, higher-
sideration of uncertainty due to parametric variability as
order methods may more effectively utilize new HPC
well as modeling error raises significant challenges. Var-
hardware through increased levels of computation per
iability and uncertainty of inputs (boundary and initial
degree of freedom.
conditions, parameters, etc.) to fluid dynamic problems
are largely unquantified. Even if estimates are available
Error control and uncertainty quantification
and/or assumed, the propagation of these uncertainties
Errors in current CFD simulations are not well understood poses a significant challenge due to the inherent cost, the
or well quantified, including errors due to spatial and tem- lack of automation and robustness of the solution pro-
poral discretization, incomplete convergence, and the phys- cess, and the poor utilization of high performance com-
ical models and parameters they embody. The lack of error

17
 puting. Even more challenging is the quantification of real time.39 This is complicated by a lack of data stand-
modeling error. This will likely require significantly ards which makes collection and analysis of results
more expensive methods (e.g., based on Bayesian ap- from different codes, researchers and organizations dif-
proaches).97 While uncertainty quantification is being ficult, time consuming and prone to error. At the same
investigated in the broad research community, most no- time, there are no robust, effective techniques for distil-
tably through DOE and NSF led programs,98 the engi- ling the important information contained in large col-
neering community, and the aerospace community in lections of CFD simulation data into reduced-order
particular, have had minimal investments to address models or meta-models that can be used for rapid pre-
these issues. dictive assessments of operational scenarios, such as
the correlation of flow conditions with vehicle perfor-
5.4 Knowledge Extraction and Visualization mance degradation or engine component failures, or
An integral part of effectively using the advanced CFD assessments of engineering tradeoffs as required in typ-
technology envisioned in 2030 is the way in which the very ical design studies.100
large amount of CFD-generated data can be harvested and 3. Merging of high fidelity CFD simulations with other
utilized to improve the overall aerodynamic design and aerodynamic data. With wind tunnel and flight-testing
analysis process. The resulting utilization of this data would still expected to play a key role in the aerospace system
include insight into pertinent flow physics, use with aero- design process, methods to merge and assimilate CFD
dynamic or multidisciplinary optimization, and generation and multidisciplinary simulation data with other multi-
of effective databases for a myriad of purposes, including fidelity experimental/computational data sources to
control law development, loads assessment, create an integrated database, including some measure
flight/performance simulation, etc. of confidence level and/or uncertainty of all (or indi-
A number of technology gaps and impediments in the area vidual) portions of the database, are required. Current-
of knowledge extraction for large-scale CFD databases and ly, the merging of large amounts of experimental and
simulations must be overcome to efficiently analyze and variable fidelity computational data is mostly carried
utilize CFD simulations in the 2030 time frame. out through experience and intuition using fairly unso-
phisticated tools. Well founded mathematically and sta-
1. Effective use of a single, high-fidelity CFD simula- tistically based approaches are required for merging
tion. As high-performance computing (HPC) systems such data,101-103 for eliminating outlier numerical solu-
become faster and more efficient, a single unsteady CFD tions, as well as experimental points, and for generally
simulation using more complicated physical models quantifying the level of uncertainties throughout the
(e.g., combustion) to solve for the flow about a complete entire database in addition to at individual data
aerospace system (e.g., airplane with full engine simula- points.104
tion, space vehicle launch sequence, aircraft in maneu-
vering flight, etc.) using a much higher number of grid 5.5 Multidisciplinary/Multiphysics Simulations
points (~10 to 100 billion) will become commonplace in and Frameworks
the 2030 time frame. Effective use (visualization and in-
We also assume that CFD capabilities in 2030 will play a
situ analysis) of these very large, single, high-fidelity
significant role in routine, multidisciplinary analysis
CFD simulations will be paramount.99 Similarly, higher-
(MDA) and optimization (MDAO) that will be typical of
order methods will likely increase in utilization during
engineering and scientific practice. In fact, in 2030 many of
this time frame, although currently the ability to visual-
the aerospace engineering problems of interest will be of a
ize results from higher order simulations is highly inade-
multidisciplinary nature and CFD will have to interface
quate. Thus, software and hardware methods to handle
seamlessly with other high-fidelity analyses including
data input/output (I/O), memory, and storage for these
acoustics, structures, heat transfer, reacting flow, radiation,
simulations (including higher-order methods) on emerg-
dynamics and control, and even ablation and catalytic reac-
ing HPC systems must improve. Likewise, effective
tions in thermal protection systems. With increasingly
CFD visualization software algorithms and innovative
available computer power and the need to simulate com-
information presentation (e.g., virtual reality) are also
plete aerospace systems, multidisciplinary simulations will
lacking.
become the norm rather than the exception. However, effec-
2. Real-time processing and display of many high- tive multidisciplinary tools and processes are still in their
fidelity CFD simulations. By the year 2030, HPC ca- infancy.
pabilities will allow for the rapid and systematic gener-
ation of thousands of CFD simulations for flow physics Limitations on multidisciplinary analyses fall under various
exploration, trend analysis, experimental test design, categories including the setup and execution of the anal-
design space exploration, etc. The main goal, therefore, yses, the robustness of the solution procedures, the dearth
is to collect, synthesize, and interrogate this large array of formal methodologies to guarantee the stability and accu-
of computational data to make engineering decisions in racy of coupled high-fidelity simulations, and the lack of

18
existing standards for multidisciplinary coupling. The result Such automation and robustness characteristics provide
tends to be one-off, laborious, and nonstandard interfaces the foundation for more complex problems that require
with other disciplines with dubious accuracy and stability. the solution of multidisciplinary simulations.
Multidisciplinary optimizations inherit all of these limita- 2. The science of multidisciplinary coupling at high
tions and suffer from additional ones of their own, includ- fidelity. Exchanges of information between Vision
ing the inability to produce accurate discipline and system- 2030 CFD and other disciplinary solvers with which
level sensitivities, the lack of quantified uncertainties in the CFD will need to interact will require assurances of
participating models, the lack of robustness in the system- both accuracy and stability. Such properties often re-
level optimization procedures, as well as very slow turna- quire the satisfaction of conservation principles to
round times. which close attention must be paid.105 Moreover, with-
The vision for 2030 MDA/O involves the seamless setup in the context of nonlinear phenomena and unsteady
and routine execution of both multidisciplinary analyses flows, the proper interfacing between CFD and other
and optimizations with codes requires significant effort and can be hard to
generalize. The development of libraries and proce-
 Rapid turnaround (hours for MDA and less than a day dures that enable high fidelity, accurate, and stable
for MDO). couplings, regardless of the mesh topologies and char-
 User-specified accuracy of coupled simulations. acteristic mesh sizes must be pursued. Such software
 Robustness of the solution methodology. may also need to be cognizant of the discretization de-
 Ability to provide sensitivity and uncertainty infor- tails of the CFD solver. Ultimately, solvers using dis-
mation. cretizations of a given accuracy (in space and in time),
 Effective leveraging of future HPC resources. when coupled to other solvers, must ensure that the ac-
curacy of the component solvers is preserved and that
For this vision to proceed forward, the development of mul- the coupling procedure does not give rise to numerical
tidisciplinary standards will be necessary, as well as the errors that may manifest themselves through solution
creation of coupling frameworks that facilitate the multidis- instabilities.106
ciplinary interactions envisioned here. Moreover, key re- 3. Availability of sensitivity information and propaga-
search challenges in multidisciplinary coupling, computa- tion of uncertainties. Vision 2030 CFD is expected to
tion of system-level sensitivities, management of uncertain- interact with other solvers (for different disciplines) in
ties in both the analyses and optimizations, hierarchical multidisciplinary analyses and optimizations. In 2030,
decomposition of the optimization problems, and both au- the state of the art is presumed to include the quantifi-
tomation and standardization processes will need to be cation of uncertainties (UQ), at the system level, aris-
overcome. ing from uncertainties in each of the participating dis-
More specifically, a number of technology gaps and imped- ciplines. In order to facilitate both optimization and UQ
iments must be tackled to enable truly multidisciplinary at the system level, Vision 2030 CFD must be able to
analyses and optimizations in the 2030 timeframe. In this provide sensitivities of multiple derived quantities of
report, we focus on the requirements that affect 2030 CFD. interest with respect to large numbers of independent
Although, by extension, we also discuss some more general parameters at reasonable computational cost. Novel
gaps and impediments that are likely to affect our vision. techniques for the propagation of uncertainties will
need to be embedded into 2030 CFD for more compre-
1. Robustness and automation of CFD analyses in hensive treatment of UQ problems. Moreover, the sup-
multidisciplinary environments. To ensure that 2030 port for system-level sensitivity information and UQ
CFD can be an integral part of routine multidiscipli- will demand the availability of derivative and UQ in-
nary, multiphysics simulations, the manpower cost re- formation related to outputs of CFD that may be uti-
quired to set up and execute such calculations must be lized by other solvers. Ensuring that these capabilities
drastically reduced. Firstly, the setup of high-fidelity are present in our Vision 2030 CFD will permit ad-
multidisciplinary analyses must be largely automated vanced analyses, optimizations, and UQ to be carried
including all operations involving surface and volume out.
grid transfers and interpolations, grid deformations 4. Standardization and coupling frameworks. Owing
and/or re-generation, information exchanges, and map- to the multitude of disciplinary solvers available for
pings to HPC environments. Secondly, the execution of coupling with 2030 CFD, the uncertainty regarding the
multiphysics simulations that involve Vision 2030 actual code structure, HPC solver architecture, and in-
CFD must include appropriate measures to ensure the ternal solution representation, it is fundamental to en-
robustness of the solution procedure, protecting against sure that multidisciplinary simulation standards (such
coupled simulation failure and including the on- as the CGNS standard107 created for CFD) are created
demand availability of all necessary modules in the so that a variety of solvers can participate in multidis-
CFD chain so that CFD failure modes are protected. ciplinary analyses and optimizations. Beyond the typi-

19
 cal codification of the inputs and outputs of a particular and shock-boundary layer interaction. Clearly, HPC ad-
physical simulation, standards for MDAO may need to vances alone will not be sufficient to solve this GC prob-
include sensitivities, uncertainties, and overall descrip- lem and improvements in algorithmic technologies or other
tions of the parameterization (possibly including the unforeseen developments will be needed to realize this
geometry itself) and the optimization problem. To ena- goal. Progress toward this goal can be measured through
ble tight coupling of diverse disciplines and codes, the the demonstration of effective hybrid RANS-LES and
data standards need to extend to include memory resi- wall-modeled LES simulations with increasing degrees of
dent information and coding structures.57,59,60 modeled versus resolved near-wall structures with increas-
ing geometric complexity. Fully optimized flow solvers
6 Technology Development Plan running on exascale computing platforms will also be criti-
cal.
To achieve our vision of CFD in 2030 and directly address
the key CFD technology shortcomings and impediments Grand Challenge Problem 2:
that currently limit the expanded use of CFD methods with-
in the aerospace analysis and design process, a comprehen- Off-design turbofan engine transient simulation. This
sive CFD development plan has been developed and is pre- case encompasses the time-dependent simulation of a
sented in this section. In order to place future technology complete engine including full-wheel rotating components,
developments within the context of our Vision for 2030 secondary flows, combustion chemistry, and conjugate
CFD, we first describe in more detail a number of Grand heat transfer. This GC will enable virtual engine testing
Challenge (GC) problems that embody the goals for CFD in and off-design characterization including compressor stall
2030. Next, a comprehensive roadmap that depicts key and surge, combustion dynamics, turbine cooling, and en-
technology milestones and demonstrations needed to sup- gine noise assessment. Similar to GC 1, demonstration of
port the GC simulations is introduced and described. An advances in accurate prediction of separated flows, com-
integrated research plan is then proposed. Finally, an over- plex geometry, sliding and adaptive meshes, and nonlinear
all research strategy with specific recommendations for unsteady flow CFD technologies will be required to
executing the plan to advance the state of the art in CFD achieve this goal. In addition, advances in the computation
simulation capability is provided. of flows of widely varying time scales, and the predictive
accuracy of combustion processes and thermal mixing, will
6.1 Grand Challenge Problems be necessary.
The intent of the GC problems is to drive the identification Grand Challenge Problem 3:
and solution of the critical CFD barriers that would lead to a
desired revolutionary CFD capability. We purposely have MDAO of a highly flexible advanced aircraft configu-
chosen GC problems that are bold, recognizing that they may ration. The increased level of structural flexibility that is
not be routinely achievable by 2030, but, if achieved, would likely to be present in future commercial aircraft configu-
represent critical step changes in engineering design capabil- rations (of the N+3 and N+4 types envisioned by NASA
ity. To this end, the GC cases are chosen to encompass the and its partners) dictates a system-level design that re-
CFD capabilities required to design and analyze advanced air quires the tight coupling of aerodynamics, structures, and
and space vehicles and systems in 2030, and represent im- control systems into a complete aero-servo-elastic analysis
portant application areas of relevance to the various NASA and design capability. This GC problem focuses on the
aeronautics and space missions. Details on each of the four multidisciplinary analysis and optimization of such config-
GC problems are given below. urations including explicit aeroelastic constraints that may
require a time-accurate CFD approach. In addition to the
Grand Challenge Problem 1: aero-servo-elastic coupling, this GC includes the integra-
tion of other disciplines (propulsion and acoustics) as well
LES of a powered aircraft configuration across the full as a full mission profile. The ultimate goal is to demon-
flight envelope. This case focuses on the ability of CFD to
strate the ability (in both MDA and MDAO) to perform
simulate the flow about a complete aircraft geometry at the
CFD-based system-level optimization of an advanced con-
critical corners of the flight envelope including low-speed
figuration that requires both steady and unsteady high-
approach and takeoff conditions, transonic buffet, and pos-
fidelity models.
sibly undergoing dynamic maneuvers, where aerodynamic
performance is highly dependent on the prediction of tur-
bulent flow phenomena such as smooth body separation

20
CASE STUDY 4: Impact of CFD Tool Development on NASA Science and Space Exploration Missions
Traditionally, the development of physics-based simulation tools for aerospace vehicle analysis and design has been the responsibility
of NASA’s Aeronautics Research Mission Directorate (ARMD), with an emphasis on solving ARMD’s aeronautics goals. However,
NASA’s science and space exploration missions rely heavily on simulation tools. CFD has played a critical role in virtually all recent past
and present NASA space vehicle programs, including Space Shuttle return to flight,1 entry-descent and landing (EDL) predictions for the
entire series of Mars landings,2 support for the recent Constellation Program3 and, more recently, for the Multipurpose Crew Vehicle
(MPCV) and the Space Launch System (SLS) programs. Throughout many of these efforts, limitations in numerical simulation tools for
space vehicle design were uncovered, requiring expensive contingency planning. For example, the Ares I development, which employed
a combination of wind-tunnel testing and CFD methods for aerodynamic database generation, found that CFD was often less reliable
and more expensive than experimental testing, resulting in limited use of CFD principally in specific regions of the flight envelope where
testing was not feasible.4 Similar conclusions were drawn by engineers developing the MPCV Orion aerodynamic database.
In a recent yearly review of NASA’s Aerosciences State-of-the-Discipline, NASA’s technical fellow for Aerosciences has assessed the
ability of the discipline to support NASA’s missions, and identified these top three technical challenges:
 Prediction of unsteady separated flows
 Aero-plume interaction prediction
 Aerothermal predictions

Accurate prediction of unsteady separated flow is critical in the design of launch vehicle sys-
tems, where buffet and aeroacoustic fluctuating pressures in the transonic regime during as-
cent can result in high vibrational environments. Currently, launch vehicle buffet environments
are obtained almost exclusively through wind tunnel testing and correlation with empirical data
at considerable expense and uncertainty, resulting in overly conservative structural mass and
reduced payload to orbit. Advanced simulation techniques such as DES are beginning to be
explored but have been found to be expensive and to require further refinement and validation.
Similarly, quantifying the aeroacoustic environment in launch vehicle design due to separated
flows and aero-plume interactions is an important consideration for flight qualification of vehicle
electronic components and passenger safety. Previous vehicle programs such as Ares I have
incurred considerable expense for the experimental determination of aeroacoustic environ-
ments, while investigations by NASA have determined that current CFD techniques are inadequate for the prediction of launch vehicle
aeroacoustic environments.5 However, the largest payoff in launch vehicle design would come from the use of CFD as a dynamic flight
simulation capability, rather than a static aerodynamic database generation tool, as is currently the case, although little effort is being
targeted toward this area. Accurate prediction of separated flows is also an important consideration for spacecraft EDL, which is com-
pounded by the need for accurate aerothermal predictions, which in turn are hindered by the need for reliable transition prediction and
the inclusion of other multiphysics considerations such as radiation and ablator performance. Accurate simulation of the aero-plume in-
teractions of the reaction-control systems for bluff body re-entry is another area where the development of accurate simulation capabili-
ties could reduce cost and uncertainties associated with complex experimental campaigns. Finally, the design and validation of space-
craft decelerators, including high-speed parachutes and deployable decelerators would benefit enormously from the development of a
reliable, nonempirical simulation capability, although this represents a complex nonlinear aero-structural problem with massive flow sep-
aration that is well beyond current capabilities.

Clearly, there is a need for better simulation tools within NASA’s science and space exploration missions, as well as within aeronautics
itself. Furthermore, many of the technological barriers are similar in both areas, such as the inability to accurately simulate separated
flows and transition and the need to harness the latest HPC hardware, while other issues are more critical to the space mission such as
aero-plume and aerothermal prediction capabilities. To overcome these deficiencies, increased coordination will be required between
NASA’s science and space exploration programs, which are driving these requirements, and NASA aeronautics, where much of the ex-
pertise for simulation method development resides. In the place of the current approach, which relies on the periodic assessment of ex-
isting simulation tools, a longer term outlook that invests in new simulation capability development for specific space programmatic ob-
jectives must be adopted.
1 Gomez, R. J., Aftosmis, M. J., Vicker, D., Meakin, R. L., Stuart, P. C., Rogers, S. E., Greathouse, J. S., Murman, S. M., Chan, W. M., Lee, D. E., Condon, G. L., and
Crain, T., Columbia Accident Investigation Board (CAIB) Final Report. Vol. II, Appendix D.8. Government Printing Office, 2003.
2 Edquist, K. T., Dyakonov, A. A., Wrightz, M. J., and Tang, C. Y., “Aerothermodynamic Design of the Mars Science Laboratory Backshell and Parachute Cone,” AIAA
paper, 2009-4078.
3 Abdol-Hamid, K. S., Ghaffari, F., and Parlette, E. B., “Overview of the Ares-I CFD Ascent Aerodynamic Data Development and Analysis based on USM3D,” AIAA
paper, 2011-15, 49th AIAA Aerospace Sciences Meeting and Exhibit, Orlando, FL, January 2011.
4 Malik, M. R. and Bushnell, D., (eds.), “Role of Computational Fluid Dynamics and Wind Tunnels in Aeronautics R&D,” NASA TP 2012-217602, September 2012.
5 “Independent Assessment of External Pressure Field Predictions Supporting Constellation Program Aeroacoustics (ITAR),” NASA Engineering and Safety Center
Report NESC-RP-07-040, September 2013.

21
 Grand Challenge Problem 4: perhaps maturing along multiple development paths, is as-
sessed to establish future viability and possible change in
Probabilistic analysis of a powered space access con- development and/or maturation strategy. Each individual
figuration. The goal of this case is to provide a complete timeline is colored by Technology Readiness Level (TRL)
description of the aerothermodynamic performance, in- in three levels: low (red), medium (yellow), and high
cluding reliable error estimates and quantified uncertainty (green). The TRL scale is used to indicate the expected
with respect to operational, material, and atmospheric pa- overall maturity level of each technology element at a spe-
rameters, for a representative space vehicle throughout its cific point in time. In general, many of the critical CFD
flight envelope. This capability will enable reliability technologies are currently at a relatively low TRL level, but
predictions and vehicle qualification in light of limited with proper research and development, mature to a high
availability of ground-based test facilities. Demonstration TRL level by 2030. Some of the CFD technologies must be
of advances in combustion modeling, off-design perfor- sequentially developed and, therefore, it is not expected that
mance, adaptive and dynamic overset meshing, unsteady all technologies will be at a high TRL in 2030. Specific
flow, hypersonic flow, CFD reliability, and reliability and details of the development plan for each technology ele-
uncertainty quantification is required. ment are given below.
Required research toward meeting these GCs is identified High Performance Computing (HPC). As mentioned pre-
in six areas, namely HPC, physical modeling, numerical viously, advances in HPC hardware systems and related
algorithms, geometry/grid generation, knowledge extrac- computer software are critically important to the advance-
tion, and MDAO, and is used to formulate the overall re- ment of the state of the art in CFD simulation, particularly for
search plan. In order to evaluate the progress of each in- high Reynolds turbulent flow simulations. Based on feedback
dividual area of the research plan, technical milestones from the user survey and from discussions during the CFD
and demonstrations are formulated with notional target workshop, we envision HPC technology advancing along
dates. While these provide a measure of progress in the two separate paths.
individual technology roadmap domains, the capability of
these combined technologies toward meeting the stated Ongoing development of exascale systems, as mentioned
GC problems must also be evaluated periodically and earlier, will continue through 2030, and represents the tech-
used to prioritize research thrusts among the various nology that will most likely provide the large increase in
technology areas. throughput for CFD simulation in the future.42,108 However,
novel technologies, such as quantum computing or molecu-
6.2 Technology Roadmap lar computing, offer a true paradigm shift in computing
The CFD technology roadmap (presented in Figure 1) is a potential and must be carefully considered at strategic
complete and concise view of the key research technologies points in the overall development plan, even though the
and capabilities that must be developed, integrated into pro- technology is at a very low TRL level today. In order to
duction CFD tools and processes, and transitioned to the aer- properly address the HPC challenge, three specific thrusts
ospace CFD user community to achieve our vision of CFD in must be supported. Firstly, current simulation software
2030. The individual elements on the roadmap were identi- must be ported to evolving and emerging HPC architectures
fied based on the results of the CFD user survey, detailed with a view toward efficiency and software maintainability.
technical discussions held during the Vision 2030 CFD Secondly, investments must be made in the development of
workshop, and from interactions among our team members. new algorithms, discretizations, and solvers that are well
suited for the massive levels of parallelism and deep
Key technology milestones, proposed technology demonstra- memory architectures anticipated in future HPC architec-
tions, and critical decision gates are positioned along time- tures.39,41 Finally, increased access to the latest large-scale
lines, which extend to the year 2030. Separate timelines are computer hardware must be provided and maintained, not
identified for each of the major CFD technology elements only for production runs, but also for algorithmic research
that comprise the overall CFD process. The key milestones and software development projects, which will be critical
indicate important advances in CFD technologies or capabili- for the design and validation of new simulation tools and
ties that are needed within each technology element. Tech- techniques.11, 45 We propose several key milestones that
nology demonstrations are identified to help verify and vali- benchmark the advances that we seek: modification of
date when technology advances are accomplished, as well as NASA and related CFD codes to efficiently execute on hi-
to validate advances toward the simulations of the GC prob- erarchical memory (GPU/co-processor) systems by 2020,
lems identified above. The technology demonstration (TD) initial evaluation of exascale performance on a representa-
entries are linked by black lines in instances when a given tive CFD problem, and a demonstration of 30 exaFLOP
TD can be used to assess CFD advances in multiple areas. performance for one or more of the proposed GC problems
Critical strategic decision gates are identified where appro- in the 2030 time frame.
priate to represent points in time where specific research,

22
TRL LOW Technology Milestone Technology Demonstration Decision Gate
MEDIUM
HIGH 2015 2020 2025 2030
Demonstrate implementation of CFD Demonstrate efficiently scaled 30 exaFLOPS, unsteady,
algorithms for extreme parallelism in CFD simulation capability on an maneuvering flight, full engine
HPC NASA CFD codes (e.g., FUN3D) exascale system simulation (with combustion)
CFD on Massively Parallel Systems
PETASCALE Demonstrate solution of a NO NO EXASCALE
representative model problem
CFD on Revolutionary Systems
YES YES
(Quantum, Bio, etc.)

Improved RST models

RANS in CFD codes Highly accurate RST models for flow separation

Unsteady, complex geometry, separated flow at

Hybrid RANS/LES NO flight Reynolds number (e.g., high lift)
Physical Modeling Integrated transition
WMLES/WRLES for complex 3D flows at appropriate Re
LES prediction

Chemical kinetics Unsteady, 3D geometry, separated flow

Chemical kinetics
in LES (e.g., rotating turbomachinery with reactions)
Combustion calculation speedup

Multi-regime
Grid convergence for a Production scalable
turbulence-chemistry
Automated robust solvers complete configuration entropy-stable solvers
Convergence/Robustness interaction model

Algorithms Scalable optimal solvers

Uncertainty Quantification (UQ) Large scale stochastic capabilities in CFD

Characterization of UQ in aerospace Reliable error estimates in CFD codes Uncertainty propagation

Large scale parallel capabilities in CFD
Automated in-situ mesh
Fixed Grid Tighter CAD coupling mesh generation
Geometry and Grid with adaptive control

Generation Adaptive Grid Production AMR in CFD codes

Creation of real-time multi-fidelity database: 1000 unsteady CFD
Simplified data
simulations plus test data with complete UQ of all data sources
representation
Integrated Databases
Knowledge Extraction
Visualization On demand analysis/visualization of a On demand analysis/visualization of a
10B point unsteady CFD simulation 100B point unsteady CFD simulation

Define standard for coupling

to other disciplines Incorporation of UQ for MDAO
MDAO
High fidelity coupling Robust CFD for MDAO simulation of an entire UQ-Enabled MDAO
techniques/frameworks complex MDAs aircraft (e.g., aero-acoustics)

Figure 1. Technology Development Roadmap

23
Concurrently, we stress the importance of closely observing we envision continued investment in RST models to 2020,
advances in revolutionary HPC technologies, such as super- including careful implementation, verification, and validation
conducting logic, new memory technologies, alternatives to of the most promising variants of these models into research
current Complementary Metal Oxide Semiconductor and production CFD codes, including hybrid RANS/LES
(CMOS) technologies with higher switching speeds and/or codes. In the 2020 time frame, a comprehensive assessment
lower power consumption (specifically for graphene, car- of the ability of these models to predict flow separation
bon nanotubes, and similar developments), quantum com- would be enabled to determine whether or not further in-
puting and molecular or DNA computing. Because these vestment is warranted.
technologies are in their infancy, we foresee decision gates
Hybrid RANS/LES methods show perhaps the most promise
in 2020, 2025, and 2030 to establish the ability of these
in being able to capture more of the relevant flow physics for
systems to solve a relevant model problem61 (i.e., typical of
complex geometries at an increasingly reasonable computa-
a Poisson problem for PDE-based problems). Implicit in
tional cost.68,111 From the user survey, the majority of survey
this strategy is the need to provide access to experimental
participants ranked the continued development of hybrid
hardware on a continual basis and to explore radical new
RANS/LES methods as the top priority in the area of turbu-
approaches to devising CFD simulation capabilities. If, at
lence modeling. However, as mentioned previously, several
any of these decision points, the technology clearly shows
issues still exist. First, the prediction of any separation that is
its expected potential, we recommend increased investment
initiated in the boundary layer will still require improvements
to accelerate the use of these machines for CFD applica-
in RANS-based methods. Second, a seamless, automatic
tions.
RANS-to-LES transition in the boundary layer is needed to
A review of current HPC trends and a forecast of future enhance the robustness of these methods. Continued invest-
capabilities are given in Appendix A. ment in hybrid RANS/LES methods to specifically address
these two critical shortcomings will be required. Additional-
Physical Modeling. Advances in the physical modeling of
ly, more effective discretizations and solvers designed specif-
turbulence for separated flows, transition, and combustion
ically for LES type problems must be sought. When com-
are critically needed to achieve the desired state of CFD in
bined with advances in HPC hardware, these three develop-
2030.1,23,24,31,39,63,64 For the advancement of turbulent flow
ments will enable continued reduction in the RANS region as
simulation, we propose three separate tracks for research:
larger resolved LES regions become more feasible. It is fully
RANS-based turbulence treatments; hybrid RANS/LES
anticipated that hybrid RANS/LES methods will become
approaches where the entire boundary layer is resolved with
viable in production mode by the 2030 timeframe for prob-
RANS-based models, and the outer flow is resolved with
lems typical of the proposed GCs. Ultimately, progress will
LES models; and LES, including both Wall-Model
be measured by the degree to which the RANS region can be
(WMLES) and Wall-Resolved (WRLES). Details on each
minimized in these simulations and the added reliability they
of the three development tracks and for transition and com-
provide in predicting complex turbulent-separated flows.
bustion modeling, are given below. Additionally, a longer
term high-risk effort should investigate radically new ap- Application of LES to increasingly complex flows is a very
proaches to physical modeling. active research area.112 At present, the TRL level of this tech-
nology is relatively low. As discussed in Case Study 2, cost
RANS-based turbulence models continue to be the standard
estimates of WRLES show scaling with Reynolds number of
approach used to predict a wide range of flows for very com-
about ReL2.5 while WMLES is about ReL1.3, with the costs
plex configurations across virtually all aerospace product
being the same at approximately ReL of 105. For the typically
categories.23,30,56,66,85 As a result, the TRL level for these
higher Reynolds numbers and aspect ratios of interest to ex-
methods is high. They are easy to use, computationally effi-
ternal aerodynamics, WRLES will be outside of a 24-hour
cient, and generally able to capture wall-bounded flows,
turnaround even on 2030 HPC environments unless substan-
flows with shear, flows with streamline curvature and rota-
tial advances are made in numerical algorithms. However,
tion, and flows with mild separation. For these reasons, as
WRLES is potentially feasible in 2030 for lower Reynolds
well as the fact that RANS models will remain as an im-
numbers and is a reasonable pursuit for many relevant aero-
portant component in hybrid RANS/LES methods, their use
space applications including many components of typical
will continue through 2030. An advanced formulation of the
aerospace turbomachinery.113 Further, the development of
RANS-based approach, where the eddy viscosity formulation
WRLES directly benefits WMLES in that the basic issues of
is replaced with the direct modeling of the Reynolds stresses,
improved HPC utilization and improved numerics are essen-
known as the Reynolds Stress Transport (RST) method,109 in
tially the same for both. WMLES, however, requires addi-
principle will be able to capture the onset and extent of flow
tional development of the wall-modeling capability114 that is
separation for a wider range of flows.110 Currently, RST
currently at a very low TRL. As such, we recommend in-
models lack robustness and are occasionally less accurate
vestments in LES with emphasis on (1) improved utilization
than standard RANS models. Solid research is needed in
of HPC including developments of numerical algorithms that
advancing RST models to production capability. To this end,
can more effectively utilize future HPC environments, and

24
(2) improved wall-modeling capability necessary for reliable Numerical Algorithms. The development of novel numeri-
WMLES. To this end, we envision waypoints to assess tech- cal algorithms will be critical to achieving the stated CFD
nology maturation: a technology demonstration of LES 2030 goals. Indeed, the proposed GCs are sufficiently ambi-
methods for complex flow physics at appropriate Reynolds tious that advances in HPC hardware alone during the next
numbers around 2020, and a GC problem involving complex 20 years will not be sufficient to achieve these goals. As
geometry and complex flows with flow separation in 2030. demonstrated in Case Study 2, even for LES of relatively
Here, as for hybrid RANS/LES models, reductions in the simple geometries, leadership class HPC hardware in 2030
wall modeled region ultimately leading to WRLES will be will be needed for 24-hour turnaround if existing algorithms
continuously sought through 2030 and beyond. are used. Thus, to tackle the proposed GCs, orders of magni-
tude improvement in simulation capabilities must be sought
Transition modeling is also a key area of investment, as an
from advances in numerical algorithms.1,39 The focus of in-
effective transition model would benefit RANS, hybrid
vestment must be on discretizations and solvers that scale to
RANS/LES, and LES (by relieving mesh requirements in the
massive levels of parallelism, that are well-suited for the
laminar and transition regions). Thus, an additional research
high-latency, deep memory hierarchies anticipated in future
thrust must be devoted toward the development of reliable
HPC hardware, and that are robust and fault tolerant.41 A
and practical transition models that can be incorporated in the
well balanced research program must provide for incremental
turbulence models being matured along each of the develop-
advances of current techniques (e.g., extending the scalability
ment tracks. The transition prediction method should be fully
of current CFD methods to the exascale level whenever pos-
automatic, and be able to account for transition occurring
sible), while at the same time investing in the fundamental
from various mechanisms such as Tollmien–Schlichting
areas of applied mathematics and computer science to devel-
waves, cross-flow instabilities, Görtler vortices, and nonline-
op new approaches with better asymptotic behavior for large-
ar interactions associated with bypass transition.
scale problems and better suitability for emerging HPC
In the area of turbulent reactive flows, investment needs to hardware.
continue toward the development of a validated, predictive,
Discretization techniques such as higher-order accurate
multiscale combustion modeling capability to optimize the
methods offer the potential for better accuracy and scalabil-
design and operation of evolving fuels for advanced engines.
ity, although robustness and cost considerations remain.89
The principal challenges are posed by the small length and
Investment must focus on removing these barriers in order to
time scales of the chemical reactions (compared to turbulent
unlock the superior asymptotic properties of these methods,
scales), the many chemical species involved in hydrocarbon
while at the same time pursuing evolutionary improvements
combustion, and the coupled process of reaction and molecu-
in other areas such as low dissipation schemes,118-120 flux
lar diffusion in a turbulent flow field. Current combustion
functions, and limiter formulations. Simultaneously, novel
modeling strategies rely on developing models for distinct
nontraditional approaches, such as Lattice-Boltzmann meth-
combustion regimes, such as non-premixed, premixed at thin
ods121,122 or other undeveloped schemes, should be investi-
reaction zone, and so forth. The predictive technology should
gated for special applications. Improved linear and nonlinear
be able to switch automatically from one regime to another,
solvers must be developed, and here as well, the focus must
as these regimes co-exist within practical devices. Further-
be on highly scalable methods that are designed to be near
more, research should continue into methods to accelerate the
optimal for the large-scale, time-implicit unsteady CFD and
calculation of chemical kinetics so that the CFD solution
MDAO simulations anticipated in the future. These may in-
progression is not limited by these stiff ordinary differential
clude the extension of well-known matrix-based techniques,
equations (ODEs). The deep research portfolios of DOE and
Krylov methods,123 highly parallel multigrid methods,124 or
the US Air Force can be leveraged to further these modeling
the development of completely novel approaches such as
needs.
systematic upscaling methods.117 Furthermore, these methods
In addition to the above thrusts, a small portion of the re- must be extensible to tightly coupled multidisciplinary prob-
search portfolio should be devoted to the investigation of lems. Investment in discretizations and solvers must also
radically novel approaches to physical modeling that may consider the potential of these methods to operate on dynam-
offer revolutionary changes in modeling capabilities. As an ically adapting meshes, to enable optimization procedures,
example, renormalization group theory115,116 has been pro- and to incorporate advanced uncertainty quantification capa-
posed as a general framework for turbulence and other mul- bilities. In many cases, adjoint technology125,126 will be re-
tiscale physical modeling, although revolutionary advances quired from the outset for all of these capabilities, but the
have not materialized specifically for turbulence modeling. potential of other more advanced technologies such as sec-
Nevertheless, advances in multiscale modeling such as sys- ond-order gradients (Hessians)127,128 should be investigated as
tematic upscaling (SU)106,117 offer the possibility for step well. Longer term, high-risk research should focus on the
changes in physical modeling capability, and should be pur- development of truly enabling technologies such as mono-
sued in a measured manner, similar to the proposed invest- tone or entropy stable schemes87,88 in combination with inno-
ments in radical HPC technologies. vative solvers on large-scale HPC hardware. The technology

25
roadmap envisions the demonstration of improved robust and nologies that enable the capabilities that will be required by
scalable solvers in the 2015-2017 timeframe, for both sec- Vision 2030 CFD applications, namely very large scale
ond-order and higher-order accurate methods. The demon- (0(1012) mesh points) parallel mesh generation, curved mesh
stration of complete configuration-grid convergence technol- elements for higher order methods,73,74 highly scalable dy-
ogy in the 2020 time frame relies on the use of robust higher- namic overset mesh technology,77,133 and in-situ anisotropic
order discretizations combined with improved scalable solv- adaptive methods for time-dependent problems. It is im-
ers and adaptive h-p refinement. Toward the 2030 time portant to realize that advances in these areas will require a
frame, it is anticipated that novel entropy stable formulations mix of investments in incremental software development,
will begin to bear fruit for industrial simulations. combined with advances in fundamental areas such as com-
putational geometry, possibly with smaller components de-
With regard to uncertainty quantification, a new thrust in the
voted to high risk disruptive ideas such as anisotropic cut-cell
area of probabilistic large-scale CFD for aerospace applica-
meshes,80 strand mesh ideas,81 and even meshless methods.82
tions should be initiated. This program can build on the sig-
Additionally, because significant technology currently re-
nificant advances already made in this area by other govern-
sides with commercial software vendors, particularly for
ment agencies,98 but provide the focus required for leverag-
CAD interfaces and access, involving these stakeholders in
ing these technologies for aerospace applications.129,130 An
the appropriate focused research programs will be critical for
initial thrust in this area should focus on enabling current
long-term success. Innovative approaches for achieving such
aerospace CFD tools with well-known uncertainty quantifi-
partnerships must be sought out, such as the formation of
cation techniques, such as sensitivity analysis and propaga-
consortiums for the definition and adoption of standards or
tion methods using adjoints and forward linearizations, non-
other potential issues such as large scale parallel licensing of
intrusive polynomial chaos methods, and other reduced-order
commercial software. The technology development roadmap
model formulations.131,132 Additionally, a concerted effort
envisions the demonstration of tight CAD coupling and pro-
should be made to characterize important aerospace uncer-
duction adaptive mesh refinement (AMR) in the 2015-2017
tainties and to make these available to the general research
time frame, followed by maturation of large-scale parallel
community for enabling relevant UQ research in these areas.
mesh generation in the 2020-2025 time frame, and leading
Improved error estimation techniques must be investigated
ultimately to fully automated in-situ mesh generation and
and developed, given the known deficiencies of current ap-
adaptive control for large-scale time-dependent problems by
proaches (including adjoint methods). This will require a
2030.
foundational program in the mathematics of error estimation
and its application to CFD software. Finally, longer term Knowledge Extraction. Petascale and exascale simulations
research must focus on statistical approaches such as Bayesi- will generate vast amounts of data and various government
an techniques for quantifying more accurately modeling and agencies such as the NSF and DOE have instituted major
other nonlinear error sources.97 The technology roadmap programs in data-driven simulation research.134,135 In order to
includes an early target date of 2015 for the characterization make effective use of large scale CFD and MDAO simula-
of typical aerospace uncertainties in order to stimulate work tions in aerospace engineering, a thrust in data knowledge
in this area. Improved error estimation techniques will be extraction should be initiated. Ideally, this should contain
gradually brought into the simulation capabilities and the three components, a visualization component, a database
state of these estimates will be assessed in the 2018 time management component, and a variable fidelity, data integra-
frame. Comprehensive uncertainty propagation techniques tion component. Methods to process and visualize very large-
including discretization error, input and parameter uncertain- scale unsteady CFD simulations in real time, including re-
ties in production-level CFD codes should be targeted for sults from higher-order discretizations, are required to sup-
2025, while the development of more sophisticated stochastic port the advanced CFD capabilities envisioned in 2030. Alt-
and Bayesian approaches will continue through the 2030 hough many of the current efforts in maturing visualization
timeframe. technology are being led by commercial vendors who con-
tinue to supply enhanced capabilities in this area, more fun-
Geometry and Grid Generation. Substantial new invest-
damental research to directly embed visualization capabilities
ment in geometry and grid generation technology will be
into production CFD tools optimized for emerging HPC plat-
required in order to meet the Vision CFD 2030 goals. In gen-
forms is needed to achieve real-time processing.99 Moreover,
eral, this area has seen very little NASA investment during
the CFD capability in 2030 must provide the analyst with a
the last decade, although it remains one of the most important
more intuitive and natural interface into the flow solution to
bottlenecks for large-scale complex simulations. Focused
better understand complex flow physics and data trends and
research programs in streamlined CAD access and interfac-
enable revolutionary capabilities such as computational steer-
ing, large-scale mesh generation, and automated optimal
ing, which could be used, as an example, for real-time virtual
adaptive meshing techniques are required. These programs
experiments or virtual flight simulation.136,137 Foreseeing the
must concentrate on the particular aspects required to make
capability of generating large databases with increasing com-
mesh generation and adaptation less burdensome and, ulti-
putational power, techniques for rapidly integrating these
mately, invisible to the CFD process, while developing tech-

26
databases, querying them in real time, and enhancing them solvers of the time, and guaranteeing coupling fidelity by the
on demand will be required, along with the ability to provide year 2020. Such demonstrations can focus on multiple aero-
reliable error estimates or confidence levels throughout all space problems of interest, including aircraft aero-
regions of the database. structural/aeroelastic analyses, aircraft aero-acoustics, ro-
torcraft aero-structural and aero-acoustic couplings, unsteady
Finally, integrating high fidelity simulation data with lower
combustion, re-entry aerothermodynamics and material re-
fidelity model data, as well as experimental data from wind
sponse, and the like. Initially, such routine MDAs would
tunnel tests, engine test rigs, or flight-test data will provide a
focus on portions of an entire vehicle (around 2020) and
powerful approach for reducing overall risk in aerospace
would transition to the treatment of the entire system around
system design.101 Techniques for building large-scale flexible
2025. A number of capabilities also must be developed in
databases are in their infancy, and range from simple soft-
order to enable MDAO with and without the presence of
ware infrastructures that manage large numbers of simulation
uncertainties (robust and reliability-based design). A major
jobs to more sophisticated reduced-order models,138 surrogate
research component that is likely to span a significant period
models, and Kriging methods.102,103 The objective of a re-
(from 2015 to 2025) is the work needed to endow industrial-
search thrust in this area should be to apply existing tech-
strength CFD solvers with both gradient calculation and un-
niques to current CFD simulation capabilities at a large scale,
certainty quantification capabilities for use in multidiscipli-
while simultaneously performing foundational research in the
nary optimization. Some of this work has been described in
development of better reduced-order models and variable
the “Numerical Algorithms” section. For the gradient/ sensi-
fidelity models that are applicable to aerospace problems and
tivity analysis capability, we envision that the CFD solver
can support embedded uncertainty quantification strategies.
will be able to compute this information for full unsteady
The technology roadmap envisions the demonstration of real-
flows for the turbulence models available at the time. Finally,
time analysis and visualization of a notional 1010-point un-
all these new capabilities must come together on a series of
steady CFD simulation in 2020, and a 1011-point simulation
MDAO grand-challenge demonstrations in the 2030
in 2025. These technology demonstrations would be an inte-
timeframe.
gral part of the GC problems designed to benchmark advanc-
es in other CFD areas. The development of reduced-order
models and other variable fidelity models will entail long-
term research and will likely remain an active research topic 7 Recommendations
past the 2030 time frame. However, the technology roadmap
In order to effectively execute the CFD development plan
envisions the periodic assessment of the state-of-the-art in
described above and achieve the goals laid out in the vision
these areas at 5 to 10 year intervals, with investment directed
of CFD in 2030, a comprehensive research strategy and set of
toward demonstrating promising approaches on large-scale
recommendations are presented. This research strategy calls
aerospace applications.
for the renewed preeminence of NASA in the area of compu-
Multidisciplinary Design and Optimization. The ability to tational sciences and aerodynamics, and calls for NASA to
perform CFD-based multidisciplinary analysis (MDA) and play a leading role in the pursuit of revolutionary simulation-
analysis/optimization (MDAO) relies on the availability of based engineering.
future capabilities that need to be developed between now
Aerospace engineering has had a long history of developing
and 2030. Pervasive and seamless MDAs (that can be rou-
technology that impacts product development well beyond
tinely exercised in industrial practice for configuration stud-
the boundaries of aerospace systems. As such, NASA is a
ies, e.g., full aero-thermo-elastic/aeroacoustic simulations of
critical force in driving technology throughout aerospace
entire airframe/propulsion systems including shielding) will
engineering directly by fulfilling its charter to “preserve the
require the development of accepted standards and APIs for
role of the United States as a leader in aeronautical and
disciplinary information and the required multidisciplinary
space science technology.”139 Computational methods are a
couplings (such as with acoustics, combustion, structures,
key example of this broad impact, as NASA has historically
heat transfer, radiation). A concerted effort is envisioned that
been a leader in the development of structural finite-element
results in a set of standards available to the community
methods, computational fluid dynamics, and applications of
around 2016. In parallel with this effort, it will also be neces-
HPC to engineering simulations. The criticality of engineer-
sary to develop high-fidelity coupling techniques that guaran-
ing-based simulation to the competitiveness of the United
tee the accuracy and stability of high fidelity, tightly coupled
States and the lack of sustained federal support have been
MDAs,106 while ensuring that the appropriate conservation
highlighted previously by the NSF37 and elsewhere.17,32
principles are satisfied with errors below acceptable thresh-
olds. This capability, together with the coupling software that NASA’s effort must be targeted toward research and tech-
includes such information transfers must be available around nology development that can make revolutionary impacts on
2018. Together, the standards and the coupling tech- simulation-based engineering in the aerospace sciences. In
niques/software would enable demonstrations of two-way particular, the current state of CFD is such that small, incre-
coupled MDAs with the best and most robust existing CFD mental improvements in existing capability have not had

27
revolutionary effects. In an environment of constrained re- program within the Aeronautical Sciences Project of the
sources, this will require that NASA evaluate its activities Fundamental Aeronautics Program (FAP) is too narrow in
with a critical eye toward supporting those efforts whose scope to address all the required technology areas in this re-
impact could be revolutionary. port. Thus, we recommend broadening and enhancing the
RCA program in several ways. The Aeronautical Sciences
To ensure that the technology plan and roadmap are as effec-
Project encompasses various subtopics including the RCA
tive as possible, we propose specific recommendations in
program, but also other areas such as materials, controls,
three broad areas: enhancement of the current Revolutionary
combustion, innovative measurements, and MDAO.
Computational Aerosciences (RCA) program, important pro-
grammatic considerations, and key strategic initiatives that We recommend that all components of subtopics focused on
taken together will help achieve the goals of our vision of computational simulation technologies be coordinated with
CFD in 2030. the RCA program. For example, numerical simulation of
combustion is an important technology that would be ill
7.1 Development of a Comprehensive Revolu- served by being isolated from the developments achieved
tionary Computational Aerosciences Program under the RCA program. Thus, we suggest joint oversight of
the numerical modeling aspects of combustion between the
Recommendation 1: NASA should develop, fund and sustain RCA program and the combustion program. Similarly, sig-
a base research and technology (R&T) development program nificant components of MDAO related to solver technology
for simulation-based analysis and design technologies. and interfacing CFD with other disciplines will benefit from
Physics-based simulation is a crosscutting technology that close interaction with the RCA program. Next, we recom-
impacts all of NASA aeronautics missions and vehicle clas- mend that the RCA program be structured around the six
ses, as evidenced by the common themes in the NAE Deca- technology areas that we have outlined in this report, namely
dal survey report.12 In addition, technologies developed with- HPC, Physical Modeling, Numerical Algorithms, Geome-
in NASA’s Aeronautics Research Mission Directorate impact try/Grid Generation, Knowledge Extraction, and MDAO.
many other aspects of the missions of various other NASA Currently, the RCA program contains technology thrust areas
directorates. Yet, until recently, there has been no systematic specifically in Numerical Algorithms and Turbulence model-
program for developing simulation technologies, and all ad- ing. Thus, the recommended structure represents a logical
vances in simulation and CFD methods had to be justified by extension of the current program, achieved by extending the
potential short-term impacts on one of the existing programs, turbulence modeling technical area to a physical modeling
or were done in response to critical simulation failures ob- technical area (i.e., adding transition modeling and combus-
served through the course of a program. This leads to the tion modeling), coordinating the relevant MDAO thrusts
preference for small improvements to existing software, with within the broader Aerosciences program, and adding the
the result that most current software is more than 20 years other required technology areas. Additionally, collaboration
old, and the initiation of any new software project cannot be in computational methods between RCA and the Controls
supported. Furthermore, investment in developing revolu- and the Structure and Materials programs should be encour-
tionary simulation technologies is out of the question within aged, while CFD validation thrusts can be expected to benefit
such a program structure due to the long fruition time re- from collaboration with the Innovative Measurements pro-
quired and distant impact on existing programs. Yet without gram. This new programmatic structure is illustrated in Fig-
a focused base R&T development program, CFD will likely ure 2.
remain stagnant. Other international government agency In the preceding section, each technical area is described in
aeronautic programs (such as DLR and ONERA) contain a detail and the required research thrusts for advancing each
base R&T component that is used to advance simulation area is provided. Naturally, individual research thrusts affect
technologies140,141 and certainly the new NASA Revolution- multiple technical areas, which in turn affect the ability to
ary Computational Aerosciences (RCA) program is a step in meet various milestones and progress toward the GC prob-
the right direction. However, NASA must ensure this pro- lems. However, for programmatic reasons it is desirable to
gram is strengthened, maintained, and expanded to cover have each individual research thrust reside within a single
investment in the critical elements required for advancing technology area. The success of this strategy relies on good
CFD and other physics-based simulation technologies as communication and interaction between the different tech-
outlined in our research roadmap. nology areas over the life of the program. A concise view of
An integrated research plan is required for the fulfillment of the proposed research program structure, including all tech-
the technology development roadmap and eventual demon- nology areas and research thrusts is given in Figure 3.
stration of the GC problems. At present, the current RCA

28
 AERONAUTICS RESEARCH MISSION DIRECTORATE (ARMD)
FUNDAMENTAL AERONAUTICS PROGRAM

AERONAUTICAL SCIENCES PROJECT

STRUCTURES AND INNOVATIVE

MATERIALS
CONTROLS COMBUSTION MDAO
MEASUREMENTS

SCIENCE MISSION
DIRECTORATE REVOLUTIONARY COMPUTATIONAL
(SMD) AEROSCIENCES (RCA)
HPC
HUMAN Physical Modeling: 
EXPLORATION AND
Turbulence
OPERATIONS
Transition
DIRECTORATE
(HEO) Combustion
Numerical Algorithms
New thrust
Geometry/Grid
Knowledge Management
Shared investment and 
technology collaboration MDAO (Interfaces/coupling)
Close coordination

Figure 2. Proposed Enhanced Revolutionary Computational Sciences Program

HPC PHYSICAL MODELING NUMERICAL ALGORITHMS

1. Increasing access to leading‐edge HPC  1. RANS turbulence modeling 1. Advances in current algorithms for HPC

hardware  2. Hybrid RANS‐LES modeling 2. Novel discretizations
2. Porting of current and future codes to  a. Improved RANS component a. Higher‐order methods
leading‐edge HPC  b. Low dissipation/dispersion schemes
b. Seamless interface
3. Radical emerging HPC technologies  c. Novel approaches (foundational)
3. LES (wall‐modeled and wall‐resolved) 3. Solvers
4. Transition a. Linear and non‐linear scalable solvers
5. Combustion b. Enhancements for MDAO and UQ
6. Radically new modeling approaches 4. UQ
a. Define aerospace uncertainties
b. Leverage known techniques
GEOMETRY AND GRID GENERATION KNOWLEDGE MANAGEMENT c. Improved error estimation techniques
d. Statistical approaches
1. CAD access and interfaces 1. Visualization
2. Large scale parallel mesh generation 2. Data‐base management
3. Adaptive mesh refinement 3. Variable fidelity models
4. Curved mesh elements for higher‐order 
methods MDAO
1. Interfaces and standards
2. Accurate and stable coupling techniques
3. UQ support and sensitivities (system‐
level)

Figure 3. Proposed New Revolutionary Computational Sciences (RCA) Program Structure

The overall program goals are driven by the GC problems, traction, combustion, MDAO), and this is partly due to the
which embody the vision of what CFD should be capable of focus on CFD technology in the current report. As can be
achieving with balanced investment over the long term, and seen, the proposed research program contains a balanced mix
provide a means for maintaining program direction and of near-term and long-term research thrusts. The overall pro-
measuring progress. While advances in all technology areas gram is also highly multidisciplinary and draws on advances
will be critical for achieving the GC problems, certain areas in disciplines at the intersection of aerospace engineering,
are described in less detail than others (e.g., knowledge ex- physics of fluids, applied mathematics, computational geom-

29
etry, computer science and statistics. Successful execution of remains beyond the scope of this report. However, what is
the program will require devising appropriate mechanisms evident is that without such an internal competence, NASA
for leveraging expertise in these diverse fields. By its very will be severely handicapped in any attempts to advance the
nature, the formulation of a comprehensive research program state of the art in physics-based simulation technologies. Ad-
of this type results in an exhaustive list of research thrusts ditionally, this recommendation is targeted to a broader audi-
that need to be addressed, and clearly prioritization of these ence at NASA than simply ARMD: given the deep reliance
individual thrusts must be performed within a limited budget on simulation-based engineering for all mission directorates
environment. The prioritization of research thrusts and the and the fact that an agency-wide coordination mechanism
prescription of funding levels must be an ongoing process exists, efforts to develop world-class, in-house simulation
and is certainly beyond the scope of this report. However, capabilities should be cooperatively pursued.
consistent mechanisms for making such decisions must be
Streamline and Improve Software Development Process-
instituted. We propose the use of periodic workshops (e.g., at
es. CFD software development at NASA has a checkered
5-year intervals) convened to measure progress toward the
history. Many of the most successful codes in use today have
GC problems, that can be used to identify the most critical
their roots in the inspiration and the devotion of a single or
technologies in need of investment, evaluate the success of
small number of researchers. In some sense, this reflects one
previous investments, and prioritize future investments.
of the strengths of NASA’s workforce and work environment
that, in the past, accorded significant scientific freedom.
7.2 Programmatic Considerations However, despite their successes, many of these codes are
still maintained by a small number of developers who strug-
Recommendation 2: NASA should develop and maintain an gle to keep up with the increasing demands of bug fixes, ap-
integrated simulation and software development infrastruc- plication support, and documentation that comes with in-
ture to enable rapid CFD technology maturation. creased usage.
To reach the goals of CFD in 2030, research and technology Today, it is well recognized that software development must
development must effectively utilize and leverage in-house be a team effort due to increasingly complex software.41 43
simulation expertise and capabilities with focused attention to While some NASA software projects (such as FUN3D) have
HPC infrastructure, software development practices, inter- successfully transitioned to a team effort model,142 there re-
faces, and standards. mains no formal structure for supporting software develop-
Maintain a World-Class In-House Simulation Capability. ment issues such as regression testing, porting to emerging
To support broad advances in CFD technology, NASA’s HPC architectures, interfacing with pre- and post-processing
simulation capability should be, in many aspects, superior to tools, general application support, and documentation. Most
the capabilities that reside with academic and industrial part- commercial software companies staff entire teams devoted to
ners and in the commercial software vendor arena. NASA’s these types of activities, thus freeing the developers to pursue
in-house simulation code development is driven by the need technology development and capability enhancements. CFD
to support NASA missions and must apply to all important software efforts at DLR143,144 and ONERA,145,146 for instance,
application regimes of relevance to the NASA ARMD mis- are known to provide continual support for dedicated soft-
sion including fixed- and rotary-wing external aerodynamics, ware engineering tasks, while various US government pro-
turbomachinery flows, combustion, aeroacoustics, and high jects such as the Department of Defense (DOD) Computa-
speed flows, as well as applications of relevance to NASA’s tional Research and Engineering Acquisition Tools and Envi-
science and space exploration missions. While NASA has ronments – Air Vehicles (CREATE-AV) program have set
excelled in many of these areas (notably fixed- and rotary- up similar capabilities including an elaborate application
wing external aerodynamics, and space vehicle entry, descent support structure.60 Furthermore, if individual NASA codes
and landing [EDL]), there are other areas such as tur- are to be applied to diverse areas such as external aerody-
bomachinery, combustion, and icing where it is believed that namics, internal turbomachinery flows, combustion, LES and
NASA’s capabilities are no longer on the cutting edge. Main- aeroacoustics, support of this type will be essential since no
taining an in-house capability is crucial for understanding the single individual can cover such a wide range of disciplines.
principal technical issues and overcoming impediments, for While there are continual cost pressures to reduce the number
investigating new techniques in a realistic setting, for sup- of CFD codes being supported, mandatory use of a single
porting NASA’s own missions, and for engaging with other code for all applications is overly constraining and even un-
stakeholders. Whether technology transfer is ultimately feasible in many cases for new technology development,
achieved through the development of production level soft- since newly developed algorithms may be ill-suited for retro-
ware that is adopted by industry, technology spin-offs to fitting into existing codes due to their data structures and
commercial software companies, or simply through realistic inherent assumptions. Thus, the creation of a formal software
demonstrations on industrial problems with accompanying support structure could provide relief and continuity to de-
publications, has been the subject of much discussion for velopers of established production codes while also facilitat-
many years within NASA and the broader community, and ing and lowering the development costs of potentially prom-

30
ising new software efforts designed either for research inves- resources toward enabling leading-edge petascale or higher
tigations or production CFD and MDAO. Moreover, this simulation capabilities.45 Lack of access to large-scale HPC
approach has the potential for cost savings due to reduction hardware on a regular and sustainable basis within NASA
in duplication of efforts between individual software devel- has led to stagnating simulation capabilities. To remedy this
opment projects. Ideally, these software components would situation, NASA, and in particular the NASA Advanced Su-
be open-source and freely distributed within the United percomputing (NAS) division, should make HPC available
States and abroad. This would mark a significant departure for large-scale runs for CFD research and technology devel-
from current NASA software practices, but could be opment. Use of HPC for large-scale problems will drive de-
achieved by isolating any sensitive capabilities in closed mand by enabling testing of more sophisticated algorithms at
source software, as is regularly done at the US Department of scale, making users more experienced and codes more scala-
Energy (DOE). ble since many issues are only uncovered through large-scale
testing. However, this approach is complicated by the fact
Emphasize CFD standards and interfaces. Many of the
that ARMD only controls a fraction of NASA’s HPC re-
impediments outlined in this report relate to the difficulty in
sources. This will require advocating the benefits of large-
accessing or exchanging information between various soft-
scale computing within NASA, either for modifying the cur-
ware components, such as CAD data for grid generation;
rent HPC usage paradigm, or for sharing resources between
AMR, post-processing data; or exchange of information be-
NASA directorates (e.g., Science Mission Directorate, Hu-
tween different components of a multidisciplinary problem.
man Exploration and Operations) with an interest in more
In many cases, the development of standardized interfaces
radical simulation capabilities. NASA ARMD must also lev-
can be used to greatly relieve these problems and facilitate
erage other national HPC facilities and enter into a discussion
further advances in CFD. As a government agency, NASA is
with the NSF, DOE and any other agencies for providing
uniquely positioned to spearhead the development and adop-
access to these systems on a regular basis for NASA objec-
tion of international standards and interfaces in various areas
tives that overlap with these agency priorities147,148. Further-
of CFD and MDAO. In particular, this is an activity that may
more, NASA should remain at the forefront in new HPC
not require significant funding in dollar terms, but will re-
quire identifying and organizing key stakeholders, develop-
ing a consensus among them, and continuing the advocacy
and support of developed standards and interfaces. At the
same time, it is important to note that frameworks and stand-
ardization can lead to significant constraints and may not be
the best solution in all cases. Thus, a large part of such an
effort must involve determining under what conditions stand-
ardization is appropriate, and then developing sufficiently technologies through the use of test platforms made available
flexible standards and building a consensus among all stake- to the research community. The recently installed D-Wave
holders. Two quantum computer149 at NASA Ames is a good example
of this, but it does not appear to be part of a concerted effort
Recommendation 3: NASA should make available and uti- to track and test HPC developments.
lize HPC systems for large-scale CFD development and test-
ing. Recommendation 4: NASA should lead efforts to develop
Access to leading-edge HPC hardware is critical for devis- and execute integrated experimental testing and computa-
ing and testing new techniques that enable more advanced tional validation campaigns.
simulations,39,41 for demonstrating the impact that CFD During the past decade, workshops to assess CFD predictive
technology enhancement can have on aerospace product capabilities have been effective in focusing attention in key
development programs,11,45 and for addressing the GC prob- areas important to the aerospace community such has drag
lems defined previously. As described in Case Study 1, prediction,23 high-lift prediction,56 and aeroelasticity,85 to
NASA’s HPC hardware is used primarily for throughput name a few (see accompanying Case Study). In most cases,
(capacity) computing rather than capability. Although the workshops involve CFD simulation of challenging flow
hardware parallelism has increased dramatically during the physics on realistic geometries. If available, experimental
last several decades, the average size of NASA CFD jobs data is used to anchor the CFD predictions. However, with
remains well below 1,000 cores, even though the NASA the exception of the Common Research Model (CRM) model
Advanced Supercomputing (NAS) division flagship system development150 and transonic test campaign,151 workshops
contains over 160,000 CPU cores and is ranked 19th in the typically rely on pre-existing experimental datasets that often
top 500 HPC installations worldwide.62 Other large HPC have an incomplete set of test data available, quality control
installations regularly allocate significant fractions of their issues, or a combination of both. Moreover, in many cases,

31
CASE STUDY 5: Community Verification and Validation Resources
As numerical simulation capabilities become more complex, verification and validation (V&V) efforts become more important but also
more difficult and time consuming. Verification is defined as the determination of whether a model is implemented correctly, whereas
validation is defined as the determination of how well the model represents physical reality. One approach to reduce this burden and en-
courage higher V&V standards and usage is through the development of community resources for V&V. As a government agency,
NASA is uniquely positioned to serve as the integrator and steward of such community resources.
An excellent example of community V&V resources can be found in the NASA Turbulence Modeling Resource web site.1. The site is
hosted by NASA, and the effort is guided by the Turbulence Model Benchmarking Working Group (TMBWG), a working group of the Flu-
id Dynamics Technical Committee of the AIAA, with contributions from NASA, academia, and industry. The objective of the site is to
provide a central resource for turbulence model verification, which includes a precise definition of commonly used turbulence models in-
cluding different model variants, and a set of verification test cases with supplied grids and sample results using different CFD codes, in-
cluding grid convergence studies. By providing a sequence of progressively highly refined meshes, many of the verification test cases
(principally in 2D) establish fully grid converged results for different CFD codes, providing a benchmark by which other codes can be
measured to verify correct implementation of the model and consistency of the discretization.
These are important prerequisites for applying implemented models to more complex cases
with confidence. At present, the site provides descriptions for 11 turbulence models, and pro-
vides four verification test cases for which the most popular models have been tested with
more than one CFD solver. The site also provides experimental data for a variety of 2D and
3D test cases in order to facilitate model validation.
During the last decade, the community workshop approach has emerged as a viable model for the validation of individual numerical
simulation tools, as well as for the assessment of the entire state of the art in specific simulation capabilities. One of the original work-
shop series, the Drag Prediction Workshop (DPW), was initiated in 20012 and has since held five workshops.3 The first workshop in
2001 was a grass-roots effort, which included substantial NASA participation, and focused mostly on comparison of CFD results for
transport aircraft transonic cruise drag prediction, with secondary emphasis on comparison to available published experimental data.
Over the years, the importance of high quality experimental data was increasingly recognized, leading to greater NASA involvement and
investment, resulting in the design, fabrication and testing of the common research model (CRM), supported by NASA, including indus-
try input, and conceived specifically for CFD validation purposes.4 Throughout this period, the DPW series has firmly established the im-
portance of discretization error as a dominant error source (often larger than turbulence modeling error) for
accurate CFD prediction of aircraft forces and moments. The DPW series has emphasized the need for care-
ful grid convergence studies, resulting in the establishment and documentation of best practices for grid gen-
eration and grid convergence studies. Each individual workshop has provided a contemporary evaluation of
the state of the art in CFD force and moment prediction, while the entire workshop series has enabled the
assessment of continual improvements in the state of the art over more than 10 years. Reduced workshop
result scatter can be correlated with evolving methodologies, increased grid sizes, and advancing computa-
tional power. The workshop series has also served to clearly identify the successes and deficiencies of cur-
rent RANS methods, with particular emphasis on the rapid degradation of RANS predictive capabilities with
increasing amounts of flow separation. Finally, the workshop series resulted in a large database of publicly
available geometries, grids, and CFD results against which new software development programs can
benchmark for more effective V&V.
The success of the DPW has spawned other workshops in related areas, such as the High-Lift Prediction
Workshop Series (HiLiftPW)5 and the Aeroelastic Prediction Workshop (AePW).6 A common feature of these
workshop series, and other community V&V resources such as the NASA Turbulence modeling Resource
web site, is that they combine the efforts of government, academia, and industry to promote advances in the
state of the art, benefiting the community at large. However, in all cases, NASA involvement and investment
has served as a key driver without which most of these endeavors would not be sustainable.
1 http://turbmodels.larc.nasa.gov/
2 Levy, D. W., Zickuhr, T., Vassberg, J., Agrawal, S., Walls, R. A., Pirzadeh, S., and Hemsch, M. J., “Data Summary from First AIAA Computational Fluid Dynamics
Drag Prediction Workshop,” AIAA Journal of Aircraft, 2003, Vol.4010.2514/1.1684
3 http://aaac.larc.nasa.gov/tsab/cfdlarc/aiaa-dpw/
4 Vassberg, J., Dehaan, M., Rivers, M., and Wahls, R. A., “Development of a Common Research Model for Applied CFD Validation Studies,” 26th AIAA Applied Aero-
dynamics Conference, 2008, 10.2514/6.2008-6919
5 http://hiliftpw.larc.nasa.gov/
6 https://c3.nasa.gov/dashlink/static/media/other/AEPW.htm

32
the geometry definition of the tested configuration must be duce a stagnating capability in the environment of shrinking
refurbished for CFD grid generation purposes. To help budgets during the last decade. These creative strategies can
achieve the vision of CFD in 2030, an integrated approach involve a wide range of partners from different directorates
involving well designed, ground-based (and perhaps flight152) within the agency, such as Space Exploration and Science, to
experiments to provide high quality datasets directly coupled other agencies such as NSF and DOE, and in terms of hard-
with CFD technology and application code verification and ware, software, and research results. As an example, the lack
validation is required. This could be used in support of both of access to HPC for NASA researchers could be addressed
CFD workshops, and the solution of GC problems, would through a potential collaboration with DOE to obtain guaran-
help focus and solidify technology development in multiple teed slices of time on their leadership class machines through
areas, and establish best practices. Moreover, with physics- a formal program that could be negotiated at an interagency
based computational modeling continuing to expand, the level, for example through the Networking and Information
need for systematic numerical validation test datasets and an Technology Research and Development (NITRD) Pro-
effective mechanism to disseminate the results of the valida- gram.147,148 In addition, many of the DOE- and DOD-
tion results are becoming paramount. NASA has both a full sponsored advances in HPC were derived from investments
range of experimental test facilities in which to collect high- in fundamental research that could be effectively leveraged
quality data, as well as the computational tools and processes by more direct NASA participation in the setup, running, and
in which to benchmark CFD capabilities. For this reason, partial sponsoring of these efforts. Finally, Memorandums Of
NASA should pursue a leadership role in developing com- Understanding (MOUs) and other vehicles for interacting
plementary experimental and computational datasets to help with foreign government agencies should be considered
guide CFD technology development. whenever possible.
Improve collaboration with industry. NASA has been at
7.3 Strategic Considerations the forefront of CFD technology for decades and was respon-
sible for introducing much of the CFD technology used in
Recommendation 5: NASA should develop, foster, and lev- industry today. At the same time, in collaboration with uni-
erage improved collaborations with key research partners versities, industry has addressed additional CFD technology
and industrial stakeholders across disciplines within the needs that are unique and essential to their business suc-
broader scientific and engineering communities. cess. These include increased emphasis in the areas of phys-
Leverage other government agencies and stakeholders ics-based modeling, rapid complex geometry and grid capa-
(US and foreign) outside of the aerospace field. Currently, bilities, managing the generation of CFD-based aerodynamic
NASA ARMD’s interaction with other government entities is design matrices, improving accuracy and robustness, and
almost exclusively focused on agencies that have a major integrating CFD databases with experimental data and other
stake in the national aeronautics enterprise such as the Feder- disciplines. The result has been a substantial reduction in
al Aviation Administration (FAA), United Stated Air Force physical testing but not uniformly across all products or flow
(USAF), and others. However, in the last decade, computa- regimes. Continued advances are required to address full-
tional science has had important visibility at the national lev- flight envelope predictions and propulsion system operating
el, through various competitiveness thrusts,17,32 and has be- conditions, reduce design cost and cycle time, reduce ground
come an important focus for various agencies such as the and flight-testing, and enable product certification through
DOE, NSF and the National Institute of Standards and Tech- analysis. Accelerated maturation of CFD technologies for all
nology (NIST).153 Therefore, it is natural for NASA ARMD, aerospace applications (e.g., external aerodynamics, tur-
which performs the bulk of the R&T in computational sci- bomachinery, space) could be better achieved with expanded
ence for the agency, to seek out and establish meaningful three-way collaboration between industry, NASA, and aca-
collaborations with these traditionally non-aerospace focused demia, beyond the current collaborations in physical testing
agencies. However, such collaborations have been sorely at NASA facilities and research through NASA Research
lacking. For example, various multi-agency studies and white Announcements (NRAs). While industry is in a unique posi-
papers are frequently published on the topic of exascale tion to provide requirements and assess the value and impact
computing,41,42,108 but surprisingly NASA has not been a par- of various CFD technologies on the aerospace industry,
ticipant in these multiagency discussions. With its limited NASA is best positioned to coordinate and advance CFD
budget, and dim prospects for improved research budgets, technologies required for maintaining competitiveness and
NASA ARMD cannot afford to “go it alone” and hope to leadership. As mentioned above, many of these technologies
make substantial progress in the important areas of computa- require substantial advances in physical modeling (e.g., tur-
tional science and simulation technology that are so im- bulence, transition, and combustion) and numerics (e.g., sta-
portant to advancing the agency’s mission in various direc- bility, accuracy, uncertainty quantification, gradient estima-
torates. Creative strategies must be devised to leverage fund- tion, adjoint methods). These are traditional NASA strengths
ing and resources with other stakeholders with similar objec- and should be re-emphasized. Further, NASA is also in a
tives, because the current approach has been shown to pro- unique position to coordinate the definition of standards for

33
data, codes, and databases that would enable more rapid gen- these throughout the report, invoking such areas as computa-
eration of next generation tools, and interfaces to industrial tional geometry for grid generation, applied mathematics for
scale CFD applications. Similarly, critical areas that have linear and nonlinear solver development, and computer sci-
mostly been left to industry (e.g., complex geometry repre- ence issues related to HPC. The specific areas of CFD, as
sentations, grid generation and 3D visualization) could also well as the broader area of MDAO, are components of the
benefit from a renewed focus by NASA, perhaps in a closer general field of computational science. CFD lies at the inter-
partnership with both the aerospace industry and other indus- section of applied math, computer science, and an application
tries that are similarly challenged, such as the automotive science area (in this case, aerodynamics), or more broadly,
industry. Finally, we advocate the creation of a NASA- aerospace engineering.1,39 It is notable that at other govern-
Industry Computational Science (CS) leadership team, which ment agencies, such as the NSF, a significant portion of fund-
will be tasked with the mission of identifying joint CS pro- ing for computational fluid dynamics comes from the math-
jects oriented toward addressing the four GC problems pre- ematical and physical sciences program, while numerical
sented in Section 6. We believe that projects with 3-year cy- solver groups at various DOE labs are staffed largely by sci-
cles accompanied by gate reviews from independent parties entists with degrees in applied math and/or computer science.
would facilitate and ensure maturation and transition of rele- The aerospace CFD and MDAO community is notoriously
vant technology to industry. insular, with most researchers having an aerospace engineer-
ing background, publishing in AIAA or similar venues, and
Emphasize basic funding in applied math and computer
with scant presence in regular computational science meet-
science. In order to advance the state-of-the-art of CFD, ad-
ings hosted by the Society for Industrial and Applied Math-
vances must also be sought in related disciplines such as ap-
ematics (SIAM), the Institute of Electrical and Electronics
plied mathematics and computer science. We have referred to

CASE STUDY 6: Sponsored Research Institutes

Currently NASA relies on a mix of internal development and external funding with academic and industrial partners through NASA Re-
search Announcements (NRA) to advance its research goals. However, additional mechanisms must be sought to more fully engage the
broader scientific community especially for computational science problems, which are both crosscutting and multidisciplinary. Spon-
sored research institutes have been used in many areas of science and engineering to further such goals. These institutes can take on
various forms and funding models, ranging from fully self-supporting autonomous institutes such as the Southwest Research Institute
(SWRI),1 university-based institutes, multi-stakeholder institutes, and government-agency based institutes. The nature, size, and funding
model of these institutes must be considered based on the objectives of the sponsoring agency or stakeholders.
The objective of a computational science based institute for NASA aeronautics would be to provide a centralized focal point for the de-
velopment of crosscutting disciplines, to engage the broader scientific community, and to execute a long-term research strategy with suf-
ficient autonomy to be free of NASA mission directorate short-term concerns. Under these conditions, the self-supporting research insti-
tute model such as SWRI is not appropriate due to the short-term pressures to continually raise research funding, and the difficulties in
maintaining agency-related focus, given the diverse and changing composition of a competitively
funded research portfolio. University-based institutes have been used successfully by a variety of
funding agencies, and are the preferred mechanism for agencies with no internal facilities of their
own, such as the National Science Foundation (NSF). During the last two decades, the NSF has
set up a number of High Performance Computing (HPC) centers at universities across the United
States, as well as various scientific institutes such as the Institute for Mathematics and its Appli-
cations (IMA) at the University of Minnesota.2 Mission agencies such as the DOE and NASA have
also followed this model occasionally, for example through support for the previous DOE ASCI
centers, NASA’s previous support of the CTR at Stanford University,3 and current DOE support
for the PSAAP centers.4 Although many of these institutes have been highly successful, such a
model may not be optimal for the considered objectives, since focused investment at specific universities is not an ideal mechanism for
engaging the broader community, while at the same time geographical separation between sponsor and university can be a detriment to
collaboration.
A number of multi-stakeholder and agency co-located research institutes with aerospace-focused computational science objectives were
used with generally favorable outcomes. CERFACS,5 located in Toulouse, France, is a research organization that aims to develop ad-
vanced methods for the numerical simulation of a wide range of large scientific and technological problems. CERFACS is organized as a
private entity with shareholders, which include government agencies ONERA, CNES, Meteo France, and corporate sponsors EADS,
SAFRAN, TOTAL, and Electricite de France (EDF). The shareholders fund the majority of research performed at CERFACS and, as a
result, jointly own research results and intellectual property. The institute employs approximately 150 people, of which 130 are technical
staff including physicists, applied mathematicians, numerical analysts, and software engineers. The institute is organized around inter-

34
 disciplinary teams that focus on the core fundamental area of numerical methods for parallel computing, combined with more applied fo-
cus areas in aerodynamics, gas turbines, combustion, climate, environmental impact, data assimilation, electromagnetism and acoustics
and multidisciplinary code coupling. The CERFACS model is interesting because it brings together common computational science
problems from different areas such as aeronautics, space, weather/climate modeling, and combustion, and includes combined govern-
ment-industrial sponsorship.
The C2A2S2E institute6 at DLR in Braunschweig Germany provides a model that is more focused on the development of computational
science for specific aeronautics applications. The institute is jointly funded by DLR, Airbus, and the German State of Lower Saxony
(Niedersachsen). The objective of the institute is to be an “interdisciplinary center of excellence in numerical aircraft simulations.” The
institute was conceived as a major new aerospace simulation center under the DLR Institute of Aerodynamics and Flow Technology in
Braunschweig, with the objective of providing a campus-like environment that brings together world-renowned experts and guest scien-
tists to stimulate top-level research in the field of numerical simulation. Another function of the institute is to provide high-end computer
simulation and visualization hardware and capabilities. C2A2S2E employs approximately 50 technical staff with expertise in applied
mathematics, computer science, and aerospace engineering.
In past years, NASA has used field-center co-located institutes such as ICASE at NASA Langley, ICOMP at NASA Glenn, and RIACS at
NASA Ames as vehicles for long-term research and to better engage the broader scientific community. Arguably, the most successful of
these was ICASE, which was created in 1972 and was supported for 30 years. The goal of ICASE was to perform long-term research in
computational science and broadly related fields that was relevant to NASA’s aeronautics mission. The institute was rather small, with
an average fluctuating total of 30 people of which approximately 25 were researchers, and with a robust visitor program of 50 to 70 peo-
ple per year. Institute funding consisted of a mix of long-term core and short-term task funding from NASA, in approximately a 60/40 ra-
tio, with a total budget of about $2M in the 1990s. While task funding was obtained directly from the supporting NASA center (Langley),
experience had shown that it was important for the core funding to be obtained from NASA Headquarters directly, in order to shield the
institute from the shorter term pressures and objectives of the supporting NASA center. A key to success was to be aware of long-term
NASA goals, but also to acquire in-depth knowledge of broader interdisciplinary research performed at other research centers and uni-
versities both within the United States and internationally. A central purpose of the visitor program was to keep abreast of emergent
technologies that the institute should be investing in to meet NASA’s long-term needs. Although shorter term in nature, task funding pro-
vided critical mass for the institute, lowering administrative costs, while at the same time tying the institute more closely to the needs of
the NASA center, and thus enabling a better long-term vision. However, pressure to grow the institute through increased task funding
needed to be resisted for the institute to retain its long-term focus. The institute was structured as a private nonprofit entity, managed by
an outside umbrella organization.
The above examples illustrate how differently structured research institutes can be used to achieve multidisciplinary and longer term re-
search goals. In most cases, it is not the direct level of funding that determines the success of the institute; rather, it is the establishment
of a structure that enables engagement of the broader scientific community and provides a long-term focus aligned with the sponsoring
agency or stakeholder goals, while shielding the institute from shorter term pressures and objectives.
1 http://www.swri.org/
2 http://www.ima.umn.edu/
3 http://nnsa.energy.gov/aboutus/ourprograms/defenseprograms/futurescienceandtechnologyprograms/asc/univpartnerships/psaap
4 http://ctr.stanford.edu/
5 http://www.cerfacs.fr/
6 http://www.dlr.de/as/en/desktopdefault.aspx/tabid-4083/6455_read-9239/

Engineers (IEEE), and the Association for Computing Ma- plines into these related fundamental areas. However, given
chinery (ACM). This trend can be noted as well in the vari- its limited budget, NASA ARMD can ill afford to start new
ous survey reports compiled for the national aerospace enter- programs in these areas. Rather, the course of action should
prise, such as the NAE decadal surveys12 and the National focus on specific areas with potentially high payoff that are
Aerospace R&D plans,13 which repeatedly refer to specific unique to the NASA mission, as well as increased leveraging
aerospace capabilities that must be developed through nu- of ongoing activities within other government programs in
merical simulation, without acknowledging the role devel- the United States and abroad. At a minimum, NASA should
opments in other related fundamental disciplines play in establish a presence at the important computational science
meeting these objectives. This should be contrasted with nu- meetings hosted by SIAM, IEEE, and ACM to keep abreast
merous reports issued by the DOE and NSF and other com- of developments in these areas.
putational science-based studies that emphasize the needed
Develop and foster collaborations with other disciplines
investment in areas at the intersection of applied math, com-
through the creation of research institutes. As mentioned
putational science and application science disci-
above, investments in applied math and computer science
plines.16,18,37,106 An essential component for advancing state-
will be critical for achieving specific advances in CFD.
of-the-art CFD will require that the research portfolio be ex-
Likewise, advances in MDAO will require investments in
panded beyond pure aeronautics and fluid mechanics disci-

35
key disciplines outside of traditional aerodynamics. Howev- would be to broaden NASA interactions with the wider sci-
er, the current makeup of NASA and the broader aerospace entific community and to foster long-term collaborations
community is such that it will be difficult to obtain the re- with experts in important complementary fields. To be effec-
quired multidisciplinary expertise from within. Furthermore, tive, the institute will require critical mass, sustained core
it will likely be difficult to ensure relevance of investments in funding over the long term, and the development of a reputa-
these more distantly related fields for maximum impact in tion for excellence. One approach to building up this facility
aeronautics. would be to start with seed funding for a sustained summer
visitor program that would grow in size over several years,
In general, NASA relies on a mix of internal development
and which can be complemented with a small group of inter-
and external funding of academia and industry through
nal researchers or longer term visitors.
NASA Research Announcements (NRAs). However, other
mechanisms are needed to engage scientists in other related
Recommendation 6: NASA should attract world-class engi-
disciplines. In the past, NASA relied on semi-academic insti-
neers and scientists.
tutes such as the Institute for Computer Applications in Sci-
ence and Engineering (ICASE), the Institute for Computa- The ability to achieve the long-term goals for CFD in 2030 is
tional Mechanics in Propulsion (ICOMP), the Research Insti- greatly dependent on having a team of highly educated and
tute for Advanced Computer Science (RIACS) set up at vari- effective engineers and scientists devoted to the advancement
ous NASA centers, and the Center for Turbulence Research of computational sciences. This is particularly critical within
(CTR) at Stanford University, to foster collaboration in spe- NASA given the de-
cific, related disciplines and to engage the broader national mographics of the cur-
and international scientific community in problems of rele- rent workforce. Oppor-
vance to the NASA aeronautics mission. During the last dec- tunities to stabilize and
ade, larger institutes such as the National Institute of Aero- expand CFD and simula-
space (NIA) and the Ohio Aerospace Institute (OAI) were tion personnel in the
created and absorbed some of these functions. However, future should be pursued
these institutes, which serve many purposes, are not devoted to enable a renewed
to specifically advance computational science issues and leadership role in CFD
were not conceived to focus principally on NASA’s longer development including
term requirements, (although they could be modified to in- researchers and developers with diverse backgrounds (physi-
clude such aspects). It is noteworthy that the small, semi- cal modeling, numerical algorithms, applied mathematics,
academic institute model has been replicated at various other computer science, software development, and various aero-
US national labs, such as the Center for Nonlinear Studies space engineering application domains). Attracting this fu-
(CNLS) at Los Alamos National Laboratory (LANL) and the ture generation of leaders will present challenges. To be suc-
Institute for Scientific Computing Research (ISCR) at Law- cessful, several suggestions are presented here. NASA cur-
rence Livermore National Laboratory (LLNL), as well as in rently has several fellowship programs in key areas through-
Europe at the European Centre for Research and Advanced out the agency, but none are specifically devoted to computa-
Training in Scientific Computation (CERFACS) in France tional science. A NASA-focused fellowship program similar
and the Center for Computer Applications in AeroSpace Sci- to the Department of Energy’s Computational Science Grad-
ence and Engineering (C2A2S2E) in Germany, with great uate Fellowship (DOE-CSGF)154 should be considered. In
success. Indeed, some of these centers include industrial addition, having opportunities for longer term visits (e.g.,
funding in their model, further tying the institute to important summer or other) for students will also be important to con-
stakeholders. One of the keys to the success of these insti- tinually attract the best aerospace talent: this is best achieved
tutes is that they are able to establish a facility and create a through formal visit programs with specific computational-
climate that fosters collaborations among government re- science goals, managed either directly through NASA or
searchers and leading academics with the goal of advancing through supported research institutes. Finally, attempts to
computational science and modeling of physics. The freedom capture future generations of computational engineers and
that is afforded to the researchers while aligning broadly with scientists can benefit from providing visiting students with
the mission of the organization serves as a magnet for attract- the opportunity to meaningfully contribute to grand-
ing the best talent from around the world, while ensuring challenge efforts that capture their imagination and are close-
relevance of the research. NASA should re-examine the role ly aligned with their values (environmental impact of com-
such institutes have played in the overall long-term success mercial aviation, personal air vehicle, larger degrees of au-
of computational methods within the agency and develop a tonomy/UASs, and the sizable interest in high-speed vehicles
new initiative to create an institute devoted to the advance- including supersonics, hypersonics, and re-entry).
ment of both CFD and MDAO for NASA aeronautics, sci-
ence, and space exploration mission directorate problems
(see accompanying Case Study). The goal of the institute

36
8 Conclusions the extended Vision 2030 CFD Study team: Joerg Ga-
blonsky, Mori Mani, Robert Narducci, Philippe Spalart, and
Despite considerable past success, today there is a general Venkat Venkatakrishnan, The Boeing Company, and Robert
feeling that CFD development for single and multidiscipli- Bush, Pratt & Whitney, for their valuable technical contribu-
nary aerospace engineering problems has been stagnant for tions in the execution of the study and in the preparation of
some time, caught between rapidly changing HPC hardware, this document. Also, we would like to acknowledge the ac-
the inability to predict adequately complex separated turbu- tive participation of the attendees at the Vision 2030 CFD
lent flows, and the difficulties incurred with increasingly study workshop held in May 2013 at the National Institute of
complex software driven by complex geometry and increas- Aerospace (NIA), especially Paul Durbin, Iowa State Univer-
ing demands for multidisciplinary simulations. In this report, sity, Sharath Girimaji, Texas A&M University, and Brian
we have provided a knowledge-based vision of what future Smith, Lockheed Martin Corporation, for their presentations
CFD capabilities could be, and indeed must be, in order to on turbulence modeling. We are grateful for additional input
radically advance the aerospace design process and enable a provided by (in no particular order): Dave Schuster (NASA),
new generation of more capable aerospace vehicles. This Michael Aftosmis (NASA), Chris Rumsey (NASA), Joe
vision was used to identify important barriers and impedi- Morrison (NASA), Jim Thomas (NASA), Manny Salas
ments, which in turn were used to formulate a long-term (NASA/ICASE retired), Jamshid Samareh (NASA), Reyn-
technology development plan. These findings were obtained aldo Gomez (NASA), Michael Hemsch (NASA), Earl Dow-
with broad community input, including a formal survey, a ell (Duke University), Ron Joslin (US Navy), David Young
community workshop, as well as through numerous informal (Boeing), Zhi Jian Wang (University of Kansas), Karthik
interactions with subject matter experts. This information Duraisamy (University of Michigan), Edwin van der Weide
was distilled into a programmatic structure and a set of rec- (University of Twente), Rob Falgout (LLNL), Laurent Cam-
ommendations that are expected to be important for advanc- bier (ONERA), Sylvie Plot (ONERA), Bijan Mohammadi
ing the state-of-the-art CFD in particular and multidiscipli- (CERFACS/Universite de Montpellier), Norbert Kroll
nary simulations in general, while also achieving the Vision. (DLR), Cord Rossow (DLR), Kazuhiro Nakahashi (JAXA)
Although some outcomes of this study point to specific tech- and Atsushi Hashimoto (JAXA). Finally, we would like to
nological solutions, many recommendations are simple and thank all who participated in the Vision 2030 CFD survey for
self-evident: robust support for basic simulation technolo- their valuable time and insightful comments.
gies, better access to leading-edge HPC hardware, better in-
ternal collaborations between aeronautics and space drivers,
better coordination with other large computational science 10 References
programs, and the need for innovative strategies to advance
the research agenda in a resource-constrained environment. 1. Malik, M. R. and Bushnell, D. M. (eds.), “Role of
Computational Fluid Dynamics and Wind Tunnels in
Many current large government agency programs in compu-
Aeronautics R&D”, NASA TP-2012-217602, Septem-
tational science can trace their roots to reports that originated
ber 2012.
from communities of experts, from the grassroots upwards,
often based on input from community workshops, commis- 2. Rubbert, P. E., “AIAA Wright Brothers Lecture: CFD
sions, or private and public panels and testimonies. We have and the Changing World of Aircraft Development”,
followed such a model in this work, with the realization that ICAS-94-0.2, September 1994.
many such endeavors are not successful on their first attempt, 3. Jameson, A., “Re-engineering the Design Process
but often require years of gestation. However, the broad Through Computation”, AIAA Journal of Aircraft, Vol.
community input, general consensus, and wide range of ex- 36, 1999, pp. 36-50, 10.2514/2.2412
pert opinions coupled with a diverse experience base from 4. Goldhammer, M. I., “Boeing 787 – Design for Optimal
academia, government, and industry that have contributed to Airplane Performance”, CEAS/KATnet Conference on
this report make it a significant advocacy document. To ulti- Key Aerodynamic Technologies, Bremen, Germany,
mately be successful, periodic reviews of CFD technology June 2005.
development like this must be undertaken to continually
drive the state of the art forward. 5. McKinney, R., “Large Eddy Simulation of Aircraft Gas
Turbine Combustors at Pratt and Whitney – Current
Experience and Next Steps”, Presentation GT2011-
46913, ASME Turbo Expo, Vancouver, Canada, June
9 Acknowledgments 2011.
This study was performed under NASA contract number 6. Lorence, C., “Engine OEM --GE Aviation Perspective-
NNL08AA16B, task number NNL12AD05T with William LEAP”, Presentation at GT2012-70198, ASME Turbo
Kleb as Contracting Officer Representative (COR) and Mu- Expo, Copenhagen, Denmark, June 2012.
jeeb Malik as Technical Monitor. The authors wish to thank

37
7. Abdol-Hamid, K. S., Ghaffari, F., and Parlette, E. B., 19. Levy, D. W., Zickuhr, T., Vassberg J.C., Agrawal, S.,
“Overview of Ares-I CFD Ascent Aerodynamic Data Walls, R. A., Pirzadeh, S., and Hemsch, M. J., “Data
Development and Analysis based on USM3D”, AIAA Summary from the First AIAA Computational Fluid
Paper 2011-15, 49th AIAA Aerospace Sciences Meet- Dynamics Drag Prediction Workshop”, AIAA Journal
ing, January 2011, 10.2514/6.2011-15 of Aircraft, Vol.40, 2003, 10.2514/1.1684.
8. Wright, M., Edquist, K., Tang, C. Y., Hollis, B., Kra- 20. Laflin, K. R., Klausmeyer, S. M., Zickuhr, T., Vass-
sa, P., and Campbell, C., “A Review of Aerothermal berg, J. C., Walls, R. A., Morrison, J. H., Brodersen, O.
Modeling for Mars Entry Missions”, AIAA Paper P., Rakowitz, M. E., Tinoco, E. N., and Godard, J-L,
2010-443, 48th AIAA Aerospace Sciences Meeting, “Data Summary from the Second AIAA Computational
January 2010, 10.2514/6.2010-443. Fluid Dynamics Drag Prediction Workshop”, AIAA
9. Gusman, M., Housman, J., and Kiris, C., “Best Practic- Journal of Aircraft, Vol.42, pp. 1165-1178, 2005,
es for CFD Simulations of Launch Vehicle Ascent with 10.2514/1.10771.
Plumes: OVERFLOW Perspective”, AIAA Paper 21. Vassberg, J. C., Tinoco, E. N., Mani, M., Brodersen, O.
2011-1054, 49th AIAA Aerospace Sciences Meeting, P., Eisfeld, B., Walls, R. A., Morrison, J. H., Zickuhr,
January 2011, 10.2514/6.2011-1054. T., Laflin, K. R., and Mavriplis, D. J., “Abridged
10. Kraft, E. M., “Integrating Computational Science and Summary of the Third AIAA Computational Fluid Dy-
Engineering with Testing to Re-engineer the Aeronau- namics Drag Prediction Workshop”, AIAA Journal of
tical Development Process”, AIAA Paper 2010-0139, Aircraft, Vol.45, pp. 781-798, 2008, 10.2514/1.30572.
48th AIAA Aerospace Sciences Meeting, January 2010, 22. Vassberg, J. C., Tinoco, E. N., Mani, M., Rider, B.,
10.2514/6.2010-139. Zickuhr, T., Levy, D. W., Brodersen, O. P., Eisfeld, B.,
11. Biswas, R., Aftosmis, M. J., Kiris, C., and Shen, B. W., Crippa, S., Walls, R. A., Morrison, J. H., Mavriplis, D.
“ Petascale Computing: Impact on Future NASA Mis- J., and Murayama, M., “Summary of the Fourth AIAA
sions”, Petascale Computing: Architectures and Algo- CFD Drag Prediction Workshop”, AIAA Paper 2010-
rithms (D. Bader, ed.), Chapman and Hall / CRC Press, 4547, 28th AIAA Applied Aerodynamics Conference,
2007. June 2010, 10.2514/6.2010-4547.
12. Kaminski, P. G. (chair), “Decadal Survey of Civil Aer- 23. Levy, D. W., Laflin, K. R., Tinoco, E. N., Vassberg, J.
onautics”, National Research Council, The National C., Mani, M., Rider, B., Rumsey, C., Wahls, R. A.,
Academies Press, ISBN 0-309-10158-1, 2006. Morrison, J. H., Broderson, O. P., Crippa, S., Mav-
riplis, D. J., and Murayama, M. “Summary of Data
13. Executive Office of the President, National Science and
from the Fifth AIAA CFD Drag Prediction Workshop”,
Technology Council, “National Plan for Aeronautics
AIAA Paper 2013-0046, 51st AIAA Aerospace Scienc-
Research and Development and Related Infrastructure”,
es Meeting, January 2012, 10.2514/6.2013-46.
December 2007,
http://www.whitehouse.gov/sites/default/files/microsite 24. Tinoco, E. N., Bogue, D. R., Kao, T.-J., Yu, N. J., Li,
s/ostp/aero-natplan-2007.pdf. P., and Ball, D. N., “Progress Toward CFD for Full
Flight Envelope”, The Aeronautical Journal, Vol. 109,
14. Hallion, R. P. (ed.), NASA’s Contributions to Aero-
pp. 451-460, October 2005.
nautics Vol. 1, National Aeronautics and Space Admin-
istration, NASA/SP-2010-570-Vol. 1, 2010, pp. 427- 25. “Responding to the Call: Aviation Plan for American
458. Leadership”, Congressionally requested report pro-
duced under the auspices of the National Institute of
15. Bishof, C. and Simon, H. D., “The implementation of
Aerospace, Hampton VA. April 2005.
the US High Performance Computing and Communica-
tion Program”, Supercomputer ’92, (H.-W. Meuer ed.) 26. Gomez, R. J., and Slotnick, J. P., “Preliminary Analysis
Springer-Verlag, 1992. of Ascent Loads on a Space Shuttle Liquid Flyback
Booster Configuration”, AIAA Paper 95-1864, 13th
16. Keyes, D. E. (chair), “A Science-Based Case for Large-
Applied Aerodynamics Conference, June 1995,
Scale Simulation”, Volume 1, Office of Science, US
10.2514/6.1995-1864.
Department of Energy, July 2003.
27. Gomez, R. J., Vicker, D., Rogers, S. E., Aftosmis, M.
17. Benioff, M. R., and Lazowska, E. D. (co-chairs),
J., Chan, W. M., Meakin, R., and Murman, S., “STS-
“Computational Science: Ensuring America’s Competi-
107 Investigation Ascent CFD Support”, AIAA Paper
tiveness. Report to the President”, President’s Infor-
2004-2226, 34th AIAA Fluid Dynamics Conference
mation Technology Advisory Committee, June 2005.
and Exhibit, June 2004, 10.2514/6.2004-2226.
18. Oden, T. (chair), “Cyber Science and Engineering: A
28. Gnoffo, P. A., “Computational Aerothermodynamics in
Report of the NSF Advisory Committee for Cyber-
Aeroassist Applications”, AIAA Journal of Spacecraft
infrastructure Task Force on Grand Challenges”, NSF
and Rockets, Vol.40, pp. 305-312, 2003,
Report, March 2011.
10.2514/2.3957.

38
29. Kless, J. and Aftosmis, M., “Analysis of Grid Fins for 40. Mavriplis, D., “Exaflops Opportunities for the NASA
Launch Abort Vehicle Using a Cartesian Euler Solver”, Fundamental Aeronautics Program”, NIA CFD Con-
AIAA Paper 2011-3555, 29th AIAA Applied Aerody- ference, Future Directions in CFD Research, A Model-
namics Conference, June 2011, 10.2514/6.2011-3666. ing and Simulation Conference, August 6-8, 2012,
30. Bartels, R. E, Chwalowski, P., Massey, S. J., Heeg, J., Hampton, VA.
and Mineck, R. E., “Computational Aeroelastic Analy- 41. Sarkar, V. (ed.), “ExaScale Software Study: Software
sis of the Ares I Crew Launch Vehicle During Ascent”, Challenges in Extreme Scale Systems”, DARPA,
AIAA Journal of Spacecraft and Rockets, Vol. 49, No. IPTO, AFRL report under contract FA8650-07-C-7724,
4, pp. 651-658, July/August 2012. September 2009.
31. Schuster, D., “The Expanding Role of Applications in 42. http://www.exascale.org
the Development and Validation of CFD at NASA”, 43. Carver, J., Heaton, D., Hochstein, L., Bartlett, R. "Self-
Proceedings of the 6th International Conference on Perceptions about Software Engineering: A Survey of
Computational Fluid Dynamics, St. Petersburg Russia, Scientists and Engineers", Computing in Science and
July 2010, (Keynote), Springer. Engineering. Vol 15(1), pp.7-11, Jan/Feb 2013.
32. “American Competitiveness Initiative: Leading the 44. Nompelis, I., Candler, G., Conti, R., “A Parallel Implic-
World in Innovation”, Domestic Policy Council, Office it CFD Code for the Simulation of Ablating Re-Entry
of Science and Technology Policy, February 2006. Vehicles”, AIAA Paper 2009-1562, 47th AIAA Aero-
33. Glotzer, S. C., Kim, S., Cumings, P. T., Deshmukh, A., space Sciences Meeting, Jan. 2009, 10.2514/6.2009-
Head-Gordon, M., Karniadakis, G., Pezold, L., Sagui, 1562.
C., and Shinozuka, M., “International Assessment of 45. INCITE Program, US Department of Energy,
Research and Development in Simulation-Based Engi- http://www.doeleadershipcomputing.org/incite-
neering and Science”, World Technology Evaluation program/
Center, Inc., URL: http://www.wtec.org/sbes/SBES-
46. Spalart, P. R. and Allmaras, S. R., "A One-Equation
GlobalFinalReport.pdf, 2009.
Turbulence Model for Aerodynamic Flows", La Re-
34. Scientific Discovery through Advanced Computing. cherche Aerospatiale, No. 1, 1994, pp. 5-21.
http://www.scidac.gov.
47. Menter, F., “Two-equation eddy-viscosity turbulence
35. Ad Hoc Committee and Technical Working Group for models for engineering applications”, AIAA Journal,
a Petascale Collaboratory for the Geosciences. “Estab- Vol.32, pp.1598-1605, 1994,10.2514/3.12149.
lishing a Petascale Collaboratory for the Geosciences:
48. Wilcox, D. C., Turbulence Modeling for CFD, DCW
Scientific Frontiers”, Report to the Geosciences Com-
Industries, 3rd edition, November 2006.
munity, UCAR/JOS., 80pp, June 2005.
49. Stock, H.W., and Haase, W., “Navier-Stokes Airfoil
36. Ad Hoc Committee and Technical Working Group for
Computations with eN Transition Prediction Including
a Petascale Collaboratory for the Geosciences. “Estab-
Transitional Flow Regions”, AIAA Journal, Vol. 38,
lishing a Petascale Collaboratory for the Geosciences:
No. 11, pp. 2059–2066, 2006, 10.2514/2.893.
Technical and Budgetary Prospectus”, Report to the
Geosciences Community, UCAR/JOS., 56pp, October 50. Langtry, R. B., Menter, F. R., “Correlation-Based Tran-
2005. sition Modeling for Unstructured Parallelized Compu-
tational Fluid Dynamics Codes”, AIAA Journal, Vol.47,
37. Oden, J. T. (chair), “Simulation-Based Engineering
pp. 2894-2906, 2009, 10.2514/1.42362.
Science – Revolutionizing Engineering Science
through Simulation”, National Science Foundation, 51. Sitaraman, J., Potsdam, M., Wissink, A., Jayaraman,
May 2006. B., Datta, A., Mavriplis, D., and Saberi, H., “Rotor
Loads Prediction Using Helios: A Multisolver Frame-
38. Mavriplis, D., “High Performance Computational En-
work for Rotorcraft Aeromechanics Analysis”, AIAA
gineering: Putting the ‘E’ Back in ‘CSE’ ”, Proceedings
Journal of Aircraft, Vol.50, pp. 478-492, 2013,
of the 21st International Conference on Parallel Compu-
10.2514/1.C031897.
tational Fluid Dynamics, Moffett Field, California, May
18−22, 2009 (Plenary), (R. Biswas, ed.), DEStech Pub- 52. Blattnig, S. R., Luckring, J. M., Morrison, J. H., Syl-
lications Inc., Lancaster PA. vester, A. J., Tripathi, R. K., Zang, T. A., “NASA
Standard for Models and Simulations: Philosophy and
39. Mavriplis, D., Darmofal, D., Keyes, D. and Turner, M.,
Requirements Overview”, AIAA Journal of Aircraft,
“Petaflops Opportunities for the NASA Fundamental
Vol.50, pp. 20-28, 2013, 10.2514/1.C000303.
Aeronautics Program”, AIAA Paper 2007-4084,18th
AIAA Computational Fluid Dynamics Conference, 53. McHale, M. P., and Friedman, J. R. (chairs), Standard
June 2007, 10.2514/6.2007-4084. for Verification and Validation in Computational Fluid
Dynamics and Heat Transfer, The American Society of

39
 Mechanical Engineers, ASME V&V 20-2009, Decem- 66. Claus, R., Townsend, S., Carney, D., Horowitz, J., and
ber 2009. Turner, M., “A Case Study of High Fidelity Engine
54. Editorial Policy Statement on Numerical and Experi- System Simulation”, AIAA Paper 2006-4971, 42nd
mental Accuracy, AIAA Journal, Vol 51, No.1, January AIAA/ASME/SAE/ASEE Joint Propulsion Confer-
2013. ence, Sacramento, CA, July 2006, 10.2514/6.2006-
4971
55. Mavriplis, D. J., Vassberg, J., Tinoco, E., Mani, M.,
Brodersen, O., Eisfeld, B., Wahls, R., Morrison, J., 67. Song, F., Haase, W., Peng, S-H., and Schwamborn, D.
Zickuhr, T., Levy, D., and Murayama, M., “Grid Quali- (eds.), Progress in Hybrid RANS-LES Modeling,
ty and Resolution Issues from the Drag Prediction Springer Press, ISBN 978-3-642-31817-7, Sept. 2011.
Workshop Series”, AIAA Journal of Aircraft, Vol. 46, 68. Spalart, P. R., “Detached-Eddy Simulation”, Annual
No. 3, pp. 935-950, March 2009. Review of Fluid Mechanics, Vol 41, pp. 181-202, 2009.
56. Rumsey, C. L., Slotnick, J. P., “Overview and Sum- 69. Batten, P., Goldberg, U., and Chakravarthy, S., “Using
mary of the Second AIAA CFD High-Lift Prediction Synthetic Turbulence to Interface RANS and LES”,
Workshop”, AIAA Paper 2014-747 52nd AIAA Aero- AIAA Paper 2003-0081, 41st AIAA Aerospace Scienc-
space Sciences Meeting, Jan. 2014, 10.2514/6.2014- es Meeting, Jan 2003, 10.2514/6.2003-81.
0747. 70. Hanjalic, K., "Will RANS Survive LES? A View of
57. Alonso, J. J., LeGresley, P., and van der Weide, E., Perspectives", J. Fluids Engineering, Vol. 27, pp. 831-
“pyMDO: A Framework for High-Fidelity Multi- 839, 2005.
Disciplinary Optimization”, AIAA Paper 2004-4480, 71. Menter, F. R., “Turbulence Modeling for Engineering
10th AIAA/ISSMO Mutidisciplinary Analysis and Op- Flows”, Ansys, Inc. 2011
timization Conference, Sept. 2004, 10.2514/6.2004-
72. Medida, S., and Baeder, J., “A New Crossflow Transi-
4480
tion Onset Criterion for RANS Turbulence Models”,
58. Martins, J. R. R. A., and Lambe, A. B., “Multidiscipli- AIAA Paper 2013-3081, 21st AIAA Computational
nary Design Optimization: A Survey of Architectures”, Fluid Dynamics Conference, Jan 2013,10.2514/6.2013-
AIAA Journal, Vol. 51, No 9, pp. 2049-2075, 2013. 3081.
59. Wissink, A., Sitaraman, J., Sankaran, V., Mavriplis, D., 73. Wang, L., Anderson, W. K., Erwin, J., and Kapadia, S.,
and Pulliam, T., “A Multi-Code Python-Based Infra- “High-order Methods for Solutions of Three-
structure for Overset CFD with Adaptive Cartesian dimensional Turbulent Flows”, AIAA Paper 2013-856,
Grids”, AIAA Paper 2008-927, 46th AIAA Aerospace 51st AIAA Aerospace Sciences Meeting, Jan 2013,
Sciences Meeting, Jan. 2008, 10.2514/6.2008-927. 10.2514/6.2013-856.
60. Meakin, R. L., Atwood, C. A., and Hariharan, N., “De- 74. Persson, P-O., Willis, D., and Peraire, J., “The Numeri-
velopment, Deployment, and Support of Multi- cal Simulation of Flapping Wings at Low Reynolds
Disciplinary, Physics-Based Simulation Software Numbers”, AIAA Paper 2010-724, 48th AIAA Aero-
Products”, AIAA Paper 2011-1104, 49th AIAA Aero- space Sciences Meeting, Jan 2010, 10.2514/6.2010-
space Sciences Meeting, Jan 2011, 10.2514/6.2011- 724.
1104.
75. Haimes, R., and Aftosmis, M. J., “On Generating High
61. Harrow, A. W., Hassidim, A., and Lloyd, S., “Quantum Quality "Water-tight" Triangulations Directly from
Algorithm for Linear Systems of Equations”, Physical CAD”, Meeting of the International Society for Grid
Review Letters, Vol. 103, October 9, 2009, 150502. Generation, (ISGG), Honolulu, HI, 2002.
62. Top 500 Supercomputer Sites; http://www.top500.org/ 76. Mavriplis, D. J., Aftosmis, M. J., and Berger, M.,
63. Fremaux, C. M. and Hall, R. M., “COMSAC: Compu- “High Resolution Aerospace Applications using the
tational Methods for Stability and Control”, NASA/CP- NASA Columbia Supercomputer”, International Jour-
2004-213028, Parts 1 and 2, 2004. nal of High Performance Computing Applications, Vol.
64. Candler, G. V., “Detached-Eddy Simulation of the 21, pp. 106-126, 2007.
Reentry-F Flight Experiment”, AIAA Journal of Space- 77. Pulliam, T. H. and Jespersen, D. C., “Large Scale Aer-
craft and Rockets, Vol.49, pp. 691-699, 2012, odynamic Calculation on Pleiades”, Proceedings of the
10.2514/1.A32128. 21st International Conference on Parallel Computational
65. Rumsey, C. L., Slotnick, J. P., Long, M., Stuever, R. Fluid Dynamics, Moffett Field, California, May 18−22,
A., and Wayman, T. R., “Summary of the First AIAA 2009, (R. Biswas, ed.), DEStech Publications Inc.,
CFD High-Lift Prediction Workshop”, AIAA Journal of Lancaster PA.
Aircraft, Vol.48, pp. 2068-2079, 2011, 78. Duffy, A. C., Hammond, D. P., and Nielsen, E. J.,
10.2514/1.C031447. “Production Level CFD Code Acceleration for Hybrid

40
 Many-Core Architectures”, NASA/TM-2012-217770, ement Methods”, Acta Numerica, Vol 10, pp. 1-102,
October 2012. May 2001.
79. Aftosmis, M. J., Berger, M. J., and Melton, J. E., “Ro- 91. Giles, M.B. and Suli, E., “Adjoint Methods for PDEs: a
bust and Efficient Cartesian Mesh Generation for Posteriori Error Analysis and Postprocessing by Duali-
Component-Based Geometry”, AIAA Journal, Vol. 36, ty”, Acta Numerica, Vol 11, pp. 145-236, Jan 2002.
No. 6, pp-952-960, 1998, 10.2514/2.464. 92. Hartman, R. and Houston, P., “Error Estimation and
80. Modisette, J., and Darmofal, D., “Toward a Robust, Adaptive Mesh Refinement for Aerodynamic Flows”,
Higher-Order Cut-Cell Method for Viscous Flows”, Von Karman Institute Lecture Series 2010-01
AIAA Paper 2010-721, 48th AIAA Aerospace Sciences CFD/ADIGMA course on hp-adaptive and hp-
Meeting, Jan 2010, 10.2514/6.2010-721. multigrid methods, 2009.
81. Katz, A., Wissink, A., Sitaraman, J., and Sankaran, V., 93. Fidkowski, K.J. and Darmofal, D.L., “Review of Out-
“Application of Strand Meshes to Complex Aerody- put-based Error Estimation and Mesh Adaptation in
namic Flow Fields”, AIAA Paper 2010-4934, 28th AI- Computational Fluid Dynamics”, AIAA Journal, Vol.
AA Applied Aerodynamics Conference, June 2010, 49, No. 4, 2011.
10.2514/6.2010-4934. 94. Venditti, D. A. and Darmofal, D. L., “Anisotropic Grid
82. Katz, A., and Jameson, A, “Meshless Scheme Based on Adaptation for Functional Outputs: Application to
Alignment Constraints”, AIAA Journal, Vol.48, pp. Two-dimensional Viscous Flows”, Journal of Compu-
2501-2511, 2010, 10.2514/1.J050127. tational Physics, Vol. 187, No. 1, pp. 22–46, 2003.
83. Nemec, M. Aftosmis, M. J., and Wintzer, M., “Adjoint- 95. Wang, Q., “Forward and adjoint sensitivity computa-
Based Adaptive Mesh Refinement for Complex Geom- tion for chaotic dynamical systems”, Journal of Com-
etries”, AIAA Paper 2008-0725, 46th AIAA Aerospace putational Physics, Vol. 235, No. 15, pp. 1-13, 2013.
Sciences Meeting, Jan 2008, 10.2514/6.2008-725. 96. Wang, Q., Gomez, S.A., Blonigan, P.J., Gregory, A.L.,
84. Wissink, A., Potsdam, M., Sankaran, V., Sitaraman, J., and Qian, E.Y., “Towards scalable parallel-in-time tur-
Yang, Z., and Mavriplis, D., “A Coupled Unstructured- bulent flow simulations”, Physics of Fluids, Vol. 25,
Adaptive Cartesian CFD Approach for Hover Predic- 110818, 2013, 10.1063/1.4819390.
tion ", Paper presented at the American Helicopter So- 97. Press, S. J. , Subjective and Objective Bayesian Statis-
ciety (AHS) Forum, Phoenix AZ, May 2010. tics: Principles, Methods and Applications, 2nd edition,
85. Heeg, J., “Overview of the Aeroelastic Prediction 2003, Wiley, New York.
Workshop”, AIAA Paper 2013-783, 51st AIAA Aero- 98. SIAM Conference on Uncertainty Quantification, Sa-
space Sciences, Jan 2013, 10.2514/6.2013-783. vannah, GA, March 31-April 3, 2014; sponsored jointly
86. Johnson, F. T., Kamenetskiy, D. S., Melvin, R. G., by the National Science Foundation (NSF) and the US
Venkatakrishnan, V., Wigton, L. B., Young, D. P., Department of Energy (DOE);
Allmaras, S. R., Bussoletti, J. E. and Hilmes, C. L., http://www.siam.org/meetings/uq14.
“Observations Regarding Algorithms Required for Ro- 99. Wang, Y., Yu, H., and Ma, K-L, “Scalable Parallel
bust CFD Codes”, Mathematical Modeling of Natural Feature Extraction and Tracking for Large Time-
Phenomena, Vol 6 (03) ,pp. 2–27, 2011. Varying 3D Volume Data”, Proceedings of EGPGV
87. Tadmor, E., “The Numerical Viscosity of Entropy Sta- 2013, May 2013, pp. 55-62.
ble Schemes for Systems of Conservation Laws. I”, 100. Willcox, K., “ Model Reduction for Large-scale Appli-
Mathematics of Computation, Vol. 49, pp. 91–103, cations in Computational Fluid Dynamics”, Real-Time
1987. PDE-Constrained Optimization, L. Biegler, O. Ghattas,
88. Carpenter, M. H., and Fisher, T. C., “High-Order En- M. Heinkenschloss, D. Keyes, and B. van Bloemen
tropy Stable Formulations for Computational Fluid Dy- Waanders, eds. SIAM Philadelphia, February 2007.
namics”, AIAA Paper 2013-2868, 21st AIAA Computa- 101. The 1st Workshop on Integration of Experimental Fluid
tional Fluid Dynamics Conference, Jan 2013, Dynamics (EFD) and Computational Fluid Dynamics
10.2514/6.2013-2868. (CFD), JAXA Special Publication SP-09-002, January
89. Kroll, N., Bieler, H., Deconinck, H., Couallier, V., van 2010.
der Ven, H.; and Sorensen, K. (Eds.), “ADIGMA – A 102. Han, Z-H., and Görtz, S., “Hierarchical Kriging Model
European Initiative on the Development of Adaptive for Variable-Fidelity Surrogate Modeling”, AIAA Jour-
High-Order Variational Methods for Aerospace Appli- nal, Vol.50, pp.1885-1896, 2012, 10.2514/1.J051354.
cations”, Notes on Numerical Fluid Mechanics and
103. Yamazaki, W., and Mavriplis, D. J., “Derivative-
Multidisciplinary Design, Vol. 11, 2010, Springer.
Enhanced Variable Fidelity Surrogate Modeling for
90. Becker, R. and Rannacher, R., “An Optimal Control
Approach to a Posteriori Error Estimation in Finite El-

41
 Aerodynamic Functions”, AIAA Journal, Vol.51, 116. Yakhot, V., “Renormalization Group for Modeling of
pp.126-137, 2013, 10.2514/1.J051633. Turbulent Flows and Turbulent Combustion”, AIAA
104. Aftosmis, M. J. and Nemec, M., “Exploring Discretiza- Paper 91-218, 29th Aerospace Sciences Meeting, 1991,
tion Error in Simulation-Based Aerodynamic Data- 10.2514/6.1991-218.
bases”, Proceedings of the 21st International Confer- 117. Brandt, A., “Multiscale Solvers and Systematic Upscal-
ence on Parallel Computational Fluid Dynamics, (R. ing in Computational Physics”, Computer Physics
Biswas, ed.), DEStech Publications Inc., Lancaster PA, Communications, Vol 169, Issues 1–3, pp. 438-441, Ju-
2009. ly 2005.
105. Farhat, C, Lesoinne, M., and LeTallec, P., “A Con- 118. Honein, A. E., and Moin, P., “Higher Entropy Conser-
servative Algorithm for Exchanging Aerodynamic and vation and Numerical Stability of Compressible Turbu-
Elastodynamic Data in Aeroelastic Systems”, AIAA lence Simulations”, Journal of Computational Physics,
Paper 98-515, 36th AIAA Aerospace Sciences Meeting Vol 201, pp. 531–545, 2004.
and Exhibit, Jan 1998, 10.2514/6.1998-515 119. Subbareddy, P. K., and Candler, G. V., “A Fully Dis-
106. “Multiphysics Simulations: Challenges and Opportuni- crete Kinetic Energy Consistent Finite-volume Scheme
ties”, Argonne National Lab Report ANL/MCS-TM- for Compressible Flows”, Journal of Computational
321, Report from Workshop sponsored by the Institute Physics, Vol 228, pp. 1347–1364, 2009.
for Computing in Science (ICiS), Park City, Utah, June 120. Allaneau, Y., and Jameson, A., “Kinetic Energy Con-
-August, 2011. serving Discontinuous Galerkin Scheme”, AIAA Paper
107. Rumsey, C., Wedan, B., Hauser, T., and Poinot, M., 2011-198, 49th AIAA Aerospace Sciences Meeting, Jan
“Recent Updates to the CFD General Notation System 2011, 10.2514/6.2011-198.
(CGNS)”, AIAA Paper 2012-1264, 50th AIAA Aero- 121. Lallemand, P. and Luo, L-S., “Theory of the Lattice
space Sciences Meeting, Jan 2012, 10.2514/6.2012- Boltzmann Method: Dispersion, Dissipation, Isotropy,
1264. Galilean Invariance, and Stability”, Physical Review E
108. Kogge, P. (Ed.), “ExaScale Computing Study: Tech- 61:6546-6562, June 2000.
nology Challenges in Achieving Exascale Systems”, 122. Fares, E. and Noelting, S., “Unsteady Flow Simulation
Contractor report for AFRL Contract No. FA8650-07- of a High-Lift Configuration Using a Lattice Boltz-
C-7724, September 2008. mann Approach”, AIAA Paper 2011-869, 49th AIAA
109. Launder, B. E., and Shima, N., “Second Moment Clo- Aerospace Sciences Meeting, Jan 2011,
sure for the Near Wall Sublayer: Development and Ap- 10.2514/6.2011-869
plication”, AIAA Journal, Vol. 27, No. 10, pp. 1319- 123. Saad, Y., Iterative Methods for Sparse Linear Systems,
1325, 1989, 10.2514/3.10267. Second Edition, SIAM, 2003.
110. Eisfeld, B., “Reynolds Stress Modeling for Complex 124. Baker, A. H., Falgout, R. D., Kolev, Tz. V., and Yang,
Aerodynamic Flows”, Presented at the European Con- U. M., “Scaling Hypre’s Multigrid Solvers to 100,000
ference on Computational Fluid Dynamics, ECCO- Cores”, High Performance Scientific Computing: Algo-
MAS CFD 2010, Lisbon, Portugal, June 14−17, 2010. rithms and Applications, M. Berry et al., eds., Springer
111. Song, F., Haase, W., Peng, S-H., and Schwamborn, D. (2012). LLNL-JRNL-479591.
(eds.), Progress in Hybrid RANS-LES Modeling, 125. Jameson, A., “Aerodynamic Design via Control Theo-
Springer Press, ISBN 978-3-642-31817-7, Sept. 2011. ry”, ICASE Report No. 88-64, November 1988, also, J.
112. Direct and Large-Eddy Simulation 9 (DLES 9), of Scientific Computing, Vol. 3, pp. 233-260, 1988.
ERCOFTAC Workshop, April 3-9, 2013, Dresden, 126. Errico, R. M., “What is an adjoint model?”, Bulletin of
Germany, http://www.dles9.org the American Meteorological Society, 8(11):2577–
113. Tucker, P. G., “Computation of Unsteady Tur- 2591, 1997.
bomachinery Flows: Part 2-LES and Hybrids”, Pro- 127. Taylor, A. C., Putko, M. M., Green, L. L., and New-
gress in Aerospace Sciences, ISSN 0376-0421, 2011. man, P. A., “Some Advanced Concepts in Discrete
114. Bodart, J., Larsson, J, and Moin, P., “Large Eddy Simu- Aerodynamic Sensitivity Analysis”, AIAA Journal,
lation of High-lift Devices”, 21st AIAA Computational Vol.41, pp. 224-1229, 2003,10.2514/2.2085.
Fluid Dynamics Conference, 2013, 10.2514/6.2013- 128. Rumpfkeil, M. P., and Mavriplis, D. J., “Efficient Hes-
2724. sian Calculations Using Automatic Differentiation and
115. Wilson, K. G., “The Renormalization Group and Criti- the Adjoint Method with Applications”, AIAA Journal,
cal Phenomena”, Rev. Mod. Phys., 55 (1983), pp. 583– Vol.48, pp. 2406-2417, 2008, 10.2514/1.J050451.
600 (1982, Nobel Prize Lecture).

42
129. Safie, F. M., “Use of probabilistic design methods for 141. Halse, B. (chair), “Strategic Research Agenda-Vol. 2”,
NASA applications”, Reliability Technology, Vol 28, Advisory Council for Aeronautics Research in Europe,
pp. 17–24, 1992. October 2004.
130. Thomson, D. E. and Fecke, T. G. “Air Force Perspec- 142. Alexandrov, N. M., Atkins, H. L., Bibb, K. L., Biedron,
tive on Probabilistic Design”, Reliability Technology, R. T., Carpenter, M. H., Gnoffo, P. A., Hammond, D.
Vol 28, pp. 25–32, 1994. P., Jones, W. T., Kleb, W. L., Lee-Rausch, E. M., Niel-
131. Shankaran, S., and Jameson, A., “Robust Optimal Con- sen, E. J., Park, M. A., Raman, V. V., Roberts, T. W.,
trol using Polynomial Chaos and Adjoints for Systems Thomas, J. L., Vatsa, V. N., Viken, S. A., White, J. A.,
with Uncertain Inputs”, AIAA Paper 2011-3069, 20th and Wood, W. A., “Team Software Development for
AIAA Computational Fluid Dynamics Conference, Aerothermodynamic and Aerodynamic Analysis and
June 2011, 10.2514/6.2011-3069. Design”, NASA TM-2003-212421, November 2003.
132. Ng, L. W-T., Huynh, D. B. P., and Willcox, K., “Mul- 143. Schwamborn, D., Gerhold, T. and Heinrich, R., “The
tifidelity Uncertainty Propagation for Optimization Un- DLR-Tau Code: Recent Applications in Research and
der Uncertainty”, 12th AIAA Aviation Technology, In- Industry”, Proceedings of the ECCOMAS CFD 2006
tegration, and Operations (ATIO) Conference and 14th Conference, (P. Wessling, E. Onate and J. Periaux
AIAA/ISSMO Multidisciplinary Analysis and Optimiza- eds.), Egmond aan Zee, The Netherlands, September
tion Conference, 2012, 10.2514/6.2012-5602. 2006.
133. Roget, B., and Sitaraman, J., “Robust and Scalable 144. Kroll, N., Rossow, C. C., and Schwamborn, D., “The
Overset Grid Assembly for Partitioned Unstructured MEGAFLOW-Project - Numerical Flow Simulation
Meshes”, AIAA Paper 2013-797, 51st AIAA Aerospace for Aircraft”, in Progress in Industrial Mathematics at
Sciences Meeting, Jan 2013, 10.2514/6.2013-797. ECMI 2004, A. DiBucchianico, R.M.M. Mattheij, M.A.
Peletier (Eds.), Springer, New York, S. 3 – 33, 2005
134. Douglas, C. and Patra, A. (chairs), “InfoSymbiot-
ics/DDDAS – The Power of Dynamic Data Driven Ap- 145. Cambier, L., Gazaix, M., Heib, S., Plot, S., Poinot, M.,
plication Systems”, Air Force Office of Scientific Re- Veuillot, J-P., Boussuge, J-F., and Montagnac, M., “An
search and the National Science Foundation, August Overview of the Multi-Purpose elsA Flow Solver”,
2010 Aerospace Lab Journal, Issue 2, March 2011.
http://www.aerospacelab-journal.org
135. Keller-McNulty, K. L., “Defense Modeling, Simula-
tion, and Analysis: Meeting the Challenge”, National 146. Cambier, L., Heib, S., and Plot, S., “The ONERA elsA
Research Council, The National Academies Press, CFD Software: Input from Research and Feedback
ISBN 0-309-10303-7, 2006. from Industry”, Mechanics & Industry, Volume 14, Is-
sue 03, pp. 159-174, Jan 2013, 10.1051/meca/2013056
136. Salas, M. D., “Digital flight: The last CFD Aeronauti-
cal Grand Challenge”, Journal of Scientific Computing, 147. The Networking and Information Technology Research
Vol 28(2-3), pp. 479–505, September 2006. and Development (NITRD) Program Strategic Plan
2012, Executive Office of the President, National Sci-
137. Schutte, A., Einarsson, G., Raichle, A., Schoning, B.,
ence and Technology Council, Office of Science and
Orlt, M., Neumann, J., Arnold, J., Monnich, W., and
Technology Policy, July 2012.
Forkert, T, “Numerical Simulation of Maneuvering
Aircraft by Aerodynamic, Flight Mechanics and Struc- 148. The Networking and Information Technology Research
tural Mechanics Coupling”, AIAA Paper 2007-1070, and Development (NITRD) Program Supplement to the
45th AIAA Aerospace Sciences Meeting, Jan 2007, President’s Budget FY2014, Executive Office of the
10.2514/6.2007-1070. President, National Science and Technology Council,
Office of Science and Technology Policy, May 2013.
138. Washabaugh, K., Amsallem, D., Zahr, M., and Farhat,
C., “Nonlinear Model Reduction for CFD Problems 149. www.dwavesys.com
Using Local Reduced-Order Bases”, AIAA Paper 150. Vassberg, J., Dehaan, M., Rivers, M., and Wahls, R.,
2012-2686, 42nd AIAA Fluid Dynamics Conference, “Development of a Common Research Model for Ap-
June 2012, 10.2514/6.2012-2686. plied CFD Validation Studies”, AIAA Paper 2008-
139. "National Aeronautics and Space Act of 1958", Public 6919, 26th AIAA Applied Aerodynamics Conference,
Law #85-568, 72 Stat., 426. Record Group 255, Na- June 2008, 10.2514/6.2008-6919.
tional Archives and Records Administration, Washing- 151. Rivers, M., and Dittberner, A., “Experimental Investi-
ton, D.C, 1958. gations of the NASA Common Research Model”, AI-
140. Dillmann, A. and Rossow, C. C., "Report of the Insti- AA Paper 2010-4218, 28th AIAA Applied Aerodynam-
tute of Aerodynamics and Flow Technology, Status ics Conference, June 2010, 10.2514/6.2010-4218.
Report 2000 - 2008”, DLR Report, Braunschweig/ 152. Yip, L. P., Vijgen, P. M. H. W., Hardin, J., and van
Gottingen, August 2008. Dam, C. P., “In-Flight Pressure Distributions and Skin

43
 Friction Measurements on a Subsonic Transport High-
Lift Wing Section”, High-Lift System Aerodynamics,
AGARD-CP-15, 1993.
153. http://www.nist.gov/scientific-computing-portal.cfm
154. http://www.krellinst.org/csgf/

44
 impact on power consumption, and heat generation, with
APPENDIX A. High Performance Compu- smaller sizes being better. Thus, forecasting feature sizes of
ting Trends and Forecast for 2030 future processors is very important. Unfortunately, the indus-
try has not always been good in that forecasting, which is one
Introduction reason why predicting where HPC technology will be in
Computational Fluid Dynamics (CFD) codes utilize High 2030 is particularly hard. For example, in 2005, the Interna-
Performance Computing (HPC) systems, so understanding tional Technology Roadmap for Semiconductors (ITRS)
where HPC technology might be in the 2030 timeframe is an forecasted a 22-nm gate length by 2008; that is, the structures
important component of creating a vision for CFD codes in in a modern processor were forecast to have features with
2030. Of course, forecasting where HPC technologies will be sizes around 22 nm. However, in 2008 the forecast date
in the future requires a significant amount of extrapolation, moved to 2011 and in 2011, it moved again to 2012. A simi-
which is especially hard in such a fast changing area as HPC. lar slip occurred for other (smaller) gate lengths (see Figure
The fastest current systems can perform more than 10 peta- 4). Note that the forecasts of the ITRS combine inputs from
FLOPS (1 petaFLOPS is 1015 floating point operations per all major chip manufacturers, equipment suppliers, and re-
second) and the HPC community is working toward systems search communities and consortia, so it represents the com-
capable of 1018 FLOPS (exaFLOPS), which are expected bined wisdom of the industry. Nevertheless, as Figure 4
sometime between 2018 and 2023. Some work is even look- shows, forecasting a key feature of even this basic compo-
ing at 1021 FLOPS (zetaFLOPS). However, reaching that nent of processors is hard. The figure points out that Moore’s
level of performance is unlikely without radically new tech- “Law” is really just an observation about engineering pro-
nologies. gress and offers no guarantee. There are serious concerns
about the longevity of this law, especially when trying to
A common, though controversial, measure of HPC systems is extrapolate to 2030.
the total number of floating point operations a given system
can perform in a second while solving a large linear system To come to our final predictions of a HPC system in 2030 we
of equations using Gaussian elimination; this is the HP go through a two-step process. We start with a prediction
LINPACK benchmark. Twice a year a list of the top 500 about a practical exascale system, likely in the 2020-2023
systems in the world against which those numbers are meas- time frame. To help with the forecasting, we incorporate
ured is published by the Top500 organization. The current some thoughts about the predictions made for a petascale
list (June 2013) is topped by the Tianhe-2 system, developed system around 2000, and how they panned out. In the second
by China’s National University of Defense Technologies, step, we further extrapolate from this base to 2030.
which achieved 33.86 petaFLOPS (quadrillions of calcula- To help create a vision for a CFD Code in 2030, it is im-
tions per second) on the LINPACK benchmark portant to describe three different classes of HPC systems
[Top500.org]. Here, we will estimate
only the peak floating-point perfor- 30
mance in terms of the maximum num-
ber of operations that can be performed
per second. We note that the perfor- 25
mance of many applications, including
Physical Gate Length, nm

CFD applications, may be more accu-

rately estimated by using sustained 20 ITRS
Forecast
memory bandwidth;1 for the purposes Year
of this summary, we assume that other
aspects of system performance, such as 2005
15
memory latency and bandwidth and 2008
integer operation performance, are pro-
vided in a similar ratio to today’s sys- 2011
10
tems. This is a significant assumption
and should be borne in mind when con-
sidering the predictions outlined in this 5
summary.
A significant measure of a processor is
the feature size of its components. The 0
smaller the features, the more elements 2007 2012 2017 2022 2027
can be placed in the same area, and Expected Manufacturing Year
hence the more powerful a processor
becomes. Feature size also has a direct Figure 4. Changing Predictions About Semiconductor Sizes

45
that are of importance for the CFD community. First are the 4. “To address the inadequate state of software productivi-
leadership class systems that reside at National Laboratories ty, there is a need to develop language systems able to
or other government agencies, such as NASA. These will be integrate software components that use different para-
used to solve the hardest, most challenging problems that digms and language dialects.”
break new ground and drive basic understanding about flow
Of these predictions, only number 2, a statement about com-
phenomena and how to simulate them. The second class of
puter hardware, proved true, as systems with a peak perfor-
systems is the type of systems large industrial companies like
mance of a petaFLOP were delivered in 2008 and achieved
Boeing, Pratt & Whitney, or Airbus use. These are in general
sustained performance in 2012.2 It is notable that in 1995, it
one to two orders of magnitude smaller than the leadership
was predicted that the clock speed of a processor would be
class systems. For example, Airbus entered a system at num-
700MHz—well under what was achieved by commodity
ber 29 in the June 2011 Top500 list, with a capability about
processors, but exactly the speed of the processors in the
33 times smaller than the number one system on that list.
IBM BlueGene/L system—a clock speed chosen as the opti-
Finally, the third class of systems consists of HPC systems
mal one for performance at a given power. Also notable is
that would be used by smaller companies, or individual de-
that programming for a petascale system was expected to
partments of larger companies. These are again one to two
need new approaches, with nine overlapping programming
orders of magnitude smaller than the systems in the second
models, including shared memory, message passing, data
class.
parallel, distributed shared memory, functional programming,
It is likely that similar technologies will be used by all of object-oriented programming, and evolution of existing lan-
these classes of systems, with the performance scaling rough- guages, considered essential.
ly with the size (number of nodes) of the system. The reason
Observe that while the predictions about hardware were rea-
that the performance is not strictly proportional to system
sonable, those about software (1, 3, and 4) were not support-
size is that the performance of the interconnect that ties all of
ed by quantitative data and turned out to be far too pessimis-
the computing elements together does not scale with the size
tic. This is not to say that there is not a software productivity
of the system, requiring either more advanced technologies at
problem, but that it has not prevented the use of petascale
larger scale or providing lower overall performance at scale.
systems, several of which are now in operation and are suc-
Another critical component of HPC capability in 2030 is the cessfully performing computations in science and engineer-
advances in software infrastructure and programming meth- ing. Given this experience, great care must be taken in mak-
odologies that will be necessary to take advantage of these ing assumptions about what software might work (if not easi-
future HPC systems. The ultimate purpose for these systems ly) and what is truly inadequate for exascale systems and
is to solve the most pressing problems in academia and in- beyond.
dustry. In particular, industrial users pursue this technology
because of the large impact on future product designs, and Exascale Background
the ability to avoid or minimize the use of other, more costly An exascale system is one capable of 1018 operations per
methods such as wind tunnels or other types of physical tests. second, roughly 100 times as fast as the most powerful sys-
tems today. Several factors suggest that an exascale system
Petascale Predictions and What Happened
will not just be a simple evolution of today’s petascale sys-
Here we consider the predictions that were made in the 1990s tems:
about the first petascale systems: which of them proved true
1. Clock rates. Clock rates for commodity logic have plat-
and which of them proved false.
eaued, in large part because of leakage currents. This
The following statements are from “Enabling Technologies implies that increasing performance will be tied to in-
for PetaFLOPS Computing” (1995): creasing parallelism, as has been true since about 2006.
It is unlikely that a new technology will be available by
1. “The software for the current generation of 100 GF ma-
2023 to provide an alternative with faster clock rates.
chines is not adequate to be scaled to a TF…”
Rapid Single Flux Quantum (RSFQ), a technology using
2. “The petaFLOPS computer is achievable at reasonable
superconducting devices, could provide 250GHz gates,
cost with technology available in about 20 years
but bringing that technology to sufficient maturity, in-
[2014].”
cluding density and cost, is unlikely within 10 years.
3. “Software technology for [Massively Parallel Proces-
With a sustained investment, it might be available by
sors] MPP’s must evolve new ways to design software
2030, but we still view this as unlikely to be a mature
that is portable across a wide variety of computer archi-
technology by 2030.
tectures. Only then can the small but important MPP
2. Power consumption. Current petascale systems are con-
sector of the computer hardware market leverage the
suming on the order of 10-20 megawatts (about 1
massive investment that is being applied to commercial
MW/PF), so scaling current technology would require 1
software for the business and commodity computer mar-
gigawatt for an exascale system (announced targets are
ket.”

46
 between 20 and 60 MW for an exascale system). Con- uniform performance or work. Adaptivity will be re-
tributors to power consumption include: quired in the algorithms and implementations.
5. Latency tolerance. References to all memory, both local
a. Processor Speed – slower processors are more pow-
and remote, will take 100s to 10,000s of cycles. For re-
er efficient (to a point); this implies even greater
mote references, much of this is speed-of-light delays,
concurrency to reach a given level of performance.
and can only be solved by making the entire machine
Processors may be in the 1 to 20 GHz range, with
physically smaller, which is unlikely in the next 10
the most likely range being 1 to 5 GHz.
years. For more local references, this is a result of the
b. Complex computing elements—features used to
memory hardware. Most novel memory techniques that
improve scalar performance, such as speculation,
may increase available memory size and/or reduce pow-
caches, and out-of-order execution—all have low
er consumption, also increase the latency for memory
power efficiency for the compute benefit they deliv-
accesses. In fact, latencies of hundreds of cycles are
er. Processing elements are likely to be simpler and
common today in microprocessors, but much of this is
optimized for particular computing patterns. For ex-
hidden from the programmer through the use of memory
ample, there may be separate scalar, vector, and
caches and prefetch hardware.
stream computing elements. GPUs already exploit
the advantages of being specialized. Sample Exascale Architecture
c. Memory, based on the current DRAM architecture,
consumes a surprising amount of power. As a result An exascale computer might look something like this:
(and combined with the need for increased parallel- Each node contains 16 1-GHz simple scalar processors, each
ism), there is likely to be both much less memory with an integrated, 1,000 core streaming processor (also 1
per processing element and much less memory to- GHz). At the memory interface is another 1,000 core stream-
tal; for example, an exascale system may have only ing processors (but with more limited operations, also at 1
10 PB of memory. GHz). This gives 128 TFLOPS per node (assuming fused
d. Moving data anywhere takes power, even within a multiply-adds, 2 per cycle). 10,000 of these nodes bring you
“node” (e.g., between local memory and the nearest to 1 exaFLOPS. The interconnect may be a combination of
processing units). This will argue for much greater electrical and optical (chosen to provide power-efficient
data locality in the choice of algorithms and their bandwidth) using high-radix routers, similar to what is used
implementation. in the Blue Waters Supercomputer at NCSA. It may also be a
3. Fault tolerance and resiliency. This is a tricky subject 3D torus (between the nodes), but allowing each link to be
and there is a great deal of misinformation about it. A used concurrently, as in the IBM Blue Gene/P systems.
common analysis is to take the failure rates of today’s Note the reliance on “stream” processing; you can think of
systems (particularly commodity cluster systems) and this as a distributed memory vector processor, where virtual-
extrapolate to a system built with the same parts but 100 ly all of the work must be suitable for the vector/stream pro-
to 1,000 times as large, based on the number of pro- cessing elements (those 1,000-core components). Today’s
cessing elements expected in an exascale system. How- graphics processing units, as well as Intel’s Xeon Phi (MIC)
ever, failure rates are more dependent on the number of chips, can be considered examples of stream processors.
parts (e.g., chips), not gates, and experience with some
of the largest systems, such as the IBM Blue Gene sys- Bisection bandwidth relative to peak performance is likely to
tems, bears this out. Parts intended for the commodity be much smaller than in today’s systems. However, there
market may accept higher rates of faults than what is ac- may be support for some critical global operations, such as a
ceptable for a large-scale, reliable exascale system. Hav- nonblocking allreduce or nonblocking barriers. These global
ing said that, there are reasons why an exascale system operations might be underappreciated, but might be particu-
may have a higher rate of faults. For example, some larly useful for special cases, for example for distributed ter-
techniques being proposed to reduce power consumption mination detection.
may raise the rate of failures. Other systems will have a different balance, for example,
4. Performance regularity and scalability. Because of the more scalar (or short vector) compute units, more nodes but
extreme degree of parallelism required, scalability will with less parallelism per node. Operation mixes may be dif-
be a major issue. Approaches that work with 1,000 or ferent, e.g., more support for integer operations at the ex-
even 10,000 nodes may fail to perform well on an ex- pense of floating point, or specialized integer units. (Such a
ascale system. In addition, several approaches under different approach is shown in Table 1 in the prediction for
consideration suggest that the performance of the com- an exascale system in 2023). However, it is very unlikely that
pute elements may be less predictable than it is now an exascale system would look much like an IBM Blue Gene,
(e.g., using dynamic frequency adjustment to improve Cray XE/XK, or IBM Productive, Easy-to-use, Reliable
power efficiency). As a result, algorithms will not be Computing System (PERCS), which is officially known as
able to decompose work based on any assumption about Power 775.3 From a functional standpoint, an exascale sys-

47
tem is likely to have heterogeneous functional elements, each languages such as UPC is very low. The reality is that creat-
optimized to different styles of computing. The closest cur- ing a truly new language is a long-term project, and is unlike-
rent systems are GPU clusters, such as RoadRunner at Los ly to be completed before the first exascale systems appear.
Alamos National Laboratory with IBM PowerXCell accel- More likely is the evolution and enhancement of existing
erators, Titan at Oak Ridge National Laboratory with NVID- languages, including extensions to MPI (e.g., MPI-3 now
IA GPUs, or Stampede at the Texas Advanced Computing provides access to shared memory regions) and to existing
Center with Intel Xeon Phi (MIC) coprocessors. However, an programming languages (e.g., OpenACC is a small extension
exascale system will be much more tightly integrated; it will to C/C++ and Fortran). There is an opportunity to develop
not have accelerators and coprocessors as loosely connected new programming systems for 2030, but only if work starts
to the rest of the system, as is the case in current offerings. soon on the necessary research.
An all-accelerator/coprocessor-like system is possible but
unlikely (consider the above sample system, but with no sca- Other Issues
lar processors). Exascale systems may well have CPU-like File input/output (I/O) is another issue. Many of the same
and accelerator/coprocessor-like elements on the same chip issues apply (power, latency, smaller ratio of space/FLOP).
as described in the first paragraph. Current approaches based on blocking I/O to a single file per
process will not be effective. At the scale of these future sys-
Programming in Exascale
tems, it will probably be necessary to have a subset of pro-
Programming languages take many years to develop. It is cessors perform I/O for groups of processors, preferably in
very unlikely that a new programming model will be devel- parallel to the computation. This is another example of spe-
oped in time for the first exascale systems. What is possible cialization, this time at the software level.
and even likely is that higher-level, special-purpose lan-
Finally, while an exascale system can act as 1,000 petascale
guages will be developed to help write efficient code for
systems (or a million terascale systems), this is unlikely to be
these systems. Applications will be a mix of current pro-
an efficient use of the system. Simple approaches to ensem-
gramming models (C/C++/Fortran), combined with Message
ble calculations that can be executed serially on a collection
Passing Interface (MPI) and OpenMP, possibly Unified Par-
of much smaller systems do not constitute an exascale calcu-
allel C (UPC) and Co-Array Fortran (CAF), now partially
lation (regardless of their scientific importance).
integrated into Fortran 2008, combined with specialty lan-
guages for particular computation styles, like Compute Uni- HPC in 2030
fied Device Architecture (CUDA) or OpenCL, algorithm and
data-structure specific code writing systems like the Tensor To extrapolate computer performance to 2030, we have made
Contraction Engine (TCE),4 and runtimes implementing oth- the following assumptions. These are for a conservative de-
er programming techniques (e.g., Intel’s Thread Building sign, and provide a likely minimum performance level for
Blocls (TBB) for intranode parallelism and CHARM++ for 2030:
general parallelism). Further developments to improve the 1. The 2012 International Technology Roadmap for Semi-
programming of accelerators and coprocessors through di- conductors (ITRS), which has predictions out to 2026,
rective-based concepts as defined in OpenACC and future can be extended to 2030.
versions of OpenMP will play an important role, especially 2. Computer organization will be similar to current systems
since they are addressing some of the portability concerns of that integrate multiple processor types, but with greater
software developers. Finding the right balance between gen- degrees of integration. Specifically, we postulate that a
eral programming language support and more limited but less system will contain nodes that integrate a fast scalar pro-
complex tools sufficient to achieve performance will be an cessor, a stream processor, and processing in memory
important part of any exascale application development. Such (PIM).
special features may include higher level programming mod- 3. Processing elements have a constant number of devices
els for the “stream” or vector computing elements or support and the size of a chip is roughly constant. Thus, the
for software and/or algorithm assisted fault tolerance. number of processing elements per chip increases with
There have been efforts to develop new approaches for pro- the square of the ratio of gate feature size.
gramming high-performance computing systems. The 4. The many engineering challenges required to achieve the
DARPA High Productivity Computing System (HPCS) pro- ITRS targets will be met.
ject had a goal of a tenfold increase in productivity through a Table 1 shows estimated performance in 2012, 2020, 2023,
combination of new programming languages and new hard- and 2030. This estimate is consistent with current leading
ware capabilities. While interesting research came out of the systems; both IBM Blue Gene/Q and Cray XK7 systems
language efforts (particularly Cray’s Chapel and IBM’s X10 achieve around 20 PetaFLOPS peak. The estimate for 2023 is
languages), these have yet to provide users with viable alter- consistent with some current DOE estimates, providing a
natives to existing models. Outside of a few specialized are- little more than 1 exaFLOPS. By assuming a significant in-
as, the adaptation of partitioned global address space (PGAS) crease in processor density, we estimate that performance

48
 Table 1. Estimated Performance for Leadership-class Systems
Clock Nodes
Feature Derived Stream PIM paral- rate GFLOPS GFLOPS GFLOPS Processor Node per Total
Year size parallelism parallelism lelism GHz FMAs (Scalar) (Stream) (PIM) per node (TFLOP) system (PFLOPS)
2012 22 16 512 0 2 2 128 1,024 0 2 1 10,000 23
2020 12 54 1,721 0 2.8 4 1,210 4,819 0 2 6 20,000 241
2023 8 122 3,873 512 3.1 4 3,026 12,006 1,587 4 17 20,000 1,330
2030 4 486 15,489 1,024 4 8 31,104 61,956 8,192 16 101 20,000 32,401
Feature size is the size of a logic gate in a semiconductor, in nanometers. Derived parallelism is the amount of concurrency, given processor cores with a
constant number of components, on a semiconductor chip of fixed size. Stream and PIM parallelism are the number of specialized processor cores for
stream and processor-in-memory processing, respectively. FMA is the number of floating-point multiply-add units available to each processor core. From
these values, the performance in GigaFLOPS is computed for each processor and node, as well as the total peak performance of a leadership-scale system.

between 16 to 32 exaFLOPS may be achievable by 2030. advances are much tougher to predict than hardware advanc-
Note that there are many assumptions in this estimate, and es.
many yet unsolved engineering problems must be overcome
Future programming models will be driven by dealing with
to maintain the targets in the ITRS roadmap. In addition, we
locality, whether they are new or extensions of existing pro-
have assumed lower clock rates in the stream and processor-
gramming models. Programmers need to be able to express
in-memory elements, reflecting a different optimization point
locality and relationships between data abstractly. NVIDIA’s
for speed and energy use.
CUDA programming language is an example of a newer
It is important to note that these predictions are sensitive to a programming model that forces programmers to deal directly
number of hard-to-forecast values. For example, energy and and explicitly with locality. This illustrates the need for the
power dissipation problems could reduce the number of pro- expression of locality while also showcasing the need for
cessing units that can be assembled into a single system, re- ways to express this information about locality more abstract-
ducing total performance.5 Conversely, new 3D fabrication ly and portable.
and packaging could increase the density of components,
As discussed above, we pointed out that the memory system
allowing even greater parallelism. The major conclusion that
will probably become much more complex, both with the
should be drawn from this table is that current trends will
introduction of processing in memory (PIM) as well as with
yield significantly faster systems than we have today, but not
more levels of memory architectures. While some of this
ones that are as fast as the past 20 years of development
complexity will be hidden from the developers, a lot of it will
would suggest.
not. The developers need to be able to express what compu-
Another important feature of an HPC system in this time ting should be done by the slower processing elements inside
frame we expect to see are even more levels of memory than the memory subsystem, and what needs to be done by the
we currently have. Current systems have up to three levels of fast scalar processor or the streaming elements. The pro-
cache, and then main memory. Systems with accelerators cessing elements within the memory subsystem will have
have additional memory attached to the accelerator. In 2030, significantly higher bandwidth to memory. One of the easiest
main memory itself might be composed of different levels, uses to imagine of this processing is to perform calculations
with portions being very fast, but small, and other portions for prefetching of data (gather), and scatter the results of cal-
larger and slower. Combined with the concept of processing culations back into the final locations in the memory subsys-
in memory, that is, having some computing capability em- tem. Because of the processing power and bandwidth envi-
bedded within the memory subsystem, this will lead to an sioned in the memory subsystems, these can be significantly
even more complex overall system. more complex than possible within the processors, which
will be especially useful for software with complex memory
Programming a 2030 HPC system access patterns, for example, unstructured CFD codes.
Software has a much longer lifespan than hardware, and as We do not believe that there will ever be a programming
pointed out earlier, the expectation is that there will be only model that completely hides the complexity of the underlying
evolutionary changes to the programming model in the 2020- HPC system from the programmer while achieving the nec-
2023 timeframe. For 2030, the likelihood that some major, essary performance. Nevertheless, we do think that great
revolutionary changes to the programming models will occur advances can be made to allow the programmer better to
is higher because of the extra development time. It is im- express her or his intent and to provide guidance to the com-
portant to point out that this is not a guarantee, as many pro- piler, runtime system, operating system, and even the under-
gramming languages and models have shown a surprising lying hardware. This will require significant research and that
level of sustainability. In addition, as we pointed out in the
discussion on the validity of petascale projections, software

49
research needs to start now for solutions to be ready in the the ability to inform the developer of the inhibitors in order to
2030 time frame. move software to the next generation of HPC systems and of
the changes to make that will allow good performance on
Related to locality is the fact that the developer community
both current and future systems will significantly increase the
needs to change their thinking away from a focus on floating
productivity of the developers and utilization of future HPC
point operations. In the 2030 time frame, FLOPS might be
systems.
considered “free,” and data movement, synchronization, and
all levels of parallelization will become the driving forces for This leads to the need for a well-trained developer workforce.
performance. We already see this to some extent with current Even application developers utilizing libraries or advanced
accelerators like the GPUs from AMD and NVIDIA. We programming models need to understand the basic principles
need to start looking at the quality of algorithms from the of parallelism and the need to utilize and express locality and
point of view of arithmetic intensity, that is work per the other demands on software by the hardware in 2030. We
memory access, rather than just looking at floating point op- also need developers that can build and optimize the lower
erations. Prime examples of algorithms that have, in general, level components of the software architectures. These devel-
relatively low arithmetic intensity are sparse linear algebra opers must have a deep understanding of the hardware archi-
methods. Utilizing the processors in the memory subsystem tectures, as well as work closely with the application devel-
could increase the arithmetic intensity by delivering the right opers to provide the high-level components they need.
data to the scalar and streaming processors in a contiguous
manner. Of course, as we see today, there will be codes cov- Alternative Computer Architectures
ering the full range from highly optimized codes that run on a Up to this point, our analysis is based on the evolution of
single architecture to codes that are highly portable, but not current technology. Such evolution has provided an exponen-
as high performing. tial increase in performance over several decades; while there
One concept to determine a “sweet-spot” at exascale is to are many challenges in continuing the same rate of increase,
develop more composable programming models. The low- the assumptions in this report represent a likely path for
level components will be highly optimized for a given archi- computer architecture. However, other device technologies
tecture, probably even by the vendor itself, and the higher could give rise to computers with very different capabilities.
level applications that build on these low-level components A few that were discussed include:
are significantly more portable. Examples of this are Intel’s 1. Quantum Computers. There is steady progress on fab-
MKL, or PETSc (Portable, Extensible Toolkit for Scientific ricating practical quantum computers, and such systems
Computation6) from Argonne National Lab. While achieving may be available in 2030. However, while a quantum
high levels of performance will never be a 100% automated computer can be used for some linear algebra calcula-
process, such a composable programming model should al- tions (after all, the Schrodinger equation involves linear
low the development of transportable software, that is soft- operators), a quantum computer is not necessarily a fast-
ware that can be ported easily from one platform to the next, er computer for CFD calculations.
while still requiring optimizations to achieve full perfor- 2. Superconducting logic. For example, Rapid Single Flux
mance. Quantum (RSFQ) logic gates were demonstrated at over
Another concept that will become important for software will 770 GHz, and 100 GHz gates are available now. A
be specialization within applications. As pointed out earlier, drawback of this device technology (as currently imple-
it might be necessary to have dedicated processes dealing mented) is significantly lower density; as an immature
with I/O in parallel to the computation. Other processes technology, cost is also an issue.
might analyze or process the data generated by the main al- 3. Low-power memory. Many novel memory designs are
gorithm on the fly, and only the processed data is actually being developed that use something other than charge to
stored. For example, visualizations of results and calculations store data. Some have much higher latencies but provide
of derived properties might be done in parallel with the main much lower power use and higher densities of storage.
calculations rather than after the main simulation finishes. On 4. Massively parallel molecular computing. This is a
a lower level, separate processes might be used to gather data very different paradigm, and uses far higher levels of
for upcoming calculations, and scatter the results of calcula- parallelism (typically 1023) but also gives up determin-
tions as mentioned above. This will be especially important ism. It is not clear how to cast CFD calculations into a
considering the extreme levels of parallelism we forecast the form suitable for molecular computing.
hardware will provide in 2030. Many of these revolutionary architectures will also require
With the increased complexity comes the need for improved revolutionary changes to the full software ecosystem, espe-
programming tools like profilers, compilers, and debuggers. cially quantum computers and molecular computing. Consid-
These need to be (relatively) easy to use and allow the devel- ering the long development timeframes for software, early
oper a deep understanding of the behavior of their codes and examples of these technologies need to be available to a larg-
the barriers to good performance on a given system. Having er developer community in the 2020-2023 time frame to en-

50
sure that these new architectures will have broad impact on ________________________
more than a very small subset of the HPC user community.
1
Moreover, while we can foresee significant progress in push- Anderson, W. K., Gropp, W. D., Kaushik, D. K., Keyes,
ing HPC technology to the next level using approaches that D. E., and Smith, B. F., “Achieving High Sustained
are more evolutionary until the 2030 timeframe, we are much Performance in an Unstructured Mesh CFD Application”,
more skeptical for the time after 2030. As we pointed out Proceedings of the ACM/IEEE SC99 Conference on High
earlier in the evolutionary approach, the semiconductor in- Performance Networking and Computing,1999, IEEE
dustry predicts feature sizes of 4 nm. This means that a fea- Computer Society.
2
ture will consist of small numbers of atoms, and significant The exact dates for these achievements depend on your
quantum effects will play a role. An end of the ability to fur- definition of peak and sustained performance.
3
ther shrink feature size is clearly in sight at that time. Even if Wikipedia article on PERCS:
the above listed revolutionary changes are not necessary to http://en.wikipedia.org/wiki/PERCS
4
reach goals for 2030, they are certainly important for allow- See: S. Hirata, J. Phys. Chem. 2003, 107 (46), 9887-9897.
5
ing us to move beyond 2030. Some technologies might not Esmaeilzadeh, H., et al., “Power challenges may end the
seem attractive now (e.g., too expensive, too slow, too hard multicore era”, Commun. ACM, 2013. 56(2): p. 93-102.
6
to program) but we need to invest in them to overcome those PETSc (Portable, Extensible Toolkit for Scientific
faults to be able to take advantage of their benefits. Computation) homepage, http://www.mcs.anl.gov/petsc/

Research Suggestions
We conclude with a list of research that needs to be started
now and in the coming years to enable our predictions of
HPC in 2030 to be realized. While not complete, this list is a
good starting point for the community. There is no order of
importance implied in this list. We need to develop and im-
plement all the items in this list and then demonstrate them
on important use cases.
 Alternate Hardware Technologies
 Quantum Computers
 Superconducting logic
 Low-power memory
 Massively parallel molecular computing
 Next generations of “traditional” processor technolo-
gies
 On-chip optics
 Advanced memory technologies (e.g., 3D memory)
 Advanced Algorithms
 Data movement aware (not FLOPS oriented) algo-
rithms
 Power aware algorithms
 Advanced Programming Models and Systems
 Locality aware, abstract programming models
 Composable programming models
 Transportable, extremely parallel programming mod-
els
 Transportable, extremely parallel HPC software
 Highly scalable and manageable data management
systems
 Compiler interfaces that allow better expressions of
developer intent
 Improved programming tools like compilers, profil-
ers and debuggers

51
 Form Approved
REPORT DOCUMENTATION PAGE OMB No. 0704-0188
The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources,
gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this
collection of information, including suggestions for reducing this burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and
Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person
shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number.
PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.
1. REPORT DATE (DD-MM-YYYY) 2. REPORT TYPE 3. DATES COVERED (From - To)
01- 03 - 2014 Contractor Report
4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER

NNL08AA16B
CFD Vision 2030 Study: A Path to Revolutionary Computational 5b. GRANT NUMBER
Aerosciences
5c. PROGRAM ELEMENT NUMBER

6. AUTHOR(S) 5d. PROJECT NUMBER

Slotnick, Jeffrey; Khodadoust, Abdollah; Alonso, Juan; Darmofal, David; 5e. TASK NUMBER
Gropp, William; Lurie, Elizabeth; Mavriplis, Dimitri
5f. WORK UNIT NUMBER

794072.02.07.02.03
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION
REPORT NUMBER
NASA Langley Research Center
Hampton, Virginia 23681

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR'S ACRONYM(S)

National Aeronautics and Space Administration NASA
Washington, DC 20546-0001
11. SPONSOR/MONITOR'S REPORT
NUMBER(S)
NASA/CR-2014-218178
12. DISTRIBUTION/AVAILABILITY STATEMENT
Unclassified - Unlimited
Subject Category 01
Availability: NASA CASI (443) 757-5802
13. SUPPLEMENTARY NOTES

Langley Technical Monitor: William L. Kleb

14. ABSTRACT
This report documents the results of a study to address the long range, strategic planning required by NASA's Revolutionary Computational Aerosciences
(RCA) program in the area of computational fluid dynamics (CFD), including future software and hardware requirements for High Performance Computing
(HPC). Specifically, the "Vision 2030" CFD study is to provide a knowledge-based forecast of the future computational capabilities required for turbulent,
transitional, and reacting flow simulations across a broad Mach number regime, and to lay the foundation for the development of a future framework and/or
environment where physics-based, accurate predictions of complex turbulent flows, including flow separation, can be accomplished routinely and efficiently in
cooperation with other physics-based simulations to enable multi-physics analysis and design. Specific technical requirements from the aerospace industrial
and scientific communities were obtained to determine critical capability gaps, anticipated technical challenges, and impediments to achieving the target CFD
capability in 2030. A preliminary development plan and roadmap were created to help focus investments in technology development to help achieve the CFD
vision in 2030.

15. SUBJECT TERMS

Aerodynamics; Aerothermodynamics; Computational Fluid Dynamics; High Performance Computing; Propulsion; Space
Exploration
16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF 18. NUMBER 19a. NAME OF RESPONSIBLE PERSON
ABSTRACT OF
a. REPORT b. ABSTRACT c. THIS PAGE PAGES STI Help Desk (email: [email protected])
19b. TELEPHONE NUMBER (Include area code)
U U U UU 58 (443) 757-5802
Standard Form 298 (Rev. 8-98)
Prescribed by ANSI Std. Z39.18