Dakota 6.11 Release Notes — (2019-11-15)

Highlight: GUI Improvements

The Dakota GUI has added many significant feature improvements over the last year.  The Dakota GUI now allows you to seamlessly browse Dakota's HDF5 database output files, as well as generate sophisticated graphical plots from HDF5 data with just a few clicks.  HDF5 browsing and plot generation can also be driven within Dakota GUI by Next-Gen Workflow, a powerful tool that allows you to construct node-based workflows for complex tasks.

Enabling / Accessing:  Dakota GUI ships with Dakota and is available for Windows, Mac, and RHEL7.

Documentation:  An enhanced version of the Dakota GUI manual now ships with the GUI, giving you easy access to a wealth of reference material for using the GUI.  The 6.11 GUI manual is also available here.

Highlight: Functional Tensor Train for low rank approximation

The C3 library (POC: Prof. A. Gorodetsky, University of Michigan) provides a new capability for low rank approximation by functional tensor train decomposition.  C3 has been integrated in Dakota, alongside other stochastic expansion methods for UQ based on projection, regression, or interpolation, enabling an array of approaches for exploiting special structure (e.g., sparsity, low rank, low intrinsic dimension) in an input-output map.

Enabling / Accessing: This capability is new and evolving and is not part of the default Dakota build.  To activate, enable HAVE_C3 in your CMake configuration.

Documentation: Current keywords are documented in the latest Reference Manual, under method function_train_uq and model surrogate global function_train. This duality reflects a new design for supporting both Iterator-based and Model-based specification of stochastic surrogate models (see "Stochastic expansions as global DataFit surrogates" below).

Improvements by Category

(Team, add a new topic for any major improvement or collection of improvements)

Interfaces

  • Batched evaluations: When batch mode is enabled by including the 'batch' interface keyword, Dakota writes multiple evaluations to a single parameters file, runs a single analysis driver for the entire batch, and expects results for the batch to be returned in a single file. The analysis driver exercises complete control over details such as the sequence in which evaluations are performed, where they are performed, and so on.
  • Preprocess Dakota input: Dakota now supports a -preproc option that will pre-process the specified input file template into a Dakota-ready input file. Requires pyprepro.py utility on PATH or specification via -preproc /path/to/preproc_tool.py. Thanks to Ken Hu and George Orient for the feature request.

Surrogate Models

  • Data import validation: (model > surrogate > global only.) Variable labels in header-annotated tabular data files are validated against expected imported variable descriptors and warnings/errors generated where possible. Imported columns are permuted to match expected variable order if possible. Each row of a tabular data file must have the correct number of columns for variables and responses. Applies to import_build_points_file and challenge_points_file.
  • Quadratic multipoint exponential approximation (QMEA): the QMEA approximation (POC: Prof. R. Canfield, Virginia Tech) has undergone refinement for managing heterogeneous training data sets, as might be expected from a sequence of trust region accept / reject decisions that incur evaluations with / without gradients, respectively.  Outlook: this surrogate extends beyond the current TANA-3 two-point capability to many-point cases, where the most recent sequence of iterates is used to construct higher-order approximations. Both optimization and reliability analysis cases are targeted.

Graphical User Interface

  • The new Dakota Visual Editor enables you to edit Dakota input files using graphical widgets instead of a text editor.
    • Methods supported in this release:  sampling, centered parameter studies, multidimensional parameter studies, list parameter studies, vector parameter studies
  • In-situ browsing of HDF database files generated by Dakota.
  • Integration between HDF database files and graphical plotting.
  • Improved graphical plotting support for centered parameter studies, optimization studies, incremental LHS studies, and Sobol method studies.
  • Improved support for screenshots of graphical plots.
  • Improved support for using the Next-Gen Workflow engine as an analysis driver for Dakota.
  • Various improvements to the Dakota text editor.
  • The Dakota GUI manual is now accessible directly from the GUI.

Optimization Methods

  • NOMAD method now supports asynchronous model evaluations for increased concurrency
  • JEGA (SOGA, MOGA) methods will now work properly with flat file initialization and asynchronous evaluations instead of hanging indefinitely. Thanks to Benoit Paillard for the bug report!
  • The Demo_TPL showing steps to add a new Optimization Third-Party Library (TPL) to Dakota along with working examples is now available in $DAKOTA_SRC/packages/external/demo_tpl and is conditionally enabled by setting -D HAVE_DEMO_TPL:BOOL=ON when building Dakota from source.  See either the README.md file within that directory or the Development Practices and Guidance section of the Developer's Manual for additional details.

UQ Methods

  • As noted in the highlights above, low rank approximation by functional tensor train (FTT) is now available in Dakota.
  • The latest individual and integrated adaptive refinement strategies for multilevel polynomial chaos and multilevel stochastic collocation are now supported as emulators underneath Bayesian calibration.

MultivariateDistribution Architecture

  • The main goal in this refactor was to modernize our random variable distribution capabilities so that we can move beyond a description of univariate marginal distributions + correlations to include other types of multivariate distributions, especially joint PDFs for data-driven workflows.
    • new Pecos::MultivariateDistribution base class provides univariate and multivariate stat routines, while Pecos::MarginalsCorrDistribution derived class covers the existing case of a std::vector<Pecos::RandomVariable> plus a correlation matrix
    • Outlook: new data-driven workflows are under development or planned, including MultivariateNormalDistribution (stubbed out), JointKDEDistribution (planned), et al.
  • New RandomVariable derived classes were added to span all types of {continuous, discrete {int,string,real}} and {design, aleatory, epistemic, state} variables, with an emphasis on templates to eliminate previous type enumeration for discrete sets and ranges.
    • This allows a more modular design for some UQ APIs (e.g. LHSDriver), where only a vector of all specified variables, a correlation matrix, and BitArray definition of active subsets need to be supplied.
    • Allows retirement of Pecos::{Aleatory,Epistemic}DistParams as well as a number of Dakota::Iterator containers for discrete set values, as all random variable data is now consolidated in one place.
      • Outlook: Discrete Askey PCE is now much closer to reality, with various discrete mappings more fully developed.
  • With Pecos::MultivariateDistribution covering statistical utilities, Pecos::{Probability,Nataf}Transformation now focuses only on the specifics of forward/inverse variable transformations.  The separation of these two concerns is much cleaner than before.

ProbabilityTransformModel

  • As enabled by the MultivariateDistribution abstraction, this release completes and fully deploys a derived RecastModel for encapsulating the Pecos probability transformation (e.g., Nataf).  This allows promotion of a probability transformation into a full-featured Model recursion.
    • Consolidates previous Dakota::NonD code into a reusable module
    • Consolidates the ProbabilityTransformation instance so that it no longer needs to be passed around among NonD helpers.
  • All Dakota::Models now contain one Pecos::MultivariateDistribution instance which describes a single probability space (original or transformed), where one starts from the original space at the bottom and then encounters transformations as one goes up the Model recursion. In this way, previous explicit management of x-space versus u-space could largely go away as a given level of the hierarchy has only one probability space (e.g., ProbabilityTransformModel consumes the x-space data from the levels below and presents only u-space data to the levels above).

Stochastic expansions as global DataFit surrogates

  • With probability transformations as part of the Model hierarchy, stochastic expansions can now be fully defined as global data fits without an associated NonD Iterator.  NonDC3FunctionTrain and C3Approximation are the first demonstrator of this new capability (either NonD Iterator or DataFit Surrogate), with PCE and SC to follow.
    • Outlook: This will simplify / unify use of stochastic surrogates in other contexts, e.g., Bayesian inference, and enable new Dakota workflows for surrogate modeling.

Output

  • HDF5
    • The types of variables (e.g. "continuous_design") were added as a dimension scale to evaluation output.
    • Results from the 'multi_start' and 'pareto_set' methods are now written.
    • Add name of the top method as a root attribute
    • Distribution parameters (e.g. mean and standard deviation of normal_uncertaint variables; lower and upper bounds of continuous_design variables) are now written out as model metadata.

Miscellaneous Enhancements and Bugfixes

  • Enh/Bug: Cantlever beam training example stress equation now properly depends on beam length. Thanks to XXX for reporting the issue and the correct derivation.
  • Bug fix: Header change to resolve Acro/Colin compilation issue with boost::signals2::last_value. Thanks to Sebastian Uharek for reporting bug and fix.
  • Bug fix: Revise JEGA EDDY_ASSERT macro for proper compilation on some platforms.
  • Bug fix: Test script dakota_test.perl will now report SKIP for tests disabled based on Dakota configuraiton options (such as missing TPLs). Thanks to XXX for reporting.
  • Bug fix: Presence of old, corrupted HDF5 files no longer causes Dakota to abort.
  • Bug fix: HDF5 model_selection behavior was fixed so that it is no longer necessary to specify 'all' to avoid a crash.
  • Bug fix: Specification of response gradients but no continuous variables no longer causes an HDF5-related crash.
  • Bug fix: pyprepro treats templates from stdin properly.
  • Bug fix: fixed issue from previous release with using the multifidelity_stoch_collocation method with gradient-enhanced interpolants.
  • Bug fix: fixed issue with output of uninitialized main effects for the case of Sobol' index-based adaptive p-refinement of polynomial chaos expansions, when variance-based decomposition is not specified.
  • Enh: Clang compiler compatibility misc. fixes to forward declare streaming operators and not use funny pointer to ostringstream data. Also misc. utilib-related fixes. Thanks to multiple users for reporting.
  • Enh: Use std::chrono for system-generated random seeds instead of Utilib; user-specified seed behavior is unchanged.
  • Enh: mpitile utility accepts a new argument, --tile-size, that allows mpitiling applications on a subset of a tile.
  • Enh: mpitile stops parsing command line options after seeing --, allowing users to pass options to their applications that match mpitile options.

Deprecated and Changed

  • Deprecated X Graphics keywords removed from example files.
  • Migrated the Continuous Integration infrastructure for MacOS to a new Jenkins build / test server, High Sierra (10.13).
  • Tabular data imports for surrogates must have correct number of cols per row.
  • Website training materials do not reflect corrected stress equation for cantilever beam.
  • MacOS Sierra (10.12) deprecated in the Jenkins Continuous Integration build/testing scripts.
  • Migrated to a new CDashboard server for posting build / test results.
  • Enabled MacOS Mojave (10.14) as a build/test alternative for the Mac platform.

    Compatibility

    • HDF5 v1.10.4 or higher is required to compile Dakota with the HDF5-based method results output enabled. HDFView may be helpful in viewing the output database, and Python h5py is necessary to run tests and examples.
    • No mandatory changes to required compilers or third-party libraries. Dakota has been smoke tested on newer gcc-7.x and gcc-8.x compilers, but not fully tested.

    Other Notes and Known Issues

    • Warning:  Intermittent Crash Bug in RHEL7 Version on Dakota GUI 6.11
      • There exists a potential application-crashing bug in the RHEL7 version of Dakota GUI that we are still investigating.  This bug occurs in the BMF model creation dialog, when you are naming variables for your model.  When you are done naming a variable, if you use the mouse to close the editor for that variable name, the GUI can potentially crash.  Instead, use the Enter key to apply your current variable name and close the editor, or the Escape key to cancel your edited variable name and close the editor.
    • HDF5 output is known to be incompatible with some types of studies involving hierarchical surrogate models.