Table of Contents

1. Introduction
2. Next-Gen Workflow Basics
3. Tutorial: Create a Nested Workflow Driver for Dakota
4. Special Topic: Manually Handling Dakota's Parameters and Results Files in a Workflow
5. Special Topic: Using Multiple Dakota Analysis Drivers in Next-Gen Workflow
6. Special Topic: Using Dakota Surrogate Files in Next-Gen Workflow

Introduction

The Dakota GUI includes a node-based workflow creation tool called "Next-Gen Workflow" (NGW from this point on). NGW is an extremely useful standalone tool for building workflows to process data, but it is primarily included as part of Dakota GUI to facilitate more complex Dakota analysis drivers. With NGW, you can fully customize the flow of information between Dakota and your simulation model using a series of nodes that are visually connected together.

Next-Gen Workflow Basics

The Palette

The Palette view provides all NGW nodes in a hierarchical folder structure, organized by topic.

If this view is not already present in your perspective, add it by going to "Window > Show View > Other… > Palette."

Settings Editor for Workflow Nodes

The Settings Editor view is used quite heavily as part of NGW, so if your perspective doesn’t have the Settings editor view yet, get it from "Window > Show View > Settings."

The contents of the Settings Editor view changes depending on what is in focus on the NGW workflow canvas. For instance, this is the view for an aprepro node:

Take a moment to familiarize yourself with the controls in this view that are common to most NGW nodes:

• Name The name of this node. It appears in the top-left corner of the node’s box on the canvas.
• Label The label of this node. This is text that appears above the node to describe what the node is doing. Feel free to edit this field to be more descriptive for your workflow.
• Properties Tab This tab contains all the fields necessary to edit this node’s behavior.
• Input Ports This tab enumerates all of this node’s input ports. You can add and remove input ports here. Removing pre-defined input ports is not recommended.
• Output Ports This tab enumerates all of this node’s output ports. You can add and remove output ports here. Removing pre-defined output ports is not recommended.

To learn more about each NGW node's configuration options, click on the small "?" button to the right of the Settings editor banner.

Running Workflows

When you have the NGW workflow canvas open in the editor area, the action bar along the top of the screen will look like this. Note the two side-by-side play buttons.

• Prompt for info and run the workflow The leftmost play button will prompt you for a parent run directory to run the workflow inside of (the default option is the IWF file’s parent directory, but you may want to run the workflow somewhere else).
• Run the workflow in the last used location The rightmost play button will not prompt you and will run the workflow in the most recent run directory (or the default, which is the IWF file’s parent directory, if this is your first run).

Tutorial: Create a Nested Workflow Driver for Dakota

This section shows you the concrete steps for creating a workflow-based Dakota analysis driver. This is meant to be an exemplary tutorial, and not the "only" way to do things. In practice, you can use any combination of nodes to get from point A to point B; in fact, you are encouraged to experiment with NGW to see what it is capable of.

We will use the classic cantilever beam problem for this example, available in the shipped Dakota example files.

Prerequisites

• Setup your system path to include Dakota! Some workflow nodes require access to the bin directory of Dakota (for example, the "dprepro" node). As a result, before you create any workflows that will be executed as analysis drivers for Dakota, make sure that Dakota's "bin" directory is available on your operating system's path ("PATH" variable on Unix, "Path" variable on Windows).
• Your cantilever.template must be marked up with DPREPRO-supported markup. Consult the section on marking up a template file if you need help with this. The following example will use DPREPRO, but you may use APREPRO instead if you wish. Just note that aprepro uses slightly different syntax.
• You must first create a new workflow using the Workflow-Based Dakota Driver wizard.

Double-click the IWF file generated by the Workflow-Based Dakota Driver wizard to get a look at the workflow canvas:

These nodes were generated thanks to the Workflow-Based Dakota Driver wizard. Now all that’s left is to create intermediate nodes that pass information between the input parameters and output responses.

Pre-Processing

Let's focus on pre-processing first. In the Palette view, expand the "Dakota" folder. Select "dprepro" and drag it onto the workflow canvas. This node is responsible for running the "dprepro" pre-processing utility that typically comes shipped with Dakota.

One of the nice things about the dprepro node is that is does not require explicit connector lines to be drawn from each parameter node into itself. Rather, the mere presence of global parameter values is enough for the dprepro node to work with. The "aprepro" node could be used in exactly the same way.

Using the Project Explorer view, not the Palette view, drag the "cantilever" executable file onto the canvas. NGW will automatically convert "cantilever" to a file node. Do the same for the "cantilever.template" file that we previously marked up.

Connect the "fileReference" output port of the cantilever.template node to the "templateFile" input port of the dprepro node. This will let dprepro know which template file to place the global parameter values in.

Simulation Model Execution

Let's focus on executing the simulation model next. Using the Palette, expand the Pipes folder and drag the "externalProcess" node onto the canvas. This node is responsible for performing the execution of our black-box simulation model.

Now, from dprepro's "outputFile" output port, drag a connector to the blank space beneath "stdIn" on the externalProcess node to create an input port also called "outputFile." Then, drag a connector from the cantilever file node’s "fileReference" output port to blank space on the externalProcess node to create an input port also called "cantilever".

Let's review what's happening here. We have provided the externalProcess node with references to two files – the "cantilever" executable file that is sitting in our workspace, and to a processed input file that doesn’t exist yet, but will be created by the dprepro node at runtime.

Next, we need to tell the externalProcess node what to do with these two files. The "cantilever" executable is actually a Python script with the .py extension removed; because of that, we’ll need to call it using Python.

Click on the externalProcess node to bring up its properties in the Settings editor view.

In the "command" field seen in the Settings editor view, type the following:

python ${cantilever} ${outputFile}

Note the syntax of dollar signs and curly brackets. These are tokens that will get replaced at runtime as appropriate. They will be replaced because the token names match the names of input ports for this node, so NGW will know what to replace the tokens with. Essentially, what we are saying here is "replace the token ${cantilever} with a path to the cantilever file, and replace the token ${outputFile} with a path to the processed input file when it gets created by dprepro node." This relieves the user from having to worry about paths to files that can potentially change as the workflow is running.

Post-Processing

Now let’s post-process the output from our cantilever executable. From the "Dakota" folder in the Palette view, drag a qoiExtractor node onto the canvas:

The "qoiExtractor" node works with "QOIs" (short for "quantity of interest"), which is simply a value extracted from a body of unstructured text. These are the values that will eventually be returned to Dakota as responses.

Our cantilever black-box simulation model only prints out to the console, so we can grab the externalProcess node's "stdout" output port, and drag a connector from it to the qoiExtractor node's "inputText" port to forward all output stream text into the qoiExtractor node:

Click on the qoiExtractor node to bring up its properties in the Settings editor view:

We need to now tell this node what quantities of interest (QOIs) to extract from the stream of input text it's going to receive. Because the qoiExtractor node will connect directly to our already-created response nodes, we're going to need to extract three QOIs from the text – “mass,” “stress,” and “displacement.”

Click on "Add QOI Extractor." In the first dialog that pops up, type in "mass" and click OK. Use the next dialog to extract "mass" from the expected output of the "cantilever" executable.

If you've never used a QOI extractor before, take a detour to learn about QOIs and how to extract them using the Dakota GUI.

Follow the same process to create QOI extractors for "stress" and "displacement."

When you’re done, your settings editor view should look something like this:

Click on Apply in the bottom-right corner of the Settings editor view to save the changes. Note that something has happened to the qoiExtractor node. For each new QOI extractor that we added, a new output port with the same name has been added to the qoiExtractor node. Each output port knows how to forward the value of the corresponding extracted QOI, so now all that remains is to connect each of these output ports to the already-created response nodes.

And we're done!

Running this workflow by itself

If you defined initial values for each of your input parameters, then at this point, you can hit one of the two play buttons on the action ribbon to see this workflow run in isolation, as well as test whether it's working as expected. It's highly recommended that you verify that the workflow behaves correctly by itself before attaching it to a Dakota study.

What happens when Dakota calls this workflow?

So, we now have a workflow that can read in parameters and return response values. This is cool, but what we want eventually is for Dakota to provide new parameters on each Dakota iteration. How do we do that?

1. We need to create a Dakota study that will drive the workflow, using the New Dakota Study wizard.
2. After that, we will need to create a second, outer workflow that knows how to launch a Dakota study that uses the workflow engine as its analysis driver.

Special Topic: Manually Handling Dakota's Parameters and Results Files in a Workflow

By default, Next-Gen Workflow manages the cycle of parameters and responses between itself and Dakota. Behind the scenes, Dakota is writing its parameter information to a file, which Next-Gen Workflow knows how to read at the beginning of your analysis driver workflow. Likewise, behind the scenes, Next-Gen Workflow is writing the output response values to a file, which Dakota knows how to read.

If you wish to manually manage the raw transfer of information for these Dakota-formatted parameters and results files, you may do so, although the resulting analysis driver workflow will be more complex.

Let's go through each of the changes:

• Use a file node to read the parameter file from Dakota: This file node must look for the file specified in your Dakota input file by the parameters_file keyword. In this example, Dakota is writing to a file called "params.txt".
• Use a dakotaParametersMap node to read the contents of the parameters file. This node will give you direct access to all the data structures that Dakota provides. Most importantly, you will have access to the key-value pairs of parameter labels and values, represented in the workflow as the node's "variables" output port, which produces a map object. You can provide this map to any of the pre-processing nodes through their inputParametersMap input port.
• Use a dakotaResultsFile node to format your quantities of interest into Dakota result file format. In this example, we can make use of the qoiMap output port from our qoiExtractor node, which provides the results of our QOI expressions as a map of key-value pairs. The dakotaResultsFile node is then responsible for translating this map into the text format that Dakota can read.
• Use a file node to write the results file for Dakota: This file node must write to the file whose name is specified by the results_file keyword in your Dakota input file. In this example, we pass the output from the dakotaResultsFile node to the dataIn port of the file node.

One last, but important note: When it comes time to make the outer workflow that executes Dakota, you must make sure to alter the properties of the dakota node:

By default, "automaticallyReadParameters" and "automaticallyWriteResults" will be checked, indicating to Next-Gen Workflow that it should handle the Dakota parameters/results files for you. Unchecking both boxes will indicate that you wish to manually handle these files yourself.

Special Topic: Using Multiple Dakota Analysis Drivers in Next-Gen Workflow

Dakota GUI provides support for hooking up multiple, workflow-based analysis drivers to Dakota studies that have multiple interface blocks. This is done by combining the features of DPREPRO and NGW.

For example, suppose we have two interface blocks in a given Dakota input file (as seen in multi-level multi-fidelity studies). In these types of Dakota studies, we interface to a low-fidelity model (LF) and a high-fidelity model (HF). Instead of providing explicit paths to a low-fidelity driver and a high-fidelity driver following each "analysis_drivers" keyword, let's add two pieces of DPREPRO markup - "{DRIVER_LF}" and "{DRIVER_HF}" (note the quotes). The text within the DPREPRO brackets is arbitrary, but the markup text must be consistent going forward.

Now, this Dakota study will not run by itself anymore, since the text now needs to be pre-processed. Let's switch over to Next-Gen Workflow to create a workflow that will run Dakota for us.

There are two things that need to happen on this workflow. The first is that we need to provide one dakotaWorkflowDriver node per analysis driver. Each dakotaWorkflowDriver node should point to the appropriate IWF file that will function as the analysis driver.

The second thing to do is to connect each dakotaWorkflowDriver node to a "dakota" workflow node using new input ports that match the text of the DPREPRO markup. For example, we should add an input port called "DRIVER_LF" to correspond to our {DRIVER_LF} markup. Then, the dakotaWorkflowDriver node that knows about the low-fidelity analysis driver should be connected to this input port. The same should be done with a "DRIVER_HF" input port and the dakotaWorkflowDriver node that knows about the high-fidelity analysis driver.

Once this is done, running the workflow will cause Dakota to execute the nested workflow analysis drivers as defined.

Special Topic: Using Dakota Surrogate Files in Next-Gen Workflow

The Dakota GUI supports replacing components of NGW workflows with Dakota-generated surrogates files. To do this, drag a "surrogate" node onto the canvas (from the Dakota folder in the Palette).

A new surrogate node will have no input or output ports until we associate it with a surrogate file. Open the Settings Editor for the node and select a Dakota surrogate file (both .bin and .txt formats are supported).

After selecting the surrogate, the node will auto-create the necessary input and output ports to interface with the surrogate as part of a workflow.

To learn more about Dakota surrogates, consult Section 8.4 of the Dakota User's Manual.