Table of Contents

1. Introduction
2. Next-Gen Workflow Basics
3. Tutorial: Create a Nested Workflow Driver for Dakota
4. Using Next-Gen Workflow and Dakota with Multiple Analysis Drivers

Introduction

The Dakota GUI includes a node-based workflow creation tool called "Next-Gen Workflow" (NGW from this point on). NGW is an extremely useful standalone tool for building workflows to process data, but it is primarily included as part of Dakota GUI to facilitate more complex Dakota analysis drivers. Before NGW, Dakota GUI could only handle very basic Dakota-to-simulation model interface mappings via the generated Python interface script. Now, you can fully customize the flow of information between Dakota and your simulation model using a series of nodes that are visually connected together.

Next-Gen Workflow Basics

The Palette

The Palette view provides all NGW nodes in a hierarchical folder structure, organized by topic.

If this view is not already present in your perspective, add it by going to "Window > Show View > Other… > Palette."

Settings Editor for Workflow Nodes

The Settings Editor view is used quite heavily as part of NGW, so if your perspective doesn’t have the Settings editor view yet, get it from "Window > Show View > Settings."

The contents of the Settings Editor view changes depending on what is in focus on the NGW workflow canvas. For instance, this is the view for a posPrePro node:

Take a moment to familiarize yourself with the controls in this view that are common to most NGW nodes:

• Name The name of this node. It appears in the top-left corner of the node’s box on the canvas.
• Type The type of this node. You should not edit this field.
• Label The label of this node. This is text that appears above the node to describe what the node is doing. Feel free to edit this field to be more descriptive for your workflow.
• Properties Tab This tab contains all the fields necessary to edit this node’s behavior.
• Input Ports This tab enumerates all of this node’s input ports. You can add and remove input ports here. Removing pre-defined input ports is not recommended.
• Output Ports This tab enumerates all of this node’s output ports. You can add and remove output ports here. Removing pre-defined output ports is not recommended.

To learn more about each NGW node's configuration options, click on the small "?" button to the right of the Settings editor banner.

Running Workflows

When you have the NGW workflow canvas open in the editor area, the action bar along the top of the screen will look like this. Note the two side-by-side play buttons.

• Prompt for info and run the workflow The leftmost play button will prompt you for a parent run directory to run the workflow inside of (the default option is the IWF file’s parent directory, but you may want to run the workflow somewhere else).
• Run the workflow in the last used location The rightmost play button will not prompt you and will run the workflow in the most recent run directory (or the default, which is the IWF file’s parent directory, if this is your first run).

Tutorial: Create a Nested Workflow Driver for Dakota

This section shows you the concrete steps for creating a workflow-based Dakota analysis driver. This is meant to be an exemplary tutorial, and not the "only" way to do things. In practice, you can use any combination of nodes to get from point A to point B; in fact, you are encouraged to experiment with NGW to see what it is capable of.

We will use the classic cantilever beam problem for this example, available in the shipped Dakota example files.

Prerequisite: Create a new workflow using the Workflow-Based Dakota Driver wizard.

Double-click the IWF file generated by the Workflow-Based Dakota Driver wizard to get a look at the workflow canvas:

These nodes were generated thanks to the Workflow-Based Dakota Driver wizard. Now all that’s left is to create intermediate nodes that pass information between the input parameters and output responses.

Pre-Processing

Let’s focus on pre-processing first. In the Palette view, expand the "Dakota" folder. Select "posPrePro" (short for "positional pre-processing") and drag it onto the workflow canvas.

The first thing we’ll want to do is connect our parameters to the posPreNode. For each parameter node, click on the output port (the right-facing arrow), drag to the blank space beneath the "inputParametersMap" input port to create a new input port for each parameter.

Now, let’s make our pre-processors. Click on the “posPrePro” node to bring up its controls in the Settings editor

The posPrePro node has some controls that are specific to editing positional pre-processors:

• templateFile Provides the path to the file that contains template text for your positional pre-processor. The template text is what will receive positional replacements at runtime. Note that this field uses relative pathing, relative to the location of your IWF workflow file.
• Import Input Ports This button is located at the bottom of the view. If you’ve already hooked up all your parameters as input ports to this node, you can use this button as a shortcut to create empty pre-processors for each of your node’s input ports, saving you from some tedium and potential typing errors.
• Positional Pre-Processors Table If you’ve used the Script-Based Dakota Driver wizard, these controls should look familiar. The table shows all the positional pre-processors you’ve defined so far. The Add/Edit/Remove buttons should be self-explanatory. Hitting "Add Pre-processor" or "Edit Pre-processor" will bring up a dialog that allows you to highlight a portion of text that will get replaced by your parameter value at runtime.

At this point, define positional pre-processors for each of your input variables. When you’re done, the Settings editor view for your posPrePro node should look something like this:

Simulation Model Execution

Let’s focus on executing the simulation model next. Using the Project Explorer view, not the Palette view, drag the "cantilever" executable file onto the canvas. NGW will automatically convert "cantilever" to a file node:

Next, using the Palette, expand the Pipes folder and drag the “externalProcess” node onto the canvas. This node is responsible for performing the execution.

Now, from posPrePro’s "outputFile" output port, drag a connector to the blank space beneath "stdIn" on the externalProcess node to create an input port also called "outputFile." Then, drag a connector from the cantilever file node’s "fileReference" output port to blank space on the externalProcess node to create an input port also called "cantilever".

Let's review what's happening here. We have provided the externalProcess node with references to two files – the "cantilever" executable file that is sitting in our workspace, and to a processed input file that doesn’t exist yet, but will be created by the posPrePro node at runtime.

Next, we need to tell the externalProcess node what to do with these two files. The "cantilever" executable is actually a Python script with the .py extension removed; because of that, we’ll need to call it using Python.

Click on the externalProcess node to bring up its properties in the settings editor view.

In the "command" field, type the following:

python ${cantilever} ${outputFile}

Note the syntax of dollar signs and curly brackets. These are tokens that will get replaced at runtime as appropriate. They will be replaced because the token names match the names of input ports for this node, so NGW will know what to replace the tokens with. Essentially, what we are saying here is "replace the token ${cantilever} with a path to the cantilever file, and replace the token ${outputFile} with a path to the processed input file when it gets created by posPrePro node." This relieves the user from having to worry about paths to files that can potentially change as the workflow is running.

Post-Processing

Now let’s post-process the output from our cantilever executable. From the "Dakota" folder in the Palette view, drag a qoiExtractor node onto the canvas:

The "qoiExtractor" node works with "QOIs" (short for "quantity of interest"), which is simply a value extracted from a body of unstructured text. These are the values that will eventually be returned to Dakota as responses.

Our cantilever black-box simulation model only prints out to the console, so we can grab the externalProcess node's "stdout" output port, and drag a connector from it to the qoiExtractor node's "inputText" port to forward all output stream text into the qoiExtractor node:

Click on the qoiExtractor node to bring up its properties in the Settings editor view:

We need to now tell this node what quantities of interest (QOIs) to extract from the stream of input text it's going to receive. Because the qoiExtractor node will connect directly to our already-created response nodes, we're going to need to extract three QOIs from the text – “mass,” “stress,” and “displacement.”

Click on "Add QOI Extractor." In the first dialog that pops up, type in "mass" and click OK. Use the next dialog to extract "mass" from the expected output of the "cantilever" executable.

If you've never used a QOI extractor before, take a detour to learn about QOIs and how to extract them using the Dakota GUI.

Follow the same process to create QOI extractors for "stress" and "displacement."

When you’re done, your settings editor view should look something like this:

Note that something has been happening while we were adding QOI extractors to this node. For each new QOI extractor added, a new output port with the same name has been added to the qoiExtractor node. Each output port knows how to forward the value of the corresponding extracted QOI, so now all that remains is to connect each of these output ports to the already-created response nodes.

And we’re done!

What happens when Dakota calls this workflow?

If you defined initial values for each of your input parameters, then at this point, you can hit one of the two play buttons on the action ribbon to see this nested workflow run in isolation (as well as test whether it's working as expected).

So, we now have a workflow that can read in parameters and return response values. This is cool by itself, but what we want eventually is for Dakota to provide new parameters on each Dakota iteration. How do we do that?

• We still need to create a Dakota study that will drive the workflow, using the New Dakota Study wizard.
• After that, we will need to create a second, outer workflow that knows how to launch a Dakota study that uses the workflow engine as its analysis driver.

DPREPRO as an Alternative to Positional Pre-Processing

In the previous tutorial, we used the "posPrePro" node to demonstrate text-based pre-processing. There are other pre-processing nodes at your disposal that can make potentially make your workflow easier to set up. For example, the "aprepro" and "dprepro" nodes perform a similar task as "posPrePro", but they do not require explicit connector lines to be drawn from each parameter node to the pre-processing node. Rather, the mere presence of global parameter values is enough for these nodes to work with. For example, using the "dprepro" node, our previous Dakota workflow example could have looked like this:

The "aprepro" node could be used in exactly the same way.

With this approach, it is necessary to markup your template file ahead of time with either APREPRO- or DPREPRO-style markup. These two pre-processing nodes require slightly different markup syntax, so be sure to consult the APREPRO and DPREPRO documentation to make sure your template file has the right kind of markup in it.

• Learn about the APREPRO pre-processing tool.
• Learn about the DPREPRO pre-processing tool (a Dakota-developed derivation of APREPRO with similar syntax) in Section 10.8 of the Dakota User's Manual.

Using Next-Gen Workflow and Dakota with Multiple Analysis Drivers

Dakota GUI provides support for hooking up multiple, workflow-based analysis drivers to Dakota studies that have multiple interface blocks. This is done by combining the features of DPREPRO and NGW.

For example, suppose we have two interface blocks in a given Dakota input file (as seen in multi-level multi-fidelity studies). In these types of Dakota studies, we interface to a low-fidelity model (LF) and a high-fidelity model (HF). Instead of providing explicit paths to a low-fidelity driver and a high-fidelity driver following each "analysis_drivers" keyword, let's add two pieces of DPREPRO markup - {DRIVER_LF} and {DRIVER_HF}. The text within the DPREPRO brackets is arbitrary, but the markup text must be consistent going forward.

Now, this Dakota study will not run by itself anymore, since the text now needs to be pre-processed. Let's switch over to Next-Gen Workflow to create a workflow that will run Dakota for us.

There are two things that need to happen on this workflow. The first is that we need to provide one dakotaWorkflowDriver node per analysis driver. Each dakotaWorkflowDriver node should point to the appropriate IWF file that will function as the analysis driver.

The second thing to do is to connect each dakotaWorkflowDriver node to a "dakota" workflow node using new input ports that match the text of the DPREPRO markup. For example, we should add an input port called "DRIVER_LF" to correspond to our {DRIVER_LF} markup. Then, the dakotaWorkflowDriver node that knows about the low-fidelity analysis driver should be connected to this input port. The same should be done with a "DRIVER_HF" input port and the dakotaWorkflowDriver node that knows about the high-fidelity analysis driver.

Once this is done, running the workflow will cause Dakota to execute the nested workflow analysis drivers as defined.