Chapter 9. Creating a Workflow (beta feature)

Starting with version 6.8, the Dakota GUI includes a graphical, node-based workflow creation tool called "Next Gen Workflow."  This feature has been provided specifically to address the need to run more complex interfaces that are driven by Dakota.  Previously, the GUI could only handle very basic Dakota-to-simulation model interface mappings via our generated Python interface script (see Chapter 6).  Now, you can fully customize the flow of information between Dakota and your simulation model using a series of blocks (i.e. nodes) visually connected together.

However, please note that Next Gen Workflow is a beta feature still under active development.  As such, setting up a workflow from scratch is a fairly complex process that can be difficult to debug.

In this chapter, we will once again use the cantilever beam example first shown in Chapter 6.  However, you'll need to download a slightly different set of files:

New Project Setup

  • Create a new Dakota project called "CantileverNestedWorkflow."
    • New > Project > Dakota Project > CantileverNestedWorkflow.
    • Set folder creation to "No, create a blank project."
    • Finish.
  • Right-click project and "Import."
    • General > Archive File
    • Choose your downloaded .zip file.
    • Finish.
    • You should have the following three files in your project:
      • cantilever (the external cantilever beam model)
      • cantilever.i (sample input file for external model)
      • dakota_cantilever_center.in (the Dakota study)

Open your Dakota study file.

  • You should notice that the analysis_drivers keyword value is set to 'SAW_DRIVER.'  SAW_DRIVER is a special driver name that gets replaced at runtime by the workflow engine, based on what the workflow engine needs the analysis driver to be.
  • Parameters files and results files must be named in order for the workflow engine to find them.  You may choose names other than "params.txt" and "results.txt," but you must be consistent when adding workflow nodes that look for these files.
  • Note that we have specified "directory_save" and "file_save" in the interface block to prevent Dakota from deleting work directories while running.  Currently, the workflow engine will run into problems if work directories are deleted in the middle of workflow execution.  You may delete the created work directories after you've finished running the workflow, in order to clean up clutter in your project.  Alternately, if you specify your interface block to cause Dakota to run in single-threaded mode and not save work directories, that approach would also work correctly (see below):
    • interface
         analysis_drivers 'SAW_DRIVER'
            fork
               parameters_file "params.txt"

               results_file "results.txt"

Workflow #1 – Run Dakota

We will create two workflows.  The first will be responsible for executing Dakota.  The second will be a "nested driver workflow," called from within Dakota via the "SAW_DRIVER" analysis driver that we specified in the input file's interface block.

Make sure to switch your GUI perspective to “Next Gen Workflow” (located at Window > Perspective > Open Perspective > Other)

To begin the first workflow...

  • Right click your Dakota project, then New > Other.
  • In the dialog that appears, select Workflow > Workflow.  Click Next.
  • Call it "RunDakota" and click Finish.

RunDakota.iwf will be added to your project.  At this time, you should see some additional views in your perspective:

DakotaGuiManual_68_Chapter9_16.png

Note that you now have a grid to create your visual workflow on, as well a Palette view that contains the nodes you can use.

Using the Palette View, select the "dakota" node under the "Dakota" folder and drag it onto your workflow editor.

  • Left-click the "dakota" node on your workflow grid.  Then, switch to the Settings view by clicking on its tab (you should already have the Settings view docked in the same view panel area as the Palette view).
  • In the Settings view editor, set this node's "dakotaPath" field to the location of Dakota on your filesystem. (just copy and paste this path from Window->Preferences->Dakota).
  • On the "analysisDriverWorkflow" field, set the value to "NestedWorkflow.iwf" (we'll create this file eventually)

Using the Palette View, select the "file" node under the "Pipes" folder and drag it onto to your workflow.

  • In the Settings view editor, set the "fileName" field for this node "dakota_cantilever_center.in" (The path is relative to your project directory, so unless you moved the Dakota input file to a different subdirectory, the text value doesn't require any additional path information).
  • A good habit to get into is to label your file nodes, to keep files straight as you build your workflow.  This can be done simply by using the Label field in the Settings view editor while you have that node selected.  For this tutorial, we will label the node with the filename "dakota_cantilever_center.in."

Pro Tip:  When working with "file" nodes, you can save yourself a couple of steps by dragging file resources directly from the project navigator resource tree onto the workflow editor view.  This has the same effect as getting a "file" node from the Palette view, but is slightly faster and more intuitive.  Doing this will also auto-label the file node for you.

On the grid, connect the "file" node's "fileReference" port to the "dakota" node's "stdin" port by left-clicking on the "fileReference" port, holding down the left mouse button and dragging to the "stdin" port.  Afterwards, your workflow should look like this.

DakotaGuiManual_68_Chapter9_17.png

This workflow will take your Dakota input file (the first node) and pass a reference to it to Dakota (the second node) for running.

Optional:  It's good to check for errors as we go.  At this stage, you may wish to test what we have so far.  Doing so won't really do anything yet (we haven't created the second workflow that does the bulk of the work).  If you run it as-is, this workflow will create a bunch of temporary files, try to run Dakota, and then get stuck.

You can run the first workflow by clicking the leftmost green play button along the ribbon in the top-left corner of the screen (disregard the play button on the right).
DakotaGuiManual_68_Chapter9_3.png

To check out what's happening, you can look at the workflow.engine.log file and RunDakota.log file to verify there are no blatant errors yet.  If anything, we should only see errors indicating that "NestedWorkflow.iwf" doesn't exist, but that's expected, because we haven't created it yet.

You can hit the red stop button to force the workflow to stop.

Workflow #2 – Nested Driver Workflow

Time to create the second workflow.  If you followed the tutorial in Chapter 6, this will essentially be recreating the work we did in that chapter (i.e. pre-processing and post-processing for the cantilever beam model).

  • File > New > Other.
  • On the dialog, select "Workflow > Workflow."  Click Next.
  • Call this workflow "NestedWorkflow."  Make sure it's placed in the same "CantileverNestedWorkflow" project we've used up to this point.  Click Finish.

NestedWorkflow.iwf will be added to your project.

Using the Palette, add the Pipes > file node to your workflow.

DakotaGuiManual_68_Chapter9_4.png

  • Set the fileName field to...
    • On Windows:
      • ${workflow.workdir}\params.txt
    • On Mac or Linux...
      • ${workflow.workdir}/params.txt

${workflow.workdir} is special property used in Next Gen Workflow to mean “the current working directory.”  Rather than hard-coding a full path to our params.txt file, including this field allows us to reach the params.txt file from potentially differently-named parent folders (for instance, when evaluation concurrency > 1).

Also note that we will have more file nodes later, so give this one a unique label like "params.txt."

Using the Palette, add the Dakota > dakotaParametersMap node to your workflow.

DakotaGuiManual_68_Chapter9_5.png

  • This node is responsible for parsing a Dakota interface parameters file and extracting all its values.  For this problem, we're only interested in the "variables" output port.
  • This node doesn’t need any configuration. 

Using the Palette, add the Dakota > posPrePro node to this workflow.

DakotaGuiManual_68_Chapter9_6.png

  • Set this node's Template File field to "cantilever.i" (remember, we imported it earlier)
  • Using the "Add Pre-processor" button, add the following pre-processing elements to your node (see Chapter 6 for more information on pre-processing this input file):
    DakotaGuiManual_68_Chapter9_7.png

Using the Palette, add the Pipes > file node to this workflow. 

DakotaGuiManual_68_Chapter9_18.png

  • This node will be used differently than our previous file nodes.  Rather than reading from an existing file, we will write all our processed cantilever input information to a new, not-yet-created file.  Use "cantilever_processed.i" (or something similar) for the fileName field.
  • Reminder:  We're still in a temporary work_dir created by Dakota, so don't forget to add the workdir variable, as follows:
    • On Windows:
      • ${workflow.workdir}\cantilever_processed.i
    • On Mac or Linux...
      • ${workflow.workdir}/cantilever_processed.i
  • Label the file node with the same name as the file you're going to write to (i.e. "cantilever_processed.i").

Using the Palette, add the Pipes > pythonScript node to this workflow.

DakotaGuiManual_68_Chapter9_9.png

  • This node will run a short Python script that will launch our cantilever beam model.
  • With this pythonScript node selected, on the Input Ports tab of the Settings view, add two new input ports to the Python node:  “f1” and “f2.”
  • In the Type column next to each new input port, set its type to "text."
  • Finally, copy-paste the following script into the script field of this node on the Properties tab:

    import subprocess

    proc = subprocess.Popen(['python', f1, f2], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

    print proc.communicate()[0]

Using the Palette, add a Pipes > file node to your workflow.  Set the fileName field to "cantilever".  Give the node a label like "modelFileNode" to distinguish it from our other file nodes.

DakotaGuiManual_68_Chapter9_4.png

  • Alternately, you can directly drag the "cantilever" file from your project navigator onto the workflow editor.  This will auto-label the node as "cantilever."

Using the Palette, add a Dakota > qoiExtractor node to your workflow.

DakotaGuiManual_68_Chapter9_10.png

  • Using the "Add QOI Extractor" button, add the following to your node (see Chapter 6 for more information on defining QOIs):
  • DakotaGuiManual_68_Chapter9_11.png

Using the Palette, add a Dakota > dakotaResultsFile node to your workflow.

DakotaGuiManual_68_Chapter9_12.png

  • This node is responsible for formatting a map of key-value pairs into the results file format that Dakota expects.
  • It does not require any configuration.

Using the Palette, add a Pipes > file node to your workflow.

DakotaGuiManual_68_Chapter9_19.png

  • Set its fileName field to:
    • On Windows:
      • ${workflow.workdir}\results.txt
    • On Mac or Linux...
      • ${workflow.workdir}/results.txt
  • This will be the file Dakota expects to read from, according to our original Dakota input file.

Connect all your nodes:

  • The params.txt file node's fileReference port connects to the DakotaParametersMap node's parametersFile port.
  • The DakotaParametersMap node's variables port connects to the PositionalPreProcessor node's inputParametersMap port.
  • The PositionalPreProcessor node's processedFileText port connects to the cantilever_processed.i file node's dataIn port.
  • The cantilever file node's fileReference port connects to the pythonScript node's f1 port.
  • The cantilever_processed.i file node's fileReference port connects to the pythonScript node's f2 port.
  • The pythonScript node's stdout port connects to the QOIExtractor node's inputText port.
  • The QOIExtractor node's qoiMap port connects to the DakotaResultsFile node's inputMap port.
  • The DakotaResultsFile node's outputText port connects to the results.txt file node's dataIn port.

When you’re done, your nested workflow should look something like this:

DakotaGuiManual_68_Chapter9_20.png

DakotaGuiManual_68_Chapter9_21.png

 

To summarize, this is what our second "nested" workflow is doing:

  • Read the Dakota parameters file:  This is handled by the params.txt file node and DakotaParametersMap node.
  • Pre-processing:  This is handled by the PositionalPreProcessor node and first cantilever_processed.i file node.
  • Model execution:  This is handled by the cantilever file node and pythonScript node.
  • Post-processing:  This is handled by the QOIExtractor node.
  • Write the Dakota results file:  This is handled by the DakotaResulsFile and final results.txt file node.

Test it out by returning to the first workflow (RunDakota.iwf) and clicking the leftmost green play button.

If it works, you should see Dakota creating work_dir directories for each of its iterations, and generating its usual output data files (such as the tabular data file).