Creating New Deadline Plugins (Oct 20, 2016)

Edwin Amsler
  • Updated
Follow

Version: Deadline 8.0

INTRODUCTION - TASK CUSTOMIZATION

As we saw previously in our overviews of the Cinema 4D and VRay DBR plugins, Deadline plugins can be customized to perform a wide variety of different tasks.

Through this blog, we will be going over the creation process of the Silhouette plugin that is being added to Deadline 8.1. Note that this plugin is an example of a Simple Plugin, which means we are using Silhouette’s command line interface to control the rendering process. Advanced Plugins can be used to do much more than just control an application via the command line, but we’ll save those for another blog entry.

By using this script as an example, we hope that it will encourage you to make your own plugins or tweak existing plugins to your liking.

INITIAL RESEARCH

Before we begin doing any work on creating a plugin, we first need to find out what we can do with the software using its command line interface. This can be done by referring to its documentation, or by printing the command line usage to the console. We’re looking for command line flags that can be used to control the rendering process, or the ability to run scripts within the program.

In the case of Silhouette, here is what we get when I run their command line tool without any arguments:

Here we see that most of the options are to override the options that are already set within the scene, and we can start by just passing in the path to the scene file we would like to render. Knowing this, we can do a quick test render to make sure this works.

We now have enough information to start making our plugin.

PLUGIN SETUP

To start development of the new plugin, we need to create the necessary files in the Deadline Repository. First, navigate to [Repository]/custom/plugins and create a new folder called Silhouette. Then within this new folder, create the following three files:

  • Silhouette.py: This is the main plugin file, and will contain the Python code that controls the rendering process.
  • Silhouette.param: This will contain the plugin configuration settings.
  • Silhouette.options: This will contain the plugin-specific job settings that can be modified after the job is submitted.

The reason we are placing it in the Repository’s custom folder is so that we don’t have to modify the one that ships with Deadline. This ensures that your script will not be overwritten when you update Deadline in the future. Our version of this plugin with the same name as the shipping version will also be used by Deadline and the shipping version will be ignored. Nifty eh? See the Scripting Overview Documentation for more information about the custom folder.

WRITING THE SCRIPT

Open the Silhouette.py file that you added to your Deadline Repository, and start by defining the basic shell of the Plugin.

from Deadline.Plugins import *
from Deadline.Scripting import *

def GetDeadlinePlugin():
    return SilhouettePlugin()

def CleanupDeadlinePlugin( deadlinePlugin ):
    deadlinePlugin.Cleanup()

class SilhouettePlugin( DeadlinePlugin ):

def __init__( self ):
    self.InitializeProcessCallback += self.InitializeProcess
    self.RenderExecutableCallback += self.RenderExecutable
    self.RenderArgumentCallback += self.RenderArgument

def Cleanup( self ):
    del self.InitializeProcessCallback
    del self.RenderExecutableCallback
    del self.RenderArgumentCallback

def InitializeProcess( self ):
    self.PluginType = PluginType.Simple

def RenderExecutable( self ):
    return ""

def RenderArgument( self ):
    return ""

The first thing to note here are the functions GetDeadlinePlugin and CleanupDeadlinePlugin. These are what Deadline uses to create and destroy the plugin when required.

The next step is to define the SilhouettePlugin class and its individual parts, starting with the init and Cleanup Functions. These functions tell Deadline what callbacks to use and then clean them up when we are done with the plugin.

Next we have the InitializeProcess function. This function does any initial setup that is needed, including telling Deadline what kind of plugin this is (Simple or Advanced), and sets up any popup or stdout handlers that are needed.

As mentioned above, we are creating a Simple plugin, meaning we are going to define an executable and some arguments for it. When Deadline renders a job using this plugin, it simply runs the command line that we define and waits for it to return.

From here, we want to run the same render test we did from the command line above. In order to do this we are going to write some code for the RenderExecutable and the RenderArgument functions.

def RenderExecutable( self ):
    return "C:\\Program Files\\SilhouetteFX\\Silhouette v5.2\\sfxcmd.exe"

def RenderArgument( self ):
    return "c:\\Users\\Grant\\Silhouette\\SilhouetteTest.sfx\\project.sfx"

SUBMITTING A TEST JOB

We can now test our plugin by submitting a test job to Deadline. To do this, we need to create the following files:

  • Job Info file: This file contains general job settings, like job name and frames to render.
  • Plugin Info file: This file contains plugin-specific job settings. In this case, it will contain settigns that our Silhouette plugin will use.

For our purposes, we’ll call these files testJobInfo.txt and testPluginInfo.txt, respectively.

Here’s the Job Info file that we’ll use:

Name=Silhouette Test
Frames=0
ChunkSize=1
Plugin=Silhouette

Since our plugin doesn’t use any plugin-specific settings yet, we can just use an empty Plugin Info file for now. We are then able to submit these now just using Deadline Command as follows.

For more information on manually submitting jobs to Deadline, and the other Job Info settings that are available, check out the Manual Job Submission Documentation.

EXPANDING THE PLUGIN

Now that we have a simple test working, we can expand this out to make it work for more than a single file using a hard coded executable. To do this, we will start by pulling the render executable from the plugin configuration.

In order to do this, we need to add some entries to the Silhouette.param file.

[About]
Type=label
Label=About
Category=About Plugin
CategoryOrder=-1
Index=0
Default=Silhouette Plugin for Deadline
Description=Not configurable

[ConcurrentTasks]
Type=label
Label=ConcurrentTasks
Category=About Plugin
CategoryOrder=-1
Index=0
Default=True
Description=Not configurable

[Silhouette_RenderExecutable_5]
Type=multilinemultifilename
Category=Render Executables
CategoryOrder=0
Index=0
Label=Silhouette 5 Executable
Description=The path to the Silhouette executable file used for rendering. Enter alternative paths on separate lines.
Default=C:\Program Files\SilhouetteFX\Silhouette v5.2\sfxcmd.exe;/Applications/SilhouetteFX/Silhouette v5.2/sfxcmd;/opt/SilhouetteFX/silhouette5/sfxcmd

The main entry above that we care about here is the Silhouette_RenderExecutable_5 setting, which defines the default executables that are to be used for Silhouette 5. Note that the type for this setting is multilinemultifilename, which means that it supports multiple executable paths separated by semicolons.

Now we can update the RenderExecutable function in the Silhouette.py plugin file:

def RenderExecutable( self ):
    version = int( self.GetPluginInfoEntry( "Version" ) )
    exe = ""
    exeList = self.GetConfigEntry( "Silhouette_RenderExecutable_" + str(version) )
    exe = FileUtils.SearchFileList( exeList )
    if( exe == "" ):
        self.FailRender( "Silhouette render executable was not found in the configured separated list \"" + exeList + "\". )
    return exe

This code will now pull what version of Silhouette to run from the Plugin Info that is submitted with the job (using the GetPluginInfoEntry function). This means that from now on we need to include the line Version=5 in our plugin info file to be able pull the correct version, otherwise this function call will fail.

Once we have the version, we get the list of executables associated with that version (using the GetConfigEntry function), which is defined by the Silhouette_RenderExecutable_5 setting in the plugin configuration file above. We then search that list for the first executable that exists on the render node, or we fail the render if none exist.

We can also do something similar for the render argument to be able to pull the project filename from the Plugin Info:

def RenderArgument( self ):
    sceneFile = self.GetPluginInfoEntryWithDefault( "SceneFile", self.GetDataFilename() )
    return sceneFile

Here we want to make sure that the user will be able to submit the project file with the job, so we develop the code to try and retrieve it from the Plugin Info. However if that does not exist we will fall back on the first Auxiliary File.

At this point we are now able to render a project file. We can test this by submitting another job using the same Job info file and using the following plugin info file:

SceneFile=C:/Users/Grant/Silhouette/SilhouetteTest.sfx/project.sfx
Version=5

We have now successfully rendered a project file using whatever settings are in the scene file. Looking back at the list of options that are available from the command line, we can start by adding the ability to choose which frames to render and what file name the output files will have.

In order to do this, we will pull the start and end frame of the current task from the job we are working on, and we will get the file name override from the Plugin Info.

def RenderArgument( self ):
    sceneFile = self.GetPluginInfoEntryWithDefault( "SceneFile", self.GetDataFilename() )
    renderArguments = "\"" + sceneFile + "\" "

    startFrame = str( self.GetStartFrame() )
    endFrame = str( self.GetEndFrame() )
    renderArguments += "-range %s-%s " % ( startFrame, endFrame )

    outputFilename = self.GetPluginInfoEntryWithDefault( "OutputFilename", "" )
    if outputFilename != "":
        renderArguments += "-file \"%s\"" % outputFilename

    return renderArguments

At this point you should be able to add any additional options that you see as necessary.

STDOUT HANDLERS

Ideally, we would like the ability to be able to update the progress of a task as it is being rendered in the Deadline Monitor, or to be able to react to something being printed to the log. In order to handle this, we add stdout handlers.

In order to use stdout handlers we need to do four things:

  • Enable stdout handlers.
  • Add the handlers to our plugin.
  • Remove the handlers when we are done.
  • Define a function to call when the stdout handler is triggered.

To do this, we update the Cleanup and InitializeProcess, and add a new handleStdOut function:

def Cleanup( self ):
    del self.InitializeProcessCallback
    del self.RenderExecutableCallback
    del self.RenderArgumentCallback
    
    # Remove the stdout handlers
    for stdoutHandler in self.StdoutHandlers:
    del stdoutHandler.HandleCallback

def InitializeProcess( self ):
    self.PluginType = PluginType.Simple
    
    # Enable stdout handling
    self.StdoutHandling = True
    
    # Define a handler
    self.AddStdoutHandlerCallback( r"test" ).HandleCallback += self.handleStdOut

def handleStdOut( self ):
    # Handle the trigger.
    pass

In this example, we want to add progress logging. We will do this by defining a handler that will trigger whenever we see a progress message in the log.

Looking back at the output from the test that we ran earlier, it appears that the lines that we care about are in the form of:

1 (50%): Frame 0 (1 of 2) - RenderingFrame 0 (1 of 2) - Writing

In this case there is already a percentage that we can easily grab using a regular expression. So let’s replace the “test” callback above with the following:

# This regular expression will catch on anything of the form (#%) where # is a number.
self.AddStdoutHandlerCallback( r".*\(([0-9]+)%\).*" ).HandleCallback += self.HandleStdoutProgress

Using this handler, we can then set the progress as follows. Note that we’re replacing the handleStdOut function above with this new HandleStdoutProgress function:

def HandleStdoutProgress( self ):
    # Set the progress to be the value of just the number
    self.SetProgress( float( self.GetRegexMatch( 1 ) ) )
    
    # Set the status message to be the full line of output.
    self.SetStatusMessage( self.GetRegexMatch( 0 ) )

So, to wrap up, we have successfully made a new simple plugin for Deadline! For our tests, we were manually building up the Job Info and Plugin Info files that are submitted with the job, but this can be streamlined by creating a Submission Script in the Deadline Monitor. However, that is a topic for another blog!

ADDITIONAL INFORMATION

In addition to the information found here, the Deadline User Manual contains useful information on creating new plugins and the information available to them, and the Deadline Scripting Reference contains documentation for all of the available classes and functions that are exposed to scripts.

There is also an ever growing GitHub project that contains multiple useful scripts that are not shipped with Deadline, including a simple example plugin.

Related articles

Comments

0 comments

Article is closed for comments.

Powered by Zendesk