How Submitters Bootstrap Themselves

Occasionally, you might hit a problem where you’ve installed an integrated submission script to send work to Deadline but it’s not working. Here, we’re going to go over the fundamentals of how most (Blender and Keyshot are exceptions) of the submitters work inside of Digital Content Creation applications (DCCs) to provide more details on what’s going on behind the scenes so you can get some intuition on how to fix them.

At a high level, the submitters are designed to bootstrap just enough code to be able to call DeadlineCommand then hand off to scripts read from the Repository to load the rest of the submitter. The smaller the footprint in the initial script, the less likely it is that users will need to make changes to their workstations during an upgrade. You can actually see the various phases of the submitters in the folder structure of the Repository’s “DeadlineRepository10\submission” folder:
The “Client” folder is the lightweight bootstrap script, the “Installers” folder contain tools to automatically copy the contents of the “Client” folder for users automatically, and the “Main” folder is where the real heavy lifting is done of building a UI and creating job info files. If you ever have problems with the installer, all of our integration documentation have manual install steps (example here for Houdini) and sometimes integrated installers can work with newer versions of a DCC but you’ll need to update the backed plugin (information on how to do that here). Our integrations are written in multiple programming languages and multiple operating systems so it’s hard to cover all cases, but let’s take a look at two examples to give you a basic idea. We’ll take the compositors for examples here so Nuke first (Python), then After Effects (JavaScript).

For Nuke, we have a few different bootstrap scripts for Nuke itself, the Frame Server, and V-Ray Standalone with Nuke. Since most will have experience with the base Nuke version we’ll focus down on that. While this guide isn’t geared to software developers, we can talk a little bit about the code at a high level. If you open DeadlineNukeClient.py for yourself you’ll see the bulk of the code (60 of 98 lines in Deadline 10.3.1.3) is focused around getting the Repository path. With that then add that to Python’s paths it uses to find client libraries and then we import SubmitNukeToDeadline from the “Main” folder and call SubmitToDeadline() to do its thing.
So, what about “menu.py”? This is special to Nuke and maybe unsurprisingly adds the “Submit Nuke to Deadline” command to Nuke’s menubar. We have a similar tiny scripts for other DCC’s like Maya’s one-line userSetup.mel which are micro scripts that sit somewhere other than the bootstrap script to... well, bootstrap the bootstrapper. Fun exceptions to the rule are everywhere! 🙂

Moving to After Effects now, the bootstrap script is DeadlineAfterEffectsClient.jsx and again we’re finding the path to the Repository, loading the contents of SubmitAEToDeadline.jsx and letting it call its own start function. They’re all-in-all pretty unexciting and you can peruse other bootstrap scripts and they’ll look about the same. So, what is exciting? The way the scripts get copied when you’re using the Remote Connection Server (RCS). It’s... a little peculiar.

For those who are technical and actually read through the bootstrapper scripts, you’ll see that there are just calls to deadlinecommand GetRepositoryFilePath <path/to/script.py> . So, how could this work with the RCS? Well, when we created the RCS (and Proxy Server that the RCS replaced) we had a hard think of how we could support the thousands of workstations that already had the bootstrap scripts installed. What we did was change how the GetRepositoryFilePath subcommand worked. In fact, what we’re doing is making a secret temporary copy of the folder or files and then return the path to the locally cached file. The location has changed a few times during Deadline 10’s lifetime, so to keep this content accurate over time we won’t include those paths here, but if you’re curious for yourself and you’re connecting to an RCS on your workstation you can call deadlinecommand GetRepositoryPath submission/AfterEffects/Main/SubmitAEToDeadline.jsx for yourself!

The final technical mystery here for those who didn’t read the bootstrapper scripts is how do we know where deadlinecommand is on the system? It’s different for all three operating systems, but we’ll start with macOS because it’s the most unusual. Both Windows and Linux rely on a system-wide environment variable called DEADLINE_PATH but Apple has changed how environment variables work so often that we simply chose not to rely on it.

On macOS workstations, we store a small file during client installation at /Users/Shared/Thinkbox/DEADLINE_PATH. It’s a single line file that contains the active Deadline installation path (this was more helpful when we released major versions more often and let people switch between Deadline 7, 8, 9, and 10). The bootstrapper scripts read this file and then do their thing.

$ cat DEADLINE_PATH 
/Applications/Thinkbox/Deadline10/Resources

As mentioned before, Windows and Linux rely on DEADLINE_PATH. Windows is the least exciting as it’s just a system-wide environment variable named DEADLINE_PATH as you may be used to. If you’ve seen one system-wide variable, you’ve seen them all. So let’s focus more on Linux! Variables here can be set in multiple places so how are Deadline’s set? Well, to make sure it’s available everywhere there’s a user shell, the client installer creates a file at /etc/profile.d/deadline-client.sh. This works since all shells run all scripts in that folder regardless of Bash, zsh, or csh. While you could add other variables in that file, it’s not recommended in case one day we need to replace that file while will lose your changes. You can make your own copy and edit that though!

This brings us to the end of our little high level adventure. We talked about the various bootstrapping phases and system configurations that make sure your submission experience is kept up to date with your chosen Repository location. If you want to learn more, sift through some of the files in the Repository. You can always install a local copy on your workstation (though use localhost for the DB hostname so your friends a co-workers don’t go talking to your Database) and reverse engineer things. It’s a fair bit of fun, and we can help out if you need. Best place for that is our forums!