This guide was written specifically to be an ammendment to any requests we might recieve for adjusting Deadline’s Python libraries. It should help systems administrators and wrangling staff understand how things work. Use with caution. :)
The typical changes we have helped customers with include:
- When an API has changed in one of our partner integration “Pipeline Tools”
- The AWS integration in Boto3 has new features
- Updating certificates that are vended inside of the certifi package need
Deadline makes heavy use of Python in it’s integrations. As it’s grown to be the industry standard glue that binds various DCCs together, we’ve leveraged Python and some of its common libraries (Boto3, requests) as well as libraries from our industry partners to communicate with pipeline services, AWS, and the DCCs themselves. If you’d like a full list, open your farm’s Repository folder and go to “[repo]\pythonsync3\pythonsync3.zip” and take a look! This guide is current to Deadline 10.2.0.10.
A word of warning: AWS Thinkbox doesn’t officially support making changes to the list of libraries here and so the Support team will have you revert back to the original deployment if there are issues. The guide should help you to figure out the Python import problems you’re having if you ever hit them.
Where do Libraries Live?
Let’s get to the detailed stuff. We ship python libraries compressed in a zip bundle and copies live in several different places. Those files are cached for performance reasons in some interesting spots you may not expect, and the compressed versions live in yet more places. Here’s a rundown of all the places Python lives and why:
Bundles: (these are those zip files, and there’s a version for Python 2 named “pythonsync.zip” and one for Python 3 “pythonsync3.zip”)
- The Repository!: This is the classic spot and in theory this is where the files will be pulled from. They’re in “pythonsync” and “pythonsync3”.
- The bin folder: This is a bit unexpected, but there’s a client-local copy that lives in the client installation directory. The reason we place it there is to ease the initial load when thousands of machines come online at at once. Especially helpful on AWS where the local caches are lost when an instance is terminated.
Caches: (where the files are extracted and loaded from)
- The local RCS cache: To ease performance pentalties of repeatedly pulling files, Deadline clients keep a local cache of the Repository files it’s asked for over the RCS. Nearly all files requested will live here, and you can go review its contents and clear it out if you like. We also create a unque cache for each RCS/Repository you’ve connect to in case they have different contents.
- The nginx cache: This is only relevant for AWS Portal, but we have an nginx server tucked away inside the Gateway that caches files for one hour. You can read more about that in our documentation.
- Install directory: There are some libraries that are extracted to the installation directory. These will be in “python” and “python3” folders. These will be loaded if the next cache doesn’t contain the imported library.
- The python cache: This is the final resting point for our python libary bundles and is how Deadline load them into memory at runtime. Because the Deadline installation folder may be write protected they live in the local user’s data folder. Examples that will be familar to those looking for the per-user deadline.ini file:
- Windows: "C:\Users\<username>\AppData\Local\Thinkbox\Deadline10\pythonAPIs\7YIEI5BVezcT33Jm38Cl3A=="
- Linux: “/home/<username>/Thinkbox/Deadline10/pythonAPIs/jdVDIxEGB4rGeWHpjEoq0A==”
- macOS: “/Users/<username>/Library/Application Support/Thinkbox/Deadline10/pythonAPIs/ehrv3b+f3rh09+YFyoF6A==”
That’s a lot of places to look! With that in mind, how do these paths get updated?
Lifetime of a Python Library
All applications are responsible for updating the caches when they open by default, with the exception of DeadlineCommand. It’s possible to have DeadlineCommand update the cache by invoking the PopulatePythonSync
subcommand. The reason for this is that we didn’t want to delay starting submitters which may call into DeadlineCommand more than five times to load farm details. At startup, Deadline applications are busy doing several things at once so the sync time is less of an impact. You’ll be able to spot the sync process in the application logs if you go looking for them.
So, let’s say the Launcher starts up and extracts the files into the user’s home folder. What’s Deadline’s PYTHONPATH environment? How is that all controlled?
We’re using Python .net under the hood, and we’re not relying on an environment for Deadline’s search paths. Why? Mainly because we’ve had issues loading our own libraries like our UI toolkit in DCCs which used a different version of PyQt. The workaround was to load up Deadline’s libraries first, then make sure to tack on PYTHONHOME to the end. We have this as our search path:
- Look for libraries that live in the user’s extracted PythonSync
- If the library doesn’t live there load it from the Deadline installation folder’s python folders
- Libraries that are defined in the Repostory’s Python Settings
The Python settings option is the recommended and supported place to put any additional libraries you need to make use of in Deadline.
How to Upgrade Libraries
There is a risk in upgrading libraries that are shipping with Deadline. If an incompaible version is used, the Deadline applications may not start and this could lead to a farm which is unable to render. It’s best to have a small test farm that’s isolated from production if you plan on making changes. Also, do not mix and match Python 2 and 3 libraries. We’ll only be working with Python 3 on a Mac in this example.
Making changes involves extracting the pythonsync.zip or pythonsync3.zip from the Repository to a temporary location, using pip3 install -t
to download a new version of the library to a new folder, then zipping the libraries back up. As an example, let’s update Deadline’s Boto3 library. Here’s the script that will prove we’re on a new version:
$ cat /tmp/boto_version.py
import boto3
def main():
print("Boto3 version {}".format(boto3.version))
This prints the following:
$ ./deadlinecommand executescriptnogui /tmp/boto_version.py
Boto3 version 1.17.112
Now let’s make a backup and extract our zip bundle:
$ cp /Applications/Thinkbox/DeadlineRepository10/pythonsync3/pythonsync3.zip /Applications/Thinkbox/DeadlineRepository10/pythonsync3/pythonsync3-backip.zip
$ mkdir /tmp/pythonsync3
$ unzip /Applications/Thinkbox/DeadlineRepository10/pythonsync3/pythonsync3.zip -d /tmp/pythonsync3/
Aft that we can update or install new libraries:
$ cd /tmp/pythonsync3
$ pip3 install --upgrade -t . boto3
$ rm -r *.dist-info
Finally, we can update the sync file. Here we want to create a new zip file instead of freshen the existing one so that Deadline will actulally sync the file:
$ zip -r9 /tmp/pythonsync3.zip *
$ mv /tmp/pythonsync3.zip /Applications/Thinkbox/DeadlineRepository10/pythonsync3/pythonsync3.zip
If we now run our script, we’ll see there’s been no change. Deadline applications on farm including workstations may have started picking up the new file but as a reminder, DeadlineCommand needs to be asked to update libraries:
$ ./deadlinecommand executescriptnogui /tmp/boto_version.py
Boto3 version 1.17.112
$ ./deadlinecommand PopulatePythonSync /Applications/Thinkbox/DeadlineRepository10/pythonsync3/pythonsync3.zip
$ ./deadlinecommand executescriptnogui /tmp/boto_version.py
Boto3 version 1.26.79
If DeadlineCommand complains that it did not update PythonSync, you should try creating a new zip file.
Comments
0 comments
Article is closed for comments.