Version: Deadline 9.0 and later
INTRODUCTION
Today, I will be demonstrating how to run "cron job" commands in Deadline. The term "cron job" comes from a software utility called Cron that allows users to run some sort of command or job on a schedule. We're going to go over two ways to implement this style of command in Deadline: Scheduled Jobs and Event Plugins.
SCHEDULED JOBS
The first method of running a cron job is by creating a Scheduled Job. Scheduled Jobs are jobs that are in the Pending state and then released at a scheduled time into the Queued state for Deadline Workers to pick up. Scheduled Jobs are great for running more complex applications because you can use any of the existing Deadline plugins to make one. Scheduled Jobs offer a lot of flexibility because you can have them release on a repeating schedule, or on certain days of the week at specific times. It's a great way to allow jobs to only run over night or on the weekend.
To create a Scheduled Job, simply submit the job as normal through the Monitor or an Integrated Submitter. Make sure to submit the job in the Suspended state, otherwise the job may run immediately. In the job list, right click on the job and select Modify Job Properties, then select Scheduling. Here you can choose to make the job repeat on an interval by selecting the 'Repeating' mode and setting the Day Interval. For example, you can set the Day Interval to 2 for the job to repeat every other day. Alternatively, you can choose the 'Custom' scheduling mode to have the job run at a specific time on a certain day of the week (eg. Wednesday mornings between 10am and 11am). Note that when a repeating job completes, it will return to the Pending state so that it can be released when it is scheduled to run again.
Scheduled Jobs do have some limitations. Scheduled Jobs run on your render nodes, which means that they might not get run immediately if your farm is saturated by higher priority jobs. Also, a Scheduled Job will run on any render node that can pick it up, so you aren't guaranteed to have the job run on the same machine. However, you can get around this with Pools and Groups. Additionally, you can make Maintenance Jobs into Scheduled Jobs. This allows you to schedule a job that runs on all of your render nodes.
EVENT PLUGINS
The second method of running a cron job is to create an Event Plugin. Event Plugins "listen" for certain Deadline events, and execute Python code when those events trigger. We want to hook into an event that happens on a regular basis, so we will choose the OnHouseCleaning event. Unlike Scheduled Jobs, Event Plugins will always run immediately when triggered. When House Cleaning occurs, our Event Plugin will be executed as part of the House Cleaning operation.
Note that if you have Deadline Pulse running in your farm, it will perform House Cleaning consistently at regular intervals. If you are not running Pulse, then House Cleaning is performed by the Workers periodically between tasks. In the latter case, House Cleaning typically won't occur at regular intervals, especially if your farm is busy. Therefore, if you need your OnHouseCleaning event to trigger at regular intervals, we highly recommending running Pulse.
As an example, we're going to write an Event Plugin that changes the priority of the Pools our Workers are in based on the time of day. Jobs in the 'OvernightTeam1' Pool will have a higher priority during the night time than Jobs in the 'Team1' Pool, and vice versa during the day.
The first thing we need to do is create a 'MyEvent' folder in the custom\events folder of our Deadline Repository. Next, we need to create a 'MyEvent.param' file in the 'MyEvent' folder, open it in a text editor, and add the following contents:
[State] Type=Enum Items=Global Enabled;Opt-In;Disabled Category=Options CategoryOrder=0 Index=0 Label=State Default=Disabled Description=How this event plug-in should respond to events. If Global, all jobs and Workers will trigger the events for this plugin. If Opt-In, jobs and Workers can choose to trigger the events for this plugin. If Disabled, no events are triggered for this plugin.
This file is used to expose configurable Event Plugin options to Deadline, but for our example, we just need to expose the option to enable or disable it. Next, create a 'MyEvent.py' file in MyEvent folder and open it in a text editor. We need to setup the basic DeadlineEventListener framework, so let's do that now. Your file should look like this:
from Deadline.Events import * from Deadline.Scripting import * import datetime import time ###################################################################### ## This is the function that Deadline calls to get an instance of the ## main DeadlineEventListener class. ###################################################################### def GetDeadlineEventListener(): return MyEvent() ###################################################################### ## This is the function that Deadline calls when the event plugin is ## no longer in use so that it can get cleaned up. ###################################################################### def CleanupDeadlineEventListener( deadlinePlugin ): deadlinePlugin.Cleanup() ###################################################################### ## This is the main DeadlineEventListener class for MyEvent. ###################################################################### class MyEvent (DeadlineEventListener): # TODO: Place code here to replace "pass" pass
Now we need to make our plugin listen on the correct event. To do that, we need to add the OnHouseCleaningCallback to the in the init function. Make sure to delete it in the Cleanup function too.
from Deadline.Events import * from Deadline.Scripting import * import datetime import time ###################################################################### ## This is the function that Deadline calls to get an instance of the ## main DeadlineEventListener class. ###################################################################### def GetDeadlineEventListener(): return MyEvent() ###################################################################### ## This is the function that Deadline calls when the event plugin is ## no longer in use so that it can get cleaned up. ###################################################################### def CleanupDeadlineEventListener( deadlinePlugin ): deadlinePlugin.Cleanup() ###################################################################### ## This is the main DeadlineEventListener class for MyEvent. ###################################################################### class MyEvent (DeadlineEventListener): def __init__( self ): # Set up the event callbacks here self.OnHouseCleaningCallback += self.OnHouseCleaning def Cleanup( self ): del self.OnHouseCleaningCallback def OnHouseCleaning( self ): pass
Now that the framework is in place we need to write the function. Let's say we want to change the priority of the Pools at 5pm and 9am. First let's check the time and see if it falls in that overnight period. Based on that, we can now loop through all the Workers and set their Pools to the appropriate setting.
An astute reader will have seen that this plugin is rather inefficient. Why? Because we set the Pools every time the House Cleaning event runs, even after we've already set them. However, there is a new feature in Deadline 9 that can help us here. In Deadline 9, Event Plugins can now have metadata associated with them. This new metadata dictionary contains key/value pairs that persist between runs of the same event. We can save a flag to show that we've set the Pools for the daytime/overnight period already.
def OnHouseCleaning( self ):
mode = self.GetMetaDataEntry( "Mode" )
if mode == None:
self.AddMetaDataEntry("Mode", "")
#default to overnight
pools = ["OvernightTeam1", "Team1"]
#if it's between 9am and 5pm change to the regular pools
if datetime.time(hour=9) <= datetime.datetime.now().time() <= datetime.time(hour=17):
if mode != "Daytime":
pools = ["Team1", "OvernightTeam1"]
self.UpdateMetaDataEntry("Mode", "Daytime")
else:
if mode != "Overnight":
self.UpdateMetaDataEntry("Mode", "Overnight")
After saving the file, the last thing we need to do is enable your new Event Plugin in Deadline. This can be done in the Deadline Monitor while in Super User Mode. Select Tools > Configure Events, and then select the MyEvent plugin from the list on the left. Here, you can set the State to Global Enabled, and press OK.
One disadvantage to using the OnHouseCleaning event for cron jobs is that it doesn't provide the complex scheduling options of a Scheduled Job. In addition, events are not re-attempted if an error occurs, but since this event will trigger on a regular interval, it is less of an issue.
CONCLUSION
While we just skimmed the surface of what's possible with Scheduled Jobs and Event Plugins, hopefully this gives you an idea of how these powerful tools can be used to enhance your pipeline. If you're looking for inspiration, check out the SPOT plugin that ships with Deadline 9. It is a good example of an Event Plugin that uses the OnHouseCleaning event, and it also takes advantage of the new metadata dictionary. Another example of a cron job style Event Plugin can be found in this blog post on Zabbix.
Comments
0 comments
Article is closed for comments.