(Re)starting Portal Infrastructure or Spot Event Plugin may fail

If using AWS Portal + Spot Fleet or Deadline within Nimble Studio, when deploying a new fleet or a new infrastructure the ResourceTracker will attempt to update (latest version is 11.4.0). If you are experiencing issues starting a Spot Fleet, please continue reading.

When the update fails, if using the Deadline Monitor it will display a popup and the following message can be seen in the console:

2021-12-22 10:56:50: Deleting rolled back Deadline AWS Resource Tracker Stack.

In CloudFormation, if you go find the ResourceTracker stack, you will see the following error:

ResourceTrackerFleetTableCleaner

UPDATE_FAILED              

Resource handler returned message: "User: arn:aws:iam::123469794321:user/AWSPortal is not authorized to perform: lambda:DeleteFunctionConcurrency on resource: arn:aws:lambda:us-east-1:123469794321:function:DeadlineResourceTracker-FleetTableCleaner because no identity-based policy allows the lambda:DeleteFunctionConcurrency action (Service: Lambda, Status Code: 403, Request ID: f0abcde-9243-4a2c-bd78-1234c0ddabcd, Extended Request ID: null)" (RequestToken: f0abcde-9243-4a2c-bd78-1234c0ddabcd, HandlerErrorCode: AccessDenied)

If using Thinkbox Deadline with Portal, simply attempt to create the infrastructure or SpotFleetRequest again and it should succeed. If using Nimble, you will need to create an inline policy and attach it to the user from the error message (in this case "arn:aws:iam::123469794321:user/AWSPortal" user) an example from the Deadline docs is here: (https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User Manual/manual/aws-portal-setup-creating-awsportal-user.html)

The JSON for the policy should be:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "lambda:DeleteFunctionConcurrency"
            ],
            "Resource": [
                "arn:aws:lambda:*:*:function:DeadlineResourceTracker*"
            ]
        }
    ]
}

If you are unsure or need help deploying these, don't hesitate to reach out either with a ticket or a phone call. The support team are available from 9-5 central.