AWS Portal Link status disconnected (red) in Deadline Monitor

You will see the status of AWS Portal become red in color in the bottom right of Deadline Monitor. This status is a result of the machine running AWS Portal no being able to connect to the Gateway instance on AWS through an SSH session using AWS Portal Link. The errors in AWS Portal Link logs will look similar to:

1586957341.633785 2020-04-15 09:29:01,633 [root] [1460] [MainThread] [ERROR] [StatusListener] Unexpected exception: Traceback (most recent call last):
  File "/opt/Thinkbox/AWSPortalLink/awsportallinklib/status_listener.py", line 166, in start
    server = server_class((ipaddress, self._listening_port), self._handler)
  File "/opt/Thinkbox/Deadline10/lib/python2.7/SocketServer.py", line 417, in __init__
    self.server_bind()
  File "/opt/Thinkbox/Deadline10/lib/python2.7/SocketServer.py", line 431, in server_bind
    self.socket.bind(self.server_address)
  File "/opt/Thinkbox/Deadline10/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 98] Address already in use

1586957341.634004 2020-04-15 09:29:01,634 [root] [1460] [MainThread] [INFO] [SSHTunnelManager] Has been told to start.
1586958003.795380 2020-04-15 09:40:03,795 [root] [1460] [MainThread] [ERROR] [SSHTunnelManager] Access denied connecting to remote host 18.219.212.153. Please see our troubleshooting docs, or contact support, for assistance.

There can be several reasons for the connection failure between AWS Portal and the Gateway instance. Two common reasons are incorrect permissions set on the Dashkey, or a corrupted or invalid Dashkey.

Solutions are listed as follows:

Dashkey Permissions Issue:

We set the permissions in C:\Program Files (x86)\Thinkbox\AWSPortalLink\ssh_keygen.bat script. To troubleshoot, run this batch file manually to see if there are any errors.

Windows:

1. Go to C:\Program Files (x86)\Thinkbox\AWSPortalLink
2. Right click on DashKey → Properties → Security
3. Click Advanced, and check the Permissions tab.
4. Ensure only one user has access to the key.

If you see "ALL APPLICATION PACKAGES" or "APPLICATION PACKAGE AUTHORITY" you'll need to move the Dashkey from the Program Files directory into a user owned folder (Desktop, My Documents, etc). Then in the C:\Program Files (x86)\Thinkbox\AWSPortalLink\awsportallink.py file replace this line:

private_key: str = os.path.join(INSTALL_LOCATION, 'DashKey')

with

private_key = "C:/Users/<Your username goes here>/Desktop/Dashkey"

Then restart the service and re-create the infrastructure if one was running.

Linux:

1. Go to /opt/Thinkbox/AWSPortalLink/
2. Run ls -l and make sure that the file has 600 as the permission set. If it does not, see the above linked steps.

Corrupted Dashkey Issue:

After correcting file permissions, if the SSH still fails, it means the DashKey has become corrupted or otherwise made invalid. You will need to create a new one. Follow below to create a new DashKey.

Windows:

1. Open Administrative Command Prompt
2. "C:\Program Files (x86)\Thinkbox\AWSPortalLink\ssh_keygen.bat"
3. "%DEADLINE_PATH%\deadlinecommand.exe" -ExecuteScriptNoGui "C:\Program Files (x86)\Thinkbox\AWSPortalLink\SaveFingerPrint.py" "C:\Program Files (x86)\Thinkbox\AWSPortalLink\DashKey.pub"

Linux:

1. sudo "/opt/Thinkbox/AWSPortalLink/ssh_keygen.sh"
2. sudo "$DEADLINE_PATH/deadlinecommand.exe" -ExecuteScriptNoGui "/opt/Thinkbox/AWSPortalLink/SaveFingerPrint.py" "/opt/Thinkbox/AWSPortalLink/DashKey.pub"

After regenerating the DashKey a new infrastructure needs to be created for the change to take effect