Credential CyberArk AIM with certs Timeout, Non-Cert works

[mgperry1]

ISSUE TYPE

• Bug Report

SUMMARY

Credential CyberArk AIM with certs Timeout, Non-Cert works.
We are able to use CuberArk AIm credentials and pull password, however when we try to add Key Cert to the call the process hangs and times out.

ENVIRONMENT

• AWX version: 9.2
• AWX install method: docker on linux
• Ansible version: 2.9.3
• Operating System: Red-Hat 7.7
• Web Browser: Chrome

STEPS TO REPRODUCE

We have setup safe and App Id in CyberArk to test pulling passwords from a CyberArk safe., The process is working fine with the Test URL(https://[awxhost]/api/v2/credentials/XX/test/) able to pull a test password. However, when l we configure to use a client key and certificate to validate the client connection the process "hangs" and nginx reports a timeout.

EXPECTED RESULTS

Test URL(https://[awxhost]/api/v2/credentials/XX/test/) Responds with with the password for the Query I pass in, this does work if I don't use Key/Cert validation.

ACTUAL RESULTS

Test URL(https://[awxhost]/api/v2/credentials/XX/test/) produces Nginx timeout

ADDITIONAL INFORMATION

I'm not sure where I can pull any logs related to the API URL Test URL(https://[awxhost]/api/v2/credentials/XX/test/). I've tried docker logs on each container and I cant see anything related to the call.

I have test the CyberArk call Certs with a basic Python Program and it works fine so I'm fairly certain that CyberArk is properly configured to respond. Include the Pyhton Code I'm using that works :

#import http.client
import http.client
import json
import ssl

Defining certificate related stuff and host of endpoint

certificate_file = '/loc/cert.pem'
certificate_secret= ''
host = 'cyberarkhost'

Defining parts of the HTTP request

request_url='/AIMWebService/api/Accounts?AppID=APP_GSF_D_Ansible_Test&Safe=GSF_D_Ansible_Test&UserName=Ansible-User'
request_headers = {
'Content-Type': 'application/json'
}
request_body_dict={
"AppID": "APP_GSF_D_Ansible_Test",
"object_query": "Safe=GSF_D_Ansible_Test;UserName=Ansible-User",
"object_query_format": "Exact",
"reason":"sss"
}

Define the client certificate settings for https connection

context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
context.load_cert_chain(certfile=certificate_file, password=certificate_secret)

Create a connection to submit HTTP requests

connection = http.client.HTTPSConnection(host, port=443, context=context)

Use connection to submit a HTTP POST request

connection.request(method="GET", url=request_url, headers=request_headers, body=json.dumps(request_body_dict))

Print the HTTP response from the IOT service endpoint

response = connection.getresponse()
print(response.status, response.reason)
data = response.read()
print(data)

Thanks Matt

[mgperry1]

Bump

[ryanpetrello]

Hey @mgperry1,

This issue tracker is for tracking feature enhancements and bugs to AWX itself.

If you need help troubleshooting an AWX install, try our mailing list or IRC channel:

http://webchat.freenode.net/?channels=ansible-awx
https://groups.google.com/forum/#!forum/awx-project

[mgperry1]

Thanks, so basically your point is I'm wasting time with what I think issue maybe and you assume I'm too stupid to have maybe have found some issue with the code base that uses SSL certs and Cyberark AIM creds.. which I assumed was part of AWX , I was just trying to get this feature working and though I might be one of the few people who are using it and could help troubleshoot , I get it not important

[gael-fuhs2]

Hi @ryanpetrello

We have the same issues that @mgperry1 on our AWX server. The AIM creds works when we use the authentication by certificate on CyberArk from AWX 9.0.0. But after upgrade to upper version AWX to 11.2.0 . We have an error on AWX and the AIM feature don't work anymore.
On the logs of CyberArk, we see an error on the request run by AWX which don't send the certificate for authentification. We test after a new reinstall from scratch of AWX 11.2.0. And we have same issues with AIM feature on AWX :(

In my opinion, it's not a problem due to AWX installation. Therefore, I have this issues after an upgrade or new installation

[ryanpetrello]

Thanks for the report @gael-fuhs2 @mgperry1,

We'll take a look.

[ryanpetrello]

The AIM creds works when we use the authentication by certificate on CyberArk from AWX 9.0.0. But after upgrade to upper version AWX to 11.2.0.

What's odd about this is that the AIM plugin hasn't really changed in a notable way in recent history:

commit b9829e2bde65da6c3c8f3c7144c87341d199527e
Author: Gabe Muniz <gmuniz@redhat.com>
Date:   Tue Mar 10 23:08:38 2020 +0000

    removed extra quotes in example

commit ce5bb9197e8c06d7a0b4e4ab06287740a116c2e4
Author: Ryan Petrello <rpetrell@redhat.com>
Date:   Wed Oct 16 15:58:35 2019 -0400

    rename the CyberArk AIM credential type

    see: https://github.com/ansible/awx/issues/4400

commit b2d84a5d8936f32cbd7699b6aab62579353fc3fb
Author: olia-dev <olia-dev@ktah.net>
Date:   Wed Jul 10 12:39:57 2019 +0200

    related #4274 - moved function 'create_temporary_fifo' to 'awx/main/utils/common.py' and referenced it in other plugins (fixe>

[ryanpetrello]

Given that you all have described that you're encountering a hang, this is the likely path of code you're stuck in:

https://github.com/ansible/awx/blob/devel/awx/main/credential_plugins/aim.py#L93-L98

That said, there isn't much exciting here - we're just writing a cert file and handing it off to the requests library to make an HTTP request.

If you can reliably encounter a hang, it would be helpful if you were able to jump in with an interactive debugger, or something like gdb or strace to figure out where specifically the code is "hanging".

For what it's worth, I do have a CyberArk AIM install with a client cert and key, and it does work for me (it doesn't hang).

[ryanpetrello]

@mgperry1

Is the CyberArk server up and responsive for you from within the awx_task container e.g.,

bash-4.4$ nc -vz your-cyberark.example.org 443
Ncat: Version 7.70 ( https://nmap.org/ncat )
Ncat: Connected to ...:443.
Ncat: 0 bytes sent, 0 bytes received in 0.24 seconds.
bash-4.4$ curl -vk https://your-cyberark.example.org

[mgperry1]

Hi ,
Yes I have actually setup a different Cred to the Same Serve(Cyberark) without a Cert file and I can do the TEST credential and have a password return, so yep its working. I have access to the awx_task container. Using Chrome I was able to to find the API URL call made when I press the TEST cred button, you can copy it to a Bash shell . If I run the Curls in the container I get the Timeout when I try with the cert and It works and gets to the Cyberark server if I dont use the Certs. I could try some debugging, I'm not sure if I can call the credenital with a test from curl as it looks like it passes in the key and cert file as part of the web call. If you have better way to do a test let me know. Thanks

[ryanpetrello]

Okay, I've dug some more, and I've produced something similar that I think is actually a limitation of urllib3 or cpython's ssl module itself. Here are some details:

urllib3/urllib3#1880

cc @jakemcdermott this probably means we can't rely on a fifo here, and we may just have to write these files to tmp space, appropriately owned by the awx user.

[ryanpetrello]

It seems to me that at some point in time, this just worked, but it looks like perhaps something changed in a newer/different version of cpython? Could be related to the move to CentOS8. Either way, it looks like we can't rely on this to work this way, so we'll probably need to remove the usage of named pipes here.

@mgperry1,

Just to confirm, does a diff like this make the issue go away for you?

diff --git a/awx/main/credential_plugins/aim.py b/awx/main/credential_plugins/aim.py
index c75d4d85aa..06b98a91f8 100644
--- a/awx/main/credential_plugins/aim.py
+++ b/awx/main/credential_plugins/aim.py
@@ -1,3 +1,6 @@
+import tempfile
+import os
+
 from .plugin import CredentialPlugin

 from urllib.parse import quote, urlencode, urljoin
@@ -10,6 +13,30 @@ from awx.main.utils import (
     create_temporary_fifo,
 )

+class CertFiles():
+
+    certfile = None
+
+    def __init__(self, cert, key):
+        self.cert = cert
+        self.key = key
+
+    def __enter__(self):
+        if not self.cert:
+            return None
+        self.certfile = tempfile.NamedTemporaryFile('wb')
+        self.certfile.write(self.cert.encode())
+        if self.key:
+            self.certfile.write(b'\n')
+            self.certfile.write(self.key.encode())
+        self.certfile.flush()
+        return str(self.certfile.name)
+
+    def __exit__(self, *args):
+        if os.path.exists(self.certfile.name):
+            os.remove(self.certfile.name)
+
+
 aim_inputs = {
     'fields': [{
         'id': 'url',
@@ -81,21 +108,14 @@ def aim_backend(**kwargs):
     request_qs = '?' + urlencode(query_params, quote_via=quote)
     request_url = urljoin(url, '/'.join(['AIMWebService', 'api', 'Accounts']))

-    cert = None
-    if client_cert and client_key:
-        cert = (
-            create_temporary_fifo(client_cert.encode()),
-            create_temporary_fifo(client_key.encode())
+    with CertFiles(client_cert, client_key) as cert:
+        res = requests.get(
+            request_url + request_qs,
+            timeout=30,
+            cert=cert,
+            verify=verify,
         )
-    elif client_cert:
-        cert = create_temporary_fifo(client_cert.encode())
-
-    res = requests.get(
-        request_url + request_qs,
-        timeout=30,
-        cert=cert,
-        verify=verify,
-    )
     res.raise_for_status()
     return res.json()['Content']

[mgperry1]

@ryanpetrello

Yes I'm a bit of noob so I wasn't sure where to make the change so I updated the aim.py with your code changes int the awx_task and awx_web containers. I then restarted the containers and tried the test and the whole thing works with the cert and key file.

I do see an excpetion in the awx_web container popping up:

/var/lib/awx/venv/awx/lib/python3.6/site-packages/urllib3/connectionpool.py:1004: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning,
Exception ignored in: <bound method _TemporaryFileCloser.__del__ of <tempfile._TemporaryFileCloser object at 0x7fe023b14a58>>
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.6/tempfile.py", line 452, in __del__
    self.close()
  File "/var/lib/awx/venv/awx/lib64/python3.6/tempfile.py", line 448, in close
    unlink(self.name)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpwkfvvam2'

Code Change: /var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/credential_plugins/aim.py

import tempfile
import os
from .plugin import CredentialPlugin

from urllib.parse import quote, urlencode, urljoin

from django.utils.translation import ugettext_lazy as _
import requests

from awx.main.utils import (
    create_temporary_fifo,
)

class CertFiles():

    certfile = None

    def __init__(self, cert, key):
        self.cert = cert
        self.key = key

    def __enter__(self):
        if not self.cert:
            return None
        self.certfile = tempfile.NamedTemporaryFile('wb')
        self.certfile.write(self.cert.encode())
        if self.key:
            self.certfile.write(b'\n')
            self.certfile.write(self.key.encode())
        self.certfile.flush()
        return str(self.certfile.name)

    def __exit__(self, *args):
        if os.path.exists(self.certfile.name):
            os.remove(self.certfile.name)


aim_inputs = {
    'fields': [{
        'id': 'url',
        'label': _('CyberArk AIM URL'),
        'type': 'string',
        'format': 'url',
    }, {
        'id': 'app_id',
        'label': _('Application ID'),
        'type': 'string',
        'secret': True,
    }, {
        'id': 'client_key',
        'label': _('Client Key'),
        'type': 'string',
        'secret': True,
        'multiline': True,
    }, {
        'id': 'client_cert',
        'label': _('Client Certificate'),
        'type': 'string',
        'secret': True,
        'multiline': True,
    }, {
        'id': 'verify',
        'label': _('Verify SSL Certificates'),
        'type': 'boolean',
        'default': True,
    }],
    'metadata': [{
        'id': 'object_query',
        'label': _('Object Query'),
        'type': 'string',
        'help_text': _('Lookup query for the object. Ex: "Safe=TestSafe;Object=testAccountName123"'),
    }, {
        'id': 'object_query_format',
        'label': _('Object Query Format'),
        'type': 'string',
        'default': 'Exact',
        'choices': ['Exact', 'Regexp']
    }, {
        'id': 'reason',
        'label': _('Reason'),
        'type': 'string',
        'help_text': _('Object request reason. This is only needed if it is required by the object\'s policy.')
    }],
    'required': ['url', 'app_id', 'object_query'],
}


def aim_backend(**kwargs):
    url = kwargs['url']
    client_cert = kwargs.get('client_cert', None)
    client_key = kwargs.get('client_key', None)
    verify = kwargs['verify']
    app_id = kwargs['app_id']
    object_query = kwargs['object_query']
    object_query_format = kwargs['object_query_format']
    reason = kwargs.get('reason', None)

    query_params = {
        'AppId': app_id,
        'Query': object_query,
        'QueryFormat': object_query_format,
    }
    if reason:
        query_params['reason'] = reason

    request_qs = '?' + urlencode(query_params, quote_via=quote)
    request_url = urljoin(url, '/'.join(['AIMWebService', 'api', 'Accounts']))

    with CertFiles(client_cert, client_key) as cert:
        res = requests.get(
            request_url + request_qs,
            timeout=30,
            cert=cert,
            verify=verify,
        )

    res.raise_for_status()
    return res.json()['Content']


aim_plugin = CredentialPlugin(
    'CyberArk AIM Central Credential Provider Lookup',
    inputs=aim_inputs,
    backend=aim_backend
)

[ryanpetrello]

Hey @mgperry1 thanks for the feedback - I can clean something up in my diff to address that error, but I don't expect it to break anything (more of a warning).

Were you able to fetch CyberArk AIM secrets using this patch? If so, I'll open a PR.

[ryanpetrello]

I then restarted the containers and tried the test and the whole thing works with the cert and key file.

Ah, I should've read more closely. I'll open a PR. Thanks for reporting this and helping test it.

[ryanpetrello]

Hey @mgperry1,

I've put up a pull request that should address this, and I'm currently testing it out (you're welcome to give it a whirl too, if you have some free time):

#7175

Once this goes in, the fix will be available in the next major version of AWX.

[kdelee]

@one-t this is the same one but for devel

[one-t]

It appears that we have coverage for the scenario already.

The existing tests passed in devel. Is that sufficient to close, @kdelee @jneedle ?

[one-t]

Verified that this has been fixed.