Cyber Deals - Get up to 65% off on CKA, CKAD, CKS, KCNA, KCSA exams and courses!

Troubleshooting Ansible Automation Platform

Automation Dashboard Manual Sync

To manually force an immediate database rollup and update your unified Automation Dashboard in Ansible Automation Platform 2.7 without waiting for the default 6-hourly cron schedule, you must use the containerized command-line utility to insert a new pending task into the metrics database. Because early builds of the platform enforce a strict NOT NULL constraint on the table’s description column, you must explicitly include the --description parameter to prevent PostgreSQL database transaction rejection.

Run the following command on your All-in-One host to instantly queue the aggregation pipeline:

podman exec automation-metrics-web metrics-service tasks create \
  --name "manual-rollup" \
  --description "Manual trigger to update dashboard data" \
  --function daily_metrics_rollup

Where,

  • podman exec automation-metrics-web metrics-service tasks: Invokes the dynamic task management executable inside your active metrics web container.
  • --name "manual-rollup": Assigns a unique name to identify this manual run inside your task history logs.
  • --description "...": Satisfies the database constraint by providing a non-null string, bypassing the early-build schema validation bug.
  • --function daily_metrics_rollup: Specifies the targeted backend function that rolls up all raw hourly micro-batches and populates the dashboard metrics tables.

Error with Self Signed SSL Cert on SCM server

Error : Peer’s certificate issuer has been marked as not trusted by the user

{
    "stderr_lines": [
        "fatal: unable to access 'https://$encrypted$:[email protected]/ansible/network-automation.git/': Peer's certificate issuer has been marked as not trusted by the user."
    ],
    "cmd": "/usr/bin/git clone --origin origin 'https://$encrypted$:[email protected]/ansible/network-automation.git' /var/lib/awx/projects/_8__network_poc",
    "_ansible_no_log": false,
    "stdout": "Cloning into '/var/lib/awx/projects/_8__network_poc'...\n",
    "changed": false,
    "invocation": {
        "module_args": {
            "force": false,
            "track_submodules": false,
            "reference": null,
            "dest": "/var/lib/awx/projects/_8__network_poc",
            "umask": null,
            "clone": true,
            "gpg_whitelist": [],
            "accept_hostkey": false,
            "update": true,
            "ssh_opts": null,
            "repo": "https://$encrypted$:[email protected]/ansible/network-automation.git",
            "bare": false,
            "archive": null,
            "refspec": null,
            "executable": null,
            "remote": "origin",
            "recursive": true,
            "separate_git_dir": null,
            "verify_commit": false,
            "depth": null,
            "version": "HEAD",
            "key_file": null
        }
    },
    "stderr": "fatal: unable to access 'https://$encrypted$:[email protected]/ansible/network-automation.git/': Peer's certificate issuer has been marked as not trusted by the user.\n",
    "rc": 128,
    "_ansible_delegated_vars": {
        "ansible_host": "localhost"
    },
    "stdout_lines": [
        "Cloning into '/var/lib/awx/projects/_8__network_poc'..."
    ],
    "msg": "fatal: unable to access 'https://$encrypted$:[email protected]/ansible/network-automation.git/': Peer's certificate issuer has been marked as not trusted by the user."
}

Solution Resolution SSL certificate validation can be prevented for Git connections originating from Tower by adding the following settings in the Tower UI at Settings » Configure Tower » Jobs in JSON format:

{
  "GIT_SSL_NO_VERIFY": "True"
}

Reference

Note : For cloning repo, use below methods

git config --global http.sslverify false
#or
export GIT_SSL_NO_VERIFY=true

Error:module ’enum’ has no attribute ‘IntFlag’

{
    "exception": "Traceback (most recent call last):\n  File \"/var/lib/awx/.ansible/tmp/ansible-tmp-1599103337.21-17-81526385316505/AnsiballZ_fortios_facts.py\", line 102, in <module>\n    _ansiballz_main()\n  File \"/var/lib/awx/.ansible/tmp/ansible-tmp-1599103337.21-17-81526385316505/AnsiballZ_fortios_facts.py\", line 17, in _ansiballz_main\n    import base64\n  File \"/usr/lib64/python3.6/base64.py\", line 9, in <module>\n    import re\n  File \"/usr/lib64/python3.6/re.py\", line 142, in <module>\n    class RegexFlag(enum.IntFlag):\nAttributeError: module 'enum' has no attribute 'IntFlag'\n",
    "_ansible_no_log": false,
    "_ansible_delegated_vars": {
        "ansible_host": "localhost"
    },
    "module_stderr": "Traceback (most recent call last):\n  File \"/var/lib/awx/.ansible/tmp/ansible-tmp-1599103337.21-17-81526385316505/AnsiballZ_fortios_facts.py\", line 102, in <module>\n    _ansiballz_main()\n  File \"/var/lib/awx/.ansible/tmp/ansible-tmp-1599103337.21-17-81526385316505/AnsiballZ_fortios_facts.py\", line 17, in _ansiballz_main\n    import base64\n  File \"/usr/lib64/python3.6/base64.py\", line 9, in <module>\n    import re\n  File \"/usr/lib64/python3.6/re.py\", line 142, in <module>\n    class RegexFlag(enum.IntFlag):\nAttributeError: module 'enum' has no attribute 'IntFlag'\n",
    "changed": false,
    "module_stdout": "",
    "rc": 1,
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error"
}

Solution

https://access.redhat.com/solutions/4282031

Error with nosuid when read/write files

sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges?

Solution

It isn’t possible to use Tower with local action to escalate to the root user. It will be necessary to alter your task to connect via SSH and then escalate to root using another user(not AWX). This is done purposefully to avoid security risks associated with our user having root level access to the system.

NOTE : It is not recommended that sudo access be given to AWX user. You’ll need to adjust your playbook to SSH rather than use a local connection.

Root Cause: The AWX service user is intentionally restricted from sudo operations.

https://access.redhat.com/solutions/3223501

Checking container logs

$ journalctl CONTAINER_NAME=<container_name>
$ journalctl CONTAINER_NAME=<container_name> -n 100 --no-pager -f

# or check other podman logs if any
$ podman logs -f automation-gateway

true | openssl s_client -connect awesome.corp.root:636 -showcerts -CAfile awesomecorp_bundle.pem

openssl x509 -in <filename.cert> -noout -subject -issuer

bundle cert cat rootca intermediate | tee bundle.pem

How to enable DEBUG mode for AAP Components

E.g. Enable DEBUB for authentication containers - in ~/aap/gateway/etc/settings.py

.
.
LOGGING['loggers']['ansible_base.authentication'] = {'level': 'DEBUG'}