Need more detailed output

[yaroslav-nakonechnikov]

Hello,
yesterday faced an issue:

TASK [splunk_deployer : Check app contents] ************************************
ok: [localhost]
Wednesday 14 February 2024  07:24:21 +0000 (0:00:00.377)       0:02:14.443 ****
Wednesday 14 February 2024  07:24:21 +0000 (0:00:00.027)       0:02:14.471 ****
FAILED - RETRYING: Install app via REST (5 retries left).
FAILED - RETRYING: Install app via REST (4 retries left).
FAILED - RETRYING: Install app via REST (3 retries left).
FAILED - RETRYING: Install app via REST (2 retries left).
FAILED - RETRYING: Install app via REST (1 retries left).

TASK [splunk_deployer : Install app via REST] **********************************
fatal: [localhost]: FAILED! => {
    "attempts": 5,
    "cache_control": "no-store, no-cache, must-revalidate, max-age=0",
    "changed": false,
    "connection": "Close",
    "content_length": "378",
    "content_type": "text/xml; charset=UTF-8",
    "date": "Wed, 14 Feb 2024 07:24:55 GMT",
    "elapsed": 0,
    "expires": "Thu, 26 Oct 1978 00:00:00 GMT",
    "redirected": false,
    "server": "Splunkd",
    "status": 400,
    "url": "https://127.0.0.1:8089/services/apps/local",
    "vary": "Cookie, Authorization",
    "warnings": [
        "Module did not set no_log for password"
    ],
    "x_content_type_options": "nosniff",
    "x_frame_options": "SAMEORIGIN"
}

MSG:

Status code was 400 and not [200, 201]: HTTP Error 400: Bad Request

PLAY RECAP *********************************************************************
localhost                  : ok=158  changed=13   unreachable=0    failed=1    skipped=66   rescued=0    ignored=0

Wednesday 14 February 2024  07:24:55 +0000 (0:00:33.988)       0:02:48.459 ****
===============================================================================
splunk_common : Restart the splunkd service - Via CLI ------------------ 58.69s
splunk_deployer : Install app via REST --------------------------------- 33.99s
splunk_common : Start Splunk via CLI ----------------------------------- 17.11s
splunk_common : Get Splunk status --------------------------------------- 9.21s
splunk_common : Set options in roleMap_SAML ----------------------------- 5.77s
splunk_common : Set options in saml ------------------------------------- 5.20s
splunk_common : Set options in RBI_SIR_Incident_Integration ------------- 1.85s
splunk_common : Set options in RBI_SIR_Reporting_Integration ------------ 1.84s
splunk_common : Set options in role_admin ------------------------------- 1.24s
Gathering Facts --------------------------------------------------------- 1.11s
splunk_common : Set node as license slave ------------------------------- 1.09s
splunk_common : Test basic https endpoint ------------------------------- 0.76s
splunk_deployer : Download remote app ----------------------------------- 0.75s
splunk_deployer : Wait for SHC to be ready ------------------------------ 0.64s
splunk_common : Check Splunk instance is running ------------------------ 0.64s
splunk_common : Setup indexer discovery for index-clustering ------------ 0.64s
splunk_common : Set options in authentication --------------------------- 0.63s
splunk_common : Set options in settings --------------------------------- 0.63s
splunk_common : Setup default tcpout group for index-clustering --------- 0.63s
splunk_deployer : Set deployer SHC key and label ------------------------ 0.62s

and the missing part - there is no name of application which it tries to install.

would be really good to see name of app in the title, for example.

[yaroslav-nakonechnikov]

it happens not only on searh-head deployer, but also in standalone instance

[yaroslav-nakonechnikov]

raised ticket in splunk.com: https://splunk.my.site.com/customer/s/case/5005a00002pmoSxAAI/deployer-fails-to-start-due-to-unknown-issue

[yaroslav-nakonechnikov]

os, with checking splunkd.log it was possible to identify that package.

but now it stuck on:
TASK [splunk_deployer : Copy installed apps to /opt/splunk/etc/shcluster/apps] ***
changed: [localhost] => (item=Splunk_TA_snow)
changed: [localhost] => (item=Splunk_Security_Essentials)
changed: [localhost] => (item=e_deployer_base)
changed: [localhost] => (item=hwf_rcdc_resource_app)
changed: [localhost] => (item=github_app_for_splunk)
changed: [localhost] => (item=lookup_editor)
changed: [localhost] => (item=Splunk_SA_Scientific_Python_linux_x86_64)
changed: [localhost] => (item=Splunk_ML_Toolkit)
changed: [localhost] => (item=DA-ESS-ContentUpdate)
changed: [localhost] => (item=phantom)
changed: [localhost] => (item=splunk_datasets_addon)
changed: [localhost] => (item=config_explorer)
changed: [localhost] => (item=splunk_app_soar)
changed: [localhost] => (item=e_deployer_deploymentmode)
changed: [localhost] => (item=splunk-rolling-upgrade)
changed: [localhost] => (item=at_rbi_search_base)
changed: [localhost] => (item=Splunk-UBA-SA-Kafka)
changed: [localhost] => (item=SplunkEnterpriseSecuritySuite)
changed: [localhost] => (item=at_rbi_forwarderinfo)
changed: [localhost] => (item=at_rbi_resmonitor)
Tuesday 27 February 2024 12:00:36 +0000 (0:00:11.730) 0:06:23.765 ******
FAILED - RETRYING: Apply shcluster bundle (60 retries left).
FAILED - RETRYING: Apply shcluster bundle (59 retries left).
FAILED - RETRYING: Apply shcluster bundle (58 retries left).
FAILED - RETRYING: Apply shcluster bundle (57 retries left).
FAILED - RETRYING: Apply shcluster bundle (56 retries left).
FAILED - RETRYING: Apply shcluster bundle (55 retries left).
FAILED - RETRYING: Apply shcluster bundle (54 retries left).
FAILED - RETRYING: Apply shcluster bundle (53 retries left).
FAILED - RETRYING: Apply shcluster bundle (52 retries left).
FAILED - RETRYING: Apply shcluster bundle (51 retries left).
FAILED - RETRYING: Apply shcluster bundle (50 retries left).
FAILED - RETRYING: Apply shcluster bundle (49 retries left).
FAILED - RETRYING: Apply shcluster bundle (48 retries left).
FAILED - RETRYING: Apply shcluster bundle (47 retries left).
FAILED - RETRYING: Apply shcluster bundle (46 retries left).
FAILED - RETRYING: Apply shcluster bundle (45 retries left).
FAILED - RETRYING: Apply shcluster bundle (44 retries left).
FAILED - RETRYING: Apply shcluster bundle (43 retries left).
FAILED - RETRYING: Apply shcluster bundle (42 retries left).

[yaroslav-nakonechnikov]

and another example of bad output:

TASK [splunk_deployer : Wait for SHC to be ready] ******************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: Exception: SHC failure, setup notcomplete. online_peers:['05BB34E3-9B8F-4916-A60A-493D4534047F', 'B7CE74CB-138D-4E4F-9A6C-B4DB791C155D']
fatal: [localhost]: FAILED! => {
    "attempts": 60,
    "changed": false,
    "rc": 1
}
 
MSG:
 
MODULE FAILURE
See stdout/stderr for the exact error
 
 
MODULE_STDERR:
 
Traceback (most recent call last):
  File "/home/splunk/.ansible/tmp/ansible-tmp-1709656969.8714278-4953-235691734405253/AnsiballZ_shc_ready.py", line 100, in <module>
    _ansiballz_main()
  File "/home/splunk/.ansible/tmp/ansible-tmp-1709656969.8714278-4953-235691734405253/AnsiballZ_shc_ready.py", line 92, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/home/splunk/.ansible/tmp/ansible-tmp-1709656969.8714278-4953-235691734405253/AnsiballZ_shc_ready.py", line 41, in invoke_module
    run_name='__main__', alter_sys=True)
  File "/usr/lib/python3.7/runpy.py", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_shc_ready_payload_nh5z9sh5/ansible_shc_ready_payload.zip/ansible/modules/shc_ready.py", line 55, in <module>
  File "/tmp/ansible_shc_ready_payload_nh5z9sh5/ansible_shc_ready_payload.zip/ansible/modules/shc_ready.py", line 50, in main
  File "/tmp/ansible_shc_ready_payload_nh5z9sh5/ansible_shc_ready_payload.zip/ansible/modules/shc_ready.py", line 37, in run
Exception: SHC failure, setup not complete. online_peers:['05BB34E3-9B8F-4916-A60A-493D4534047F', 'B7CE74CB-138D-4E4F-9A6C-B4DB791C155D']
 
 
PLAY RECAP *********************************************************************
localhost                  : ok=137  changed=20   unreachable=0    failed=1    skipped=64   rescued=0    ignored=0

and in that case in splunkd.log there are no expected issues. Only deep investigation helps to solve it