All Apps and Add-ons

Splunk Add-on for VMware: Fetching inventory successfully but failing to authenticate for data pull

Path Finder

Hi All,

Facing an issue. Just got Splunk Add-on for VMware installed on my dev environment and not able to fetch data. What happens:
1. Plugged in Splunk user - all validated, green checkmark.
2. Plugged in vCenter hostname, username+password - all validated, green checkmark and fetched the inventory (all included ESXi servers).

When I enable data pulling, nothing happens. Checking internal logs, found this message:

2019-10-19 17:28:51,087 ERROR [ta_vmware_collection_worker://alpha:25993] Problem with hydra worker ta_vmware_collection_worker://alpha:25993: ('Could not login to target=%s with username=%s after num_retries=%s', 'vcenter.host', 'domain\\vcenter_user', 3)
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_worker.py", line 618, in run
    self.assignJobToHandler(cur_job)
  File "/opt/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_worker.py", line 452, in assignJobToHandler
    session = self.getSessionForTarget(job_tuple.target, config["username"], config["realm"])
  File "/opt/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_worker.py", line 420, in getSessionForTarget
    session_stanza = self.updateSessionStanza(session_stanza, target, username, realm)
  File "/opt/splunk/etc/apps/SA-Hydra/bin/hydra/hydra_worker.py", line 349, in updateSessionStanza
    session = self.loginToTarget(target, username, password, realm)
  File "/opt/splunk/etc/apps/Splunk_TA_vmware/bin/ta_vmware_collection_worker.py", line 55, in loginToTarget
    raise Exception("Could not login to target=%s with username=%s after num_retries=%s", target, user, retry_count)
Exception: ('Could not login to target=%s with username=%s after num_retries=%s','vcenter.host', 'domain\\vcenter_user', 3)

Of course, vcenter.host and domain\vcenter.user are correct values. Dev instance is a single instance so avoiding any DCN, time sync, etc. issues. The weirdest thing is, how comes it gets the inventory but then fails to login. On the same host, netapp works perfectly. Any help/hints appreciated. Thank you.

Splunk: v7.1.4
Splunk Add-on for VMware: v3.4.5
vCenter: v6.5

Update 20-10-2019.
Found in the documentation that user shall be used in format of user@domain, changed that. Behaviour is still the same.

0 Karma
1 Solution

Path Finder

Went into every single log file, went through all python scripts. Found a dependency to read "/tmp/suds/version" file/folder. Splunk runs as splunk user, that folder was owned by other user from who knows when thus it was failing. As soon as I have allowed read permission, all suddenly started to work. Just 3 days of effort down the drain 🙂 anyway, works now! Hope it helps someone else 🙂

View solution in original post

0 Karma

Path Finder

Went into every single log file, went through all python scripts. Found a dependency to read "/tmp/suds/version" file/folder. Splunk runs as splunk user, that folder was owned by other user from who knows when thus it was failing. As soon as I have allowed read permission, all suddenly started to work. Just 3 days of effort down the drain 🙂 anyway, works now! Hope it helps someone else 🙂

View solution in original post

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!