Update: (problems still persist...)
I've removed my inputs.conf entry in my universal forwarders /opt/splunkforwarder/etc/system/local/inputs.conf and moved them into the TA's directory in /opt/splunkforwarder/etc/apps/Splunk_TA_paloalto/local/inputs.conf
My apps inputs.conf contains the same entries that I had under Splunks ~etc/system/local/inputs.conf:
[udp://5514]
connection_host = ip
sourcetype = pan:log
index = pan_logs
no_appending_timestamp = true
pan:log is thus far the ONLY sourcetype being created, although at least some of my raw logs show signs that they should have gotten tagged as sourcetype pan:threat, see sample below
Jan 20 11:02:31 PALO-ALTO1.somedomain.local 1,2016/01/20 11:02:31,007801001089,THREAT,url,0,2016/01/20 11:02:31,192.168.2.100,74.125.226.186,0.0.0.0,0.0.0.0,Guest Outbound,,,google-base,vsys1,Guest Trusted,Guest Untrusted,ethernet1/10,ethernet1/9,PAN Log Forwarding,2016/01/20 11:02:31,263889,1,55319,80,0,0,0xc000,tcp,alert,"googleads.g.doubleclick.net/pagead/gen_204?id=wfocus&gqid=la-fVtmbFYKLFGL_s-AD&qqid=CNXIw-fhuMoCFdEsTFodTv0Dlg&bglotd=1",(9999),web-advertisements,informational,client-to-server,938841822,0x0,192.168.0.0-192.168.255.255,US,0,text/html,0,,,5,,,,,,,,0
I've confirmed that I am indeed receiving traffic (tcpdump), and verified data sent to Indexers, and finally, verified that the Indexers are also receiving this traffic (also via tcpdump).
My /opt/splunk/var/log/splunk/paloalto_ta_installer.log on my indexer contains the following:
2016-01-15 13:18:51,507 [INFO] Splunk App for Palo Alto Networks Dependency Manager: Exiting...
2016-01-19 17:46:25,073 [INFO] Splunk App for Palo Alto Networks Dependency Manager: Starting...
2016-01-19 17:46:25,130 [INFO] dependency Splunk_TA_paloalto not found - installing...
2016-01-19 17:46:25,130 [ERROR] unable to copy /opt/splunk/etc/apps/SplunkforPaloAltoNetworks/install/Splunk_TA_paloalto to /opt/splunk/etc/appsSplunk_TA_paloalto
2016-01-19 17:46:25,130 [ERROR] cannot copy tree '/opt/splunk/etc/apps/SplunkforPaloAltoNetworks/install/Splunk_TA_paloalto': not a directory
Traceback (most recent call last):
File "/opt/splunk/etc/slave-apps/SplunkforPaloAltoNetworks/bin/scripted_inputs/deploy_splunk_ta_paloalto.py", line 40, in install_dependency
dir_util.copy_tree(src, dst)
File "/opt/splunk/lib/python2.7/distutils/dir_util.py", line 128, in copy_tree
"cannot copy tree '%s': not a directory" % src
DistutilsFileError: cannot copy tree '/opt/splunk/etc/apps/SplunkforPaloAltoNetworks/install/Splunk_TA_paloalto': not a directory
2016-01-19 17:46:25,137 [INFO] Splunk App for Palo Alto Networks Dependency Manager: Exiting...
My indexer DOES NOT contain an /opt/splunk/etc/slave-apps/Splunk_TA_paloalto directory, however my search heads do contain /opt/splunk/etc/apps/Splunk_TA_paloalto; perhaps not installed correctly, or at all?
Thank you
... View more