My home lab setup involves a handful of VMs sending data to an all-in-one instance of splunk. I had initially started with configuring things via the GUI, however I'm beginning to try writing TAs from scratch, and I'm thoroughly stumped by this problem. I have several forwarders which I've assigned TAs to, whose only content is an inputs.conf defining a file to monitor (squid access.log, bind query.log, selinux audit.log).
The content of splunkd.log and metrics.log indicate that the files are being monitored; the output of bin/splunk inputstatus shows the file position marker is updating - but no data for these files ever arrives at the indexer (confirmed via tcpdump, so it's not that it's arriving but not being indexed for some reason). Here is an example of one input:
[monitor:///var/log/squid/access.log] sourcetype = squid:access index = squid disabled = false
I have not made any changes to whitelists/blacklists, that's all default. What am I missing?
Does uForwarder have read access to those files?
Does you have a outputs.conf file located under /local/ or is it under default/?
Yes, uForwarder has read permissions (it's running as root currently), and outputs.conf is delivered as another app. Pretty sure it's functional though, otherwise how would the metrics/splunkd.log etc be getting sent?
~# /opt/splunkforwarder/bin/splunk list forward-server
Configured but inactive forwards:
Let's start simple:
Do you have the index set up on the indexer?
Do you see any alerts regarding getting data for a non-configured index?
If you run a real time all time search for index=squid, do you see events coming in with the correct timestamps being extracted?
I would also suggest checking your "all-in-one" splunk instance to verify that the index "proxy" is set up and ENABLED.
I've had similar issues before where the index was disabled on the indexer.
I've also had issues when enabling the inputs on the forwarder before creating the index. Creating and enabling the index (proxy in your case) should always be step 1
Other than that i would also check if the sourcetype (squid:access) is defined on the "all-in-one" splunk instance and create a sourcetype with that name if it does not exist.
I also assume you have tried restarting the splunk service on the forwarders after you assigned the TA's to them?