Getting Data In
Highlighted

Why are the forwarders not sending anything other than _internal/metrics data?

New Member

My home lab setup involves a handful of VMs sending data to an all-in-one instance of splunk. I had initially started with configuring things via the GUI, however I'm beginning to try writing TAs from scratch, and I'm thoroughly stumped by this problem. I have several forwarders which I've assigned TAs to, whose only content is an inputs.conf defining a file to monitor (squid access.log, bind query.log, selinux audit.log).

The content of splunkd.log and metrics.log indicate that the files are being monitored; the output of bin/splunk inputstatus shows the file position marker is updating - but no data for these files ever arrives at the indexer (confirmed via tcpdump, so it's not that it's arriving but not being indexed for some reason). Here is an example of one input:

[monitor:///var/log/squid/access.log]
sourcetype = squid:access
index = squid
disabled = false

I have not made any changes to whitelists/blacklists, that's all default. What am I missing?

0 Karma
Highlighted

Re: Why are the forwarders not sending anything other than _internal/metrics data?

SplunkTrust
SplunkTrust
0 Karma
Highlighted

Re: Why are the forwarders not sending anything other than _internal/metrics data?

Path Finder

Does uForwarder have read access to those files?
Does you have a outputs.conf file located under /local/ or is it under default/?

0 Karma
Highlighted

Re: Why are the forwarders not sending anything other than _internal/metrics data?

New Member

Yes, uForwarder has read permissions (it's running as root currently), and outputs.conf is delivered as another app. Pretty sure it's functional though, otherwise how would the metrics/splunkd.log etc be getting sent?

Also:
~# /opt/splunkforwarder/bin/splunk list forward-server
Active forwards:
$splunk_ip:9997
Configured but inactive forwards:
None

0 Karma
Highlighted

Re: Why are the forwarders not sending anything other than _internal/metrics data?

Splunk Employee
Splunk Employee

Let's start simple:
Do you have the index set up on the indexer?
Do you see any alerts regarding getting data for a non-configured index?
If you run a real time all time search for index=squid, do you see events coming in with the correct timestamps being extracted?

0 Karma
Highlighted

Re: Why are the forwarders not sending anything other than _internal/metrics data?

Explorer

I would also suggest checking your "all-in-one" splunk instance to verify that the index "proxy" is set up and ENABLED.

I've had similar issues before where the index was disabled on the indexer.
I've also had issues when enabling the inputs on the forwarder before creating the index. Creating and enabling the index (proxy in your case) should always be step 1

Other than that i would also check if the sourcetype (squid:access) is defined on the "all-in-one" splunk instance and create a sourcetype with that name if it does not exist.

I also assume you have tried restarting the splunk service on the forwarders after you assigned the TA's to them?

0 Karma