Getting Data In

Why are the forwarders not sending anything other than _internal/metrics data?

jdsl
Loves-to-Learn

My home lab setup involves a handful of VMs sending data to an all-in-one instance of splunk. I had initially started with configuring things via the GUI, however I'm beginning to try writing TAs from scratch, and I'm thoroughly stumped by this problem. I have several forwarders which I've assigned TAs to, whose only content is an inputs.conf defining a file to monitor (squid access.log, bind query.log, selinux audit.log).

The content of splunkd.log and metrics.log indicate that the files are being monitored; the output of bin/splunk inputstatus shows the file position marker is updating - but no data for these files ever arrives at the indexer (confirmed via tcpdump, so it's not that it's arriving but not being indexed for some reason). Here is an example of one input:

[monitor:///var/log/squid/access.log]
sourcetype = squid:access
index = squid
disabled = false

I have not made any changes to whitelists/blacklists, that's all default. What am I missing?

0 Karma

Anonymous
Not applicable

Does uForwarder have read access to those files?
Does you have a outputs.conf file located under /local/ or is it under default/?

0 Karma

jdsl
Loves-to-Learn

Yes, uForwarder has read permissions (it's running as root currently), and outputs.conf is delivered as another app. Pretty sure it's functional though, otherwise how would the metrics/splunkd.log etc be getting sent?

Also:
~# /opt/splunkforwarder/bin/splunk list forward-server
Active forwards:
$splunk_ip:9997
Configured but inactive forwards:
None

0 Karma

bosburn_splunk
Splunk Employee
Splunk Employee

Let's start simple:
Do you have the index set up on the indexer?
Do you see any alerts regarding getting data for a non-configured index?
If you run a real time all time search for index=squid, do you see events coming in with the correct timestamps being extracted?

0 Karma

mortf
Explorer

I would also suggest checking your "all-in-one" splunk instance to verify that the index "proxy" is set up and ENABLED.

I've had similar issues before where the index was disabled on the indexer.
I've also had issues when enabling the inputs on the forwarder before creating the index. Creating and enabling the index (proxy in your case) should always be step 1

Other than that i would also check if the sourcetype (squid:access) is defined on the "all-in-one" splunk instance and create a sourcetype with that name if it does not exist.

I also assume you have tried restarting the splunk service on the forwarders after you assigned the TA's to them?

0 Karma

MuS
SplunkTrust
SplunkTrust
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...