Getting Data In

How to troubleshoot why 1 of 2 files is no longer getting indexed after updating glibc and restarting our heavy forwarders using Splunk 5.0.11?

Sqig
Path Finder

Hi. This is regarding Splunk 5.0.11 Universal Forwarder and Heavy Forwarder.

We rebooted 2 Heavy Forwarders today (after updating glibc) and now, we are only seeing 1 of the 2 files that each of our Universal Forwarders reads and forwards.

I know the data is not getting to the Indexer (or at least is not getting indexed).

Is there a best practice for determining whether or not data is at least getting TO the heavy forwarder?

I did try adding crcSalt= in the stanza in inputs.conf on the universal forwarder that specifies the data we're missing, just in case something cropped up.

Thanks for any suggestions to help us get started with this one...

0 Karma
1 Solution

Sqig
Path Finder

I ended up working with Splunk Support on this one. For reasons neither of us can pinpoint, it seems that rebooting the whole server that the Heavy forwarder ran on stopped proper timestamp parsing for the events from that one logfile.

Explicitly specifying the timezone resolved the issue.

View solution in original post

Sqig
Path Finder

I ended up working with Splunk Support on this one. For reasons neither of us can pinpoint, it seems that rebooting the whole server that the Heavy forwarder ran on stopped proper timestamp parsing for the events from that one logfile.

Explicitly specifying the timezone resolved the issue.

Sqig
Path Finder

A little more info: I saw references to my missing sourcetype in the metrics.log on one of the source servers. Also, hitting /services/admin/inputstatus/TailingProcessor%3AFileStatus on a source server showed the files I am looking for as having been read to 100% completion.

And I checked the per_sourcetype_thruput on the indexers, and that seems to show no difference in volume from previous days. However, I absolutely do NOT see any indication in the index itself that the data is present. I even searched index=* just to be sure.

Now I'm really puzzled....

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...