Getting Data In

How to troubleshoot why 1 of 2 files is no longer getting indexed after updating glibc and restarting our heavy forwarders using Splunk 5.0.11?

Sqig
Path Finder

Hi. This is regarding Splunk 5.0.11 Universal Forwarder and Heavy Forwarder.

We rebooted 2 Heavy Forwarders today (after updating glibc) and now, we are only seeing 1 of the 2 files that each of our Universal Forwarders reads and forwards.

I know the data is not getting to the Indexer (or at least is not getting indexed).

Is there a best practice for determining whether or not data is at least getting TO the heavy forwarder?

I did try adding crcSalt= in the stanza in inputs.conf on the universal forwarder that specifies the data we're missing, just in case something cropped up.

Thanks for any suggestions to help us get started with this one...

0 Karma
1 Solution

Sqig
Path Finder

I ended up working with Splunk Support on this one. For reasons neither of us can pinpoint, it seems that rebooting the whole server that the Heavy forwarder ran on stopped proper timestamp parsing for the events from that one logfile.

Explicitly specifying the timezone resolved the issue.

View solution in original post

Sqig
Path Finder

I ended up working with Splunk Support on this one. For reasons neither of us can pinpoint, it seems that rebooting the whole server that the Heavy forwarder ran on stopped proper timestamp parsing for the events from that one logfile.

Explicitly specifying the timezone resolved the issue.

Sqig
Path Finder

A little more info: I saw references to my missing sourcetype in the metrics.log on one of the source servers. Also, hitting /services/admin/inputstatus/TailingProcessor%3AFileStatus on a source server showed the files I am looking for as having been read to 100% completion.

And I checked the per_sourcetype_thruput on the indexers, and that seems to show no difference in volume from previous days. However, I absolutely do NOT see any indication in the index itself that the data is present. I even searched index=* just to be sure.

Now I'm really puzzled....

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...