Figure out when a log entry was sent from a forwarder?

[hajducko]

We're having an issue where a log entry isn't being indexed by the indexer until several hours after the log entry was written.

The log entry has a timestamp of 2/17 21:15 and goes all the way back to 2/17 20:33. However, our indexer shows ( via indexed_time ) that it didn't index the events until 2/18 2:16 AM.

I need to able to determine why that it is - is it forwarder lag? Did they misconfigure something? Or is it indexer lag?

As far as indexer lag, we have SoS installed, but according to it, the indexer wasn't experiencing any issues - none of the queues were filled but at this point, I have no data I can give the customer about why this would have happened.

Which brings me to the question - is there someway to determine when the forwarder saw/sent the event or when the indexer received ( not indexed ) the event?

If I could tell that, at least it would help me point at the forwarder or the indexer and narrow the investigation down, but I don't know of anyway to determine that information.

[yannK]

They are no timestamp to know when the events was read from the log file.
The only one you have is the _indextime if when the indexer parsed it. With it you can evaluate the delay between the event timestamp and the indextime

source=mysource host=myhost | delay=_indextime-_time | table _time delay date_zone _raw

So first of all :

• check the timezone applied
• check the thruput limit on the forwarder http://answers.splunk.com/answers/29538/maxkbps-option-and-limiting-a-forwarders-rate-of-thruput

[hajducko]

The timezone and forwarder thruput don't appear to be the issue - the issue appears to be a file descriptor issue. The host in question is generating 124,000+ individual log files per hour.

[hajducko]

Yeah, I already used the indexed time to determine that we're experiencing several hours of delay between the timestamp of the event and the time when the event was eventually indexed. I don't have access to the forwarding host, so I'm having the user get the TZ and thruput items, but was hoping for some other way of diagnosing this or pointing to which side of the issue was the problem.