Getting Data In
Highlighted

How to get last timestamp parsed from host?

Communicator

I'm working in an environment where the light forwarders watching windows eventlog inputs are configured for many different timezones.

As i found in another question, this is a bit of a problem because the windows eventlog inputs don't include timezone info with their timezones.

Is there a quick query that can show me the last timestamp received from a host?

I am thinking that this might not be in metrics log because that might only contain info about how much the server parsed at that time. I'm looking for "Last timestamp from all hosts" in a way that doesn't have to sort through the raw results of every single event.

Tags (2)
0 Karma
Highlighted

Re: How to get last timestamp parsed from host?

Communicator

I was hoping that i could do this by calculating a large difference in metadata restults.

|metadata type=hosts | eval drift=lastTime - recentTime

But in my environment, it seems this difference if always 0. -_-

0 Karma
Highlighted

Re: How to get last timestamp parsed from host?

Splunk Employee
Splunk Employee

this should do:

| metadata type=hosts | table host, lastTime  

Also, you might want to refer to this in case you want recentTime instead of lastTime...

.gz

0 Karma
Highlighted

Re: How to get last timestamp parsed from host?

Communicator

I tried evaling the difference between recentTime and lastTime, but i think it is trusting the timestamp send by the light forwarder. which is the whole problem because that timestamp is incorrect due to not including timezone.

0 Karma
Highlighted

Re: How to get last timestamp parsed from host?

Splunk Employee
Splunk Employee

drift=0 means your data is coming in live and you do not have archived data coming in. that, is a good thing, since its kind of saying your data is being timestamped correctly..

0 Karma
Highlighted

Re: How to get last timestamp parsed from host?

Communicator

Yah, but i know that is incorrect. I have live data coming in from a windows forwarded that gets timestamped 8 hours in the past.

0 Karma
Highlighted

Re: How to get last timestamp parsed from host?

Communicator

It is trusting the lightForwarder and setting the index time "lastTime" as the timestamp set by the lightForwarder, which is the oldness.

0 Karma
Highlighted

Re: How to get last timestamp parsed from host?

Communicator

The best i can come up with is just reviewing the metadata lastTime on hosts i /think/ should have send data recently but might have incorrect timezone extrapolation. This is difficult because i have >1000 hosts that send correct time, but may be low volume, and maybe 100 that do not.

| metadata type=hosts | convert timeformat="%y-%m-%d %H:%M:%S" ctime(lastTime) as mytime | table host,lastTime,mytime

Weird thing is that metadata's lastTime is being set to the time provided by the light forwarder... not the time of indexing on my indexer. Perhaps this is a bug. Documentation describes lastTime as the last time the indexer saw an event from the host. I read this as "the last time data was sent from this host", but maybe that can't apply for light or heavy forwarders.

0 Karma
Highlighted

Re: How to get last timestamp parsed from host?

Splunk Employee
Splunk Employee

then why do you not add a props.conf to tell the indexer to format time correctly: TZ=...

0 Karma
Highlighted

Re: How to get last timestamp parsed from host?

Communicator

Heh. A bit of a circular problem, that. The point of this search is to help identify hosts sending with the incorrect time, as happens when you have windows lightForwarders in with multiple timezone settings.

0 Karma