Data Summary Report Events for a Host but searching the Host in the Shearch Head Doesn't Show Events

[israelgutierrez]

Hi

We have weird behavior, in the Data Summary Screen on the Search Head, we see a Host reporting events, when clic on the host searching for the details, the Search Head shows 0 results

This is happening only with 1 host, other hosts from the same index and same sourcetype didn't present the same issue.

Any Ideas?

[martin_mueller · Accepted solution]

Search over all time - could be old data / misrecognized timestamp.

[martin_mueller]

All your data is from 2013, so searching today won't find it.

The metadata command powering the data summary is a bit more lenient with time ranges so it finds those old events.

[israelgutierrez]

Thanks, The type of log is different from the others and the time it was bad recognized

[martin_mueller]

There should be three timestamps for that host, what are they?

[israelgutierrez]

FirstTime
30/3/2013 14:06:56 GMT-6:00

LastTime
30/3/2013 17:08:12 GMT-6:00

RecentTime
30/3/2016 17:08:27 GMT-6:00

[martin_mueller]

Okay, this?

| metadata type=hosts index=*

[israelgutierrez]

Yes, on that one the host appears with 766 Events

[martin_mueller]

Hmm... to drill deeper, run this over today:

| tstats count where index=* by index host sourcetype

[israelgutierrez]

Doing that search the host we are looking for didn't appear

[martin_mueller]

Yeah, last log received - as in time it was indexed. The timestamp extracted from the events may be different.

[israelgutierrez]

Ok, for that Got It, after a couple of reviews we are seeing that the _time of the log and the _indexed time are consistent so we are extracting the time correctly

But Sadly we know that host it have events today, and in the "All time search" the logs for today doesn't shows, but the Data Summary it shows updates

[martin_mueller]

Okay... made sure the search runs over a sufficiently large time range?

[israelgutierrez]

The Summary screen shows Last Update 3-30-16 2:34:26.000 PM

But searching for that Host on Today, shows 0 results

[martin_mueller]

Have there been calls to the delete command?

[lib_systems]

I am experiencing the same issue as the original post, however, I have been issuing the delete command to remove test data, etc. When I issue the delete command it confirms that the correct number of events have been deleted, yet days later the Data Summary still shows event counts for hosts which no longer have associated events indexed. Please advise.

[martin_mueller]

The data summary is not updated by the delete command.

[lib_systems]

And so how does one update the Data Summary after using the delete command? The documentation glosses over this point.

[martin_mueller]

You don't. It updates itself when the bucket ages out.

[lib_systems]

What's the logic behind having this discrepancy? I don't administer the application so I don't yet know what the retention period is, but, why would someone want the Data Summary to continue to show results for data which is no longer present for the next X days/weeks/months?

[martin_mueller]

The metadata command uses data that doesn't get updated when events are marked for deletion. Deleting events isn't intended to happen at all in Splunk, so it's not that big of a deal.
The metadata has many more shortcomings, for example it doesn't honour time ranges accurately but rather has an all-or-nothing-per-bucket approach giving you seriously inaccurate data even without the delete command.

If you want accurate summaries, use tstats.