Deployment Architecture

Data Summary Report Events for a Host but searching the Host in the Shearch Head Doesn't Show Events

israelgutierrez
Path Finder

Hi

We have weird behavior, in the Data Summary Screen on the Search Head, we see a Host reporting events, when clic on the host searching for the details, the Search Head shows 0 results

This is happening only with 1 host, other hosts from the same index and same sourcetype didn't present the same issue.

Any Ideas?

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Search over all time - could be old data / misrecognized timestamp.

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust

Search over all time - could be old data / misrecognized timestamp.

martin_mueller
SplunkTrust
SplunkTrust

All your data is from 2013, so searching today won't find it.

The metadata command powering the data summary is a bit more lenient with time ranges so it finds those old events.

0 Karma

israelgutierrez
Path Finder

Thanks, The type of log is different from the others and the time it was bad recognized

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

There should be three timestamps for that host, what are they?

0 Karma

israelgutierrez
Path Finder

FirstTime
30/3/2013 14:06:56 GMT-6:00

LastTime
30/3/2013 17:08:12 GMT-6:00

RecentTime
30/3/2016 17:08:27 GMT-6:00

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Okay, this?

| metadata type=hosts index=*
0 Karma

israelgutierrez
Path Finder

Yes, on that one the host appears with 766 Events

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Hmm... to drill deeper, run this over today:

| tstats count where index=* by index host sourcetype
0 Karma

israelgutierrez
Path Finder

Doing that search the host we are looking for didn't appear

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Yeah, last log received - as in time it was indexed. The timestamp extracted from the events may be different.

0 Karma

israelgutierrez
Path Finder

Ok, for that Got It, after a couple of reviews we are seeing that the _time of the log and the _indexed time are consistent so we are extracting the time correctly

But Sadly we know that host it have events today, and in the "All time search" the logs for today doesn't shows, but the Data Summary it shows updates

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Okay... made sure the search runs over a sufficiently large time range?

0 Karma

israelgutierrez
Path Finder

The Summary screen shows Last Update 3-30-16 2:34:26.000 PM

But searching for that Host on Today, shows 0 results

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Have there been calls to the delete command?

0 Karma

lib_systems
Path Finder

I am experiencing the same issue as the original post, however, I have been issuing the delete command to remove test data, etc. When I issue the delete command it confirms that the correct number of events have been deleted, yet days later the Data Summary still shows event counts for hosts which no longer have associated events indexed. Please advise.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

The data summary is not updated by the delete command.

0 Karma

lib_systems
Path Finder

And so how does one update the Data Summary after using the delete command? The documentation glosses over this point.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

You don't. It updates itself when the bucket ages out.

0 Karma

lib_systems
Path Finder

What's the logic behind having this discrepancy? I don't administer the application so I don't yet know what the retention period is, but, why would someone want the Data Summary to continue to show results for data which is no longer present for the next X days/weeks/months?

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

The metadata command uses data that doesn't get updated when events are marked for deletion. Deleting events isn't intended to happen at all in Splunk, so it's not that big of a deal.
The metadata has many more shortcomings, for example it doesn't honour time ranges accurately but rather has an all-or-nothing-per-bucket approach giving you seriously inaccurate data even without the delete command.

If you want accurate summaries, use tstats.

Get Updates on the Splunk Community!

This Week's Community Digest - Splunk Community Happenings [9.26.22]

Get the latest news and updates from the Splunk Community here! Upcoming User Group Events! 👏 Check ...

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...