Splunk Enterprise

Please advise. How do I Check for data gaps, broken events & incorrectly indexed data in Splunk Ent. Thank u

SamHTexas
Builder

As a deep dive into my data sources / data integrity. I need to learn what SPLs /Apps needs to be used for this purpose. I appreciate your help.

Labels (1)
Tags (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

That sounds very site-specific.  Tell us more about your needs.  What constitutes a data gap?  What is a broken event?  What do you mean by "incorrectly indexed data"?

---
If this reply helps you, an upvote would be appreciated.
0 Karma

SamHTexas
Builder

Thank u for your message. Broken events are events that looks like they are there but don't make sense - they are incomplete, same with incorrect indexed data. I get errors that are caused when someone re-starts Splunk instances in the middle of the day or Indexers lose communication with each other or there are Internet lapses. I hope I made sense. Is there a way to see incomplete events & why they occurred ? Thank u

Tags (1)
0 Karma

codebuilder
Influencer

If you want to see gaps in data ingestion, such as days or hours where no data came in you can run this:

| tstats count where index=your_index_name by _time


Then just click on "visualization" and you'll get a nice graph of event count over a timeline (controlled by your date/time picker).

You can drill down further on the search to visualize by day, hour, seconds, etc.

----
An upvote would be appreciated and Accept Solution if it helps!

SamHTexas
Builder

Thank u. So what is Data gap per your experience. I did get bar code & data but could not make sense of it. Does data gap really mean no data was ingested? What causes that per your knowledge? 

Tags (1)
0 Karma

codebuilder
Influencer

The bars represent the number of events indexed by Splunk on a given day/time and is based on the range you selected with the date/time picker. The timestamp is shown below the bars. You can left click/hold on the graph and drag across it to drill down into smaller time ranges.

To me, a data gap would mean no events received for some period of time. That would indicate a problem with the forwarders, hosts, the indexers, etc.

Gaps can be normal, such as on weekends or holidays, for example. It just depends on your specific environment as @richgalloway  mentioned.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!