As a deep dive into my data sources / data integrity. I need to learn what SPLs /Apps needs to be used for this purpose. I appreciate your help.
That sounds very site-specific. Tell us more about your needs. What constitutes a data gap? What is a broken event? What do you mean by "incorrectly indexed data"?
Thank u for your message. Broken events are events that looks like they are there but don't make sense - they are incomplete, same with incorrect indexed data. I get errors that are caused when someone re-starts Splunk instances in the middle of the day or Indexers lose communication with each other or there are Internet lapses. I hope I made sense. Is there a way to see incomplete events & why they occurred ? Thank u
If you want to see gaps in data ingestion, such as days or hours where no data came in you can run this:
| tstats count where index=your_index_name by _time
Then just click on "visualization" and you'll get a nice graph of event count over a timeline (controlled by your date/time picker).
You can drill down further on the search to visualize by day, hour, seconds, etc.
Thank u. So what is Data gap per your experience. I did get bar code & data but could not make sense of it. Does data gap really mean no data was ingested? What causes that per your knowledge?
The bars represent the number of events indexed by Splunk on a given day/time and is based on the range you selected with the date/time picker. The timestamp is shown below the bars. You can left click/hold on the graph and drag across it to drill down into smaller time ranges.
To me, a data gap would mean no events received for some period of time. That would indicate a problem with the forwarders, hosts, the indexers, etc.
Gaps can be normal, such as on weekends or holidays, for example. It just depends on your specific environment as @richgalloway mentioned.