Getting Data In

Why are the number of events indexed in Splunk less than the number of events in the monitored log file?

royimad
Builder

We are monitoring a file name X.log witch contain similar structure for events starting by a date format. The number of events that were indexed in Splunk are less than the number of events in the file. Is there any bug in indexing ? what logs should i look at ?

0 Karma

woodcock
Esteemed Legend

It is also possible that your timestamping is off and some of your events are showing up way at the wrong time (even in the future). Try this search run for ALL TIME (you have to use "ALL TIME" because that is the only way to search for events "in the future):


index=myIndex sourcetype=mySourceType | eval bytes=len(_raw) | stats count sum(bytes) by source

This will allow you to compare against your sourcefiles with this linux CLI:


cat myFile | wc

If the bytes match, then all of your file is going into Splunk. If the "count" equals "lines" then all the lines are broken correctly.

0 Karma

dflodstrom
Builder

If your events are single line it is possible that Splunk has combined some of them accidentally. You can configure event line breaking by following the guidance here --> http://docs.splunk.com/Documentation/Splunk/6.2.2/Data/Indexmulti-lineevents

In your case you may be most interested in the BREAK_ONLY_BEFORE parameter.

Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!