topic Re: Why are certain log events not getting indexed in Splunk 5.0.4 and how to troubleshoot? in Monitoring Splunk

Why are certain log events not getting indexed in Splunk 5.0.4 and how to troubleshoot?

strive — Wed, 27 Aug 2014 12:23:18 GMT

Hi,

One of our customers is using Splunk 5.0.4. The log files are forwarded to indexer using Splunk Universal Forwarder.

The log in flow is like this:

Splunk UF on Devices --> Splunk UF in the product --> Indexer

The issue is: At times, some log events are not getting indexed and this leads to data inaccuracy in our metrics. Recently when they reported this issue, i took log files from them and indexed them in my local test bed. I was able to replicate the issue. Out of 5000 log events, 7 events did not enter the index. Similarly in other log file, out of 5085 log events, 13 events did not enter the index.

I checked following:

1. If log event length is on the higher side -- answer is No.

2. If some unreasonable junk characters are present in the log event -- answer is No.

3. If the log events are duplicate of other log events -- answer is No.

Could you suggest some pointers for me to troubleshoot this issue. Why some specific log lines are not getting indexed?

Note: This is not happening all the time. In last two weeks this has happened twice for around 10 log files.

Thanks

Strive

Re: Why are certain log events not getting indexed in Splunk 5.0.4 and how to troubleshoot?

jbouch03 — Wed, 27 Aug 2014 13:53:14 GMT

Are they the same sourcetype or different? Also, is there any commonality among the events that are not getting indexed?

Re: Why are certain log events not getting indexed in Splunk 5.0.4 and how to troubleshoot?

strive — Wed, 27 Aug 2014 14:52:13 GMT

They are from same sourcetype. There is no commanlity.

Re: Why are certain log events not getting indexed in Splunk 5.0.4 and how to troubleshoot?

strive — Wed, 27 Aug 2014 20:48:22 GMT

Created a log file using the missing events alone and tried indexing this file. The events are not getting indexed, there are no errors in splunkd.log (enabled debug mode and checked). Manually verified every field in the log file, it all looks fine.

Re: Why are certain log events not getting indexed in Splunk 5.0.4 and how to troubleshoot?

somesoni2 — Wed, 27 Aug 2014 21:08:12 GMT

Would it be possible for you to share those events which are not getting indexed? (may after masking sensitive information), Also, the sourcetype definition (props.conf)?

Re: Why are certain log events not getting indexed in Splunk 5.0.4 and how to troubleshoot?

strive — Mon, 28 Sep 2020 17:26:07 GMT

Link to files
https://www.dropbox.com/s/5g8q4d40j5mwf2b/my_data.13.13.13.13_20140823_114500_1501?dl=0

[my_source_type]
SHOULD_LINEMERGE = false
TRANSFORMS-include = some transforms
TIME_PREFIX=^([^\t]*\t){2}
MAX_TIMESTAMP_LOOKAHEAD=35

Re: Why are certain log events not getting indexed in Splunk 5.0.4 and how to troubleshoot?

MuS — Thu, 28 Aug 2014 12:14:08 GMT

any nullQueue in any transforms.conf which could interfere here? check with btool

Re: Why are certain log events not getting indexed in Splunk 5.0.4 and how to troubleshoot?

pradeepkumarg — Thu, 28 Aug 2014 13:31:18 GMT

Are the log files rolling? If so, check if the events are being missed for some reason while the log is being rolled.

Re: Why are certain log events not getting indexed in Splunk 5.0.4 and how to troubleshoot?

MuS — Thu, 28 Aug 2014 13:34:23 GMT

try to index the events again while running this script http://blogs.splunk.com/2011/01/02/did-i-miss-christmas-2/

Re: Why are certain log events not getting indexed in Splunk 5.0.4 and how to troubleshoot?

strive — Thu, 28 Aug 2014 17:21:40 GMT

The log files are not rolling.
We have set nullQueue for headers. This wont interfere with these log lines.

Re: Why are certain log events not getting indexed in Splunk 5.0.4 and how to troubleshoot?

strive — Sun, 31 Aug 2014 17:58:24 GMT

The log files had secondary header line starting with words s-ip|#Fields.

If the log lines had any field value(s) with s-ip as substring then those log lines were stripped off.
We had to modify our transforms.conf configurations to address this issue.