Why are certain log events not getting indexed in Splunk 5.0.4 and how to troubleshoot?

[strive]

Hi,

One of our customers is using Splunk 5.0.4. The log files are forwarded to indexer using Splunk Universal Forwarder.

The log in flow is like this:

Splunk UF on Devices --> Splunk UF in the product --> Indexer

The issue is: At times, some log events are not getting indexed and this leads to data inaccuracy in our metrics. Recently when they reported this issue, i took log files from them and indexed them in my local test bed. I was able to replicate the issue. Out of 5000 log events, 7 events did not enter the index. Similarly in other log file, out of 5085 log events, 13 events did not enter the index.

I checked following:

1. If log event length is on the higher side -- answer is No.

2. If some unreasonable junk characters are present in the log event -- answer is No.

3. If the log events are duplicate of other log events -- answer is No.

Could you suggest some pointers for me to troubleshoot this issue. Why some specific log lines are not getting indexed?

Note: This is not happening all the time. In last two weeks this has happened twice for around 10 log files.

Thanks

Strive

[strive · Accepted solution]

The log files had secondary header line starting with words s-ip|#Fields.

If the log lines had any field value(s) with s-ip as substring then those log lines were stripped off.
We had to modify our transforms.conf configurations to address this issue.

[strive]

The log files are not rolling.
We have set nullQueue for headers. This wont interfere with these log lines.

[MuS]

try to index the events again while running this script http://blogs.splunk.com/2011/01/02/did-i-miss-christmas-2/

[pradeepkumarg]

Are the log files rolling? If so, check if the events are being missed for some reason while the log is being rolled.

[MuS]

any nullQueue in any transforms.conf which could interfere here? check with btool

[strive]

Link to files
https://www.dropbox.com/s/5g8q4d40j5mwf2b/my_data.13.13.13.13_20140823_114500_1501?dl=0

[my_source_type]
SHOULD_LINEMERGE = false
TRANSFORMS-include = some transforms
TIME_PREFIX=^([^\t]*\t){2}
MAX_TIMESTAMP_LOOKAHEAD=35

[somesoni2]

Would it be possible for you to share those events which are not getting indexed? (may after masking sensitive information), Also, the sourcetype definition (props.conf)?

[strive]

Created a log file using the missing events alone and tried indexing this file. The events are not getting indexed, there are no errors in splunkd.log (enabled debug mode and checked). Manually verified every field in the log file, it all looks fine.

[strive]

They are from same sourcetype. There is no commanlity.

[jbouch03]

Are they the same sourcetype or different? Also, is there any commonality among the events that are not getting indexed?