topic Re: Event mismatch in data. event count is higher in splunk than actual log count in server log file. in Getting Data In

How to troubleshoot event mismatch in data?

gowthammahes — Mon, 15 May 2023 12:58:04 GMT

Hi Team,
Recently, I have configured splunk in my project to monitoring the application logs. I could find there is some log count mismatch between log file in server and event count in splunk logs. it is not happening in all time only some times like 2 or 3 times in a month then remaining days the event count is matching with log file count in server. Could you please share suggestion to troubleshoot the issue.

Splunk enterprise licensed version: 9.0.3

server kernel: Linux red hat

Universal forwarder version: 9.0.3

server kernel: Linux red hat

Example: Log file size is 500MB and total log count in log file is 1520713 and total event count in splunk after indexing is 1520794 which is higher than the server log file.

logs count in application log file = 1520713

event count in splunk search = 1520794 which is higher than actual log file.

I have verified the splunkd logs and there is no error.

verified limits conf and props ocnf as well and there is no specific config related to it.

index conf:

[monitor:///app/log/audit.log]
index = xxxx
disabled = false
ignoreOlderThan = 7d
recursive = false

limits.conf:

[thruput]
maxKBps = 512

Re: Event mismatch in data. event count is higher in splunk than actual log count in server log file.

richgalloway — Fri, 12 May 2023 13:06:40 GMT

This can happen if Splunk treats a single log entry as a multiple events because the line breaking settings in props.conf are incorrect. I understand you verified the props.conf, but it's possible that 2 or 3 times a month a log entry is created which does not match the props.conf settings.

If you are using indexer acknowledgement then duplication of events is very possible. It will happen when a UF re-sends events after an ack was lost or arrived late.

Re: Event mismatch in data. event count is higher in splunk than actual log count in server log file.

gowthammahes — Fri, 12 May 2023 13:50:48 GMT

HI @richgalloway ,

Thank you for the quick response. We dont have indexer acknowledgement and in props.conf we have only below configuration.

[source::/app/log/audit.log]
sourcetype = audit_log

Re: Event mismatch in data. event count is higher in splunk than actual log count in server log file.

richgalloway — Fri, 12 May 2023 13:59:43 GMT

That's not much for a prop.conf stanza. It's hard to be certain without seeing sample data, but you may need additional settings. Consider adding the "Magic Six" attributes.

TIME_PREFIX TIME_FORMAT MAX_TIMESTAMP_LOOKAHEAD SHOULD_LINEMERGE LINE_BREAKER TRUNCATE