How to troubleshoot event mismatch in data?

[gowthammahes]

Hi Team,
Recently, I have configured splunk in my project to monitoring the application logs. I could find there is some log count mismatch between log file in server and event count in splunk logs. it is not happening in all time only some times like 2 or 3 times in a month then remaining days the event count is matching with log file count in server.  Could you please share suggestion to troubleshoot the issue.

Splunk enterprise licensed version: 9.0.3

server kernel: Linux red hat

Universal forwarder version: 9.0.3

server kernel: Linux red hat

Example: Log file size is 500MB and total log count in log file is  1520713 and total event count in splunk after indexing is 1520794 which is higher than the server log file. 

logs count in application log file = 1520713 

event count in splunk search = 1520794  which is higher than actual log file. 

I have verified the splunkd logs and there is no error.

verified limits conf and props ocnf as well and there is no specific config related to it. 

index conf:

[monitor:///app/log/audit.log]
index = xxxx
disabled = false
ignoreOlderThan = 7d
recursive = false

limits.conf:

[thruput]
maxKBps = 512

[richgalloway]

This can happen if Splunk treats a single log entry as a multiple events because the line breaking settings in props.conf are incorrect.  I understand you verified the props.conf, but it's possible that 2 or 3 times a month a log entry is created which does not match the props.conf settings.

If you are using indexer acknowledgement then duplication of events is very possible.  It will happen when a UF re-sends events after an ack was lost or arrived late.

[gowthammahes]

HI @richgalloway ,

Thank you for the quick response. We dont have indexer acknowledgement and in props.conf we have only below configuration.

[source::/app/log/audit.log]
sourcetype = audit_log

[richgalloway]

That's not much for a prop.conf stanza.  It's hard to be certain without seeing sample data, but you may need additional settings.  Consider adding the "Magic Six" attributes.

TIME_PREFIX
TIME_FORMAT
MAX_TIMESTAMP_LOOKAHEAD
SHOULD_LINEMERGE
LINE_BREAKER
TRUNCATE