Getting Data In

How to troubleshoot event mismatch in data?

gowthammahes
Path Finder

Hi Team,
Recently, I have configured splunk in my project to monitoring the application logs. I could find there is some log count mismatch between log file in server and event count in splunk logs. it is not happening in all time only some times like 2 or 3 times in a month then remaining days the event count is matching with log file count in server.  Could you please share suggestion to troubleshoot the issue.

Splunk enterprise licensed version: 9.0.3

server kernel: Linux red hat

Universal forwarder version: 9.0.3

server kernel: Linux red hat

Example: Log file size is 500MB and total log count in log file is  1520713 and total event count in splunk after indexing is 1520794 which is higher than the server log file. 

logs count in application log file = 1520713 

event count in splunk search = 1520794  which is higher than actual log file. 

I have verified the splunkd logs and there is no error.

verified limits conf and props ocnf as well and there is no specific config related to it. 

index conf:

[monitor:///app/log/audit.log]
index = xxxx
disabled = false
ignoreOlderThan = 7d
recursive = false

limits.conf:

[thruput]
maxKBps = 512

 

Labels (2)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

This can happen if Splunk treats a single log entry as a multiple events because the line breaking settings in props.conf are incorrect.  I understand you verified the props.conf, but it's possible that 2 or 3 times a month a log entry is created which does not match the props.conf settings.

If you are using indexer acknowledgement then duplication of events is very possible.  It will happen when a UF re-sends events after an ack was lost or arrived late.

---
If this reply helps you, Karma would be appreciated.

gowthammahes
Path Finder

HI @richgalloway ,

Thank you for the quick response. We dont have indexer acknowledgement and in props.conf we have only below configuration.

[source::/app/log/audit.log]
sourcetype = audit_log

0 Karma

richgalloway
SplunkTrust
SplunkTrust

That's not much for a prop.conf stanza.  It's hard to be certain without seeing sample data, but you may need additional settings.  Consider adding the "Magic Six" attributes.

TIME_PREFIX
TIME_FORMAT
MAX_TIMESTAMP_LOOKAHEAD
SHOULD_LINEMERGE
LINE_BREAKER
TRUNCATE
---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...