Splunk Enterprise

Why is Splunk ingesting additional information from the source, when there is no actual data present in the source ?

Hemnaath
Motivator

Hi All,  We have concern raised by one of our application team as they could see incorrect data in their dashboard, When validated the same by looking into the source of the file where the splunk is reading it, we noticed that there is no actual data present in the log source.

Problem:  Getting incorrect data ingested into Splunk in the status field value

[13/Apr/2022:06:33:03 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8444 status=2
[13/Apr/2022:04:30:01 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=2
[12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=2
[12/Apr/2022:09:11:37 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8444 status=2

Actual data present in the application server

 Path:/var/mware/logs/xxx/localhost_access_log.2022-04-12.11.log

[12/Apr/2022:11:10:26 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=216 response=1
[12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=216 response=1
[12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=219 response=1
[12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=216 response=0
[12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=219 response=0
[12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=216 response=1

Monitoring Stanza details:

[monitor:///var/mware/logs/*/*localhost*]
sourcetype = access_combined
index = test
disabled = 0
ignoreOlderThan = 1d
blacklist=\.(gz)$

Splunkd.log : There is no significant ERROR|WARN|INFO related to this issue found.

So could you please guide me what will be the reason why Splunk is ingesting an incorrect information when there is no actual data present in the source and also guide me how to troubleshoot this issue.

Labels (2)
Tags (1)
0 Karma
Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Dynamic formatting from XML events

This challenge was first posted on Slack #puzzles channelFor a previous puzzle, I needed a set of fixed-length ...

Enter the Agentic Era with Splunk AI Assistant for SPL 1.4

  🚀 Your data just got a serious AI upgrade — are you ready? Say hello to the Agentic Era with the ...

Stronger Security with Federated Search for S3, GCP SQL & Australian Threat ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...