Splunk Enterprise

Why is Splunk ingesting additional information from the source, when there is no actual data present in the source ?

Hemnaath
Motivator

Hi All,  We have concern raised by one of our application team as they could see incorrect data in their dashboard, When validated the same by looking into the source of the file where the splunk is reading it, we noticed that there is no actual data present in the log source.

Problem:  Getting incorrect data ingested into Splunk in the status field value

[13/Apr/2022:06:33:03 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8444 status=2
[13/Apr/2022:04:30:01 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=2
[12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=2
[12/Apr/2022:09:11:37 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8444 status=2

Actual data present in the application server

 Path:/var/mware/logs/xxx/localhost_access_log.2022-04-12.11.log

[12/Apr/2022:11:10:26 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=216 response=1
[12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=216 response=1
[12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=219 response=1
[12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=216 response=0
[12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=219 response=0
[12/Apr/2022:11:10:27 +0000] fip="10.X.X.X" ip="10.X.X.X" method="POST" url="/xxx/service/decrypt" port=8443 status=200 size=216 response=1

Monitoring Stanza details:

[monitor:///var/mware/logs/*/*localhost*]
sourcetype = access_combined
index = test
disabled = 0
ignoreOlderThan = 1d
blacklist=\.(gz)$

Splunkd.log : There is no significant ERROR|WARN|INFO related to this issue found.

So could you please guide me what will be the reason why Splunk is ingesting an incorrect information when there is no actual data present in the source and also guide me how to troubleshoot this issue.

Labels (2)
Tags (1)
0 Karma
Get Updates on the Splunk Community!

Application management with Targeted Application Install for Victoria Experience

  Experience a new era of flexibility in managing your Splunk Cloud Platform apps! With Targeted Application ...

Index This | What goes up and never comes down?

January 2026 Edition  Hayyy Splunk Education Enthusiasts and the Eternally Curious!   We’re back with this ...

Splunkers, Pack Your Bags: Why Cisco Live EMEA is Your Next Big Destination

The Power of Two: Splunk + Cisco at "Ludicrous Scale"   You know Splunk. You know Cisco. But have you seen ...