Getting Data In

Real source IP of event

Builder

Hi,

We uncovered a problem with two forwarders using the same host-value. At first, we were baffled by the fact that Splunk was showing events that weren't in the log file itself. After looking in the _internal index, I noticed an unknown server sending events to the indexer.

The cause of the problem is that the unknown server has been created by cloning the original server, so both servers have the same host-value in their respective inputs.conf-files.

So now we have a problem where the index is contaminated with correct data from the incorrect host, using the same host-value. I would like to delete the incorrect data, but how can I find the data? The field sourceHost apparently doesn't exist outside of the _internal index.

Tags (2)
0 Karma
1 Solution

Builder

Just want to return to this to report the conclusions.

I contacted Splunk support about this and apparently, the event is indexed using the hostname reported by the Splunk forwarder. Once it's indexed, there is no way of knowing the originating host, unless you can identify the information in the event itself, eg. source path, sourcetype, etc.

View solution in original post

Builder

Just want to return to this to report the conclusions.

I contacted Splunk support about this and apparently, the event is indexed using the hostname reported by the Splunk forwarder. Once it's indexed, there is no way of knowing the originating host, unless you can identify the information in the event itself, eg. source path, sourcetype, etc.

View solution in original post

Builder

So it sounds like you have duplicate events. I would run a search and then use the dedup command with a common field then just pipe to delete and that should take care of those events.

0 Karma

Builder

Since host2 is identifying itself as host1, it is hard to distinguish the false and irrelevant data from the real data that actually comes from host1.

So I'm wondering if there is a way of disregarding the host-field in the event and searching for the real IP address that the event originates from? Then I could delete all the incorrect events from host2. Now it seems I have to re-index.

0 Karma

Builder

Oops... Seems I described my problem very poorly.

No, there are no duplicate events. There are events from two hosts, host1.example.com and host2.example.com. Host2 has been created as a clone of host1. Host1 is the production host, whereas host2 has been built to be used later.

Since host2 is a clone of host1, it has this line in inputs.conf:
host = host1.example.com

Host2 is running the same software as host1, but nobody is using host2 and host2 does not have DB access. Therefore any events generated on host2 are irrelevant, but appear alarming.

0 Karma