topic Splunk is ingesting archive files of syslog data that has already been ingested. in Getting Data In

Splunk is ingesting archive files of syslog data that has already been ingested.

hrithiktej — Thu, 07 Sep 2017 10:31:36 GMT

our setup 2 SH, 1 deployment server, 1 license server and 2 indexers , our two indexers are also syslog servers and they read the input file directly from syslog folder for indexing i suspect Splunk is ingesting archive files of syslog data that has already been ingested How do i verify this ?

Re: Splunk is ingesting archive files of syslog data that has already been ingested.

gcusello — Thu, 07 Sep 2017 10:43:57 GMT

Hi
you can run a search like this

index=your_index
| stats count by source

and verify if the eventcount for each source is the same of the file or not.

otherwise you can run

index=your_index
| stats count by _raw
| where count>1

if there are results, you have duplicated events.

You can also verify if your monitored servers are configured to send to both the hosts or not.
The best way is to have a Load Balancer between monitored servers and Indexers to be sure of syslog flow ingestion.

Splunk can run a syslog server, and it's efficient (if you have quick disks), why do you use a syslog server on your indexers?

Bye.
Giuseppe

Re: Splunk is ingesting archive files of syslog data that has already been ingested.

hrithiktej — Thu, 07 Sep 2017 10:56:18 GMT

Thanks
I ran index=* | stats count by _raw | where count>1 for last 24hrs and I see more than 4 hundred thousand events.

We do have a f5 load balancer before the indexer/syslog cluster and I do not know why our previous splunk admin set the indexers and syslog servers on same box.

Re: Splunk is ingesting archive files of syslog data that has already been ingested.

gcusello — Thu, 07 Sep 2017 11:09:35 GMT

It could not be a problem to have both Indexers and syslog servers on the same machines,it depends only by the load that they have to manage, if you have few events to ingest and index you can leave them on Indexers, if instead you have many events it's surely better to have two dedicated Heavy Forwarders for syslog ingestion.
Anyway per the log duplication, you have to check your F5 configuration and what is the destination host of your monitored servers, maybe they are configurated to directly send to both the Indexers addresses instead Virtual IP.
Bye.
Giuseppe
N.B.: if your satisfied by this answer, please accept or upvote it

Re: Splunk is ingesting archive files of syslog data that has already been ingested.

hrithiktej — Thu, 07 Sep 2017 12:40:21 GMT

From the query and the results that i told you earlier do you mean to say we have duplication of data issue? and I checked the f5 config we have both our indexers mapped to a virtual IP and all the monitored hosts like FWs and switches have this VIP entry in their logging for syslog setup.

Re: Splunk is ingesting archive files of syslog data that has already been ingested.

gcusello — Thu, 07 Sep 2017 12:56:29 GMT

Only one additional check, run:
index=* | stats count by host _raw
to verify if the doubled log is sent by one or two hosts (it's really difficoult that the same log is sent by different hosts!).

About the reason of this, the only possibility is that there are two sources for the same data, two sources because F5 sends to both the indexers or because there are two inputs.confs.

Verify F5 configuration.
Then check your inputs configurations to understand if there are more inputs for the same source.

Bye.
Giuseppe

Re: Splunk is ingesting archive files of syslog data that has already been ingested.

hrithiktej — Thu, 07 Sep 2017 13:18:00 GMT

Thanks! when i run index=* | stats count by host _raw i do see multiple logs for same ip however the count is 1

Re: Splunk is ingesting archive files of syslog data that has already been ingested.

gcusello — Sun, 10 Sep 2017 05:56:14 GMT

Are your hostnames expressed as IPs?
If you didn't configured hostname run the same search using IP instead host.
The scope of this search is exclude same logs that arrive from different hosts.
bye.
Giuseppe

Re: Splunk is ingesting archive files of syslog data that has already been ingested.

hrithiktej — Mon, 11 Sep 2017 09:37:11 GMT

Thanks Even when I run index=* | stats count by IP _raw I do see multiple logs for the same IP. Is this Duplication?