topic Re: Forwarder keeps input duplicate event in Getting Data In

Forwarder keeps input duplicate event

slasyang — Thu, 19 Aug 2021 08:37:01 GMT

Hi,

I have a log server with universal forwarder and some Linux server,

and I set a cronjob to make those Linux server upload their /var/log/secure and /var/log/messages to log server every 10 mins, and universal forwarder will monitor them.

But every time, when linux server upload their log files to log server,

universal forwarder will index not only difference part but entire files,

and it caused a lot of waste of license.

here is my inputs.conf

[monitor://D:\Log\Linux\*\messages]
sourcetype = message
index = os
host_segment = 3

How can I fix it?

Re: Forwarder keeps input duplicate event

jwalthour — Thu, 19 Aug 2021 08:58:37 GMT

How are you sending the Linux logs to your Windows log server? What exactly is the content of your cron job on your Linux servers?

Re: Forwarder keeps input duplicate event

slasyang — Thu, 19 Aug 2021 09:09:08 GMT

First, It copy secure log and messages log to a temp folder,

then use FTP command to PUT the file to windows log server.

Re: Forwarder keeps input duplicate event

jwalthour — Thu, 19 Aug 2021 09:37:11 GMT

And each time you FTP the entire log file to the Windows log server or just the additions?

Re: Forwarder keeps input duplicate event

slasyang — Thu, 19 Aug 2021 09:47:30 GMT

I FTP the entire log file to overwrite the last one.

In my cognition,

forwarder will continue to index event from the end of the last file,

not from the first line.

Re: Forwarder keeps input duplicate event

jwalthour — Thu, 19 Aug 2021 12:35:37 GMT

Except that you’re not just adding events to an existing file; you’re recreating the file each time you overwrite it. I’d need to know more details, but something about what you’re doing triggers Splunk to see it as a new file. I’d advise consulting this troubleshooting guide below to figure out why Splunk is seeing it as a new file and, possibly, how to prevent it.

https://wiki.splunk.com/Community:Troubleshooting_Monitor_Inputs

I’d also recommend you get a proper syslog server set up to accomplish this task in the best way possible.

If this solved your problem, please mark it as the solution.

Re: Forwarder keeps input duplicate event

slasyang — Fri, 20 Aug 2021 04:45:58 GMT

Thanks for your recommendation,

I think I understand the cause of this problem.

I'll try another way to index those log file.