topic Re: Indexing same log file multiple times issue in Getting Data In

Indexing same log file multiple times issue

gyarici — Thu, 07 May 2015 14:37:55 GMT

Hi All,

I have a question regarding indexing log file. I am using one application and monitoring events online.I have no issue with this but after midnight, this file is zipped and renamed inside the same directory by the application.

Example:
Original file name is : a.log and become: a-070515.log.zip

Therefore Splunk indexing the same file(s) double time. (a.log and a-070515.log.zip)

I need to monitor event online
If I ignore indexing a.log only, I cannot follow the events online.
If I ignore indexing a-xxxx.log , there is risk to be indexed on time (?) -network connection issue etc..

How can I figure out this easily?

Thanks

Gokhan

Re: Indexing same log file multiple times issue

woodcock — Thu, 07 May 2015 14:44:47 GMT

Modify your inputs.conf to include a whiltelist/blacklist:

whitelist=\.log$
blacklist=\.gz$

I believe that the .log files are only being processed once and that the second copy is from the .gz file.

Re: Indexing same log file multiple times issue

gyarici — Fri, 08 May 2015 08:03:47 GMT

Thanks for the answer. I have some doubts.

If all a.log files are not indexed before midnight due to network issue or other possible scenarios (these files are zipped after midnight), Splunk forwarder somehow caches this data to send it to the Splunk server later OR I lost the data to be indexed due to name(a.log.zip) as blacklist.

Re: Indexing same log file multiple times issue

woodcock — Fri, 08 May 2015 14:42:35 GMT

Research "crcSalt=<SOURCE>". By default, Splunk does not use the name of the file to see if it has forwarded this file or not; it uses a CRC checksum of the first and last bytes of the file. So by default (i.e. unless you TOLD Splunk NOT TO), Splunk will ignore a file that is renamed and not reforward it. However, if you compress it and you have not told splunk NOT to forward the compressed files, it will happily forward the compressed file's contents.

Re: Indexing same log file multiple times issue

gyarici — Thu, 28 May 2015 11:41:27 GMT

Thanks for the advice. I did it with using whitelist/blacklist. I also revised the wildcards according to link below.
http://docs.splunk.com/Documentation/Splunk/6.2.3/Data/Specifyinputpathswithwildcards
Now everything is ok.

Re: Indexing same log file multiple times issue

gyarici — Thu, 28 May 2015 11:47:55 GMT

whitelist = a.log$
blacklist = .*zip$