Hi All,
I have a question regarding indexing log file. I am using one application and monitoring events online.I have no issue with this but after midnight, this file is zipped and renamed inside the same directory by the application.
Example:
Original file name is : a.log and become: a-070515.log.zip
Therefore Splunk indexing the same file(s) double time. (a.log and a-070515.log.zip)
How can I figure out this easily?
Thanks
Gokhan
Modify your inputs.conf to include a whiltelist/blacklist:
whitelist=\.log$
blacklist=\.gz$
I believe that the .log files are only being processed once and that the second copy is from the .gz file.
Thanks for the answer. I have some doubts.
If all a.log files are not indexed before midnight due to network issue or other possible scenarios (these files are zipped after midnight), Splunk forwarder somehow caches this data to send it to the Splunk server later OR I lost the data to be indexed due to name(a.log.zip) as blacklist.
Research "crcSalt=<SOURCE>". By default, Splunk does not use the name of the file to see if it has forwarded this file or not; it uses a CRC checksum of the first and last bytes of the file. So by default (i.e. unless you TOLD Splunk NOT TO), Splunk will ignore a file that is renamed and not reforward it. However, if you compress it and you have not told splunk NOT to forward the compressed files, it will happily forward the compressed file's contents.
Thanks for the advice. I did it with using whitelist/blacklist. I also revised the wildcards according to link below.
http://docs.splunk.com/Documentation/Splunk/6.2.3/Data/Specifyinputpathswithwildcards
Now everything is ok.
whitelist = a.log$
blacklist = .*zip$
Modify your inputs.conf to include a whiltelist/blacklist:
whitelist=\.log$
blacklist=\.gz$
I believe that the .log files are only being processed once and that the second copy is from the .gz file.