Getting Data In

Indexing same log file multiple times issue

gyarici
Path Finder

Hi All,

I have a question regarding indexing log file. I am using one application and monitoring events online.I have no issue with this but after midnight, this file is zipped and renamed inside the same directory by the application.

Example:
Original file name is : a.log and become: a-070515.log.zip

Therefore Splunk indexing the same file(s) double time. (a.log and a-070515.log.zip)

  1. I need to monitor event online
  2. If I ignore indexing a.log only, I cannot follow the events online.
  3. If I ignore indexing a-xxxx.log , there is risk to be indexed on time (?) -network connection issue etc..

How can I figure out this easily?

Thanks

Gokhan

0 Karma
1 Solution

woodcock
Esteemed Legend

Modify your inputs.conf to include a whiltelist/blacklist:

whitelist=\.log$
blacklist=\.gz$

I believe that the .log files are only being processed once and that the second copy is from the .gz file.

View solution in original post

0 Karma

gyarici
Path Finder

Thanks for the answer. I have some doubts.

If all a.log files are not indexed before midnight due to network issue or other possible scenarios (these files are zipped after midnight), Splunk forwarder somehow caches this data to send it to the Splunk server later OR I lost the data to be indexed due to name(a.log.zip) as blacklist.

0 Karma

woodcock
Esteemed Legend

Research "crcSalt=<SOURCE>". By default, Splunk does not use the name of the file to see if it has forwarded this file or not; it uses a CRC checksum of the first and last bytes of the file. So by default (i.e. unless you TOLD Splunk NOT TO), Splunk will ignore a file that is renamed and not reforward it. However, if you compress it and you have not told splunk NOT to forward the compressed files, it will happily forward the compressed file's contents.

0 Karma

gyarici
Path Finder

Thanks for the advice. I did it with using whitelist/blacklist. I also revised the wildcards according to link below.
http://docs.splunk.com/Documentation/Splunk/6.2.3/Data/Specifyinputpathswithwildcards
Now everything is ok.

0 Karma

gyarici
Path Finder

whitelist = a.log$
blacklist = .*zip$

0 Karma

woodcock
Esteemed Legend

Modify your inputs.conf to include a whiltelist/blacklist:

whitelist=\.log$
blacklist=\.gz$

I believe that the .log files are only being processed once and that the second copy is from the .gz file.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

&#x1f342; Fall into November with a fresh lineup of Community Office Hours, Tech Talks, and Webinars we’ve ...

Transform your security operations with Splunk Enterprise Security

Hi Splunk Community, Splunk Platform has set a great foundation for your security operations. With the ...

Splunk Admins and App Developers | Earn a $35 gift card!

Splunk, in collaboration with ESG (Enterprise Strategy Group) by TechTarget, is excited to announce a ...