Getting Data In

Indexing same log file multiple times issue

gyarici
Path Finder

Hi All,

I have a question regarding indexing log file. I am using one application and monitoring events online.I have no issue with this but after midnight, this file is zipped and renamed inside the same directory by the application.

Example:
Original file name is : a.log and become: a-070515.log.zip

Therefore Splunk indexing the same file(s) double time. (a.log and a-070515.log.zip)

  1. I need to monitor event online
  2. If I ignore indexing a.log only, I cannot follow the events online.
  3. If I ignore indexing a-xxxx.log , there is risk to be indexed on time (?) -network connection issue etc..

How can I figure out this easily?

Thanks

Gokhan

0 Karma
1 Solution

woodcock
Esteemed Legend

Modify your inputs.conf to include a whiltelist/blacklist:

whitelist=\.log$
blacklist=\.gz$

I believe that the .log files are only being processed once and that the second copy is from the .gz file.

View solution in original post

0 Karma

gyarici
Path Finder

Thanks for the answer. I have some doubts.

If all a.log files are not indexed before midnight due to network issue or other possible scenarios (these files are zipped after midnight), Splunk forwarder somehow caches this data to send it to the Splunk server later OR I lost the data to be indexed due to name(a.log.zip) as blacklist.

0 Karma

woodcock
Esteemed Legend

Research "crcSalt=<SOURCE>". By default, Splunk does not use the name of the file to see if it has forwarded this file or not; it uses a CRC checksum of the first and last bytes of the file. So by default (i.e. unless you TOLD Splunk NOT TO), Splunk will ignore a file that is renamed and not reforward it. However, if you compress it and you have not told splunk NOT to forward the compressed files, it will happily forward the compressed file's contents.

0 Karma

gyarici
Path Finder

Thanks for the advice. I did it with using whitelist/blacklist. I also revised the wildcards according to link below.
http://docs.splunk.com/Documentation/Splunk/6.2.3/Data/Specifyinputpathswithwildcards
Now everything is ok.

0 Karma

gyarici
Path Finder

whitelist = a.log$
blacklist = .*zip$

0 Karma

woodcock
Esteemed Legend

Modify your inputs.conf to include a whiltelist/blacklist:

whitelist=\.log$
blacklist=\.gz$

I believe that the .log files are only being processed once and that the second copy is from the .gz file.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...