topic Re: Rolled over issue with same header in Getting Data In

Rolled over issue with same header

jitsinha — Mon, 28 Sep 2020 20:04:21 GMT

I have couple of files in a directory as below

Out.log
Out_15.05.20_14.32.33.log
Out_15.05.21_07.06.45.log
Out_15.05.21_10.00.27.log

All of this files are having same header hence Splunk is ignoring all new files since rolled over files are having same header and same size.

Now I am planning is to use *initCrcLength * property and set it as 1000 Splunk will ignore the header part.

One issue with this is that Splunk will re-index all the files.

Can anyone please suggest how to ignore the older files??/

Re: Rolled over issue with same header

miteshvohra — Mon, 25 May 2015 07:33:16 GMT

Here is the doc link that explains how Splunk handles log file rotation.

On the other hand, you can also add ignoreOlderThan = stanza in inputs.conf file, with value mentioned as {number}{unit} (without brackets). For example, "7d" indicates one week. Valid units are "d" (days), "h" (hours), "m" (minutes), and "s" (seconds).

Let us know what worked for you so others visiting this post can learn/re-use.

Regards, Mitesh.

Re: Rolled over issue with same header

jitsinha — Mon, 25 May 2015 09:08:31 GMT

Thanks for your response. The system in place roll over files based on date/size, whichever is earlier.
Hence ignoreOlderThan will not work properly.

Sorry I might not be clear earlier.

The issues I am referring to is more related to header being same across all the files.

So for an example say I got one file A.log for today.

In this situation if I start to monitor the directory containing the file, Splunk will only pick A.log file and will index it.

But tomorrow when a new file will be created,today's file will be renamed to A.20150525.log but since the new file and old file are having same header the new file will be ignored and for that matter all the files for all the consecutive days.

Now the indexing issue has been fixed by setting initCrcLength = 1000, but Splunk has reindexed all the older files again.

How to stop this re-indexing??