whole file is re-indexed on addition of new events

[newbie2tech]

Hi All,

I need to monitor a log file which is pipe delimited, events get written onto log as the day progresses.

On my sandbox am trying to mimic the addition of new events by adding some events manually using vi editor, vi the file , add some lines at the bottom and save it [:wq!] , however this process is re-indexing the complete file.

I am using initCrcLength = 2500 in my props.conf. I am not using crcSALT and my log does not have any header

Any suggestions on how to avoid the file from completely re-indexing

Below is the log sample.

hostname|cluster_name|11/26/17 00:43:19|AB- 1|INFO| Retail.getCategoryListCodesFromProperties() retail Code List to show the link ::[02756, 2127]
hostname|cluster_name|11/26/17 00:49:28|AB-No Memory|object|||||||123467|123123123|01
hostname|cluster_name|11/26/17 00:51:42|AB-No Memory|object|||||||123455|123123123|00
hostname|cluster_name|11/26/17 01:04:28|AB-No Memory|object|||||||111111|123123123|01
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Header><wsse:Security xmlns:wsse="http://docs.basis-open.org/wss/2004/01/basis-2011-wss-wss-secext-1.0.xsd"><wsse:UsernameToken xmlns:wsse="http://docs.basis-open.org/wss/2004/01/basis-2011-wss-wss-secext-1.0.xsd" xmlns:wsu="http://docs.basis-open.org/wss/2004/01/basis-2011-wss-wss-utility-1.0.xsd"></soapenv:Body></soapenv:Envelope>
hostname|cluster_name|11/26/17 01:06:42|AB-No Memory|object|||||||222222|123123123|00
hostname|cluster_name|11/26/17 01:19:28|AB-No Memory|object|||||||333333|123123123|01
hostname|cluster_name|11/26/17 01:21:42|AB-No Memory|object|||||||555555|123123123|10
hostname|cluster_name|11/26/17 01:34:28|AB-No Memory|object|||||||777777|123123123|11
hostname|cluster_name|11/26/17 01:36:42|AB-No Memory|object|||||||111111|123123123|10

[sandyIscream]

Could you please share your inputs.conf

But my suggestion is first change the CrcSalt text into something else and then push the changes again from the deployment server.

If this also doesn't work then delete your fishbucket of your forwarder and check

[skoelpin]

The issue is with your fishbucket on the forwarder..

https://www.splunk.com/blog/2008/08/14/what-is-this-fishbucket-thing.html

[traxxasbreaker]

The initCrcLength setting is known to cause data to be reindexed when it is applied. Did you only see it reindexed once right after applying that update, or do you see it reindexed every time an event is added?

[newbie2tech]

i was having same issue even before introducing the initCrclength, as it was re-indexing whole file i tried initCrclength to avoid it but it did not help.