topic Re: Indexing time is too long in Getting Data In

Indexing time is too long

Kawtar — Tue, 29 Sep 2020 19:46:36 GMT

Hello,

In my props.conf, I added , BREAK_ONLY_BEFORE= regex AND LINE_BREAKER_REGEX , and I see that time of indexing is too long, the universal forwarder detect the files but it index it 4 ou 5 min after, but when I removed BREAK_ONLY_BEFORE and LINE_BREAKER_REGEX from the config file: props.conf, It indexed very quick . Any explications plz ?

Thank you

Re: Indexing time is too long

somesoni2 — Tue, 29 May 2018 17:51:03 GMT

It means your event parsing configurations are no efficient and causing delay in indexing (mostly delay in parsing layer). Please share your current props.conf entry for your sourcetype (assuming it was setup in Index/heavy forwarder) and some sample log entries. Based on that, Splunker's here can suggest you efficient config to put in your props.conf.

Re: Indexing time is too long

richgalloway — Tue, 29 Sep 2020 19:46:45 GMT

How complex are your regex strings? Have you tried only one of BREAK_ONLY_BEFORE and LINE_BREAKER_REGEX?

Re: Indexing time is too long

Kawtar — Fri, 01 Jun 2018 09:39:59 GMT

Thank you for your response I'll share the config

Re: Indexing time is too long

Kawtar — Tue, 29 Sep 2020 19:45:06 GMT

I noticed when I use BREAK_ONLY_BEFORE Indexing time is too long but when I use LINE_BREAKER_REGEX fast than the parameter break_only_before, can you confirm that ?

Re: Indexing time is too long

richgalloway — Tue, 29 Sep 2020 19:48:31 GMT

I haven't noticed this so I can't confirm it. I rarely use BREAK_ONLY_BEFORE so I can't say if it's slower than LINE_BREAKER. If only BREAK_ONLY_BEFORE works for your data and its performance is bad, I suggest you open a support case.