topic Indexers props.conf in Getting Data In

Indexers props.conf

Jarohnimo — Sat, 10 Aug 2019 11:45:36 GMT

From Splunk it's said it's best to do your custom Field extractions at search time. So the only extractions you do on your indexers are date/time field extractions and what else?

Can someone provide me a well written, efficient props.conf file for your index time extractions for IIS or Tomcat sourcetype logs?

I'm pulling in HUGE iis logs from about 40 servers many logs larger that 1gb in size and we are noticing a delay of up to 2+ hours for those logs. I understand increasing pipelines on indexer or UFs, I also understand upping the maxkb limit in limits.conf

My purpose here is to determine if there's a more efficient way to get my data in via the configurations files. Am I doing something wrong or neglecting the best practices?

On my indexers props.conf
I have the timezone set to UTC and that's it. All my custom Field extractions are on my search heads via props and transforms. I'd imagine there's more i can do to help out my indexer via props.conf

Can someone provide an example and explain the key value pairs within their stanza. Thanks

Re: Indexers props.conf

DavidHourani — Sat, 10 Aug 2019 13:00:40 GMT

Hey @Jarohnimo,

Best way to find well written props.conf for any data source is to find the splunk built TA. Have a look here for IIS
https://splunkbase.splunk.com/app/3185/

And here for Tomcat:
https://splunkbase.splunk.com/app/2911/

You can grab the app and take the props.conf from there.

Cheers,
David

Re: Indexers props.conf

Jarohnimo — Sat, 10 Aug 2019 17:57:46 GMT

Thanks David, I have these currently set on my search heads but curious as to what makes sense to add explicitly to time date / parsing that may improve indexing.

Generally I don't want the entire app on my indexer as that will add to index time (slow resources)

Is setting the time zone all I need to do for these source types? There's a lot of options for time date field parsing

Re: Indexers props.conf

Jarohnimo — Wed, 30 Sep 2020 01:40:52 GMT

For example, I'm pulling huge logs. And I read that due to the large log set event processing can be clogged on the forwarders.

I saw this bit in an article I was reading

For optimal performance of your data, you can set the following settings for your sourcetype in props.conf:

DATETIME_CONFIG
MAX_TIMESTAMP_LOOKAHEAD
TIME_PREFIX
TIME_FORMAT
SHOULD_LINEMERGE
ANNOTATE_PUNCT

Should I be doing this on the inexers?

Re: Indexers props.conf

DavidHourani — Wed, 30 Sep 2020 01:42:36 GMT

Yes! Exactly.
Those settings are referred to as the magic 6 and should be configured for all your sourcetypes.

So yeah make sure you have the six of them in props.conf and drop all the search time configs : TIME_PREFIX, TIME_FORMAT, MAX_TIMESTAMP_LOOKAHEAD, SHOULD_LINEMERGE, LINE_BREAKER and TRUNCATE.

Re: Indexers props.conf

DavidHourani — Wed, 14 Aug 2019 07:01:59 GMT

@jarohnimo, do you need any more help on this issue ? If not could you please accept the answer to close it down ?