Archive
Highlighted

Indexers props.conf

Builder

From Splunk it's said it's best to do your custom Field extractions at search time. So the only extractions you do on your indexers are date/time field extractions and what else?

Can someone provide me a well written, efficient props.conf file for your index time extractions for IIS or Tomcat sourcetype logs?

I'm pulling in HUGE iis logs from about 40 servers many logs larger that 1gb in size and we are noticing a delay of up to 2+ hours for those logs. I understand increasing pipelines on indexer or UFs, I also understand upping the maxkb limit in limits.conf

My purpose here is to determine if there's a more efficient way to get my data in via the configurations files. Am I doing something wrong or neglecting the best practices?

On my indexers props.conf
I have the timezone set to UTC and that's it. All my custom Field extractions are on my search heads via props and transforms. I'd imagine there's more i can do to help out my indexer via props.conf

Can someone provide an example and explain the key value pairs within their stanza. Thanks

Tags (1)
0 Karma
Highlighted

Re: Indexers props.conf

SplunkTrust
SplunkTrust

Hey @Jarohnimo,

Best way to find well written props.conf for any data source is to find the splunk built TA. Have a look here for IIS
https://splunkbase.splunk.com/app/3185/

And here for Tomcat:
https://splunkbase.splunk.com/app/2911/

You can grab the app and take the props.conf from there.

Cheers,
David

0 Karma
Highlighted

Re: Indexers props.conf

Builder

Thanks David, I have these currently set on my search heads but curious as to what makes sense to add explicitly to time date / parsing that may improve indexing.

Generally I don't want the entire app on my indexer as that will add to index time (slow resources)

Is setting the time zone all I need to do for these source types? There's a lot of options for time date field parsing

0 Karma
Highlighted

Re: Indexers props.conf

Builder

For example, I'm pulling huge logs. And I read that due to the large log set event processing can be clogged on the forwarders.

I saw this bit in an article I was reading

For optimal performance of your data, you can set the following settings for your sourcetype in props.conf:

DATETIMECONFIG
MAX
TIMESTAMPLOOKAHEAD
TIME
PREFIX
TIMEFORMAT
SHOULD
LINEMERGE
ANNOTATE_PUNCT

Should I be doing this on the inexers?

0 Karma
Highlighted

Re: Indexers props.conf

SplunkTrust
SplunkTrust

Yes! Exactly.
Those settings are referred to as the magic 6 and should be configured for all your sourcetypes.

So yeah make sure you have the six of them in props.conf and drop all the search time configs : TIMEPREFIX, TIMEFORMAT, MAXTIMESTAMPLOOKAHEAD, SHOULDLINEMERGE, LINEBREAKER and TRUNCATE.

0 Karma
Highlighted

Re: Indexers props.conf

SplunkTrust
SplunkTrust

@jarohnimo, do you need any more help on this issue ? If not could you please accept the answer to close it down ?

0 Karma