topic How to check performance implications for props.conf changes (parsing and merging pipeline)? in Getting Data In

How to check performance implications for props.conf changes (parsing and merging pipeline)?

jnguy_qmulos — Mon, 22 Jul 2019 19:18:31 GMT

Is it possible to check the performance of the parsing and merging pipeline when making changes to props.conf for a particular source or sourcetype?

We currently only have line_breaker set for a particular source and would like to make recommendations to improve performance by including the props.conf changes that are part of Splunk's best practices like:

 LINE_BREAKER
 MAX_TIMESTAMP_LOOKAHEAD
 TIME_PREFIX
 TIME_FORMAT
 SHOULD_LINEMERGE
 TRUNCATE

I also looked at the MC/DMC under the Indexing tab but it wasn't much help.

I tried digging through metrics.log and ran a search like index=_internal host=indexer source="/opt/splunk/var/log/splunk/metrics.log" processor=linebreaker OR processor=aggregator
and I came up with some data I would possibly be interested in but there is no distinction of which source or sourcetype the info belongs to. I assume it's an aggregated number that includes all sources and sourcetypes.

07-22-2019 14:45:16.947 -0400 INFO Metrics - group=pipeline, name=parsing, processor=linebreaker, cpu_seconds=0, executes=97, cumulative_hits=1706601

I also ran a search for the source I was interested in (forescout) by running index=_internal host=indexer source="/opt/splunk/var/log/splunk/metrics.log" forescout and I came across logs from metrics.log that were part of the forescout index, source, and sourcetype. I saw groups like:

 per_index_thruput
 per_sourcetype_thruput
 per_source_thruput
 thruput

But I read from Splunk docs - Aboutmetricslog (I can't post links) that the thruput messages relate to the size of the "raw" items flowing through the data pipeline when it reaches the indexing pipeline, so this all takes place after the parsing and merging pipeline, so it's not of any help to me.

If anyone has any ideas, please let me know!

Thanks.

7/23 edit:
I came up with:

index=forescout 
| eval latency=(_indextime-_time) 
| eval day=strftime(_time,"%b/%d")
| stats avg(latency), min(latency), max(latency) BY day

It's not exactly what I'm looking for but I think it will provide some insight into what I am trying to achieve.

Re: How to check performance implications for props.conf changes (parsing and merging pipeline)?

woodcock — Sat, 27 Jul 2019 17:01:41 GMT

Check out this slide deck:
https://conf.splunk.com/files/2016/slides/jiffy-lube-quick-tune-up-for-your-splunk-environment.pdf

Re: How to check performance implications for props.conf changes (parsing and merging pipeline)?

jnguy_qmulos — Fri, 16 Aug 2019 13:02:45 GMT

The PDF you provided has some really good information. I am looking for something like the graph on slide 32, where we can maybe compare something like that before and after making the changes to props.conf for a particular sourcetype. Really want to see how/if performance improved from making the props.conf changes. I know Splunk says that it will but I was wondering if it was possible to come up with some actual metrics to back up the statement.

Re: How to check performance implications for props.conf changes (parsing and merging pipeline)?

woodcock — Fri, 16 Aug 2019 14:06:03 GMT

Contact the authors.

Re: How to check performance implications for props.conf changes (parsing and merging pipeline)?

jnguy_qmulos — Fri, 16 Aug 2019 14:19:34 GMT

I'll give that a shot, thanks!