Use transforms.conf to share LINE_BREAKER and TIME_ settings for indexer?

[richard_g_curry]

I am working on a large set of log that Splunk will monitor for a 3rd party app which has nearly 2 dozen logs to be monitored. Many of these logs have similar formatting so my props.conf for the Indexers have a lot of repetition. I know I can use a transforms.conf with props.conf on my Search Heads to share common field extractions but I don't see anything in the doc that specifically says I can do this for LINE_BREAKER or TIME_ formatting parameters. Clearly it is because these settings are not valid in the transforms.conf.

Is it possible that this could be setup in a future version of Splunk that these parameters could be shared rather than having them repeated several times on the file? This could be done either by providing equivalent settings in transforms.conf (but that doesn't really make logical sense as you consider what the transforms.conf is meant to do) or by allow for a "global" in the props.conf file. The latter does seem like the more logical possibility. What does the Splunk gurus think?

[somesoni2 · Accepted solution]

As far as I know transforms.conf doesn't contain attributes to handle event-breaks/timestamp identification.

In props.conf file, there is a <default> stanza. This is the place where you can define the global eventbreak/timestamp related configuration which will be applied to all the sourcetypes, unless sourcetype overrides it.

I did some testing and it does work. (see example below) Any data input configured with any of the sourcetype mentioned below now follows the settings defined in <default> stanza.

Note: Since you've changed the default splunk setting, you need to ensure that any new sourcetype which don't follow these setting should have explicit configuration defined.

e.g.
props.conf

    [default]
    SHOULD_LINEMERGE = True
    BREAK_ONLY_BEFORE = \<Data\>
    TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q

[deviceinfo]
NO_BINARY_CHECK = 1
pulldown_type = 1

[errorlog]
NO_BINARY_CHECK = 1
pulldown_type = 1

[apachelog]
NO_BINARY_CHECK = 1
pulldown_type = 1

[exchangelog]
NO_BINARY_CHECK = 1
pulldown_type = 1

[commonlog]
NO_BINARY_CHECK = 1
pulldown_type = 1

[richard_g_curry]

All but [exchangelog] will use the [default] settings and it will use those defined with its stanza. Cool.

And thanks! (8-D)

Rick

[richard_g_curry]

[default]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = <Data>
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q

[deviceinfo]
NO_BINARY_CHECK = 1
pulldown_type = 1

[errorlog]
NO_BINARY_CHECK = 1
pulldown_type = 1

[apachelog]
NO_BINARY_CHECK = 1
pulldown_type = 1

[exchangelog]
SHOULD_LINEMERGE = False
LINE_BREAKER = ([\r\n]+)\d{2}-\d{2}-\d{4}\s+\d{2}:
TIME_PREFIX = ^\w+:\s+
TIME_FORMAT = %m-%d-%Y %H:%M:%S,%3N
NO_BINARY_CHECK = 1
pulldown_type = 1

[commonlog]
NO_BINARY_CHECK = 1
pulldown_type = 1

SEE NEXT COMMENT

[richard_g_curry]

I somehow missed the [default] spec option. Then again, I focused on what was in the doc for props.conf ( http://docs.splunk.com/Documentation/Splunk/6.1.2/Admin/Propsconf?r=searchtip ). So using your example, let's assume that the [exchangelog] required different LINE_BREAKER and TIME_FORMAT options, then it would look like this:
(SEE NEXT COMMENT)

[martin_mueller]

Yeah, transforms.conf can't work for this because it's applied after event breaking and line merging in the indexing pipeline - too late then.