Getting Data In

Use transforms.conf to share LINE_BREAKER and TIME_ settings for indexer?

richard_g_curry
Explorer

I am working on a large set of log that Splunk will monitor for a 3rd party app which has nearly 2 dozen logs to be monitored. Many of these logs have similar formatting so my props.conf for the Indexers have a lot of repetition. I know I can use a transforms.conf with props.conf on my Search Heads to share common field extractions but I don't see anything in the doc that specifically says I can do this for LINE_BREAKER or TIME_ formatting parameters. Clearly it is because these settings are not valid in the transforms.conf.

Is it possible that this could be setup in a future version of Splunk that these parameters could be shared rather than having them repeated several times on the file? This could be done either by providing equivalent settings in transforms.conf (but that doesn't really make logical sense as you consider what the transforms.conf is meant to do) or by allow for a "global" in the props.conf file. The latter does seem like the more logical possibility. What does the Splunk gurus think?

0 Karma
1 Solution

somesoni2
Revered Legend

As far as I know transforms.conf doesn't contain attributes to handle event-breaks/timestamp identification.

In props.conf file, there is a <default> stanza. This is the place where you can define the global eventbreak/timestamp related configuration which will be applied to all the sourcetypes, unless sourcetype overrides it.

I did some testing and it does work. (see example below) Any data input configured with any of the sourcetype mentioned below now follows the settings defined in <default> stanza.

Note: Since you've changed the default splunk setting, you need to ensure that any new sourcetype which don't follow these setting should have explicit configuration defined.

e.g.
props.conf

    [default]
    SHOULD_LINEMERGE = True
    BREAK_ONLY_BEFORE = \<Data\>
    TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q

[deviceinfo]
NO_BINARY_CHECK = 1
pulldown_type = 1

[errorlog]
NO_BINARY_CHECK = 1
pulldown_type = 1

[apachelog]
NO_BINARY_CHECK = 1
pulldown_type = 1

[exchangelog]
NO_BINARY_CHECK = 1
pulldown_type = 1

[commonlog]
NO_BINARY_CHECK = 1
pulldown_type = 1

View solution in original post

somesoni2
Revered Legend

As far as I know transforms.conf doesn't contain attributes to handle event-breaks/timestamp identification.

In props.conf file, there is a <default> stanza. This is the place where you can define the global eventbreak/timestamp related configuration which will be applied to all the sourcetypes, unless sourcetype overrides it.

I did some testing and it does work. (see example below) Any data input configured with any of the sourcetype mentioned below now follows the settings defined in <default> stanza.

Note: Since you've changed the default splunk setting, you need to ensure that any new sourcetype which don't follow these setting should have explicit configuration defined.

e.g.
props.conf

    [default]
    SHOULD_LINEMERGE = True
    BREAK_ONLY_BEFORE = \<Data\>
    TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q

[deviceinfo]
NO_BINARY_CHECK = 1
pulldown_type = 1

[errorlog]
NO_BINARY_CHECK = 1
pulldown_type = 1

[apachelog]
NO_BINARY_CHECK = 1
pulldown_type = 1

[exchangelog]
NO_BINARY_CHECK = 1
pulldown_type = 1

[commonlog]
NO_BINARY_CHECK = 1
pulldown_type = 1

richard_g_curry
Explorer

All but [exchangelog] will use the [default] settings and it will use those defined with its stanza. Cool.

And thanks! (8-D)

Rick

richard_g_curry
Explorer

[default]
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE = <Data>
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3Q

[deviceinfo]
NO_BINARY_CHECK = 1
pulldown_type = 1

[errorlog]
NO_BINARY_CHECK = 1
pulldown_type = 1

[apachelog]
NO_BINARY_CHECK = 1
pulldown_type = 1

[exchangelog]
SHOULD_LINEMERGE = False
LINE_BREAKER = ([\r\n]+)\d{2}-\d{2}-\d{4}\s+\d{2}:
TIME_PREFIX = ^\w+:\s+
TIME_FORMAT = %m-%d-%Y %H:%M:%S,%3N
NO_BINARY_CHECK = 1
pulldown_type = 1

[commonlog]
NO_BINARY_CHECK = 1
pulldown_type = 1

SEE NEXT COMMENT

0 Karma

richard_g_curry
Explorer

I somehow missed the [default] spec option. Then again, I focused on what was in the doc for props.conf ( http://docs.splunk.com/Documentation/Splunk/6.1.2/Admin/Propsconf?r=searchtip ). So using your example, let's assume that the [exchangelog] required different LINE_BREAKER and TIME_FORMAT options, then it would look like this:
(SEE NEXT COMMENT)

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Yeah, transforms.conf can't work for this because it's applied after event breaking and line merging in the indexing pipeline - too late then.

Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...