topic Re: Cannot overwrite sourcetype and source from _raw in Getting Data In

Cannot overwrite sourcetype and source from _raw

seanwong — Mon, 28 Sep 2020 09:40:50 GMT

Hi All,

I'm having a transforms.conf and props.conf override issue.

inputs.conf:

[tcp://10000]

connection_host = dns

index = myindex

props.conf:

[source::tcp:10000]

MAX_EVENTS = 10000

TRUNCATE = 100000

BREAK_ONLY_BEFORE = ^host

TRANSFORMS-all=setHost, setSource, setSourceType

transforms.conf:

[setHost]

DEST_KEY = MetaData:Host

REGEX = ^host=([a-z0-9-]+)$

FORMAT = host::$1

[setSource]

SOURCE_KEY = _raw

DEST_KEY = MetaData:Source

REGEX = ^source=(.*)$

FORMAT = source::$1

[setSourceType]

SOURCE_KEY = _raw

DEST_KEY = MetaData:Sourcetype

REGEX = ^sourcetype=(.*)$

FORMAT = sourcetype::$1

So, the transformation setHost gets applied, but setSource and setSourceType doesnt.

Any ideas?

dshpritz — Mon, 28 Sep 2020 09:40:53 GMT

Splunk is treating the data in _raw as one large string. Instead of using the "^" with the regexes, try using "\n", so:

[setSourceType]
SOURCE_KEY = __raw

DEST_KEY = MetaData:Sourcetype

REGEX = \nsourcetype=(.*)$

FORMAT = sourcetype::$1

seanwong — Thu, 16 Jun 2011 02:39:36 GMT

With the explanation of it being treated as one large string, i then assumed splunk might treating it as a literal string ''.

Just in case the greedy quantifier of * was eating too much, i also modified my regex to be:

REGEX = \nsource=([a-zA-Z0-9-.]+)

Thanks dshpritz!