Cannot overwrite sourcetype and source from _raw

[seanwong]

Hi All,

I'm having a transforms.conf and props.conf override issue.

inputs.conf:

[tcp://10000]

connection_host = dns

index = myindex

props.conf:

[source::tcp:10000]

MAX_EVENTS = 10000

TRUNCATE = 100000

BREAK_ONLY_BEFORE = ^host

TRANSFORMS-all=setHost, setSource, setSourceType

transforms.conf:

[setHost]

DEST_KEY = MetaData:Host

REGEX = ^host=([a-z0-9-]+)$

FORMAT = host::$1

[setSource]

SOURCE_KEY = _raw

DEST_KEY = MetaData:Source

REGEX = ^source=(.*)$

FORMAT = source::$1

[setSourceType]

SOURCE_KEY = _raw

DEST_KEY = MetaData:Sourcetype

REGEX = ^sourcetype=(.*)$

FORMAT = sourcetype::$1

So, the transformation setHost gets applied, but setSource and setSourceType doesnt.

Any ideas?

data is being sent via tcpsocket and a sample is like so:

host=test-devdb01

sourcetype=SESSIONS

source=myscript.sh

test-devdb01|itmscmd|SESSIONS|ACTIVE=1

test-devdb01|itmscmd|SESSIONS|ACTIVE=1

test-devdb01|itmscmd|SESSIONS|ACTIVE=1

test-devdb01|itmscmd|SESSIONS|ACTIVE=1

test-devdb01|itmscmd|SESSIONS|ACTIVE=1

test-devdb01|itmscmd|SESSIONS|ACTIVE=1

host=test-devdb01 Options| sourcetype=tcp-raw Options| source=tcp:1567 Options

[dshpritz · Accepted solution]

Splunk is treating the data in _raw as one large string. Instead of using the "^" with the regexes, try using "\n", so:

[setSourceType]
SOURCE_KEY = __raw

DEST_KEY = MetaData:Sourcetype

REGEX = \nsourcetype=(.*)$

FORMAT = sourcetype::$1

[seanwong]

With the explanation of it being treated as one large string, i then assumed splunk might treating it as a literal string ''.

Just in case the greedy quantifier of * was eating too much, i also modified my regex to be:

REGEX = \nsource=([a-zA-Z0-9-.]+)

Thanks dshpritz!