Getting Data In

Cannot overwrite sourcetype and source from _raw

seanwong
Explorer

Hi All,

I'm having a transforms.conf and props.conf override issue.

inputs.conf:

[tcp://10000]

connection_host = dns

index = myindex

props.conf:

[source::tcp:10000]

MAX_EVENTS = 10000

TRUNCATE = 100000

BREAK_ONLY_BEFORE = ^host

TRANSFORMS-all=setHost, setSource, setSourceType

transforms.conf:

[setHost]

DEST_KEY = MetaData:Host

REGEX = ^host=([a-z0-9-]+)$

FORMAT = host::$1

[setSource]

SOURCE_KEY = _raw

DEST_KEY = MetaData:Source

REGEX = ^source=(.*)$

FORMAT = source::$1

[setSourceType]

SOURCE_KEY = _raw

DEST_KEY = MetaData:Sourcetype

REGEX = ^sourcetype=(.*)$

FORMAT = sourcetype::$1

So, the transformation setHost gets applied, but setSource and setSourceType doesnt.

Any ideas?

data is being sent via tcpsocket and a sample is like so:

host=test-devdb01

sourcetype=SESSIONS

source=myscript.sh

test-devdb01|itmscmd|SESSIONS|ACTIVE=1

test-devdb01|itmscmd|SESSIONS|ACTIVE=1

test-devdb01|itmscmd|SESSIONS|ACTIVE=1

test-devdb01|itmscmd|SESSIONS|ACTIVE=1

test-devdb01|itmscmd|SESSIONS|ACTIVE=1

test-devdb01|itmscmd|SESSIONS|ACTIVE=1

host=test-devdb01 Options| sourcetype=tcp-raw Options| source=tcp:1567 Options

0 Karma
1 Solution

dshpritz
SplunkTrust
SplunkTrust

Splunk is treating the data in _raw as one large string. Instead of using the "^" with the regexes, try using "\n", so:

[setSourceType]
SOURCE_KEY = __raw

DEST_KEY = MetaData:Sourcetype

REGEX = \nsourcetype=(.*)$

FORMAT = sourcetype::$1

View solution in original post

dshpritz
SplunkTrust
SplunkTrust

Splunk is treating the data in _raw as one large string. Instead of using the "^" with the regexes, try using "\n", so:

[setSourceType]
SOURCE_KEY = __raw

DEST_KEY = MetaData:Sourcetype

REGEX = \nsourcetype=(.*)$

FORMAT = sourcetype::$1

seanwong
Explorer

With the explanation of it being treated as one large string, i then assumed splunk might treating it as a literal string ''.

Just in case the greedy quantifier of * was eating too much, i also modified my regex to be:

REGEX = \nsource=([a-zA-Z0-9-.]+)

Thanks dshpritz!

0 Karma
Get Updates on the Splunk Community!

Updated Data Type Articles, Anniversary Celebrations, and More on Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

A Prelude to .conf25: Your Guide to Splunk University

Heading to Boston this September for .conf25? Get a jumpstart by arriving a few days early for Splunk ...

4 Ways the Splunk Community Helps You Prepare for .conf25

.conf25 is right around the corner, and whether you’re a first-time attendee or a seasoned Splunker, the ...