[Forwarder] extract field from flow tcp 514

[SuperMisterT]

Hi,

I have a data stream on the forwarder, streaming on the 514. the data is correctly indexed. But I would like to extract/build some fields from the _raw.
In search head, i try with rex field. it works but it's too long for user.
So, i want to do it on forwarder before indexation.

Example:
_raw: <150> 2021-06-01: 00: 05: 12 localhost blue car=porsche,959 .....
i want build this fields for begining:
carbrand : porsche


inputs.conf
[tcp://my_hostname_client:514]
index = car_park
sourcetype = sale

First WAY: only in props.conf
[sale] # i try something
EXTRACT-testsale = ^.*car=(?<carbrand>.*)\,$

Second WAY: props + transforms

In props.conf
[sale]
REPORT-testsale = extract-cardata

And in transforms.conf
[extract-cardata]
REGEX = ^.*car=(.*)\,$
FORMAT = carbrand::$1


So, is-it possible to extract field in the _raw on the forwarder from  tcp flow 514 ?
If yes, where are my mistakes in my conf?

Thks for your returns and help.
Best regards.

[scelikok]

Hİ @SuperMisterT,

You mean FW as Universal Forwarder or Heavy Forwarder? If it is Universal Forwarder, you should put these props.conf and transforms.conf settings to indexers.

[venkatasri]

@SuperMisterT 

Your rex is not extracting as i have tested you can replace it with following , you can test your rex using this link - https://regex101.com/r/lz6C69/1

And i noticed car= is KV pair splunk shall Auto extract , have you tried searching in verbose mode? Otherwise try below.

[sale] # i try something
EXTRACT-testsale = ^.*car=(?<carbrand>[^,]+)

First WAY: props conf  should work fine however you should deploy it your SH as EXTRACT is search-time conf.

---

Hope this helps!

[SuperMisterT]

Thanks for your answer.

In fact my example sample wasn't my real data.

My _raw is like a long string and i try to extract some parts receive by udp 514.

I want extract and build a custom field at index time (so on the forwarder).

i follow this doc https://docs.splunk.com/Documentation/Splunk/8.2.1/Data/Configureindex-timefieldextraction it doesnt work.

I try a simple rex:

REGEX = (\d)

FORMAT = my_test::"$1"   

Damned

[venkatasri]

@SuperMisterT 

Deploy following conf to indexer/Heavy-forwarder replace the regex that i have placed with working one.

#prop. conf
[sales]
TRANSFORM-extract = extract-car

#transforms.conf
[extract-car]
REGEX = ^.*car=(?<carbrand>[^,]+)
FORMAT = carbrand::$1

[SuperMisterT]

Sorry,

I try to place this syntaxe:

#prop. conf
[sales]
TRANSFORMS-extract = extract-car # you wrote a typo 🙂

#transforms.conf
[extract-car]
REGEX = (?<carbrand>\d) # very simple regexp just to try new field's creation (the string has some digits)
FORMAT = carbrand::$1

I delete data in index and sourcetype.
I restart splunkd on the forwarder. Done.

In searchhead, i've some events but the new field doesn't appaer. No error in log.
😞

[venkatasri]

Can you try this make sure your regex works, you can try using <your_search> | rex "(?<carbrand>\d)"

#prop. conf deploy to HF
[sales]
TRANSFORMS-extract = extract-car

#transforms.conf deploy to HF
[extract-car]
REGEX = (?<carbrand>\d)
FORMAT = carbrand::$1
WRITE_META = true

#fields.conf deploy to search-head in distributed set-up
[carbrand]
INDEXED = true

[SuperMisterT]

Right, so i apply this code on the FW and SH.

But it doesnt work. The conf's indexer clustering simple.

I try on SH:

index = xyz sourcetype = test001 | rex field=_raw  "(?<carbrand>\d)"

It works: i've a new field in "Interesting fields" 🙂

But, with props and transforms on FW and fields on SH, it doesn''t work. I can't create a new field on TCP 514 event at index time....

question: during an indexation (just before) of tcp event 514, _raw contains data  or not ?

Any idea?

[isoutamo]

Hi

I think that you should add also WRITE_META = true to transforms.conf. Also you should add fields.conf to your SH nodes.

https://docs.splunk.com/Documentation/Splunk/8.2.1/Data/Configureindex-timefieldextraction

r. Ismo

[SuperMisterT]

I add on SH this key. But no effect.

[SuperMisterT]

SOLVED

I build an instance HeavyForwarder instead of Universal.

It works only on HF.

👏

[venkatasri]

@SuperMisterT yes as i mentioned in conf comments where to deploy what it works on HF or indexer..