topic [Forwarder] extract field from flow tcp 514 in Splunk Search

[Forwarder] extract field from flow tcp 514

SuperMisterT — Wed, 11 Aug 2021 06:48:45 GMT

Hi,

I have a data stream on the forwarder, streaming on the 514. the data is correctly indexed. But I would like to extract/build some fields from the _raw.
In search head, i try with rex field. it works but it's too long for user.
So, i want to do it on forwarder before indexation.

Example:
_raw: <150> 2021-06-01: 00: 05: 12 localhost blue car=porsche,959 .....
i want build this fields for begining:
carbrand : porsche

inputs.conf
[tcp://my_hostname_client:514]
index = car_park
sourcetype = sale

First WAY: only in props.conf
[sale] # i try something
EXTRACT-testsale = ^.*car=(?<carbrand>.*)\,$

Second WAY: props + transforms

In props.conf
[sale]
REPORT-testsale = extract-cardata

And in transforms.conf
[extract-cardata]
REGEX = ^.*car=(.*)\,$
FORMAT = carbrand::$1

So, is-it possible to extract field in the _raw on the forwarder from tcp flow 514 ?
If yes, where are my mistakes in my conf?

Thks for your returns and help.
Best regards.

Re: [Forwarder] extract field from flow tcp 514

venkatasri — Wed, 11 Aug 2021 09:40:46 GMT

@SuperMisterT

Your rex is not extracting as i have tested you can replace it with following , you can test your rex using this link - https://regex101.com/r/lz6C69/1

And i noticed car= is KV pair splunk shall Auto extract , have you tried searching in verbose mode? Otherwise try below.

[sale] # i try something EXTRACT-testsale = ^.*car=(?<carbrand>[^,]+)

First WAY: props conf should work fine however you should deploy it your SH as EXTRACT is search-time conf.

---

Hope this helps!

Re: [Forwarder] extract field from flow tcp 514

SuperMisterT — Wed, 11 Aug 2021 09:48:33 GMT

Thanks for your answer.

In fact my example sample wasn't my real data.

My _raw is like a long string and i try to extract some parts receive by udp 514.

I want extract and build a custom field at index time (so on the forwarder).

i follow this doc https://docs.splunk.com/Documentation/Splunk/8.2.1/Data/Configureindex-timefieldextraction it doesnt work.

I try a simple rex:

REGEX = (\d)

FORMAT = my_test::"$1"

Damned

Re: [Forwarder] extract field from flow tcp 514

venkatasri — Wed, 11 Aug 2021 10:02:21 GMT

@SuperMisterT

Deploy following conf to indexer/Heavy-forwarder replace the regex that i have placed with working one.

#prop. conf [sales] TRANSFORM-extract = extract-car #transforms.conf [extract-car] REGEX = ^.*car=(?<carbrand>[^,]+) FORMAT = carbrand::$1

Re: [Forwarder] extract field from flow tcp 514

SuperMisterT — Thu, 12 Aug 2021 07:21:05 GMT

Sorry,

I try to place this syntaxe:

#prop. conf
[sales]
TRANSFORMS-extract = extract-car # you wrote a typo 🙂

#transforms.conf
[extract-car]
REGEX = (?<carbrand>\d) # very simple regexp just to try new field's creation (the string has some digits)
FORMAT = carbrand::$1

I delete data in index and sourcetype.
I restart splunkd on the forwarder. Done.

In searchhead, i've some events but the new field doesn't appaer. No error in log.
😞

Re: [Forwarder] extract field from flow tcp 514

isoutamo — Thu, 12 Aug 2021 10:03:30 GMT

I think that you should add also WRITE_META = true to transforms.conf. Also you should add fields.conf to your SH nodes.

https://docs.splunk.com/Documentation/Splunk/8.2.1/Data/Configureindex-timefieldextraction

r. Ismo

Re: [Forwarder] extract field from flow tcp 514

venkatasri — Thu, 12 Aug 2021 10:27:40 GMT

Can you try this make sure your regex works, you can try using <your_search> | rex "(?<carbrand>\d)"

#prop. conf deploy to HF [sales] TRANSFORMS-extract = extract-car #transforms.conf deploy to HF [extract-car] REGEX = (?<carbrand>\d) FORMAT = carbrand::$1 WRITE_META = true #fields.conf deploy to search-head in distributed set-up [carbrand] INDEXED = true

Re: [Forwarder] extract field from flow tcp 514

SuperMisterT — Thu, 12 Aug 2021 12:14:39 GMT

Right, so i apply this code on the FW and SH.

But it doesnt work. The conf's indexer clustering simple.

I try on SH:

index = xyz sourcetype = test001 | rex field=_raw "(?<carbrand>\d)"

It works: i've a new field in "Interesting fields" 🙂

But, with props and transforms on FW and fields on SH, it doesn''t work. I can't create a new field on TCP 514 event at index time....

question: during an indexation (just before) of tcp event 514, _raw contains data or not ?

Any idea?

Re: [Forwarder] extract field from flow tcp 514

SuperMisterT — Thu, 12 Aug 2021 12:54:22 GMT

I add on SH this key. But no effect.

Re: [Forwarder] extract field from flow tcp 514

scelikok — Fri, 13 Aug 2021 04:44:31 GMT

Hİ @SuperMisterT,

You mean FW as Universal Forwarder or Heavy Forwarder? If it is Universal Forwarder, you should put these props.conf and transforms.conf settings to indexers.

Re: [Forwarder] extract field from flow tcp 514

SuperMisterT — Fri, 13 Aug 2021 07:45:08 GMT

SOLVED

I build an instance HeavyForwarder instead of Universal.

It works only on HF.

👏

Re: [Forwarder] extract field from flow tcp 514

venkatasri — Fri, 13 Aug 2021 08:14:32 GMT

@SuperMisterT yes as i mentioned in conf comments where to deploy what it works on HF or indexer..