topic Re: How to only index events that contain specific fields? in Getting Data In

How to only index events that contain specific fields?

templier — Tue, 22 Nov 2016 13:03:28 GMT

Hello, all.

I know that my question's not a unique, but I want to ask it 🙂
I have a netflow text log on a server with a universal forwarder installed.

I don't want to index this entire log. I only want to index fields containing a certain key. For example, I can provide a few strings:

{"timestamp":"2016-11-22T15:42:17.037821+0300","flow_id":268878859621513,"event_type":"netflow","src_ip":"11.11.11.11","src_port":22,"dest_ip":"22.22.22.22","dest_port":44206,"proto":"TCP","app_proto":"ssh","netflow":{"pkts":8,"bytes":2230,"start":"2016-11-22T15:41:14.611465+0300","end":"2016-11-22T15:41:14.638311+0300","age":0},"tcp":{"tcp_flags":"1a","syn":true,"psh":true,"ack":true}}
{"timestamp":"2016-11-22T15:44:18.013133+0300","flow_id":720902685008782,"event_type":"netflow","src_ip":"157.55.130.156","src_port":40032,"dest_ip":"22.22.22.22","dest_port":3166,"proto":"UDP","netflow":{"pkts":2,"bytes":126,"start":"2016-11-22T15:39:17.402318+0300","end":"2016-11-22T15:39:17.527073+0300","age":0}}
{"timestamp":"2016-11-22T15:44:16.025489+0300","flow_id":265292561318767,"event_type":"netflow","src_ip":"22.22.22.22","src_port":41979,"dest_ip":"33.33.33.33","dest_port":443,"proto":"TCP","app_proto":"tls","netflow":{"pkts":40,"bytes":14432,"start":"2016-11-22T15:41:05.983919+0300","end":"2016-11-22T15:43:14.286741+0300","age":129},"tcp":{"tcp_flags":"1b","syn":true,"fin":true,"psh":true,"ack":true}}

As you can see, we have a different field - proto and app_proto. I only want to index data with these specific fields in Splunk. For example, I only need events with proto":"TCP", or maybe proto":"TCP" and (or) app_proto":"ssh"

Can you help my with this case? I read the manual, but I can't understand the principle of the implementation of this.

Thanks!

Re: How to only index events that contain specific fields?

gcusello — Tue, 22 Nov 2016 13:11:22 GMT

Hi templier,
for my knowledge, you can filter your events discarding those events that don't contain your strings, but it isn't possible take only a part of each event that contains one of your strings.
Every way to take only events that contain your strings, you have to configure:

props.conf

 [your_sourcetype]
 TRANSFORMS-set-nullqueue=set_nullqueue,set_OK

transforms.conf

 [set_nullqueue]
 REGEX=.
 DEST_KEY=queue
 FORMAT=nullQueue

 [set_OK]
 REGEX=regex1|regex2|regex3
 DEST_KEY = queue
 FORMAT = indexQueue

Bye.
Giuseppe

Re: How to only index events that contain specific fields?

templier — Tue, 29 Sep 2020 11:54:39 GMT

Hello, Giuseppe.
Thx for you answer.

Tried do this, my files:
transforms.conf
[set_nullqueue]
REGEX=\S*UDP\S*
DEST_KEY=UDP
FORMAT=nullQueue

[set_OK]
REGEX=\S*ssh\S*
DEST_KEY = queue
FORMAT = indexQueue

props.conf unchanged, only set my sourcetype

And nothing new in result. I write to indexer lines contained UDP

Re: How to only index events that contain specific fields?

gcusello — Tue, 29 Sep 2020 11:54:42 GMT

What do you mean when you say props.conf unchanged: Do you used my props.conf?

Male this test inverting Order in TRANSFORMS command TRANSFORMS-set-nullqueue=set_OK,set_nullqueue

Are you sure of your regex?

Bye.
Giuseppe

Re: How to only index events that contain specific fields?

mrgibbon — Tue, 22 Nov 2016 22:36:31 GMT

A space in regex is \s not \S, try replacing that.

Re: How to only index events that contain specific fields?

templier — Wed, 23 Nov 2016 09:36:25 GMT

I test it on splunk field extraction - work great.
Maybe solution in uninstall universal forwarder and install a heavy forwarder?

Re: How to only index events that contain specific fields?

templier — Wed, 23 Nov 2016 09:36:48 GMT

Try, nothing new

Re: How to only index events that contain specific fields?

rodrigorsilva — Wed, 23 Nov 2016 10:26:51 GMT

Exactly, this needs to be done on a heavy forwarder.

If interested, I would adjust the regular expression:

transforms.conf
[setnull]
REGEX = (\"proto\":\"UDP\")
DEST_KEY = queue
FORMAT = nullQueue

[setok]
REGEX = (\"proto\":\"TCP\")|(\"app_proto\":\"ssh\")
DEST_KEY = queue
FORMAT = nullQueue

props.conf
[your_sourcetype]
TRANSFORMS-set = setnull, setok

Rodrigo Ribeiro

Re: How to only index events that contain specific fields?

gcusello — Wed, 23 Nov 2016 10:46:43 GMT

Hi templier,
Filtering using props.conf and transforms.conf it's a part of parsing phase done on your indexers.
It isn't a good idea to use an heavy forwarder in all your servers!
Bye.
Giuseppe

Re: How to only index events that contain specific fields?

rodrigorsilva — Wed, 23 Nov 2016 11:03:59 GMT

Perfect Cusello.

As informed by our colleague cusello, can be done by indexers, but in fact can not be done in a universal forwarder 🙂

Tks

Rodrigo Ribeiro

Re: How to only index events that contain specific fields?

templier — Wed, 23 Nov 2016 13:25:39 GMT

Hello,
It's work.
And now I have more experience in this theme.
Can you to issue this post as an Answer, rather than a comment? I mark it 🙂
Many thanks.

Re: How to only index events that contain specific fields?

templier — Wed, 23 Nov 2016 13:27:15 GMT

Yeap, thanks to you and Giuseppe for information and live example of this solution

Re: How to only index events that contain specific fields?

gcusello — Wed, 23 Nov 2016 13:44:42 GMT

Hi templier,
Filtering using props.conf and transforms.conf it's a part of parsing phase done on your indexers.
It isn't a good idea to use an heavy forwarder in all your servers!
Bye.
Giuseppe

Re: How to only index events that contain specific fields?

gcusello — Wed, 23 Nov 2016 13:45:15 GMT

If you like, accept my answer.
Thank you.
Bye.
Giuseppe