Getting Data In
Highlighted

How to only index events that contain specific fields?

Communicator

Hello, all.

I know that my question's not a unique, but I want to ask it 🙂
I have a netflow text log on a server with a universal forwarder installed.

I don't want to index this entire log. I only want to index fields containing a certain key. For example, I can provide a few strings:

{"timestamp":"2016-11-22T15:42:17.037821+0300","flow_id":268878859621513,"event_type":"netflow","src_ip":"11.11.11.11","src_port":22,"dest_ip":"22.22.22.22","dest_port":44206,"proto":"TCP","app_proto":"ssh","netflow":{"pkts":8,"bytes":2230,"start":"2016-11-22T15:41:14.611465+0300","end":"2016-11-22T15:41:14.638311+0300","age":0},"tcp":{"tcp_flags":"1a","syn":true,"psh":true,"ack":true}}
{"timestamp":"2016-11-22T15:44:18.013133+0300","flow_id":720902685008782,"event_type":"netflow","src_ip":"157.55.130.156","src_port":40032,"dest_ip":"22.22.22.22","dest_port":3166,"proto":"UDP","netflow":{"pkts":2,"bytes":126,"start":"2016-11-22T15:39:17.402318+0300","end":"2016-11-22T15:39:17.527073+0300","age":0}}
{"timestamp":"2016-11-22T15:44:16.025489+0300","flow_id":265292561318767,"event_type":"netflow","src_ip":"22.22.22.22","src_port":41979,"dest_ip":"33.33.33.33","dest_port":443,"proto":"TCP","app_proto":"tls","netflow":{"pkts":40,"bytes":14432,"start":"2016-11-22T15:41:05.983919+0300","end":"2016-11-22T15:43:14.286741+0300","age":129},"tcp":{"tcp_flags":"1b","syn":true,"fin":true,"psh":true,"ack":true}}

As you can see, we have a different field - proto and app_proto. I only want to index data with these specific fields in Splunk. For example, I only need events with proto":"TCP", or maybe proto":"TCP" and (or) app_proto":"ssh"

Can you help my with this case? I read the manual, but I can't understand the principle of the implementation of this.

Thanks!

Highlighted

Re: How to only index events that contain specific fields?

Legend

Hi templier,
for my knowledge, you can filter your events discarding those events that don't contain your strings, but it isn't possible take only a part of each event that contains one of your strings.
Every way to take only events that contain your strings, you have to configure:

props.conf

 [your_sourcetype]
 TRANSFORMS-set-nullqueue=set_nullqueue,set_OK

transforms.conf

 [set_nullqueue]
 REGEX=.
 DEST_KEY=queue
 FORMAT=nullQueue

 [set_OK]
 REGEX=regex1|regex2|regex3
 DEST_KEY = queue
 FORMAT = indexQueue

Bye.
Giuseppe

View solution in original post

Highlighted

Re: How to only index events that contain specific fields?

Communicator

Hello, Giuseppe.
Thx for you answer.

Tried do this, my files:
transforms.conf
[setnullqueue]
REGEX=\SUDP\S
DEST
KEY=UDP
FORMAT=nullQueue

[set_OK]
REGEX=\S*ssh\S*
DEST_KEY = queue
FORMAT = indexQueue

props.conf unchanged, only set my sourcetype

And nothing new in result. I write to indexer lines contained UDP

0 Karma
Highlighted

Re: How to only index events that contain specific fields?

Contributor

A space in regex is \s not \S, try replacing that.

0 Karma
Highlighted

Re: How to only index events that contain specific fields?

Communicator

Try, nothing new

0 Karma
Highlighted

Re: How to only index events that contain specific fields?

Legend

What do you mean when you say props.conf unchanged: Do you used my props.conf?

Male this test inverting Order in TRANSFORMS command TRANSFORMS-set-nullqueue=setOK,setnullqueue

Are you sure of your regex?

Bye.
Giuseppe

0 Karma
Highlighted

Re: How to only index events that contain specific fields?

Communicator

I test it on splunk field extraction - work great.
Maybe solution in uninstall universal forwarder and install a heavy forwarder?

0 Karma
Highlighted

Re: How to only index events that contain specific fields?

Communicator

Exactly, this needs to be done on a heavy forwarder.

If interested, I would adjust the regular expression:

transforms.conf
[setnull]
REGEX = (\"proto\":\"UDP\")
DEST_KEY = queue
FORMAT = nullQueue

[setok]
REGEX = (\"proto\":\"TCP\")|(\"app_proto\":\"ssh\")
DEST_KEY = queue
FORMAT = nullQueue

props.conf
[your_sourcetype]
TRANSFORMS-set = setnull, setok

Rodrigo Ribeiro

0 Karma
Highlighted

Re: How to only index events that contain specific fields?

Legend

Hi templier,
Filtering using props.conf and transforms.conf it's a part of parsing phase done on your indexers.
It isn't a good idea to use an heavy forwarder in all your servers!
Bye.
Giuseppe

0 Karma
Highlighted

Re: How to only index events that contain specific fields?

Communicator

Perfect Cusello.

As informed by our colleague cusello, can be done by indexers, but in fact can not be done in a universal forwarder 🙂

Tks

Rodrigo Ribeiro

0 Karma