topic Re: My Indexer is not processing "Keep specific events and discard the rest" in Splunk Dev

My Indexer is not processing "Keep specific events and discard the rest"

GersonGarcia — Thu, 31 Jan 2019 19:29:48 GMT

Hello all,
I have one app that generates a lot of data and it is killing my license. We need this data for sensitive customer only.
So, I followed
https://docs.splunk.com/Documentation/Splunk/6.5.2/Forwarding/Routeandfilterdatad
Keep specific events and discard the rest:
My props.conf is:

[metrics-app-gateway]
TRANSFORMS-set= setnull,setparsing

And trasnsforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX=lasconpr01vmw08.las.ssnsgs.net
DEST_KEY = queue
FORMAT = indexQueue

But for some reason it is not working. When I comment out #TRANSFORMS-set= in props.conf, I ingested the file properly.
How can I troubleshoot this? How come it is not passing by REGEX?

Thank you
Gerson Garcia.

Re: My Indexer is not processing "Keep specific events and discard the rest"

chrisyounger — Thu, 31 Jan 2019 19:36:47 GMT

Hi @GersonGarcia

Does the string lasconpr01vmw08.las.ssnsgs.net exist in the _raw message? This won't match the MetaData fields

Secondly, this props and transforms are parse time (unless using indexed extractions) that means they need to go on the Indexer, or heavy forwarder if the data goes through a heavy forwarder.

Hope this helps.

Re: My Indexer is not processing "Keep specific events and discard the rest"

GersonGarcia — Thu, 31 Jan 2019 19:43:09 GMT

Ah, that may be the reason. Is there any way I can use MetaData field something like:
REGEX=host::lasconpr01vmw08.las.ssnsgs.net

The props.conf and transforms.conf are in the Indexer

Thank you.

Re: My Indexer is not processing "Keep specific events and discard the rest"

chrisyounger — Thu, 31 Jan 2019 19:49:45 GMT

This should work

[host::lasconpr01vmw08.las.ssnsgs.net]
TRANSFORMS-set= setparsing

[host::<other hosts>]
TRANSFORMS-set= setnull

Note you will have to set the other_hosts yourself.

Re: My Indexer is not processing "Keep specific events and discard the rest"

GersonGarcia — Thu, 31 Jan 2019 20:04:46 GMT

But hold on, if I do that, I will exclude everything else (all apps and logs):

[host::<other hosts>]
TRANSFORMS-set= setnull

Right?

Re: My Indexer is not processing "Keep specific events and discard the rest"

chrisyounger — Thu, 31 Jan 2019 20:06:44 GMT

Yes that's right. So you should set it very carefully. Sorry I should have made that clearer.

Re: My Indexer is not processing "Keep specific events and discard the rest"

GersonGarcia — Thu, 31 Jan 2019 20:10:23 GMT

I can't do that, I just need to stop index this particular stanza [metrics-app-gateway] for all hosts except lasconpr01vmw08.las.ssnsgs.net.

Re: My Indexer is not processing "Keep specific events and discard the rest"

chrisyounger — Thu, 31 Jan 2019 20:17:06 GMT

OK give this a try. Use the configs from your original message but add SOURCE_KEY like so

[setparsing]
REGEX=lasconpr01vmw08.las.ssnsgs.net
DEST_KEY = queue
SOURCE_KEY = MetaData:Host
FORMAT = indexQueue

Re: My Indexer is not processing "Keep specific events and discard the rest"

GersonGarcia — Thu, 31 Jan 2019 21:37:43 GMT

It didn't work. It is not finding the REGEX. In the documentation I found:

Data:Host : The host associated with the event.
The value must be prefixed by "host::"

What does it mean?

In my inputs.conf I have:

[monitor:///usr/ssn/gateway/logs/metrics.log]
host = lasconpr01vmw08.las.ssnsgs.net
sourcetype = metrics-app-gateway
_meta = ssnservice::CONED-PROD01
index = ssn

Should I add:

[monitor:///usr/ssn/gateway/logs/metrics.log]
host = lasconpr01vmw08.las.ssnsgs.net
sourcetype = metrics-app-gateway
_meta = ssnservice::CONED-PROD01 host::lasconpr01vmw08.las.ssnsgs.net
index = ssn