topic Re: Filter events for specific keywords in Getting Data In

Filter events for specific keywords

keishamtcs — Wed, 20 Feb 2019 14:30:11 GMT

Hi,

I have some set of events that has keywords like "inbound message" and "outbound message". the events looks something like this .

on the indexer side, i have created as such but it is not working. how to resolve this ?

In transforms :

[test]
REGEX = Inbound
DEST_KEY = queue
FORMAT = indexQueue

props

[testsource]
TRANSFORMS-set= test

Re: Filter events for specific keywords

markusspitzli — Wed, 20 Feb 2019 14:48:34 GMT

Hey.

You better put everything else into the nullQueue. I tested this with the following testdata:

2019-02-20 14:12:45.642 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Inbound Message.
2019-02-20 14:12:45.643 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Outbound Message.
2019-02-20 14:12:45.647 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | gugus Message.
2019-02-20 14:12:45.644 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Inbound Message.
2019-02-20 14:12:45.645 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Outbound Message.
2019-02-20 14:12:45.648 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | gugus Message.
2019-02-20 14:12:45.646 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Inbound Message.

transforms.conf

[test]
REGEX = ^((?!Inbound Message|Outbound Message).)*$
DEST_KEY=queue
FORMAT=nullQueue

props.conf

[testsource]
SHOULD_LINEMERGE = false
TRUNCATE = 100000
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
TIME_FORMAT=%Y-%m-%d %T.%3N
TIME_PREFIX=^
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}
TRANSFORMS-set = test

Re: Filter events for specific keywords

keishamtcs — Wed, 20 Feb 2019 15:00:15 GMT

Hi,

The problem is those events which i don't need does not have any proper syntax or keyword.
I will need to filter only with events that has Inbound Message or Outbound Message.

Re: Filter events for specific keywords

markusspitzli — Wed, 20 Feb 2019 15:15:33 GMT

Sorry I put the wrong regex. You have to put everything in the nullQueue exept of the Inbound or Outbound Message like this regex:
^((?!Inbound Message|Outbound Message).)*$

Re: Filter events for specific keywords

keishamtcs — Wed, 20 Feb 2019 16:25:58 GMT

Hi,

It is not working. It is still indexing all the data.

Re: Filter events for specific keywords

markusspitzli — Wed, 20 Feb 2019 16:31:06 GMT

could you please add some example logs? thx

Re: Filter events for specific keywords

ddrillic — Wed, 20 Feb 2019 16:40:59 GMT

Maybe -

REGEX = (Inbound Message|Outbound Message)

Re: Filter events for specific keywords

markusspitzli — Wed, 20 Feb 2019 21:24:43 GMT

I just tried it with a few sample logs. See the updated config from my original comment.

Re: Filter events for specific keywords

lakshman239 — Thu, 21 Feb 2019 11:30:58 GMT

Pls change the filters as below

In transforms :

# send everything to null queue except the ones we want
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = (Inbound|inbound|Outbound|outbound)
DEST_KEY = queue
FORMAT = indexQueue

props

[testsource]
TRANSFORMS-set= setnull, setparsing

Re: Filter events for specific keywords

keishamtcs — Thu, 21 Feb 2019 14:17:00 GMT

Hi lakshman239

i tried your options but unfortunately it is not working.

regards

Re: Filter events for specific keywords

keishamtcs — Thu, 21 Feb 2019 14:18:06 GMT

Hi,

Tried but it is not working.

Regards

Re: Filter events for specific keywords

lakshman239 — Thu, 21 Feb 2019 14:56:21 GMT

I assume you restarted splunk after changing props/transforms and looking for any new events in the index. right? Are you seeing any errors or its not filtering any events?

Re: Filter events for specific keywords

keishamtcs — Thu, 21 Feb 2019 15:04:33 GMT

Hi,

yes i did. I didn't get any error. It simply does not filter the data at all.

Regards

Re: Filter events for specific keywords

keishamtcs — Tue, 29 Sep 2020 23:24:16 GMT

Hi,

When i used something like this as mentioned above it works a bit but i guess the LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2} needs to be modified a bit to filter only the exact event.

[test]
REGEX = ^((?!Inbound Message|Outbound Message).)*$
DEST_KEY=queue
FORMAT=nullQueue

[testsource]
SHOULD_LINEMERGE = false
TRUNCATE = 100000
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
TIME_FORMAT=%Y-%m-%d %T.%3N
TIME_PREFIX=^
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}
TRANSFORMS-set = test

Re: Filter events for specific keywords

lakshman239 — Fri, 22 Feb 2019 09:20:28 GMT

Pls test with this and let me know. [ you may need restarts]
updated line breaker to include complete string.

[testsource]
SHOULD_LINEMERGE = false
TRUNCATE = 999999
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
TIME_FORMAT=%Y-%m-%d %T.%3N
TIME_PREFIX=^
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.\d{3}
TRANSFORMS-set = test

https://docs.splunk.com/Documentation/Splunk/7.2.4/SearchReference/Commontimeformatvariables

Re: Filter events for specific keywords

keishamtcs — Tue, 26 Feb 2019 11:55:46 GMT

Hi,

Yes..i will try with this.

Regards