Filter events for specific keywords

[keishamtcs]

Hi,

I have some set of events that has keywords like "inbound message" and "outbound message". the events looks something like this .

2010-02-20 14:12:45.642 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Inbound Message. I want such event only to be index.

on the indexer side, i have created as such but it is not working. how to resolve this ?

In transforms :

[test]
REGEX = Inbound
DEST_KEY = queue
FORMAT = indexQueue

props

[testsource]
TRANSFORMS-set= test

[lakshman239]

Pls change the filters as below

In transforms :

# send everything to null queue except the ones we want
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = (Inbound|inbound|Outbound|outbound)
DEST_KEY = queue
FORMAT = indexQueue

props

[testsource]
TRANSFORMS-set= setnull, setparsing

[keishamtcs]

Hi lakshman239

i tried your options but unfortunately it is not working.

regards

[lakshman239]

I assume you restarted splunk after changing props/transforms and looking for any new events in the index. right? Are you seeing any errors or its not filtering any events?

[keishamtcs]

Hi,

yes i did. I didn't get any error. It simply does not filter the data at all.

Regards

[keishamtcs]

Hi,

When i used something like this as mentioned above it works a bit but i guess the LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2} needs to be modified a bit to filter only the exact event.

[test]
REGEX = ^((?!Inbound Message|Outbound Message).)*$
DEST_KEY=queue
FORMAT=nullQueue

[testsource]
SHOULD_LINEMERGE = false
TRUNCATE = 100000
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
TIME_FORMAT=%Y-%m-%d %T.%3N
TIME_PREFIX=^
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}
TRANSFORMS-set = test

[lakshman239]

Pls test with this and let me know. [ you may need restarts]
updated line breaker to include complete string.

[testsource]
SHOULD_LINEMERGE = false
TRUNCATE = 999999
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
TIME_FORMAT=%Y-%m-%d %T.%3N
TIME_PREFIX=^
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.\d{3}
TRANSFORMS-set = test

https://docs.splunk.com/Documentation/Splunk/7.2.4/SearchReference/Commontimeformatvariables

[keishamtcs]

Hi,

Yes..i will try with this.

Regards

[ddrillic]

Maybe -

REGEX = (Inbound Message|Outbound Message)

[keishamtcs]

Hi,

Tried but it is not working.

Regards

[markusspitzli]

Hey.

You better put everything else into the nullQueue. I tested this with the following testdata:

2019-02-20 14:12:45.642 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Inbound Message.
2019-02-20 14:12:45.643 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Outbound Message.
2019-02-20 14:12:45.647 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | gugus Message.
2019-02-20 14:12:45.644 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Inbound Message.
2019-02-20 14:12:45.645 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Outbound Message.
2019-02-20 14:12:45.648 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | gugus Message.
2019-02-20 14:12:45.646 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Inbound Message.

transforms.conf

[test]
REGEX = ^((?!Inbound Message|Outbound Message).)*$
DEST_KEY=queue
FORMAT=nullQueue

props.conf

[testsource]
SHOULD_LINEMERGE = false
TRUNCATE = 100000
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
TIME_FORMAT=%Y-%m-%d %T.%3N
TIME_PREFIX=^
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}
TRANSFORMS-set = test

[keishamtcs]

Hi,

The problem is those events which i don't need does not have any proper syntax or keyword.
I will need to filter only with events that has Inbound Message or Outbound Message.

[markusspitzli]

Hi

Sorry I put the wrong regex. You have to put everything in the nullQueue exept of the Inbound or Outbound Message like this regex:
^((?!Inbound Message|Outbound Message).)*$

[keishamtcs]

Hi,

It is not working. It is still indexing all the data.

[markusspitzli]

I just tried it with a few sample logs. See the updated config from my original comment.

[markusspitzli]

could you please add some example logs? thx