Getting Data In

Filter events for specific keywords

keishamtcs
Explorer

Hi,

I have some set of events that has keywords like "inbound message" and "outbound message". the events looks something like this .

2010-02-20 14:12:45.642 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Inbound Message. I want such event only to be index.

on the indexer side, i have created as such but it is not working. how to resolve this ?

In transforms :

[test]
REGEX = Inbound
DEST_KEY = queue
FORMAT = indexQueue

props

[testsource]
TRANSFORMS-set= test

0 Karma

lakshman239
SplunkTrust
SplunkTrust

Pls change the filters as below

In transforms :

# send everything to null queue except the ones we want
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = (Inbound|inbound|Outbound|outbound)
DEST_KEY = queue
FORMAT = indexQueue

props

[testsource]
TRANSFORMS-set= setnull, setparsing

keishamtcs
Explorer

Hi lakshman239

i tried your options but unfortunately it is not working.

regards

0 Karma

lakshman239
SplunkTrust
SplunkTrust

I assume you restarted splunk after changing props/transforms and looking for any new events in the index. right? Are you seeing any errors or its not filtering any events?

0 Karma

keishamtcs
Explorer

Hi,

yes i did. I didn't get any error. It simply does not filter the data at all.

Regards

0 Karma

keishamtcs
Explorer

Hi,

When i used something like this as mentioned above it works a bit but i guess the LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2} needs to be modified a bit to filter only the exact event.

[test]
REGEX = ^((?!Inbound Message|Outbound Message).)*$
DEST_KEY=queue
FORMAT=nullQueue

[testsource]
SHOULD_LINEMERGE = false
TRUNCATE = 100000
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
TIME_FORMAT=%Y-%m-%d %T.%3N
TIME_PREFIX=^
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}
TRANSFORMS-set = test

0 Karma

lakshman239
SplunkTrust
SplunkTrust

Pls test with this and let me know. [ you may need restarts]
updated line breaker to include complete string.

[testsource]
SHOULD_LINEMERGE = false
TRUNCATE = 999999
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
TIME_FORMAT=%Y-%m-%d %T.%3N
TIME_PREFIX=^
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.\d{3}
TRANSFORMS-set = test

https://docs.splunk.com/Documentation/Splunk/7.2.4/SearchReference/Commontimeformatvariables

0 Karma

keishamtcs
Explorer

Hi,

Yes..i will try with this.

Regards

0 Karma

ddrillic
Ultra Champion

Maybe -

REGEX = (Inbound Message|Outbound Message)
0 Karma

keishamtcs
Explorer

Hi,

Tried but it is not working.

Regards

0 Karma

markusspitzli
Communicator

Hey.

You better put everything else into the nullQueue. I tested this with the following testdata:

2019-02-20 14:12:45.642 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Inbound Message.
2019-02-20 14:12:45.643 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Outbound Message.
2019-02-20 14:12:45.647 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | gugus Message.
2019-02-20 14:12:45.644 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Inbound Message.
2019-02-20 14:12:45.645 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Outbound Message.
2019-02-20 14:12:45.648 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | gugus Message.
2019-02-20 14:12:45.646 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Inbound Message.

transforms.conf

[test]
REGEX = ^((?!Inbound Message|Outbound Message).)*$
DEST_KEY=queue
FORMAT=nullQueue

props.conf

[testsource]
SHOULD_LINEMERGE = false
TRUNCATE = 100000
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
TIME_FORMAT=%Y-%m-%d %T.%3N
TIME_PREFIX=^
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}
TRANSFORMS-set = test
0 Karma

keishamtcs
Explorer

Hi,

The problem is those events which i don't need does not have any proper syntax or keyword.
I will need to filter only with events that has Inbound Message or Outbound Message.

0 Karma

markusspitzli
Communicator

Hi

Sorry I put the wrong regex. You have to put everything in the nullQueue exept of the Inbound or Outbound Message like this regex:
^((?!Inbound Message|Outbound Message).)*$

0 Karma

keishamtcs
Explorer

Hi,

It is not working. It is still indexing all the data.

0 Karma

markusspitzli
Communicator

I just tried it with a few sample logs. See the updated config from my original comment.

0 Karma

markusspitzli
Communicator

could you please add some example logs? thx

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...