Getting Data In

Filter events for specific keywords

keishamtcs
Explorer

Hi,

I have some set of events that has keywords like "inbound message" and "outbound message". the events looks something like this .

2010-02-20 14:12:45.642 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Inbound Message. I want such event only to be index.

on the indexer side, i have created as such but it is not working. how to resolve this ?

In transforms :

[test]
REGEX = Inbound
DEST_KEY = queue
FORMAT = indexQueue

props

[testsource]
TRANSFORMS-set= test

0 Karma

lakshman239
SplunkTrust
SplunkTrust

Pls change the filters as below

In transforms :

# send everything to null queue except the ones we want
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = (Inbound|inbound|Outbound|outbound)
DEST_KEY = queue
FORMAT = indexQueue

props

[testsource]
TRANSFORMS-set= setnull, setparsing

keishamtcs
Explorer

Hi lakshman239

i tried your options but unfortunately it is not working.

regards

0 Karma

lakshman239
SplunkTrust
SplunkTrust

I assume you restarted splunk after changing props/transforms and looking for any new events in the index. right? Are you seeing any errors or its not filtering any events?

0 Karma

keishamtcs
Explorer

Hi,

yes i did. I didn't get any error. It simply does not filter the data at all.

Regards

0 Karma

keishamtcs
Explorer

Hi,

When i used something like this as mentioned above it works a bit but i guess the LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2} needs to be modified a bit to filter only the exact event.

[test]
REGEX = ^((?!Inbound Message|Outbound Message).)*$
DEST_KEY=queue
FORMAT=nullQueue

[testsource]
SHOULD_LINEMERGE = false
TRUNCATE = 100000
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
TIME_FORMAT=%Y-%m-%d %T.%3N
TIME_PREFIX=^
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}
TRANSFORMS-set = test

0 Karma

lakshman239
SplunkTrust
SplunkTrust

Pls test with this and let me know. [ you may need restarts]
updated line breaker to include complete string.

[testsource]
SHOULD_LINEMERGE = false
TRUNCATE = 999999
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
TIME_FORMAT=%Y-%m-%d %T.%3N
TIME_PREFIX=^
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.\d{3}
TRANSFORMS-set = test

https://docs.splunk.com/Documentation/Splunk/7.2.4/SearchReference/Commontimeformatvariables

0 Karma

keishamtcs
Explorer

Hi,

Yes..i will try with this.

Regards

0 Karma

ddrillic
Ultra Champion

Maybe -

REGEX = (Inbound Message|Outbound Message)
0 Karma

keishamtcs
Explorer

Hi,

Tried but it is not working.

Regards

0 Karma

markusspitzli
Communicator

Hey.

You better put everything else into the nullQueue. I tested this with the following testdata:

2019-02-20 14:12:45.642 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Inbound Message.
2019-02-20 14:12:45.643 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Outbound Message.
2019-02-20 14:12:45.647 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | gugus Message.
2019-02-20 14:12:45.644 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Inbound Message.
2019-02-20 14:12:45.645 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Outbound Message.
2019-02-20 14:12:45.648 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | gugus Message.
2019-02-20 14:12:45.646 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Inbound Message.

transforms.conf

[test]
REGEX = ^((?!Inbound Message|Outbound Message).)*$
DEST_KEY=queue
FORMAT=nullQueue

props.conf

[testsource]
SHOULD_LINEMERGE = false
TRUNCATE = 100000
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
TIME_FORMAT=%Y-%m-%d %T.%3N
TIME_PREFIX=^
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}
TRANSFORMS-set = test
0 Karma

keishamtcs
Explorer

Hi,

The problem is those events which i don't need does not have any proper syntax or keyword.
I will need to filter only with events that has Inbound Message or Outbound Message.

0 Karma

markusspitzli
Communicator

Hi

Sorry I put the wrong regex. You have to put everything in the nullQueue exept of the Inbound or Outbound Message like this regex:
^((?!Inbound Message|Outbound Message).)*$

0 Karma

keishamtcs
Explorer

Hi,

It is not working. It is still indexing all the data.

0 Karma

markusspitzli
Communicator

I just tried it with a few sample logs. See the updated config from my original comment.

0 Karma

markusspitzli
Communicator

could you please add some example logs? thx

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...