Getting Data In

Filter events for specific keywords

keishamtcs
Explorer

Hi,

I have some set of events that has keywords like "inbound message" and "outbound message". the events looks something like this .

2010-02-20 14:12:45.642 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Inbound Message. I want such event only to be index.

on the indexer side, i have created as such but it is not working. how to resolve this ?

In transforms :

[test]
REGEX = Inbound
DEST_KEY = queue
FORMAT = indexQueue

props

[testsource]
TRANSFORMS-set= test

0 Karma

lakshman239
SplunkTrust
SplunkTrust

Pls change the filters as below

In transforms :

# send everything to null queue except the ones we want
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = (Inbound|inbound|Outbound|outbound)
DEST_KEY = queue
FORMAT = indexQueue

props

[testsource]
TRANSFORMS-set= setnull, setparsing

keishamtcs
Explorer

Hi lakshman239

i tried your options but unfortunately it is not working.

regards

0 Karma

lakshman239
SplunkTrust
SplunkTrust

I assume you restarted splunk after changing props/transforms and looking for any new events in the index. right? Are you seeing any errors or its not filtering any events?

0 Karma

keishamtcs
Explorer

Hi,

yes i did. I didn't get any error. It simply does not filter the data at all.

Regards

0 Karma

keishamtcs
Explorer

Hi,

When i used something like this as mentioned above it works a bit but i guess the LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2} needs to be modified a bit to filter only the exact event.

[test]
REGEX = ^((?!Inbound Message|Outbound Message).)*$
DEST_KEY=queue
FORMAT=nullQueue

[testsource]
SHOULD_LINEMERGE = false
TRUNCATE = 100000
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
TIME_FORMAT=%Y-%m-%d %T.%3N
TIME_PREFIX=^
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}
TRANSFORMS-set = test

0 Karma

lakshman239
SplunkTrust
SplunkTrust

Pls test with this and let me know. [ you may need restarts]
updated line breaker to include complete string.

[testsource]
SHOULD_LINEMERGE = false
TRUNCATE = 999999
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
TIME_FORMAT=%Y-%m-%d %T.%3N
TIME_PREFIX=^
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\.\d{3}
TRANSFORMS-set = test

https://docs.splunk.com/Documentation/Splunk/7.2.4/SearchReference/Commontimeformatvariables

0 Karma

keishamtcs
Explorer

Hi,

Yes..i will try with this.

Regards

0 Karma

ddrillic
Ultra Champion

Maybe -

REGEX = (Inbound Message|Outbound Message)
0 Karma

keishamtcs
Explorer

Hi,

Tried but it is not working.

Regards

0 Karma

markusspitzli
Communicator

Hey.

You better put everything else into the nullQueue. I tested this with the following testdata:

2019-02-20 14:12:45.642 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Inbound Message.
2019-02-20 14:12:45.643 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Outbound Message.
2019-02-20 14:12:45.647 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | gugus Message.
2019-02-20 14:12:45.644 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Inbound Message.
2019-02-20 14:12:45.645 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Outbound Message.
2019-02-20 14:12:45.648 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | gugus Message.
2019-02-20 14:12:45.646 | INFO | qtp413909515-1424 - /aaaaaaaaaaaaaaa | uuid:aaaaaaaa | vice.InServiceSOAP.InServicePort | 74 - org.apache.cxf.cxf-core - 3.0.4.redhat-621169 | Inbound Message.

transforms.conf

[test]
REGEX = ^((?!Inbound Message|Outbound Message).)*$
DEST_KEY=queue
FORMAT=nullQueue

props.conf

[testsource]
SHOULD_LINEMERGE = false
TRUNCATE = 100000
MAX_TIMESTAMP_LOOKAHEAD = 30
NO_BINARY_CHECK = true
TIME_FORMAT=%Y-%m-%d %T.%3N
TIME_PREFIX=^
LINE_BREAKER=([\r\n]+)\d{4}-\d{2}-\d{2}
TRANSFORMS-set = test
0 Karma

keishamtcs
Explorer

Hi,

The problem is those events which i don't need does not have any proper syntax or keyword.
I will need to filter only with events that has Inbound Message or Outbound Message.

0 Karma

markusspitzli
Communicator

Hi

Sorry I put the wrong regex. You have to put everything in the nullQueue exept of the Inbound or Outbound Message like this regex:
^((?!Inbound Message|Outbound Message).)*$

0 Karma

keishamtcs
Explorer

Hi,

It is not working. It is still indexing all the data.

0 Karma

markusspitzli
Communicator

I just tried it with a few sample logs. See the updated config from my original comment.

0 Karma

markusspitzli
Communicator

could you please add some example logs? thx

0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...