How to onboard specific events and discard the rest in props.conf and transforms.conf

[sahityasweety]

How to keep specific events and discard the rest in props.conf and transforms.conf

We are Receiving large amount of data which is onboarded to splunk via tar files.
We dont require monitoring all the events.,we would need only some events with some data to be monitored and rest all files/sources needed to sent into nullqueue.

Please give me some insights on it.

Thanks in advance.

[sahityasweety]

Hi Paul,

Thankyou for your response,i have checked the link that you've given.
I have tried with that, but that is not working for me.

For eg: I want to onboard the data where it has "some message" in the events and rest to discard in the below event.
Could you please suggest any solution for this

2023-01-31 10:39:58 message1
2023-01-31 10:40:01 message2
2023-01-31 10:40:08 message3
2023-01-31 10:40:08 message4
2023-01-31 10:40:00 some message
2023-01-31 10:40:01 some message in between
2023-01-31 10:40:01 some message in between
2023-01-31 10:40:01 some message in between
2023-01-31 10:40:01 message5
2023-01-31 10:40:01 message5

[marnall]

PaulPanther's link https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Filter_event_data... Is where you want to go.

Under the "Keep specific events and discard the rest" section, you can find stanzas for props.conf and transforms.conf files that you can place in an app on your indexing machines. Setting the regex of the setparsing stanza to "some message" would give you only the events containing that "some message", and discard the rest.

# In props.conf
[source::/your/log/file/path]
TRANSFORMS-set= setnull,setparsing

# In transforms.conf:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = some message
DEST_KEY = queue
FORMAT = indexQueue

(It is assumed that you already have a working inputs.conf file to get the logs into your indexing machines. You can also set the stanza name in the props.conf file to use your log sourcetype)

[PickleRick]

It is described in the "route and filter data" document you've been pointed to.

One important thing that people often misunderstand at first - if you configure multiple transforms in one transform groups - all of them are executed in sequence. So you must define a transform redirecting all events to nullQueue (dropping them) and only after that have a transform sending chosen events to indexQueue.

[PaulPanther]

Check out: Route and filter data - Splunk Documentation

If you have more specific questions about your data just ask.