topic Re: Filter data in log files in Splunk Enterprise

Filter data in log files

JagsP — Wed, 23 Oct 2024 23:36:30 GMT

Hello everyone ,

I want to filter data for a specific keyword "Snapshot created successfully " from a log file but i am getting other events also along with the searched keywords. My entries in props.conf and transform.conf is as below :

props.conf [sourcetype] TRANSFORMS-filter = stanza transforms.conf [stanza] REGEX = "Snapshot created successfully" DEST_KEY = queue FORMAT = indexqueue

Is there any issue here ?

Re: Filter data in log files

PaulPanther — Wed, 21 Aug 2024 07:58:18 GMT

If you wanna filter out all other events please try

props.conf


[sourcetype]
TRANSFORMS-filter = setnull,stanza


transforms:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue


[stanza]
REGEX = "Snapshot created successfully"
DEST_KEY = queue
FORMAT = indexQueue

Re: Filter data in log files

JagsP — Mon, 26 Aug 2024 09:59:26 GMT

I am trying out this . I will let you know whether it worked ! Thanks .

Re: Filter data in log files

JagsP — Tue, 08 Oct 2024 10:44:32 GMT

It did not work 😞 It is still giving all the events other than the expected one.

Re: Filter data in log files

PaulPanther — Tue, 08 Oct 2024 13:16:26 GMT

Where have youe configured these settings? And how do you pull/push the data to your indexer? Are there any HeavyForwarders involved?

Feel free to share an sample event with us.

Re: Filter data in log files

JagsP — Tue, 22 Oct 2024 11:09:42 GMT

This is my error string " successful, returned exit code '0'" but apart from this error , other events are also getting returned. Please help in creating the regex for this as i am new to Regex and also how can we create or take help from where to create regex?

Re: Filter data in log files

JagsP — Wed, 23 Oct 2024 06:07:19 GMT

Yes , Heavy forwarders are being used but how it will impact the data filtration ?

Re: Filter data in log files

PickleRick — Wed, 23 Oct 2024 07:18:24 GMT

It is important where you put your settings.

Parsing is done on the first "heavy" component in event's path to indexers. So if you have a HF as an intermediate forwarder, you need to put your props/transforms there.

Of course you will still be getting already indexed events during searching index-time transforms are applied only for the new events.

Re: Filter data in log files

JagsP — Wed, 23 Oct 2024 08:14:25 GMT

Is there any issue with the below settings ? Also is the Regex wrong here ?

[sourcetype]
TRANSFORMS-filter = setnull,stanza


transforms:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue


[stanza]
REGEX = "Snapshot created successfully"
DEST_KEY = queue
FORMAT = indexQueue

Re: Filter data in log files

Bhumi — Wed, 23 Oct 2024 10:21:06 GMT

Hello @JagsP

1. What is the dataflow ? For Eg: UF->HF->Indexer and where have you placed your configurations.

2. Also, share the sample event , so accordingly I can help you with regex part.

Re: Filter data in log files

JagsP — Wed, 23 Oct 2024 23:36:05 GMT

props.conf : [sap_failure] TRANSFORMS-filter = setnull,stanza transform.conf : [setnull] REGEX = . DEST_KEY = queue FORMAT = nullQueue [stanza] REGEX = "successful, returned exit code '0'" DEST_KEY = queue FORMAT = indexQueue

Content:

[16/Sep/2024:02:00:36 -05:00] INFO : [PID:0185317:Backup:360] [1] successful, returned exit code '0'
[16/Sep/2024:02:00:36 -05:00] DEBUG: [PID:0185317:SAPHANA:648] SQLSTRING: 'BACKUP DATA FOR FULL SYSTEM CLOSE SNAPSHOT BACKUP_ID 1726470003540 SUCCESSFUL 'hana_data_FHL__2024-09-16T070003-1779190Z''

Here in content if you see i have other events which is not having the desired text string . I need to filter all other events .

has context menu

Re: Filter data in log files

Bhumi — Wed, 23 Oct 2024 12:52:30 GMT

Hello @JagsP

Please update the [stanza] regex as follows:

REGEX = successful\, returned exit code \'0\'

Note: Make sure to place the configurations correctly otherwise it will not work .

For eg: If your data flow is UF->HF->Indexer, parsing occurs at the full enterprise instance, with the HF being the first layer where this takes place.

Additionally, here is the documentation for routing and filtering data, which will help you understand the core concepts.
https://docs.splunk.com/Documentation/Splunk/9.3.1/Forwarding/Routeandfilterdatad

For learning and writing regex for your data, you can use the platform https://regex101.com/

If this resolves your issue, please accept and upvote the answers.

Happy Splunking!

Re: Filter data in log files

PickleRick — Wed, 23 Oct 2024 20:31:07 GMT

Wait a second. Does your raw data contain the string in quotes or without them?