Filter data in log files

[JagsP]

Hello everyone , 

I want to filter data for a specific keyword "Snapshot created successfully " from a log file but i am getting other events also along with the searched keywords. My entries in props.conf and transform.conf is as below :

props.conf
[sourcetype]
TRANSFORMS-filter = stanza

transforms.conf

[stanza]
REGEX = "Snapshot created successfully"
DEST_KEY = queue
FORMAT = indexqueue

Is there any issue here ?

[Bhumi]

Hello @JagsP 

Please update the [stanza] regex as follows:

REGEX = successful\, returned exit code \'0\'

Note: Make sure to place the configurations correctly otherwise it will not work .

For eg: If your data flow is UF->HF->Indexer, parsing occurs at the full enterprise instance, with the HF being the first layer where this takes place. 

Additionally, here is the documentation for routing and filtering data, which will help you understand the core concepts.
https://docs.splunk.com/Documentation/Splunk/9.3.1/Forwarding/Routeandfilterdatad

For learning and writing regex for your data, you can use the platform  https://regex101.com/

If this resolves your issue, please accept and upvote the answers.

Happy Splunking!

[Bhumi]

Hello @JagsP 

1. What is the dataflow ? For Eg: UF->HF->Indexer and where have you placed your configurations. 

2. Also, share the sample event , so accordingly I can help you with regex part. 

[JagsP]

props.conf :

[sap_failure]
TRANSFORMS-filter = setnull,stanza



transform.conf :

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue



[stanza]
REGEX = "successful, returned exit code '0'"
DEST_KEY = queue
FORMAT = indexQueue

Content:

[16/Sep/2024:02:00:36 -05:00] INFO : [PID:0185317:Backup:360] [1] successful, returned exit code '0'
[16/Sep/2024:02:00:36 -05:00] DEBUG: [PID:0185317:SAPHANA:648] SQLSTRING: 'BACKUP DATA FOR FULL SYSTEM CLOSE SNAPSHOT BACKUP_ID 1726470003540 SUCCESSFUL 'hana_data_FHL__2024-09-16T070003-1779190Z''

Here in content if you see i have other events which is not having the desired text string . I need to filter all other events .

has context menu

[PaulPanther]

If you wanna filter out all other events please try

props.conf

[sourcetype]
TRANSFORMS-filter = setnull,stanza


transforms:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[stanza]
REGEX = "Snapshot created successfully"
DEST_KEY = queue
FORMAT = indexQueue

 

[JagsP]

It did not work 😞 It is still giving all the events other than the expected one.

[JagsP]

I am trying out this . I will let you know whether it worked ! Thanks .

[PaulPanther]

Where have youe configured these settings? And how do you pull/push the data to your indexer? Are there any HeavyForwarders involved?

Feel free to share an sample event with us.

[JagsP]

Yes , Heavy forwarders are being used but how it will impact the data filtration ?

[PickleRick]

It is important where you put your settings.

Parsing is done on the first "heavy" component in event's path to indexers. So if you have a HF as an intermediate forwarder, you need to put your props/transforms there.

Of course you will still be getting already indexed events during searching index-time transforms are applied only for the new events.

[JagsP]

Is there any issue with the below settings ? Also is the Regex wrong here ?

[sourcetype]
TRANSFORMS-filter = setnull,stanza


transforms:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[stanza]
REGEX = "Snapshot created successfully"
DEST_KEY = queue
FORMAT = indexQueue

 A 

[PickleRick]

Wait a second. Does your raw data contain the string in quotes or without them?

[JagsP]

This is my error string " successful, returned exit code '0'" but apart from this error , other events are also getting returned. Please help in creating the regex for this as i am new to Regex and also how can we create or take help from where to create regex?