Splunk Enterprise

Filter data in log files

JagsP
Explorer

Hello everyone , 

I want to filter data for a specific keyword "Snapshot created successfully " from a log file but i am getting other events also along with the searched keywords. My entries in props.conf and transform.conf is as below :

 

props.conf
[sourcetype]
TRANSFORMS-filter = stanza

transforms.conf

[stanza]
REGEX = "Snapshot created successfully"
DEST_KEY = queue
FORMAT = indexqueue

Is there any issue here ?

Labels (1)
0 Karma

Bhumi
Explorer

Hello @JagsP 

Please update the [stanza] regex as follows:

REGEX = successful\, returned exit code \'0\'

Note: Make sure to place the configurations correctly otherwise it will not work .

For eg: If your data flow is UF->HF->Indexer, parsing occurs at the full enterprise instance, with the HF being the first layer where this takes place. 

Additionally, here is the documentation for routing and filtering data, which will help you understand the core concepts.
https://docs.splunk.com/Documentation/Splunk/9.3.1/Forwarding/Routeandfilterdatad

For learning and writing regex for your data, you can use the platform  https://regex101.com/

If this resolves your issue, please accept and upvote the answers.

Happy Splunking!

0 Karma

Bhumi
Explorer

Hello @JagsP 

1. What is the dataflow ? For Eg: UF->HF->Indexer and where have you placed your configurations. 

2. Also, share the sample event , so accordingly I can help you with regex part. 

0 Karma

JagsP
Explorer
props.conf :

[sap_failure]
TRANSFORMS-filter = setnull,stanza



transform.conf :

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue



[stanza]
REGEX = "successful, returned exit code '0'"
DEST_KEY = queue
FORMAT = indexQueue



Content:

[16/Sep/2024:02:00:36 -05:00] INFO : [PID:0185317:Backup:360] [1] successful, returned exit code '0'
[16/Sep/2024:02:00:36 -05:00] DEBUG: [PID:0185317:SAPHANA:648] SQLSTRING: 'BACKUP DATA FOR FULL SYSTEM CLOSE SNAPSHOT BACKUP_ID 1726470003540 SUCCESSFUL 'hana_data_FHL__2024-09-16T070003-1779190Z''

 

Here in content if you see i have other events which is not having the desired text string . I need to filter all other events .

 

has context menu

0 Karma

PaulPanther
Motivator

If you wanna filter out all other events please try

props.conf


[sourcetype]
TRANSFORMS-filter = setnull,stanza


transforms:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[stanza]
REGEX = "Snapshot created successfully"
DEST_KEY = queue
FORMAT = indexQueue
 
0 Karma

JagsP
Explorer

It did not work 😞 It is still giving all the events other than the expected one.

0 Karma

JagsP
Explorer

I am trying out this . I will let you know whether it worked ! Thanks .

0 Karma

PaulPanther
Motivator

Where have youe configured these settings? And how do you pull/push the data to your indexer? Are there any HeavyForwarders involved?

Feel free to share an sample event with us.

0 Karma

JagsP
Explorer

Yes , Heavy forwarders are being used but how it will impact the data filtration ?

0 Karma

PickleRick
SplunkTrust
SplunkTrust

It is important where you put your settings.

Parsing is done on the first "heavy" component in event's path to indexers. So if you have a HF as an intermediate forwarder, you need to put your props/transforms there.

Of course you will still be getting already indexed events during searching index-time transforms are applied only for the new events.

0 Karma

JagsP
Explorer

Is there any issue with the below settings ? Also is the Regex wrong here ?

[sourcetype]
TRANSFORMS-filter = setnull,stanza


transforms:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[stanza]
REGEX = "Snapshot created successfully"
DEST_KEY = queue
FORMAT = indexQueue

 A 

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Wait a second. Does your raw data contain the string in quotes or without them?

0 Karma

JagsP
Explorer

This is my error string " successful, returned exit code '0'" but apart from this error , other events are also getting returned. Please help in creating the regex for this as i am new to Regex and also how can we create or take help from where to create regex?

Tags (1)
0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...