Getting Data In

How to drop specific lines from an event not the whole event

Mfmahdi
Explorer

we are getting WAF log and the events are very big we need to drop some lines from the events that has no meaningful value not the whole event.

@gcusello 

thank you in advance.

Labels (3)
0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

Hi @Mfmahdi,

you could truncate your events defining the max lenght of each event using the TRUNCATE option in props.conf.

Otherwise you could define a regex to exclude from each event the part that you don't want.

You should use the SEDCMD command in props.conf

For more infos see at https://docs.splunk.com/Documentation/Splunk/9.2.0/admin/Propsconf

Ciao.

Giuseppe

View solution in original post

0 Karma

kiran_panchavat
Contributor

@Mfmahdi Example: 

SEDCMD:

[your_sourcetype]
SEDCMD-drop_unwanted_lines = <regex>

Truncate:

[your_sourcetype]
TRUNCATE = 10000

Adjust the value to your desired maximum event length.

SEDCMD-<class> = <sed script>
* Only used at index time.
* Commonly used to anonymize incoming data at index time, such as credit
card or social security numbers. For more information, search the online
documentation for "anonymize data."
* Used to specify a sed script which Splunk software applies to the _raw
field.
* A sed script is a space-separated list of sed commands. Currently the
following subset of sed commands is supported:
* replace (s) and character substitution (y).
* Syntax:
* replace - s/regex/replacement/flags
* regex is a perl regular expression (optionally containing capturing
groups).
* replacement is a string to replace the regex match. Use \n for back
references, where "n" is a single digit.
* flags can be either: g to replace all matches, or a number to
replace a specified match.
* substitute - y/string1/string2/
* substitutes the string1[i] with string2[i]
* No default.

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Mfmahdi,

you could truncate your events defining the max lenght of each event using the TRUNCATE option in props.conf.

Otherwise you could define a regex to exclude from each event the part that you don't want.

You should use the SEDCMD command in props.conf

For more infos see at https://docs.splunk.com/Documentation/Splunk/9.2.0/admin/Propsconf

Ciao.

Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @Mfmahdi ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...