Getting Data In

send to nullqueue events which have more than 100 lines

robertosegantin
Path Finder

I have an XML file which has events made by many rows.
I would like to send to null queue the events which have more than 100 rows.
How can I do that?

0 Karma
1 Solution

DavidHourani
Super Champion

Hi @robertosegantin,

I see three ways to go about this:

1- The easiest way is to set TRUNCATE limit in props.conf for your sourcetype in order to avoid having more than a fixed number of a characters per event. This however will not completely remove the event but merely filter part of it.

2- (Recommended) Use a combination of regexp to identify the long events and move them to the nullqueue:
https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Discard_specific_...

3- Use a regular expression to select the useful info from the long 100 line event, and discard the rest. You can find this here:
https://answers.splunk.com/answers/735219/index-selected-lines-in-a-multiline-event.html

I think 2 is best suited for what you're trying to achieve. You can use a regex to count the number of lines if needed, something like : (.*(\n|\r)){100}, then send to null queue anything that matches.

Let me know if that helps.

Cheers,
David

View solution in original post

0 Karma

DavidHourani
Super Champion

Hi @robertosegantin,

I see three ways to go about this:

1- The easiest way is to set TRUNCATE limit in props.conf for your sourcetype in order to avoid having more than a fixed number of a characters per event. This however will not completely remove the event but merely filter part of it.

2- (Recommended) Use a combination of regexp to identify the long events and move them to the nullqueue:
https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Discard_specific_...

3- Use a regular expression to select the useful info from the long 100 line event, and discard the rest. You can find this here:
https://answers.splunk.com/answers/735219/index-selected-lines-in-a-multiline-event.html

I think 2 is best suited for what you're trying to achieve. You can use a regex to count the number of lines if needed, something like : (.*(\n|\r)){100}, then send to null queue anything that matches.

Let me know if that helps.

Cheers,
David

0 Karma

robertosegantin
Path Finder

Hi @DavidHourani ,

thanks for your answer.
I tried:

== props.conf ==
[my]
CHARSET = UTF-8
KV_MODE = xml
MAX_EVENTS = 100000
MAX_TIMESTAMP_LOOKAHEAD = 28
NO_BINARY_CHECK = true
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%4Q
TIME_PREFIX = ^[
category = Custom
disabled = false
pulldown_type = true
TRANSFORMS-nullqueue_more_than_100_lines = nullqueue_more_than_100_lines

== transforms.conf ==
[nullqueue_more_than_100_lines]
REGEX = (.*(\n|\r)){100,}
DEST_KEY = queue
FORMAT = nullQueue

But it does not send events with more than 100 lines to nullqueue.
I also tried with TRUNCATE=5000, but is does not work, too

Cheers,
Roberto

0 Karma

DavidHourani
Super Champion

Hi @robertosegantin, Is this working now ?

0 Karma

robertosegantin
Path Finder

Hi @DavidHourani,
sorry for late replay.
The problem is that "TRUNCATE" and "TRANSFORMS" seem to work on single line, and they work before "SHOULD_LINEMERGE", which is true by default.
In this way, when Splunk merges 150 xml lines into one event, the "TRUNCATE" and "TRANSFORMS" option work on single line, which has less than 100 lines (is only one!) and is less than 5000 byte

Have I done some mistake?

Thanks!

0 Karma

DavidHourani
Super Champion

Hi @robertosegantin,

Go for SHOULD_LINEMERGE = false and set a new line breaker which is the closing stanza for your xml. It should do the trick.

0 Karma

robertosegantin
Path Finder

Hi @DavidHourani,

found the golden path! 😄

[my_sourcetype]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
CHARSET=UTF-8
MAX_TIMESTAMP_LOOKAHEAD=45
disabled=false
LINE_BREAKER =(<\/s:Envelope>)
TIME_FORMAT=%Y-%m-%d %H:%M:%S.%4Q
TIME_PREFIX=(<\/s:Envelope>[\r\n]+)?[
TRUNCATE=3000

Thanks for your help!

0 Karma

DavidHourani
Super Champion

You're welcome!

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...