Getting Data In

send to nullqueue events which have more than 100 lines

Path Finder

I have an XML file which has events made by many rows.
I would like to send to null queue the events which have more than 100 rows.
How can I do that?

0 Karma
1 Solution

SplunkTrust
SplunkTrust

Hi @robertosegantin,

I see three ways to go about this:

1- The easiest way is to set TRUNCATE limit in props.conf for your sourcetype in order to avoid having more than a fixed number of a characters per event. This however will not completely remove the event but merely filter part of it.

2- (Recommended) Use a combination of regexp to identify the long events and move them to the nullqueue:
https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Discard_specific_...

3- Use a regular expression to select the useful info from the long 100 line event, and discard the rest. You can find this here:
https://answers.splunk.com/answers/735219/index-selected-lines-in-a-multiline-event.html

I think 2 is best suited for what you're trying to achieve. You can use a regex to count the number of lines if needed, something like : (.*(\n|\r)){100}, then send to null queue anything that matches.

Let me know if that helps.

Cheers,
David

View solution in original post

0 Karma

SplunkTrust
SplunkTrust

Hi @robertosegantin,

I see three ways to go about this:

1- The easiest way is to set TRUNCATE limit in props.conf for your sourcetype in order to avoid having more than a fixed number of a characters per event. This however will not completely remove the event but merely filter part of it.

2- (Recommended) Use a combination of regexp to identify the long events and move them to the nullqueue:
https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Discard_specific_...

3- Use a regular expression to select the useful info from the long 100 line event, and discard the rest. You can find this here:
https://answers.splunk.com/answers/735219/index-selected-lines-in-a-multiline-event.html

I think 2 is best suited for what you're trying to achieve. You can use a regex to count the number of lines if needed, something like : (.*(\n|\r)){100}, then send to null queue anything that matches.

Let me know if that helps.

Cheers,
David

View solution in original post

0 Karma

Path Finder

Hi @DavidHourani ,

thanks for your answer.
I tried:

== props.conf ==
[my]
CHARSET = UTF-8
KV_MODE = xml
MAX_EVENTS = 100000
MAX_TIMESTAMP_LOOKAHEAD = 28
NO_BINARY_CHECK = true
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%4Q
TIME_PREFIX = ^[
category = Custom
disabled = false
pulldown_type = true
TRANSFORMS-nullqueue_more_than_100_lines = nullqueue_more_than_100_lines

== transforms.conf ==
[nullqueue_more_than_100_lines]
REGEX = (.*(\n|\r)){100,}
DEST_KEY = queue
FORMAT = nullQueue

But it does not send events with more than 100 lines to nullqueue.
I also tried with TRUNCATE=5000, but is does not work, too

Cheers,
Roberto

0 Karma

SplunkTrust
SplunkTrust

Hi @robertosegantin, Is this working now ?

0 Karma

Path Finder

Hi @DavidHourani,
sorry for late replay.
The problem is that "TRUNCATE" and "TRANSFORMS" seem to work on single line, and they work before "SHOULD_LINEMERGE", which is true by default.
In this way, when Splunk merges 150 xml lines into one event, the "TRUNCATE" and "TRANSFORMS" option work on single line, which has less than 100 lines (is only one!) and is less than 5000 byte

Have I done some mistake?

Thanks!

0 Karma

SplunkTrust
SplunkTrust

Hi @robertosegantin,

Go for SHOULD_LINEMERGE = false and set a new line breaker which is the closing stanza for your xml. It should do the trick.

0 Karma

Path Finder

Hi @DavidHourani,

found the golden path! 😄

[my_sourcetype]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
CHARSET=UTF-8
MAX_TIMESTAMP_LOOKAHEAD=45
disabled=false
LINE_BREAKER =(<\/s:Envelope>)
TIME_FORMAT=%Y-%m-%d %H:%M:%S.%4Q
TIME_PREFIX=(<\/s:Envelope>[\r\n]+)?[
TRUNCATE=3000

Thanks for your help!

0 Karma

SplunkTrust
SplunkTrust

You're welcome!

0 Karma