topic Re: send to nullqueue events which have more than 100 lines in Getting Data In

send to nullqueue events which have more than 100 lines

robertosegantin — Tue, 06 Aug 2019 08:11:19 GMT

I have an XML file which has events made by many rows.
I would like to send to null queue the events which have more than 100 rows.
How can I do that?

Re: send to nullqueue events which have more than 100 lines

DavidHourani — Tue, 06 Aug 2019 08:42:22 GMT

Hi @robertosegantin,

I see three ways to go about this:

1- The easiest way is to set TRUNCATE limit in props.conf for your sourcetype in order to avoid having more than a fixed number of a characters per event. This however will not completely remove the event but merely filter part of it.

2- (Recommended) Use a combination of regexp to identify the long events and move them to the nullqueue:
https://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Discard_specific_events_and_keep_the_rest

3- Use a regular expression to select the useful info from the long 100 line event, and discard the rest. You can find this here:
https://answers.splunk.com/answers/735219/index-selected-lines-in-a-multiline-event.html

I think 2 is best suited for what you're trying to achieve. You can use a regex to count the number of lines if needed, something like : (.*(\n|\r)){100}, then send to null queue anything that matches.

Let me know if that helps.

Cheers,
David

Re: send to nullqueue events which have more than 100 lines

robertosegantin — Wed, 30 Sep 2020 01:37:40 GMT

Hi @DavidHourani ,

thanks for your answer.
I tried:

== props.conf ==
[my]
CHARSET = UTF-8
KV_MODE = xml
MAX_EVENTS = 100000
MAX_TIMESTAMP_LOOKAHEAD = 28
NO_BINARY_CHECK = true
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%4Q
TIME_PREFIX = ^[
category = Custom
disabled = false
pulldown_type = true
TRANSFORMS-nullqueue_more_than_100_lines = nullqueue_more_than_100_lines

== transforms.conf ==
[nullqueue_more_than_100_lines]
REGEX = (.*(\n|\r)){100,}
DEST_KEY = queue
FORMAT = nullQueue

But it does not send events with more than 100 lines to nullqueue.
I also tried with TRUNCATE=5000, but is does not work, too

Cheers,
Roberto

Re: send to nullqueue events which have more than 100 lines

DavidHourani — Wed, 14 Aug 2019 07:03:23 GMT

Hi @robertosegantin, Is this working now ?

Re: send to nullqueue events which have more than 100 lines

robertosegantin — Mon, 19 Aug 2019 08:16:55 GMT

Hi @DavidHourani,
sorry for late replay.
The problem is that "TRUNCATE" and "TRANSFORMS" seem to work on single line, and they work before "SHOULD_LINEMERGE", which is true by default.
In this way, when Splunk merges 150 xml lines into one event, the "TRUNCATE" and "TRANSFORMS" option work on single line, which has less than 100 lines (is only one!) and is less than 5000 byte

Have I done some mistake?

Thanks!

Re: send to nullqueue events which have more than 100 lines

DavidHourani — Mon, 19 Aug 2019 10:17:13 GMT

Hi @robertosegantin,

Go for SHOULD_LINEMERGE = false and set a new line breaker which is the closing stanza for your xml. It should do the trick.

Re: send to nullqueue events which have more than 100 lines

robertosegantin — Wed, 30 Sep 2020 01:52:02 GMT

Hi @DavidHourani,

found the golden path! 😄

[my_sourcetype]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
CHARSET=UTF-8
MAX_TIMESTAMP_LOOKAHEAD=45
disabled=false
LINE_BREAKER =(<\/s:Envelope>)
TIME_FORMAT=%Y-%m-%d %H:%M:%S.%4Q
TIME_PREFIX=(<\/s:Envelope>[\r\n]+)?[
TRUNCATE=3000

Thanks for your help!

Re: send to nullqueue events which have more than 100 lines

DavidHourani — Thu, 22 Aug 2019 18:56:03 GMT

You're welcome!