How does splunk reassamble chunks of data coming from UFs in an indexer cluster ?

[yoho]

By default, UFs are sending chunks of 64kB data and spread these over multiple indexers. But indexers are supposed to reassemble these chunks so that they can break lines, delimit events and extract timestamps. I don't see clearly in the documentation how this process is working.

Suppose we have 3 events. End of event1 + beginning of event2 are sent in a first chunk (chunk1); end of event2 and beginning of event3 are sent in a second chunk (chunk2). When indexers apply linebreaking rules, each of them ends up with a part of event2. They know about it because they have a leftover (partially delimited event) but what happens next ?

Also how is LINE_BREAKER_LOOKBEHIND in props.conf working exactly ? Why do you need to go back on the previous chunk of data for a certain number of bytes ?

[PavelP]

Hello @yoho,

please consult https://docs.splunk.com/Documentation/Splunk/8.0.3/Data/Configureeventlinebreaking

the UF should break events on file's EOL or splitt events using EVENT_BREAKER (default [\r\n]).

Do you have wrongly splitted events?

here is a documentation on EVENT_BREAKER:

# Use the following to define event boundaries for multi-line events
# For single-line events, the default settings should suffice

EVENT_BREAKER = <regular expression>
* A regular expression that specifies the event boundary for a
  universal forwarder to use to determine when it can send events
  to an indexer.
* The regular expression must contain a capturing group
  (a pair of parentheses that defines an identified sub-component
  of the match.)
* When the UF finds a match, it considers the first capturing group
  to be the end of the previous event, and the end of the capturing group
  to be the beginning of the next event.
* At this point, the forwarder can then change the receiving indexer
  based on these event boundaries.
* This setting is only active if you set 'EVENT_BREAKER_ENABLE' to
  "true", only works on universal forwarders, and
  works best with multiline events.
* Default: "\r\n"

[yoho]

Hello PaveIP, thanks for your answer.

I already knew about this parameter but it's disabled by default. [ By the way, note this documentation is partly wrong: the default value must be a capturing group. I reported it to splunk which acknowledged it as a documentation mistake and the true value will be mentioned in the next documentation release : "([\r\n]+)". ] .

If you read props.conf :

EVENT_BREAKER_ENABLE = <boolean>
* Whether or not a universal forwarder (UF) uses the 'ChunkedLBProcessor'
  data processor to improve distribution of events to receiving
  indexers for a given source type.
* When set to true, a UF splits incoming data with a
  light-weight chunked line breaking processor ('ChunkedLBProcessor')
  so that data is distributed fairly evenly amongst multiple indexers.
* When set to false, a UF uses standard load-balancing methods to
  send events to indexers.
* Use this setting on a UF to indicate that data
  should be split on event boundaries across indexers, especially
  for large files.
* This setting is only valid on universal forwarder instances.
* Default: false

So by default, the UF doesn't break events on file's EOL but "a UF uses standard load-balancing methods to send events to indexers.".

The standard load-balancing method is based on autoLBVolume and autoLBFrequency (see outputs.conf) and chunks of data (outputs.conf : "Non-parsing forwarders, such as universal forwarders, send blocks, which can be up to 64KB.").

So like I mention, by default, one indexer may receive a chunk of data and another indexer will receive the next chunk. My question is "how do they get reassembled ?".

[PavelP]

@yoho thank you for the correction, you're right.

I've checked internal training documentation, event splits for multi-line data are mentioned as "potential side-effects", the only solution provided is enabling of event breaker on the UF.
My assumption is that there are no "event reassembling" on the Indexer, since there are no any "sequence number" or similar.

I've just tested in my lab and pause log writing for a while (emulation slow IO), checked with tcpdump that the chunk was sent to the indexer, checked with list forward-server that the UF switched from IDX1 to IDX2, then continued to write the line. This produced two different events:

IDX2: 16/04/2020 20:27:19 a=11111
IDX1: 16/04/2020 20:27:19 a=1

Not quite a case which you described, but for me it seems both UF and IDX not so intellegent as I hoped.

I would be happy if somebody can disproof this 🙂

[yoho]

@PaveIP, interesting information.

Events I'm ingesting are indeed potentially multiline (95% of them are single line) and I break events based on the date. Here's the relevant props.conf stanza.

[db.command.trace]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)\w+\s+\w+\s+\d+\s+\d+:\d+:\d+\.\d+\s
#LINE_BREAKER_LOOKBEHIND = 4096
MAX_TIMESTAMP_LOOKAHEAD = 40
TIME_PREFIX = ^
#Wed Mar 25 04:05:00.978 2020
TIME_FORMAT = %a %b %e %H:%M:%S.%Q %Y

I get errors from indexers regarding date/time extraction although when I search for events, I don't see any problem. I have not configured event breaker on the UF but I believe I should, I'll give it a try.

But we are also ingesting many other source types, including multiline ones and we've never experienced such problem before so I was wondering how it works under normal situations.

Thanks for the test but I don't completely understand how it's done. Is your event multiline ? Yes, a slow inputs will generally not work well with multiline. Splunk will send an event as soon as it reaches "\n"+EOF , so you need to play with the following parameter in inputs.conf:

multiline_event_extra_waittime = <boolean>
* By default, the file monitor sends an event delimiter when:
  * It reaches EOF of a file it monitors and
  * The last character it reads is a newline.
* In some cases, it takes time for all lines of a multiple-line event to
  arrive.
* Set to "true" to delay sending an event delimiter until the time that the
  file monitor closes the file, as defined by the 'time_before_close' setting,
  to allow all event lines to arrive.
* Default: false