Can Forwarder forward a multiline messages?

[yossig124]

I use only Splunk Universal Forwarder on my clients with third party server. I recieve the message from Forwarder line by line and not as one message (Forwarder splits messages by new line).
Is there a way to configure Forwarder not to split messages by new line and to keep the multiline message as one message?

[DavidHourani · Accepted solution]

Hi @yossig124,

Where are you forwarding to ? The universal forwarder does not apply any linebreaking to the event, this happens on the receiving side.

Cheers,
David

[yossig124]

Thanks for the response.
We are using the Splunk Universal Forwarder to forward messages to the ELK Logstash component.
If I understand correctly all the multi line configuration settings mentioned above apply only to the indexer which means we would have to use a heavy forwarder.
Is this understanding correct? If so is there any other possible solution for our current scenario?

[DavidHourani]

The well even if you use cooked data from an HF and send to elastic it wouldnt be multiline as this info is relevant at index time for splunk indexers. You will need to regroup the data with logstash, can logstash do that ?

[yossig124]

Logstash indeed has a Multiline Plugin which is supposed to be able regroup the message. We were however reluctant to use it as we have read that it's behavior can be problematic and inconsistent.
The hope was that Splunk allows configuring the forwarder to support multiline but I suppose that if this is not the case we will have to consider using this plugin after all.
Thanks again.

[DavidHourani]

Yeah, so logstash has similar functionalities as an HF since it can group forwarded TCP events together... simce your UF is simply sending out TCP with no line breaking indication youll most probably need to do that with Logstash.. and Im sure Splunk wouldnt invest in interoperability for multiline forwarding with logstash since data going to ELK means less license cost..so yes that plugin is your solution.

You could also use a syslog server, have it write the logs to files and just use a file beats on it... It could be easier to maintain and configure

[yossig124]

We are not using Indexer.
What do you propose to do in this case?

[ddrillic]

The following from Configure event line breaking explains it -

-- How Splunk software determines event boundaries

Splunk software determines event boundaries in two steps:

1. Line breaking, which uses the LINE_BREAKER attribute regular expression value to split the incoming stream of bytes into separate lines. By default, the LINE_BREAKER is any sequence of newlines and carriage returns (that is, ([\r\n]+)).

2. Line merging, which only occurs when you configure the SHOULD_LINEMERGE setting to "true" (the default). This step uses all the other line merging settings (for example, BREAK_ONLY_BEFORE, BREAK_ONLY_BEFORE_DATE, MUST_BREAK_AFTER, etc.) to merge the previously separated lines into events.

If the second step does not run (because you set the SHOULD_LINEMERGE attribute to "false"), then the events are the individual lines that the LINE_BREAKER attribute determines. The first step is relatively efficient, while the second is relatively slow. Appropriate use of the LINE_BREAKER regular expression can produce the results you want in the first step. This is valuable if a significant amount of your data consists of multiline events.

I had a similar thread at How can we index an entire XML document as one event?

The solution by @somesoni2 worked perfectly well -

     [yoursourcetype]
     LINE_BREAKER = ([\r\n]+)(?=\<mlcpMetricsModel )
     SHOULD_LINEMERGE = false
     TIME_PREFIX = \<reportDate\>
     TIME_FORMAT = %Y-%m-%dT%H:%M:%S
     MAX_TIMESTAMP_LOOKAHEAD = 19

LINE_BREAKER = ([\r\n]+)(?=\<mlcpMetricsModel ) does the work here and the logic provided by SHOULD_LINEMERGE is disabled by setting it to false.

[yossig124]

We are not using Indexer.
What do you propose to do in this case?