Why is our universal forwarder is truncating/adding extra line breaks to events output over TCP?

[petehmrc]

We have a universal forwarder set up to forward incoming messages to logstash, TCP -> forwarder -> TCP:

outputs.conf:

[tcpout] 
defaultGroup = logstash

[tcpout:logstash]
server=localhost:7777 
sendCookedData = false 
useACK = true

And inputs.conf:

 [tcp://:9997] 
 sourcetype=_json

 [monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
 disabled = true

 [monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
 disabled = true

 [monitor://$SPLUNK_HOME/var/log/splunk/license_audit.log]
 disabled = true

If we connect our load tester directly to logstash, then our messages appear correctly formatted. If we go via the universal formatter then messages are coming through with extra line breaks. It's as if Splunk isn't able to tell where one event stops and another begins.

Interestingly, if we send the same event multiple times, it seems to get truncated at the same point each time. All our messages originating from the load testing tool are json formatted, newline seperated. We have also tried terminating messages with EOT characters but to no avail.

What could be causing this? Is there a key piece of information/documentation we are missing?

[dorlevi]

Years later, and I'm experiencing the same problem.

Were you able to fix this?

[jamie00171]

Hi @dorlevi 

Have you tired setting EVENT_BREAKER in props.conf on the UF? You'd also need to set EVENT_BREAKER_ENABLE = true and it might be worth looking at setting TRUNCATE if you have large events. 

(I'm assuming you're forwarding to something other than Splunk).

Thanks,

Jamie

[dorlevi]

@jamie00171 

Upon further testing, it seems that doing

EVENT_BREAKER_ENABLE = true
EVENT_BREAKER = "([\r\n]+)"

Works, but only if accompanied by also doing

force_local_processing = true

Are the event breaker flags ignored if there's no force local processing?

[dorlevi]

Hi,

I just tried adding EVENT_BREAKER_ENABLE = true, as well as also trying to add TRUNCATE = 0 in a different test, to no avial.

I'm basically trying to simply forward a logfile, unaltered, to a 3rd party tool, and am seeing what appears to be random line breaks in the file, sometimes even in the middle of a word.

So far my config is -

outputs.conf

[tcpout:fastlane]
server = host.docker.internal:6996
sendCookedData = false

inputs.conf

[monitor:///opt/splunkforwarder/etc/splunk.version]
_TCP_ROUTING = default-autolb-group

[monitor:///opt/splunkforwarder/var/log/splunk/metrics.log]
_TCP_ROUTING = default-autolb-group

[monitor:///opt/splunkforwarder/var/log/splunk/splunkd.log]
_TCP_ROUTING = default-autolb-group

[monitor:///var/log/test_file.log]
disabled = false
sourcetype = my_source_type
_TCP_ROUTING = fastlane

And after you suggestion, added the following to props.conf

[my_source_type]
EVENT_BREAKER_ENABLE = true
EVENT_BREAKER = ([\r\n]+)
TRUNCATE = 0

So far, no success in having just the plain file being forwarded, as mentioned I'm having random line breaks throughout.

I am running this UF in a docker container on my local machine as I'm still testing this, but I doubt this is an issue here because if I'm using the same UF to forward the file to my local Splunk Enterprise node, I'm seeing the events exactly as they should be.

[fairje]

I think you want the props.conf on the receiving server (not the forwarder), to specify your newline character as the end of the log. The default for this is a regex pattern on "\r\n", but depending on how the log comes in this may not be the case. Below is a snippet from the props.conf specification file:
http://docs.splunk.com/Documentation/Splunk/6.3.1/admin/Propsconf

LINE_BREAKER = <regular expression>
* Specifies a regex that determines how the raw text stream is broken into
  initial events, before line merging takes place. (See the SHOULD_LINEMERGE
  attribute, below)
* Defaults to ([\r\n]+), meaning data is broken into an event for each line,
  delimited by any number of carriage return or newline characters.

You might also be able to get the data to cooperate on the forwarder itself by specifying that the type of data is "JSON" see this section in the props.conf:

* This feature and all of its settings apply at input time, when data is
  first read by Splunk.  The setting is used on a Splunk system that has
  configured inputs acquiring the data.

# Special characters for Structured Data Header Extraction:
# Some unprintable characters can be described with escape sequences. The
# attributes that can use these characters specifically mention that
# capability in their descriptions below.
# \f : form feed       byte: 0x0c
# \s : space           byte: 0x20
# \t : horizontal tab  byte: 0x09
# \v : vertical tab    byte: 0x0b

INDEXED_EXTRACTIONS = < CSV|W3C|TSV|PSV|JSON >
* Tells Splunk the type of file and the extraction and/or parsing method
  Splunk should use on the file.
  CSV  - Comma separated value format
  TSV  - Tab-separated value format
  PSV  - pipe "|" separated value format
  W3C  - W3C Extended Extended Log File Format
  JSON - JavaScript Object Notation format
* These settings default the values of the remaining settings to the
  appropriate values for these known formats.
* Defaults to unset.

So you can either fix this on the receiving side or the sending side.

[petehmrc]

As I mentioned in the question, we aren't forwarding on to splunk, but logstash. Secondly, we are using a universal forwarder... we have tried your suggestion in props.conf but I don't think this is checked by UF as it does not parse the data. Either way it did not resolve the problem : /

When I say messages are being truncated, I mean that the forwarder is adding extra line breaks in... I'm guessing it fills up some kind of buffer before flushing everything it has out as a single message perhaps? As such we can't use any workarounds that rely on line breaks.

[fairje]

Oh, sorry, for some reason I didn't register that Logstash was another product (which is basically a fork of ELK, right?). What you may want to do is run a Heavy Forwarder so you can collect the data and then forward it on via syslog. Which I don't think you can do from a UF. You should be able to register the Enterprise instance as either the free license or the forwarding license since I assume you are not paying for Splunk since you are using ELK. Just make sure to turn off indexing on your Heavy Forwarder and have it dump the data via syslog to ELK.

Otherwise I think I am getting outside my depth here since I am not familiar enough with ELK to help further.

[petehmrc]

No worries, your help was much appreciated 🙂

[woodcock]

What is in your props.conf?

[petehmrc]

I don't currently have any overrides other than the default. I had assumed that as the universal forwarder does not parse the data, props.conf was not relevant but I've just googled it and read to the contrary.

Is there a particular setting you think we should apply?