Why is linemerging not working with Http Event Collector (HEC)?

[yarinm]

Hey,

We're trying to use Splunk HEC (+fluentd) and our existing linemerge rules aren't applied to events pushed using HEC.
We have a Splunk forwarder that pushes the same data and the linemerge rules properly applied to them.

Am I missing anything? Does HEC ignore merge rules ?

[marycordova]

Are you sending JSON data or "raw" data? I have JSON data that is a little off, so it uses the raw endpoint instead and I set the KV_MODE to json in props.conf.

http://docs.splunk.com/Documentation/Splunk/7.2.0/Data/FormateventsforHTTPEventCollector#Event_parsi...

http://docs.splunk.com/Documentation/Splunk/7.2.0/RESTREF/RESTinput#services.2Fcollector.2Fraw

Also...there are some things you cannot do on a Universal Forwarder that you can on a Heavy Forwarder regarding props and transforms.

So:

1. maybe try a Heavy Forwarder
2. maybe try the raw HEC and a combination of props settings (also on a Heavy Forwarder?)

[yarinm]

@marycordovacaa The current official plugin by splunk for fluentd doesn't support the raw API https://github.com/splunk/fluent-plugin-splunk-hec

We're currently sending using the /event endpoint.

After some testing, if I use the raw endpoint (and batch the events) together the LINEMERGE rule applies. If I use the event endpoint (with batching) it ignores it completely. I was afraid that I'll have to resort to merging the lines with fluentd..

Any other suggestions?

[marycordova]

Doesn't using the raw endpoint solve your problem? Can you just use that or does some other issue arise?

[yarinm]

The official splunk-hec fluentd plugin doesn't support the /raw endpoint at the moment..

[marycordova]

maybe submit a feature request or a bug...i dont know if its trivial to edit the plugin to use the raw endpoint or not...and then of course updates could be likely to break the customization

[benazir]

does anyone fix this issue?