topic Re: Why is linemerging not working with Http Event Collector (HEC)? in Getting Data In

Why is linemerging not working with Http Event Collector (HEC)?

yarinm — Tue, 06 Nov 2018 13:52:02 GMT

Hey,

We're trying to use Splunk HEC (+fluentd) and our existing linemerge rules aren't applied to events pushed using HEC.
We have a Splunk forwarder that pushes the same data and the linemerge rules properly applied to them.

Am I missing anything? Does HEC ignore merge rules ?

Re: Why is linemerging not working with Http Event Collector (HEC)?

marycordova — Tue, 06 Nov 2018 19:42:39 GMT

Are you sending JSON data or "raw" data? I have JSON data that is a little off, so it uses the raw endpoint instead and I set the KV_MODE to json in props.conf.

http://docs.splunk.com/Documentation/Splunk/7.2.0/Data/FormateventsforHTTPEventCollector#Event_parsing

http://docs.splunk.com/Documentation/Splunk/7.2.0/RESTREF/RESTinput#services.2Fcollector.2Fraw

Also...there are some things you cannot do on a Universal Forwarder that you can on a Heavy Forwarder regarding props and transforms.

So:

maybe try a Heavy Forwarder
maybe try the raw HEC and a combination of props settings (also on a Heavy Forwarder?)

Re: Why is linemerging not working with Http Event Collector (HEC)?

yarinm — Wed, 07 Nov 2018 07:33:40 GMT

@marycordovacaa The current official plugin by splunk for fluentd doesn't support the raw API https://github.com/splunk/fluent-plugin-splunk-hec

We're currently sending using the /event endpoint.

After some testing, if I use the raw endpoint (and batch the events) together the LINEMERGE rule applies. If I use the event endpoint (with batching) it ignores it completely. I was afraid that I'll have to resort to merging the lines with fluentd..

Any other suggestions?

Re: Why is linemerging not working with Http Event Collector (HEC)?

marycordova — Wed, 07 Nov 2018 18:09:41 GMT

Doesn't using the raw endpoint solve your problem? Can you just use that or does some other issue arise?

Re: Why is linemerging not working with Http Event Collector (HEC)?

yarinm — Wed, 07 Nov 2018 20:56:10 GMT

The official splunk-hec fluentd plugin doesn't support the /raw endpoint at the moment..

Re: Why is linemerging not working with Http Event Collector (HEC)?

marycordova — Thu, 08 Nov 2018 00:30:04 GMT

maybe submit a feature request or a bug...i dont know if its trivial to edit the plugin to use the raw endpoint or not...and then of course updates could be likely to break the customization

Re: Why is linemerging not working with Http Event Collector (HEC)?

benazir — Wed, 18 Sep 2019 08:19:18 GMT

does anyone fix this issue?