Getting Data In
Highlighted

Why is linemerging not working with Http Event Collector (HEC)?

New Member

Hey,

We're trying to use Splunk HEC (+fluentd) and our existing linemerge rules aren't applied to events pushed using HEC.
We have a Splunk forwarder that pushes the same data and the linemerge rules properly applied to them.

Am I missing anything? Does HEC ignore merge rules ?

0 Karma
Highlighted

Re: Why is linemerging not working with Http Event Collector (HEC)?

SplunkTrust
SplunkTrust

Are you sending JSON data or "raw" data? I have JSON data that is a little off, so it uses the raw endpoint instead and I set the KV_MODE to json in props.conf.

http://docs.splunk.com/Documentation/Splunk/7.2.0/Data/FormateventsforHTTPEventCollector#Event_parsi...

http://docs.splunk.com/Documentation/Splunk/7.2.0/RESTREF/RESTinput#services.2Fcollector.2Fraw

Also...there are some things you cannot do on a Universal Forwarder that you can on a Heavy Forwarder regarding props and transforms.

So:

  1. maybe try a Heavy Forwarder
  2. maybe try the raw HEC and a combination of props settings (also on a Heavy Forwarder?)
0 Karma
Highlighted

Re: Why is linemerging not working with Http Event Collector (HEC)?

New Member

@marycordovacaa The current official plugin by splunk for fluentd doesn't support the raw API https://github.com/splunk/fluent-plugin-splunk-hec

We're currently sending using the /event endpoint.

After some testing, if I use the raw endpoint (and batch the events) together the LINEMERGE rule applies. If I use the event endpoint (with batching) it ignores it completely. I was afraid that I'll have to resort to merging the lines with fluentd..

Any other suggestions?

0 Karma
Highlighted

Re: Why is linemerging not working with Http Event Collector (HEC)?

SplunkTrust
SplunkTrust

Doesn't using the raw endpoint solve your problem? Can you just use that or does some other issue arise?

0 Karma
Highlighted

Re: Why is linemerging not working with Http Event Collector (HEC)?

New Member

The official splunk-hec fluentd plugin doesn't support the /raw endpoint at the moment..

0 Karma
Highlighted

Re: Why is linemerging not working with Http Event Collector (HEC)?

SplunkTrust
SplunkTrust

maybe submit a feature request or a bug...i dont know if its trivial to edit the plugin to use the raw endpoint or not...and then of course updates could be likely to break the customization

0 Karma
Highlighted

Re: Why is linemerging not working with Http Event Collector (HEC)?

Explorer

does anyone fix this issue?

0 Karma