Getting Data In

Why is linemerging not working with Http Event Collector (HEC)?

yarinm
Explorer

Hey,

We're trying to use Splunk HEC (+fluentd) and our existing linemerge rules aren't applied to events pushed using HEC.
We have a Splunk forwarder that pushes the same data and the linemerge rules properly applied to them.

Am I missing anything? Does HEC ignore merge rules ?

marycordova
SplunkTrust
SplunkTrust

Are you sending JSON data or "raw" data? I have JSON data that is a little off, so it uses the raw endpoint instead and I set the KV_MODE to json in props.conf.

http://docs.splunk.com/Documentation/Splunk/7.2.0/Data/FormateventsforHTTPEventCollector#Event_parsi...

http://docs.splunk.com/Documentation/Splunk/7.2.0/RESTREF/RESTinput#services.2Fcollector.2Fraw

Also...there are some things you cannot do on a Universal Forwarder that you can on a Heavy Forwarder regarding props and transforms.

So:

  1. maybe try a Heavy Forwarder
  2. maybe try the raw HEC and a combination of props settings (also on a Heavy Forwarder?)
@marycordova
0 Karma

yarinm
Explorer

@marycordovacaa The current official plugin by splunk for fluentd doesn't support the raw API https://github.com/splunk/fluent-plugin-splunk-hec

We're currently sending using the /event endpoint.

After some testing, if I use the raw endpoint (and batch the events) together the LINEMERGE rule applies. If I use the event endpoint (with batching) it ignores it completely. I was afraid that I'll have to resort to merging the lines with fluentd..

Any other suggestions?

0 Karma

marycordova
SplunkTrust
SplunkTrust

Doesn't using the raw endpoint solve your problem? Can you just use that or does some other issue arise?

@marycordova
0 Karma

yarinm
Explorer

The official splunk-hec fluentd plugin doesn't support the /raw endpoint at the moment..

0 Karma

marycordova
SplunkTrust
SplunkTrust

maybe submit a feature request or a bug...i dont know if its trivial to edit the plugin to use the raw endpoint or not...and then of course updates could be likely to break the customization

@marycordova
0 Karma

benazir
Explorer

does anyone fix this issue?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...