How to Load Balanced HEC entrypoint with indexer ack

[emallinger]

Hi !

I wonder how to correct the following behaviour.

Here's my architecture :

1 dns entry point load balancing between 2 forwarders on port 8088 for Http Event Collector (HEC).

behind that 1 indexer (monoinstance).

indexer ack activated for one collect serie (one index with one sourcetype).

When sending event, an IdAck is answered back to check if the event is correctly received by the indexer.

Problem :

2 different events can have the same ackID !

I suppose it is because of the load balancing and each ackID list is linked to each forwarder.

As the query is balanced, I cannot know if I will be answerd be the fw1 or 2.

Event1 is processed by fw1 with idAck = 7

Event2 is processed by fw2 with idAck = 7 (also !!)

When asking for indexing ack status for idAck7 : my query can be processed by fw1 or 2, but the answer cannot be meaningful because I don't know which event I'm asking about.

How do we go around this behaviour ?

Does this mean I can't load balance the entrypoint in front of the forwarder ? In this case, how am I supposed to allow high availability of the service ?

Thank your a lot in advance for you insights

Ema

[emallinger · Accepted solution]

Hi all,

Problem solved, finally. It was all jumbled on conf files (too many tests with too many confs).

Wiping out and starting anew did the trick.

[emallinger]

Hello,

I made some progress :  in order to use persistence, the load balancer need to be able to read the flow.

So instead of puting the certificate on the forwarder, I put it on the load balancer.

At least, now curl to the load balancer works fine.

Except the connexion is rejected : I keep having this message :

*  SSL certificate verify ok.
* OpenSSL SSL_write: Connexion ré-initialisée par le correspondant, errno 104
* Failed sending HTTP POST request
* Connection #0 to host splunk-hec.test left intact
curl: (55) OpenSSL SSL_write: Connexion ré-initialisée par le correspondant, errno 104

I'm lost as to how I'm supposed to configure the inputs.conf/server.conf on the forwarders.

Could'nt find any doc with examples about that, yet this is a validated HEC tier in splunk validated architecture documents.

Does anyone have a suggestion ?

Thanks in advance,

Ema

[emallinger]

Hi,

I read that doc page and yes, ackId seems to be unique per forwarder.

I'm sending from one channel only to 2 fw via one load balancer.

How am I supposed to know which event the status of one ackId is refering to when I don't know where I'm going to end (hence which forwarder) ?

How is resolved the need of High Availability for the endpoint (that's why I have a LB in front of my fw) with indexer ack ?

Thanks,

Ema

[VatsalJagani]

@emallinger  - I think you need to try on both forwarders also because they are sitting behind a load balancer.

[emallinger]

Hi,

Yes, but with 2 answers, how am I supposed to know which one is correct ?

I think it might be necessary to know which forwarder sent the ackId, in order to be able to identify the info I need.

Suppose I do.

Yet, I could keep asking the LB and never go back again to the forwarder which sent the ackID (hypothetically).

I don't think this reasonning can be used as a fix.

We might be missing some point in the "how to do that"..

Regards,

Ema

[emallinger]

Hi again,

use persistent session in the LB might be an option.

Was it tested that way ?

Thanks

Ema

[emallinger]

Hi,

Tested with cookie persistence and sticky persistence : could not make the thing work.

I do not know what's wrong.

There is not Server Hello sent to answer the Client Hello in any of the tries.

I'm lost...

Any idea ?

Thanks !

Eglantine

[VatsalJagani]

@emallinger - I believe AckID should be unique per forwarder.

Or if I misunderstood your question, then maybe what you are looking for is channel ID. Please read about it here - https://docs.splunk.com/Documentation/Splunk/8.2.6/Data/AboutHECIDXAck

I hope this helps!!!