Splunk HEC Integration. Where to enable? Deployment Server or Log Server?

[lloydknight]

Hello,

Here's our Splunk setup:

3 Indexers (not clustered)
1 Search Head/Deployment Server
1 Log Server (acts like Heavy Forwarder)

All Splunk instances are reporting to Deployment Server and indexes are app-based.
Log Server is basically a log server where other integrations that doesn't require Forwarder use this to dump their logs and we push an application from Deployment server to monitor the logs in the Log server.

Questions:
1. Where should I enable HEC?
This will be my first time using an HEC and I'm not sure if I'll enable it to Log server or Deployment Server.

2. If I will enable it to our Log Server, will there be API logs that will be dumped/generated somewhere that I can just monitor using deployed monitor apps from Deployment Server?

3. If I will enable it to our Deployment Server, my concern here is if it will generate actual logs which should be done in Log Server and if it will not generate logs on the server, what is the behavior of the API logs upon enabling it to Deployment server?

4. Will the API logs directly be indexed on the defined index? (i.e. index=api_logs)

5. High level explanation for Indexer Acknowledgement?

Can someone kindly answer all my 5 questions?

PS.
Already read this btw:
http://dev.splunk.com/view/event-collector/SP-CAAAE73

Much appreciated!

[skalliger · Accepted solution]

Hi,

you might want to read this about HEC: https://www.splunk.com/blog/2017/03/30/syslog-ng-and-hec-scalable-aggregated-data-collection-in-splu...

Should answer 1-4 (1. syslog, 2. I don't get your question, but the blog should clear that, 3. In general, you want to forward your splunk data to your indexers anyways, see link below, 4. yes).

Indexer acknowledgment works like this: a fowarder sends data to the indexer. An indexer has to acknowledge the data before the forwarder sends other data. In case of no ACK, the data will be re-sent. An indexer acknowledges data after it has been written to the disk.

Additional links:
https://docs.splunk.com/Documentation/Splunk/6.6.0/DistSearch/Forwardsearchheaddata
https://docs.splunk.com/Documentation/Splunk/6.6.0/Forwarding/Protectagainstlossofin-flightdata

Skalli

[lloydknight]

big help. thanks