HEC configuration for AWS logs

[Karthikeya]

We have a architecture of 3 site multi cluster which contains 6 indexers (2 in each site), 3 search heads (one in each), 1 Deployment server, 2 CMs(active and standby), Deployer, and these instances residing in AWS cloud and we are in Splunk enterprise 9.1.4

Our requirement is AWS logs will be pushed to our Splunk environment. So they are pushing Cloudwatch logs from Kinesis data streams and Lambda configured (no idea on all these). But they asked our team to create HEC token and give token details and endpoint details to them in order to push logs to Splunk. They don't want Amazon firehouse add-on here. They prefer only through HEC.

Now my doubt is where to configure this HEC token? I want all my 6 indexers to be load balances data receiving from AWS (considering it will be huge data). I have gone through HEC docs but it is unclear. Can someone help me with the step by step procedure how can I acheive it? Like where to create HEC token and all these?

Thank you.

[PickleRick]

As usual, you have a relatively simply sounding problem which might not turn out to be so simple.

As @marnall already pointed out - the HEC input is just an input on the component you define it on.

So you can have multiple options here.

1) Create a HEC input on each indexer using the same HEC token and let the sources load-balance their requests between the receivers. But that requires the sources which can actively do load-balancing.

2) Deploy an external HTTP LB which will distribute HTTP requests to HEC inputs defined on indexers (again - with the same HEC token).

3) Create a HF with a HEC input which will receive data and distribute it to indexers using normal s2s load-balancing mechanisms

4) Create multiple HFs with HEC input and either LB between them on source or set up a HTTP LB in front of them.

Each of those scenarios have their own pros and cons (simplicity, cost, maintainability, resilience).

[Karthikeya]

Hi @PickleRick , thanks for the options.

I think we are going with the 1st option. we are going to create AWS ELB for load balancing. But I am confused how to create HEC token on all indexers with the same token? My approach is this please correct me if I am wrong...

Going to configure HEC on cluster manager first.. Then take those inputs.conf from etc/apps/http_input/local/ inputs.conf and paste it under etc/manager-apps/hec_input/local/inputs.conf and paste it here. But won't enable http in CM. (disabled = 1 for http) But http:inputname will be enabled (disabled = 0) will it still index HEC data in cluster manager (which should not do). Then push this configurations (app with inputs.conf) through configuration bundle to all my indexers so that each indexer will receive this inputs.conf. additionally should I need to go to each indexer and give [http] disabled = 0 to enable HEC in each indexer in order to receive data? Please confirm and correct me if I am wrong anywhere 

[PickleRick]

You don't define HEC input on CM. CM will not be ingesting your HEC events.

You define HEC input (either just a token if you have the general HTTP input already enabled or enable whole HTTP input, define TLS parameters and define a token - for maintainability I'd split it into two separate apps - one with general HEC input settings, another with a token definition for your particular input needs) with CM in an app which is pushed to indexers.

So the only thing you should do (as with pretty much everything regarding configuring your indexers) is creating an app in the manager-apps directory and pushing it with a new configuration bundle to your indexers. I'm not sure if enabling HTTP input in general in case you hadn't had one already will not require you to rolling-restart your indexers (and that's why I prefer a dedicated HF-layer in front of indexers so that changes to ingestion process and parsing do not cause indexer restarts but that's a topic for another time.

[Karthikeya]

@PickleRick @two separate apps - one with general HEC input settings, another with a token definition for your particular input needs) with CM in an app which is pushed to indexers...

So here I created two apps under etc/master-apps and in one enabled http and in other given token and other parameters. So once we push this to indexers.. HEC will be enabled right in indexers? Or do we need to edit each indexer etc/apps/http_input/local/inputs.conf and give [http] disabled = 0 in order to enable? Or already pushed app will do the similar work?

[PickleRick]

You should not need to enable the inputs separately on each indexer. That's what the CM-pushed apps are for.

[isoutamo]

As @PickleRick said I suppose that best option is just set up at least two separate HF to manage actual HEC inputs. Then add LB before those. 

Still you should use those two apps: one for enable HEC interface and another for actual HEC token and maybe props and transforms conf if you need to manipulate those events. 

In long run it could be even easier to manage each token in it's own app, but this it totally depending on your needs and which kind of environment you have (e.g. several dev, test, stage, UAT, prod and several integrations going on at same time).

Anyhow don't use indexers as HEC receivers as there are too many times when those needs to do rolling restarts when you are managing those tokens!

And You could generate a valid token with uuidgen command in any linux node or there are also some web pages too for this.

[Karthikeya]

@isoutamo @PickleRick @what could be the consequences for HEC data if indexers get rolling restarts everytime? Loss of data? Loss of token? Please explain

[PickleRick]

Ok. Let's get it out into the open - rolling restart is one of the basic concepts of managing Splunk environments, multisite cluster is one of the more complicated setups. You're trying to manage the latter without understaing the former. I understand that you might simply have inherited an environment and "someone has to" take care of it but I'd urge you to take some proactive steps to learn how to do stuff properly. Not just as asking case by case for help, but getting some more organized training.

Having said that - rolling restart is a mechanism with which you automatically restart clustered indexers or search heads but not all at once. That's why it's called rolling. The process is managed by either CM in case of indexers or captain in case of search heads and only part of your components is restarted at the same time until the whole layer is restarted. How big this part is is configurable (if I remember correctly it's 20% by default meaning that at any given time 20% of your indexers can be down due to the ongoing rolling restart).

On its own it doesn't cause any configuration changes (although it's typically triggered by them) so no "token loss" whatever that could mean.

I can imagine some badly engineered environments where rolling restart would cause interruptions in data ingestion but in a well designed setup it shouldn't.

There are two most important effects of rolling restarts of indexers.

1. Your data availability during restart is reduced so the searches spawned during that time might return wrong/incomplete results.

2. Since Splunk limits some replication-related activities during rolling restart you have reduced resiliency in that time and your environment is more prone to data lose should anything like server crash happen during this process.

Additionally, roling restart as any restart if done too often can create many small buckets.

[isoutamo]

Probably most important thing is that rolling restarts affects your searches every time when this happens even there is options to avoid it. The most important thing is that it affects all alerts, reports etc which are running when indexers or SH in SHC are restarted by rolling restart.

The implementation is somehow broken still. There are ideas to fix this in ideas.splunk.com, but no estimates if splunk have plan and capabilities to fix it.

[Karthikeya]

@isoutamo @PickleRick so everytime I push configuration bundle from CM to indexers, rolling restart will be happened everytime for indexers? 

[isoutamo]

This depends on what changes you are deploying. Some needs restart and son don’t. You can found more information from docs.splunk.com. 

Here is a basic information of clustering https://docs.splunk.com/Documentation/Splunk/9.4.1/Indexer/Basicclusterarchitecture

You should read and try to understand what it means. Unfortunately that doc doesn’t tell everything as it takes too much space and to be honest most of us don’t need to know all those details.

[PickleRick]

I wouldn't call it broken. Short of rebalancing buckets between restarting each single indexer (which is completely ridiculous) you can't make sure that every bucket is searchable throughout the rolling restart process.

If you have RF=SF>=2, then it's all just a matter of reassigning primaries and maybe restarting only one indexer at a time (which can be a long process if you have a big setup). But what if you have RF=SF=1? (yes, I've seen such setups).

So yes, it's a bit frustrating but I wouldn't call it broken.

[Karthikeya]

@PickleRick we already have HF configured in our environment. So how to do it?

[PickleRick]

As I said before in another thread - this is something that should be best discussed with your local Splunk Partner.

I can tell you that you can just enable HEC input on the HF and send all the data there so that the HF distributes it to the indexers but there are other possible issues with this setup. Since you probably have just one HF, you're gonna be creating a SPOF. It might also not be able to process all the load you're gonna push on it.

That's why at the start I said that this problem only looks easy. There are several approaches to solving it which might look right from the technical point of view, meaning that each of them "should work" but some of them might be better from the business point of view given specific circumstances unique to your situation and environment.

[marnall]

The HTTP Event Collector won't do load balancing itself, so you will need to set up a load balancer in front of the indexers.

One way you could set up the HEC token is to take a Splunk server with a web interface (probably not the indexers), go to Settings->Data inputs->HTTP Event Collector, then click the "New Token" button. Go through the menu specifying your desired input name, sourcetype, index, etc. This will generate an inputs.conf stanza for the HTTP input. You can then open the inputs.conf file and copy this stanza to each of your indexers to ensure they have the same token.

(Remaining instructions assume your indexers are running Linux)

For me, the inputs.conf file was generated in /opt/splunk/etc/apps/launcher/local, because I went to the HTTP Event Collector web interface from the main Splunk Enterprise screen.

The stanza will look like this: (with different values, of course)

[http://inputname]
disabled = 0
host = yourhostname
index = main
indexes = main
source = inputsourcetype
token = fe2cfed6-664a-4d75-a79d-41dc0548b9de

Of course, you should change the host value for each indexer or remove the host line so that the host value is decided on startup.

Then, create a new file on each indexer at:

/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf

Containing this text:

[http]
disabled = 0

This will enable the HTTP event collector on the indexers.

You can check that the HTTP event listener is opening the port on the indexer by using netstat:

netstat -apn | grep 8088

[Karthikeya]

@marnall Where should I create HEC token through web interface? In cluster manager or deployment server?

And do we need to copy inputs.conf which is generated initially to each of the indexers?

And once we copy it do we need to remove the data input created initially because of we don't remove data will index to that component also right? Please confirm?