How to set up a heavy forwarder to forward data to Splunk Cloud?

[Dayane_tr]

I didn't find the cloud documentation very clear...

Do I need to install splunk enterprise separately to have heavy for warder and then configure my splunk cloud license?

Do I need to ask splunk support for an enterprise license?
After all, how do I configure a heavy forwarder? And what address do I put in Universal forwarder? From the IP or hostname cloud?

I've read the following threads and it gets more and more confused:

https://www.splunk.com/en_us/resources/videos/splunk-cloud-tutorial.html

https://community.splunk.com/t5/Getting-Data-In/How-to-set-up-a-heavy-forwarder-to-forward-data-to-S...

https://docs.splunk.com/Documentation/SplunkCloud/8.2.2202/Admin/WindowsGDI  Step2

Can you help me please?

Regards

[Roy_9]

@Dayane_tr  You could follow the below steps to set up a HF which can be used for filtering/routing the logs.

1. Spun up a machine and install Splunk Enterprise license(full blown) by downloading it from splunk website.

2.Since you want to send this logs to Splunk cloud, you need to download UF credentials package from splunk cloud SH and deploy it under /opt/splunk/etc/apps where this package has SSL cert info and all the indexers addresses.

3. Open a FW connection between the HF and the splunk cloud indexer hosts(host names available in the above downloaded UF credential package)  to listen on 9997 so that it wont interrupt indexing.

That's it, you are good to go.

[Dayane_tr]

Hello @Roy_9 

Thank you very much! I've seen this solution but they say you don't need to install SPLUNK enterprise. Just have an HF with the app provided on SPLUNK cloud, which will send it to the cloud. And the UF installed in on primeses (Linux/Windows machines).

Some here said they don't need SPLUNK enterprise so I tell you this. I saw in several forums this question of mine and some just closed with no solution.

thanks

[SinghK]

Heavy forwarders do not need license they come in with a forwarders license which you can enable. 

now on a hf  you can install addons specific to data you want to gather and create inputs to gather data. 
there has to be a outputs config for forwarder to forward data to indexers in splunk cloud.

universal forwarder is entirely diff forwarder and easiest to configure. Install that on a windows or Linux box you want to monitor for logs or metrics data and add and configure respective Addons and point it to a forwarding location.

and you should be all set.

[Dayane_tr]

in short I need to install a Heavy Forwarder. I know that universal installs on the machine on primese for monitoring! but I want the UF to forward it to the HF and it forward it to the cloud… I don't know if it was clear. How can I install an HF? does it need splunk enterprise or can it install it by itself on a server?
Sorry, I'm new and it's my first contact with SPLUNK cloud!

I just want an HF to forward my data to SPLUNK CLOUD and then know how to activate the license (when downloading the “app” the .spl file)

Regards

[gcusello]

Hi @Dayane_tr,

it's a best practice to have at least two Heavy Forwarders to use as concentrators for the other forwarders (Universal or Heavy)of your inrastructure.

Heavy Forwarders are a complete Splunk Enterprise installayion that forwardrs all its logs to Splunk Cloud.

The thing that @SinghK is trying to explain is that you don't need it but it's a best practice to avoid to open too many connections between your servers and Splunk Cloud.

In addition, best practices hint to use at least two HFs to avoid Single Point of Failures.

About the license, you can configure your HFs using the Forwarder License so you don't need a Splunk Enterprise license (for more infos see at https://docs.splunk.com/Documentation/Splunk/8.2.6/Admin/TypesofSplunklicenses#Forwarder_license).

Ciao.

Giuseppe

[Dayane_tr]

Hi @gcusello 

Many thanks for your explanation. The part of the SPLUNK cloud license to activate an enterprise has been explained…. now what I need to know is: install Heavy Forwarder alone on a server or on the same server as Enterprise? I want to keep only 1 HF because there are few resources “on primeses” that I will monitor in the splunk cloud, the rest is in the cloud, for example Office 365… this one connects directly to the cloud.

Regards 🙂 

[gcusello]

Hi @Dayane_tr,

the Heavy Forwarder is the only instance you need to concentrate and forwarder logs to Splunk Cloud, you don't need another instance (the one that you call "Enterprise").

As @PickleRick hinted, as concentrator you could also use a Universal Forwarder, in this way you could have the resources for two machines to avoid a Single Points of Failure.

huse of an HF is also a good idea if you have to manage the UFs in your network because, if you have less than 50 UFs, you could use the HF Concentrator also as Deployment Server.

If instead you have to manage more than 50 UFs, you have to use a dedicated Deployment Server and you could use one or (better) tow UFs as Concentrors.

Ciao.

Giuseppe

[Dayane_tr]

Hi, @gcusello 

I have less than 50 UF as most services connect directly to the cloud (Of365 and Azure).

It would only be the firewall and anti virus on primeses... I would like to have an HF to send to the cloud instead of sending from the UF right. From what I read, the HF consolidates the data better and has technologies that are not compatible with the UF.

Thanks

[gcusello]

Hi @Dayane_tr,

I don't know where you read these things.

Anyway, HFs and UFs are different components:

HFs are full Splunk instances, with web interface and the possibility (usually not used) to index data, they cook data, in other words they make the jobs in the marging and parsing phases, for this reason you cannot filter or modify data on Indexers if you have an HF before.

UFs are limited Splunk instances with the only target to input logs, they don't elaborate them, they take logs and send them to Indexers (eventually passing through othe UFs or HFs).

Obviously you systems are directly connected to O365 and Azure, but usually to send data to Splunk Cloud is Splunk hints as best practice to have at least two HFs or UFs to concentrate logs and avoid to open a connection between all your systems and Splunk Cloud, but anyway is not recommended but possible.

At the end, you need also to manage configurations of your UFs, the best way to do this is using a Deployment Servers.

DS is a Full Splunk instances that makes only this role, otherwise you have to manually distribute configurationa to all Forwarders and it isn't a good idea!

If you have more than 50 clients to manage DS must be a dedicated system, if less, you could also use une Splunk full instace that make this role and also another (e.g. like HF).

For this reason I hinted that you could use an HF as concentrator and DS.

I hope to be clear.

If you want to know more things about Splunk architectures, I hint to follow a free course:

https://education.splunk.com/free

https://www.youtube.com/watch?v=lJtm27sg2FU

https://education.splunk.com/course/splunk-infrastructure-overview

Ciao.

Giuseppe

[PickleRick]

No, it's not like that.

HF is really a full instance of on-premise Splunk Enterprise. The only difference is the fact that it doesn't do local indexing (and might use a different license if installed that way).

The difference in your case would be that this intermediate HF would do all the ingest-stage activity except for the writing to indexes themselves. So you'd have to have all the apps that are responsible for proper event breaking, timestamp parsing and any other index-time activity for the sources "passing through" this HF.

Splunk simply does all this "heavy" stuff at first "heavy" component (HF or indexer) it encounters on the event's path. So if your UFs send directly to indexers, the indexers do the dirty work. If you have UF sending to HF and those send to indexers, it's the HFs which do the dirty work. The downside of using HFs in front of indexers is that the data in parsed format - ready for writing to indexes - as they are sent pre-processed from the HFs to indexers are much bigger than the actual raw data. So you use up much more bandwidth. That's why Splunk advises strongly against using HFs unless they're absolutely needed.

And the two cases when they are absolutely needed are:

1) when you need to have modular inputs since they rely on python included with the full Splunk Enterprise installation and not included in UF

2) when you need to filter your events before sending them out to indexers, because you can't do all this parsing and transforming stuff on UFs

For simple aggregating data from multiple sources and sending them to indexers (in your case - to your Splunk Cloud instance) UF is perfectly sufficient.

[Dayane_tr]

Hello, @PickleRick

Okay, I think I understand what you mean.

But UF is what we install on servers and desktops, right? Correct me if I'm talking nonsense.

My client had an HF and UF at the same time, hence all this confusion. I am taking over the project because the old company hired by the client is no longer in action.

I created a free license for testing and installed UF on my windows machine but I am not getting data. What am I doing wrong?

Thank you very much 🙂

[PickleRick]

You don't need a HF just to forward data. An intermediate UF will do just fine.

[Dayane_tr]

Hi, @PickleRick

UF is installed directly on physical machines. I would like to have the HF to consolidate the data correctly. As a practice, it is not recommended to send directly from the UF to the cloud, despite the fact that the data is encrypted.

The question is whether to have the HF in a separate environment, that is, only it is installed in a Debian for example, because in this documentation it talks about the configuration but for Splunk enterprise:

https://docs.splunk.com/Documentation/SplunkCloud/8.2.2202/Forwarding/Deployaheavyforwarder

This documentation talks about how to configure Splunk Cloud to get data from Windows:

https://docs.splunk.com/Documentation/SplunkCloud/8.2.2202/Admin/WindowsGDI#Step_2:_Set_up_your_Splu...

But it says it needs Splunk enterprise.

https://docs.splunk.com/Documentation/SplunkCloud/8.2.2202/Data/UsingforwardingagentsCloud

"If you want to set up a heavy forwarder to send data in Splunk Cloud Platform, request a deployment server license from Splunk support to allow them to carry out functions above and beyond what is covered by the forwarder license. See Data collection in the Splunk Cloud Platform Service Description."
"The main difference between a universal forwarder and a heavy forwarder is that the heavy forwarder contains the full parsing pipeline, performing the identical functions an indexer performs, without writing and indexing events on disk."

Do you understand why so much doubt?

I just want to send data to the splunk cloud in the most secure way and what was reported by some colleagues is that an HF is needed to make this "conversation" with the cloud and not install UF on the machines on primeses directly to the cloud.

Regards

[PickleRick]

UF is just what it says - a Forwarder - it collects data from inputs, sends it to output(s), does a tiny little bit of work if configured for indexed extractions (that's one exception from the rule that no parsing whatsoever happens on a UF) and that's it.

I've never heard about the rule "don't send directly to cloud". On the contrary - Splunk advises to send directly from UFs to the destination component (indexers or the cloud) instead of aggregating separate data streams of network level since it helps with load-balancing across multiple indexers. One of the inputs can be a "splunk input" which will simply listen for connection from other UFs. I've had several configurations like that - you use them in situations where you have a separated site and are allowed only limited network connectivity. Then most of the network communication is intra-site and only the outgoing connection from the UF would be outbound-destined. But if you needed to use an app which creates modular inputs (like DBX or Checkpoint OPSEC LEA) in such site, you'd need a HF.

As long as you have your connection between forwarder (of any kind - HF or UF) and the indexers properly configured with TLS, there's no concern of "greater security" in case of any kind of forwarder. The protocol is the same, the authentication mechanisms are the same. There's no difference here.

UF is what we install on desktops and servers to fetch data with. But UFs can listen on network ports and can run scripted inputs (like some inputs privided by TA_windows add-on).

As to why your setup is now working - hard to say without a relevant config parts and logs 😉

[Dayane_tr]

Hello @PickleRick

I sent you a private message... 

Can you help me?

Thanks

[PickleRick]

Post your inputs.conf and outputs.conf (redacted of course since you probably don't want to disclose your cloud instance details) and excerpt of splunkd.log concerning outputs and/or inputs, we'll see if there's anything reasonable there.

[Dayane_tr]

Hello @PickleRick 

I did the following... I installed the UF on my windows machine, I installed the license file (splunkclouduf.spl), I determined the index of the windows events and I can now receive it in my cloud trial environment.

Can I install an HF on a debian and when I install the UF on the local machines send it to the HF and the HF send it to the cloud?

Thanks

[PickleRick]

It's not a license file, because UF doesn't need any external means of licensing. It's an app with definitions of output parameters. It's created for you to make forwarders deployment easier because you could get the same results configuring those parameters manually. See https://docs.splunk.com/Documentation/Forwarder/8.2.6/Forwarder/ConfigSCUFCredentials

If you want to install a multi-tiered event-collection environment you must install this app only on the forwarder (either UF or HF) which will be sending directly to the cloud. On it you'd have to also configure inputs to receive events from your other forwarders. And then you configure other forwarders to send events to this central forwarder.

[Dayane_tr]

Hello @PickleRick 

Come on... the client has it in its structure (O365, Azure, Fortigate, SentinelOne).

Azure and Office 365 communicate directly with the cloud.
Fortigate and SentinelOne I wanted to upload to a debian server that forwards to the cloud. Do you have any structure drawings you can share?

Second: I installed UF on my local Windows 10 machine and installed the "app", which is just a configuration file(this in my cloud trial framework), if I were to install UF on another machine I would have to install this APP together on all local machines ? or is this app only installed on a "deployment server"?

This is "app" is it installed only once? That is the question.

Thank you so far.

[PickleRick · Accepted solution]

As @gcusello said - it's a bit complicated to advise a proper architecture for a particular case. There are many conditions and border cases which might need to be taken into account. As I see, there are various source systems which need to be connected (and for each there should be a propper connection type chosen and configured). It looks like a fully blown deployment project for which you should contact either Professional Services or your local Splunk partner who employs skilled architects which will do that with you.