Our setup is as follows:
Our on prem servers have universal forwarders on them and forwarder to the HF which then sends to splunk cloud.
We are starting to spin up EC2 instances in AWS and want to do the same monitoring, so UF installed on the instance and forwarding to splunk cloud.
My question is how do we do this?
It seems a bit daft to send our logs back to our on-premis HF to then send to the cloud.
So should we create a HF in our AWS VPC and point all our ec2 instances towards that?
How has everyone else tackled this issue?
You should be able to set up a UF in AWS the same way you did for your on-prem HF. If doesn't matter if it's UF or HF as the outputs.conf settings are the same. You will, however, need to check your AWS security groups to make sure the UF is allowed to connect to Splunk Cloud.
As an aside, are you sure you need the intermediate HF in your on-prem space? It's a bottleneck, single point of failure, and impairs performance.
Okay I was thinking we can just use a UF instead.
I agree about the bottleneck and single point of failure but we were told it is best practice to point towards an HF before sending to the cloud.
It is also where all our SaaS add-ons are configured so we do need it in some capacity.
If you want to use UF then you can directly send data to Splunk cloud but the UF will not parse the data as it will only forward the data to the Splunk cloud indexer and for that, you have to just put the config in outputs.conf of UF and in this case parsing and indxing will be done by Splunk cloud indexer.
If you will use HF only then it will be a better option, As it will parse the data and will send it to Splunk cloud for indexing and in this case we don't have to use UF and need to put the same config in outputs.conf as per option1.