Troubleshooting Log Forwarding to Splunk Cloud from Private Network Machines

[rahusri2]

Hello,

I have a requirement to collect and monitor logs from several machines running in a private network. These machines are generating logs that need to be sent to Splunk Cloud for monitoring.

Here's what I've done so far:

1. Installed Universal Forwarder: I have installed the Splunk Universal Forwarder on each machine that generates logs.

2. Configured Forwarding: I used the command ./splunk add forward-server prd-xxx.splunkcloud.com:9997 to set the server address for forwarding logs to Splunk Cloud.

3. Set Up Monitoring: I added the directory to be monitored with the command ./splunk add monitor /var/log.

However, I'm unable to see any logs on the Splunk Cloud dashboard at "prd-xxx.splunkcloud.com:9997". I have a question regarding port 9997; it seems that this port should be open on Splunk Cloud, but I don't see an option to configure this in Splunk Cloud as there is no "Settings > Forwarding and Receiving > Receive data" section available.

How can I resolve this issue and ensure that logs are properly sent to and visible on Splunk Cloud?

Thanks.

[rahusri2]

Hello @kiran_panchavat,

Thanks for explaining this in very details, thanks for your time. Really appreciated.

[kiran_panchavat · Accepted solution]

@rahusri2 

Install the forwarder credentials on individual forwarders in *nix

From your Splunk Cloud Platform instance, go to Apps > Universal Forwarder.
Click Download Universal Forwarder Credentials.
Note the location where the credentials package splunkclouduf.spl has been downloaded.
Copy the file to a temporary directory, this is usually your "/tmp" folder.
Install the splunkclouduf.spl app by entering the following in command line: $SPLUNK_HOME/bin/splunk install app /tmp/splunkclouduf.spl.
When you are prompted for a user name and password, enter the user name and password for the Universal Forwarder. The following message displays if the installation is successful: App '/tmp/splunkclouduf.spl' installed.
Restart the forwarder to enable the changes by entering the following command: ./splunk restart.

I hope this helps, if any reply helps you, you could add your upvote/karma points to that reply, thanks.

[kiran_panchavat]

@rahusri2 Please check this documentation 

https://docs.splunk.com/Documentation/Forwarder/9.4.0/Forwarder/ConfigSCUFCredentials 

I hope this helps, if any reply helps you, you could add your upvote/karma points to that reply, thanks.

[kiran_panchavat]

@rahusri2 

1. Configure the `inputs.conf` file on your forwarders to monitor the `/var/log` directory and create an index on the indexers. 

2. Download the `outputs.conf` file (Splunk Cloud Platform universal forwarder credentials package )from Splunk Cloud.
- If there is no intermediate forwarder, you can directly apply the file to your universal forwarders.
- If you are using an intermediate forwarder, download the file from Splunk Cloud and apply it to the heavy forwarder or intermediate forwarder.

3. If you have a deployment server, retrieve the `outputs.conf`(Splunk Cloud Platform universal forwarder credentials package) file from Splunk Cloud and push it to the forwarders using the deployment server. If you do not have a deployment server and prefer to implement the configuration directly, you can apply it manually to the forwarders.

4. Restart the Splunk instance to apply the changes.

**Note:**

1. Ensure that the firewall rules between your on-premises environment and Splunk Cloud are properly configured.

2. A Splunk Cloud Platform receiving port is configured and enabled by default.

I hope this helps, if any reply helps you, you could add your upvote/karma points to that reply, thanks.

[kiran_panchavat]

@rahusri2 

When you work with forwarders to send data to Splunk Cloud Platform, you must download an app that has the credentials specific to your Splunk Cloud Platform instance. You install the forwarder credentials app on your universal forwarder, heavy forwarder, or deployment server, and it lets you connect to Splunk Cloud Platform.

I hope this helps, if any reply helps you, you could add your upvote/karma points to that reply, thanks.