topic Windows universal forwarder to Splunk Cloud issues in Getting Data In

Windows universal forwarder to Splunk Cloud issues

AndyC1 — Fri, 23 Apr 2021 08:33:26 GMT

I am trialing the Splunk Cloud software and having read through all the information on how to setup universal forwarders i've reached an impasse.

I believe i have setup the forwarder correctly: -

installed forwader
incoprorated .spl certificate
added logs to monitor
added the forward-server details
restarted splunk.

I have opened ports 8089 and 9997 inbound/outbound to ensure not firewall blocking traffic.

The documentation then seems to indicate that in the Splunk Cloud UI should see under under Settings --> Forwarding & Receiving option or a Forwarder under Data Inputs.

I don't see either and as such can setup a data source.

Could anyone advise if i have missed a step somewhere on client side universal forwarder setup or whether it is something within Splunk Cloud i have failed to do?

Re: Windows universal forwarder to Splunk Cloud issues

aasabatini — Fri, 23 Apr 2021 12:04:14 GMT

Hi @AndyC1

is it a windows or linux forwarder? have you defined the inputs.conf on your forwarder?

if yes, can you share the inputs.conf and outputs.conf stanza?

Re: Windows universal forwarder to Splunk Cloud issues

edgarrity — Fri, 23 Apr 2021 12:15:49 GMT

@AndyC1 , I found the setup of forwarders to the cloud tricky. However, when I followed the step-by-step process in https://docs.splunk.com/Documentation/SplunkCloud/8.1.2103/Admin/WindowsGDI it worked for me.

Re: Windows universal forwarder to Splunk Cloud issues

ayush1906 — Fri, 23 Apr 2021 12:33:37 GMT

hi there,

according to the docs

When you work with forwarders to send data to Splunk Cloud, you must download an app that has the credentials specific to your Splunk Cloud instance. You install the forwarder credentials app on your universal forwarder, heavy forwarder, or deployment server, and it lets you connect to Splunk Cloud.

If everything is correct try following steps:

try doing telnet to the cloud instance from your splunk forwarder

telnet <IP> <port>

telnet 192.168.1.1 9997

and/or on your forwarder server run following commands

/splunkforwarder/bin/splunk list forward-server ( if all settings okay, it should come under Active forwards else Configured but inactive forwards)

/splunkforwarder/bin/splunk show deploy-poll ( will show the deployment server configured)

/splunkforwarder/bin/splunk list monitor (will list the files that splunk is watching)

also try doing tail or scan the end lines of splunkforwarder splunkd logs

/splunkforwarder/var/log/splunk/splunkd.log

Ps: in windows you can use cmd to run splunk CLI commands, instead / use \ for paths.

Re: Windows universal forwarder to Splunk Cloud issues

AndyC1 — Fri, 23 Apr 2021 16:36:53 GMT

Ayush,

Thank you for these suggestions.

Regards the deployment server it suggests you can set up a universal forwarder on a windows server to forward direct to Splunk Cloud that shouldn't need an enterprise Splunk to act as a deployment server is this correct? Or does the Cloud version become the deployment server in this scenario?

Checked the logs and actually ma seeing loads of below errors appearing.

04-23-2021 16:46:07.058 +0100 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected

Will try and test telnet connectivity next week as will need to open up ports and install.

splunk list forward-server

Active forwards: inputs.prd-p-vk6k0.splunkcloud.com:9997 (ssl) Configured but inactive forwards: prd-p-vk6k0.splunkcloud.com:9997

splunk show deploy-poll
Deployment Server URI is set to "prd-p-vk6k0.splunkcloud.com:8089".

splunk list monitor

Monitored Directories:
$SPLUNK_HOME\var\log\splunk
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\audit.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\btool.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\conf.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\dfm_stderr.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\dfm_stdout.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\first_install.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\health.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\license_usage.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\mongod.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\remote_searches.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\scheduler.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\search_messages.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\searchhistory.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd-utility.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd_access.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd_ui_access.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\wlm_monitor.log
$SPLUNK_HOME\var\log\splunk\license_usage_summary.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\license_usage_summary.log
$SPLUNK_HOME\var\log\splunk\metrics.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log
$SPLUNK_HOME\var\log\splunk\splunk_instrumentation_cloud.log*
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunk_instrumentation_cloud.log
$SPLUNK_HOME\var\log\splunk\splunkd.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log
$SPLUNK_HOME\var\log\watchdog\watchdog.log*
C:\Program Files\SplunkUniversalForwarder\var\log\watchdog\watchdog.log
$SPLUNK_HOME\var\run\splunk\search_telemetry\*search_telemetry.json
$SPLUNK_HOME\var\spool\splunk\...stash_new
Monitored Files:
$SPLUNK_HOME\etc\splunk.version
D:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\PELMAX761DEVSVR\SystemErr.log
D:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\PELMAX761DEVSVR\SystemOut.log
D:\IBM\WebSphere\AppServer\profiles\Dmgr01\logs\dmgr\SystemErr.log
D:\IBM\WebSphere\AppServer\profiles\Dmgr01\logs\dmgr\SystemOut.log

Re: Windows universal forwarder to Splunk Cloud issues

AndyC1 — Fri, 23 Apr 2021 16:38:32 GMT

Hi Ed,

Yes this is the same document i'm working off I think i must be inadvertently missign a step or missing one one thinking it's not needed, didn't do anythign with Deployment Server pre-req as thought the Cloud version didn't need when universal forwarder setup directly on a windows server manually?

Re: Windows universal forwarder to Splunk Cloud issues

AndyC1 — Fri, 23 Apr 2021 16:41:04 GMT

Hi assabatini,

It is a windows server, I will have to check path locations for the .conf file and will post once have them though won't be until Monday now

Re: Windows universal forwarder to Splunk Cloud issues

edgarrity — Fri, 23 Apr 2021 16:48:40 GMT

My first attempt through the step-by-step using a Deployment Server to configure a Heavy Forwarder to send data to the cloud failed. I wound up with a Heavy Forwarder that could not provide the Web UI. So on my second attempt I just installed the forwarder config directly on the Heavy Forwarder and that was successful.