Getting Data In

Windows universal forwarder to Splunk Cloud issues

AndyC1
Observer

I am trialing the Splunk Cloud software and having read through all the information on how to setup universal forwarders i've reached an impasse.

I believe i have setup the forwarder correctly: -

  1. installed forwader
  2. incoprorated .spl certificate
  3. added logs to monitor
  4. added the forward-server details
  5. restarted splunk.

I have opened ports 8089 and 9997 inbound/outbound to ensure not firewall blocking traffic.

The documentation then seems to indicate that in the Splunk Cloud UI should see under under Settings --> Forwarding & Receiving option or a Forwarder under Data Inputs.

I don't see either and as such can setup a data source.

Could anyone advise if i have missed a step somewhere on client side universal forwarder setup or whether it is something within Splunk Cloud i have failed to do?

 

Labels (2)
0 Karma

ayush1906
Path Finder

hi there,

according to the docs

When you work with forwarders to send data to Splunk Cloud, you must download an app that has the credentials specific to your Splunk Cloud instance. You install the forwarder credentials app on your universal forwarder, heavy forwarder, or deployment server, and it lets you connect to Splunk Cloud.

If everything is correct try following steps:

try doing telnet to the cloud instance from your splunk forwarder

telnet <IP> <port>

telnet 192.168.1.1 9997

 

and/or on your forwarder server run following commands

/splunkforwarder/bin/splunk list forward-server   ( if all settings okay, it should come under Active forwards else Configured but inactive forwards)

/splunkforwarder/bin/splunk show deploy-poll    ( will show the deployment server configured)

/splunkforwarder/bin/splunk list monitor  (will list the files that splunk is watching)

 

also try doing tail or scan the end lines of splunkforwarder splunkd logs

/splunkforwarder/var/log/splunk/splunkd.log

 

Ps: in windows you can use cmd to run splunk CLI commands, instead / use \ for paths.

0 Karma

AndyC1
Observer

Ayush,

Thank you for these suggestions.

Regards the deployment server it suggests you can set up a universal forwarder on a windows server to forward direct to Splunk Cloud that shouldn't need an enterprise Splunk to act as a deployment server is this correct? Or does the Cloud version become the deployment server in this scenario?

Checked the logs and actually ma seeing loads of below errors appearing.

04-23-2021 16:46:07.058 +0100 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected

Will try and test telnet connectivity next week as will need to open up ports and install.

splunk list forward-server

Active forwards: inputs.prd-p-vk6k0.splunkcloud.com:9997 (ssl) Configured but inactive forwards: prd-p-vk6k0.splunkcloud.com:9997

splunk show deploy-poll
Deployment Server URI is set to "prd-p-vk6k0.splunkcloud.com:8089".

splunk list monitor

Monitored Directories:
$SPLUNK_HOME\var\log\splunk
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\audit.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\btool.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\conf.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\dfm_stderr.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\dfm_stdout.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\first_install.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\health.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\license_usage.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\mongod.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\remote_searches.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\scheduler.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\search_messages.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\searchhistory.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd-utility.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd_access.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd_ui_access.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\wlm_monitor.log
$SPLUNK_HOME\var\log\splunk\license_usage_summary.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\license_usage_summary.log
$SPLUNK_HOME\var\log\splunk\metrics.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log
$SPLUNK_HOME\var\log\splunk\splunk_instrumentation_cloud.log*
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunk_instrumentation_cloud.log
$SPLUNK_HOME\var\log\splunk\splunkd.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log
$SPLUNK_HOME\var\log\watchdog\watchdog.log*
C:\Program Files\SplunkUniversalForwarder\var\log\watchdog\watchdog.log
$SPLUNK_HOME\var\run\splunk\search_telemetry\*search_telemetry.json
$SPLUNK_HOME\var\spool\splunk\...stash_new
Monitored Files:
$SPLUNK_HOME\etc\splunk.version
D:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\PELMAX761DEVSVR\SystemErr.log
D:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\PELMAX761DEVSVR\SystemOut.log
D:\IBM\WebSphere\AppServer\profiles\Dmgr01\logs\dmgr\SystemErr.log
D:\IBM\WebSphere\AppServer\profiles\Dmgr01\logs\dmgr\SystemOut.log

0 Karma

edgarrity
Explorer
@AndyC1 , I found the setup of forwarders to the cloud tricky. However, when I followed the step-by-step process in https://docs.splunk.com/Documentation/SplunkCloud/8.1.2103/Admin/WindowsGDI it worked for me.
0 Karma

AndyC1
Observer

Hi Ed,

 

Yes this is the same document i'm working off I think i must be inadvertently missign a step or missing one one thinking it's not needed, didn't do anythign with Deployment Server pre-req as thought the Cloud version didn't need when universal forwarder setup directly on a windows server manually?

0 Karma

edgarrity
Explorer
My first attempt through the step-by-step using a Deployment Server to configure a Heavy Forwarder to send data to the cloud failed. I wound up with a Heavy Forwarder that could not provide the Web UI. So on my second attempt I just installed the forwarder config directly on the Heavy Forwarder and that was successful.
0 Karma

aasabatini
Builder

Hi @AndyC1 

is it a windows or linux forwarder? have you defined the inputs.conf on your forwarder?

if yes, can you share the inputs.conf and outputs.conf stanza?

 

0 Karma

AndyC1
Observer

Hi assabatini,

It is a windows server, I will have to check path locations for the .conf file and will post once have them though won't be until Monday now

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!