I am trialing the Splunk Cloud software and having read through all the information on how to setup universal forwarders i've reached an impasse.
I believe i have setup the forwarder correctly: -
I have opened ports 8089 and 9997 inbound/outbound to ensure not firewall blocking traffic.
The documentation then seems to indicate that in the Splunk Cloud UI should see under under Settings --> Forwarding & Receiving option or a Forwarder under Data Inputs.
I don't see either and as such can setup a data source.
Could anyone advise if i have missed a step somewhere on client side universal forwarder setup or whether it is something within Splunk Cloud i have failed to do?
hi there,
according to the docs
When you work with forwarders to send data to Splunk Cloud, you must download an app that has the credentials specific to your Splunk Cloud instance. You install the forwarder credentials app on your universal forwarder, heavy forwarder, or deployment server, and it lets you connect to Splunk Cloud.
If everything is correct try following steps:
try doing telnet to the cloud instance from your splunk forwarder
telnet <IP> <port>
telnet 192.168.1.1 9997
and/or on your forwarder server run following commands
/splunkforwarder/bin/splunk list forward-server ( if all settings okay, it should come under Active forwards else Configured but inactive forwards)
/splunkforwarder/bin/splunk show deploy-poll ( will show the deployment server configured)
/splunkforwarder/bin/splunk list monitor (will list the files that splunk is watching)
also try doing tail or scan the end lines of splunkforwarder splunkd logs
/splunkforwarder/var/log/splunk/splunkd.log
Ps: in windows you can use cmd to run splunk CLI commands, instead / use \ for paths.
Ayush,
Thank you for these suggestions.
Regards the deployment server it suggests you can set up a universal forwarder on a windows server to forward direct to Splunk Cloud that shouldn't need an enterprise Splunk to act as a deployment server is this correct? Or does the Cloud version become the deployment server in this scenario?
Checked the logs and actually ma seeing loads of below errors appearing.
04-23-2021 16:46:07.058 +0100 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
Will try and test telnet connectivity next week as will need to open up ports and install.
splunk list forward-server
Active forwards: inputs.prd-p-vk6k0.splunkcloud.com:9997 (ssl) Configured but inactive forwards: prd-p-vk6k0.splunkcloud.com:9997
splunk show deploy-poll
Deployment Server URI is set to "prd-p-vk6k0.splunkcloud.com:8089".
splunk list monitor
Monitored Directories:
$SPLUNK_HOME\var\log\splunk
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\audit.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\btool.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\conf.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\dfm_stderr.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\dfm_stdout.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\first_install.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\health.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\license_usage.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log.1
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\mongod.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\remote_searches.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\scheduler.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\search_messages.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\searchhistory.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd-utility.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd_access.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd_ui_access.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\wlm_monitor.log
$SPLUNK_HOME\var\log\splunk\license_usage_summary.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\license_usage_summary.log
$SPLUNK_HOME\var\log\splunk\metrics.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log
$SPLUNK_HOME\var\log\splunk\splunk_instrumentation_cloud.log*
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunk_instrumentation_cloud.log
$SPLUNK_HOME\var\log\splunk\splunkd.log
C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log
$SPLUNK_HOME\var\log\watchdog\watchdog.log*
C:\Program Files\SplunkUniversalForwarder\var\log\watchdog\watchdog.log
$SPLUNK_HOME\var\run\splunk\search_telemetry\*search_telemetry.json
$SPLUNK_HOME\var\spool\splunk\...stash_new
Monitored Files:
$SPLUNK_HOME\etc\splunk.version
D:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\PELMAX761DEVSVR\SystemErr.log
D:\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\PELMAX761DEVSVR\SystemOut.log
D:\IBM\WebSphere\AppServer\profiles\Dmgr01\logs\dmgr\SystemErr.log
D:\IBM\WebSphere\AppServer\profiles\Dmgr01\logs\dmgr\SystemOut.log
Hi Ed,
Yes this is the same document i'm working off I think i must be inadvertently missign a step or missing one one thinking it's not needed, didn't do anythign with Deployment Server pre-req as thought the Cloud version didn't need when universal forwarder setup directly on a windows server manually?
Hi @AndyC1
is it a windows or linux forwarder? have you defined the inputs.conf on your forwarder?
if yes, can you share the inputs.conf and outputs.conf stanza?
Hi assabatini,
It is a windows server, I will have to check path locations for the .conf file and will post once have them though won't be until Monday now