Why am I not able to get data to Splunk Enterprise from another VM?

[druvakumar]

I've installed Splunk Enterprise on one VM and installed Universal Forwarder on another VM and I followed all the setup and all ports are opened but not able to get data onto Splunk Enterprise. I installed Universal Forwarder on the Splunk Server and followed the setup, where the data is passing to Splunk Enterprise.
Could someone help me here that what is the issue to get logs data from another machine?

[druvakumar · Accepted solution]

Anyway now it is working 🙂
I just uninstalled Splunk and UF then reinstalled Splunk and UF. Now I'm able to see my Forwarder instance.
but something weird, when I installed first time it didn't work now I followed the same process to install and setup the UF and it is working.

[druvakumar]

Thank you very much for your replies 🙂

[robgora_deloitt]

Sounds like you need to check a few things.

First validate that your Splunk Indexer is set to listen on port 9997.
Validate that port 9997 is open on your firewall for the System, for your Firewall Appliance and also on your UF System.
Validate that your outputs.conf on the UF has the correct Indexers and ports set.

Use something like Telnet on the UF System to validate that you can connect to the Indexer over port 9997.

Definitely check your splunkd.log on the UF and also on the Indexer.

Are you using a Deployment Server to manage your UFs? If so, can you see the UF check in with the Deployment Server?

[druvakumar]

my outputs.conf content is below:

#   Version 7.0.3
# DO NOT EDIT THIS FILE!
# Changes to default files will be lost on update and are difficult to
# manage and support.
#
# Please make any changes to system defaults by overriding them in
# apps or $SPLUNK_HOME/etc/system/local  
# (See "Configuration file precedence" in the web documentation).
#
# To override a specific setting, copy the name of the stanza and
# setting to the file where you wish to override it.

[tcpout:group1]
server=10.5.210.187:9997

[tcpout]
maxQueueSize = auto
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry|_jira)
forwardedindex.filter.disable = false
indexAndForward = true
autoLBFrequency = 30
autoLBVolume = 0
blockOnCloning = true
compressed = false
disabled = false
dropClonedEventsOnQueueFull = 5
dropEventsOnQueueFull = -1
heartbeatFrequency = 30
maxFailuresPerInterval = 2
secsInFailureInterval = 1
maxConnectionsPerIndexer = 2
forceTimebasedAutoLB = false
sendCookedData = true
connectionTimeout = 20 
readTimeout = 300
writeTimeout = 300 
tcpSendBufSz = 0
ackTimeoutOnShutdown = 30
useACK = false
blockWarnThreshold = 100
sslQuietShutdown = false

# The following provides modern TLS configuration that guarantees forward-
# secrecy and efficiency. This configuration drops support for old Splunk
# versions (Splunk 5.x and earlier).
# To add support for Splunk 5.x set sslVersions to tls and add this to the
# end of cipherSuite:
#     DHE-RSA-AES256-SHA:AES256-SHA:DHE-RSA-AES128-SHA:AES128-SHA
# and this, in case Diffie Hellman is not configured:
#     AES256-SHA:AES128-SHA
sslVersions = tls1.2
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256
ecdhCurves = prime256v1, secp384r1, secp521r1

[syslog]
type = tcp
priority = <13>
maxEventSize = 1024

[robgora_deloitt]

Are you running all of these from the Default folders? All custom changes to these files should be in the local folder. These all look to be the original files but with slight changes. The reason for this is that when you do an upgrade, files in the default folders will be overwritten. Nothing in the local folders will be overwritten.

Let me look over what you have here. But I would correct this part first if that is the case. If these are just a copy of what is in the default, I would get rid of these and only put the stanzas that you need in your inputs.conf under local and the same for your outputs.conf.

Also have you looked at the btool to see if there are any issues with the outputs.conf or your inputs.conf?

[druvakumar]

my inputs.confs content is below:

Version 7.0.3

DO NOT EDIT THIS FILE!

Changes to default files will be lost on update and are difficult to

manage and support.

Please make any changes to system defaults by overriding them in

apps or $SPLUNK_HOME/etc/system/local

(See "Configuration file precedence" in the web documentation).

To override a specific setting, copy the name of the stanza and

setting to the file where you wish to override it.

This file contains possible attributes and values you can use to

configure inputs, distributed inputs and file system monitoring.

[default]
index = default
_rcvbuf = 1572864
host = $decideOnStartup
evt_resolve_ad_obj = 0
evt_dc_name=
evt_dns_name=

[blacklist:$SPLUNK_HOME\etc\auth]

[monitor://$SPLUNK_HOME\var\log\splunk]
index = _internal

[monitor://C:\Program Files\Atlassian\Application Data\JIRA\log]
index = _jira

[monitor://$SPLUNK_HOME\var\log\splunk\license_usage_summary.log]
index = _telemetry

[monitor://$SPLUNK_HOME\etc\splunk.version]
_TCP_ROUTING = *
index = _internal
sourcetype=splunk_version

[batch://$SPLUNK_HOME\var\spool\splunk]
move_policy = sinkhole
crcSalt =

[batch://$SPLUNK_HOME\var\spool\splunk...stash_new]
queue = stashparsing
sourcetype = stash_new
move_policy = sinkhole
crcSalt =

[fschange:$SPLUNK_HOME\etc]

poll every 10 minutes

pollPeriod = 600

generate audit events into the audit index, instead of fschange events

signedaudit=true
recurse=true
followLinks=false
hashMaxSize=-1
fullEvent=false
sendEventMaxSize=-1
filesPerDelay = 10
delayInMills = 100

[udp]
connection_host=ip

[tcp]
acceptFrom=*
connection_host=dns

[splunktcp]
route=has_key:_replicationBucketUUID:replicationQueue;has_key:_dstrx:typingQueue;has_key:_linebreaker:indexQueue;absent_key:_linebreaker:parsingQueue
acceptFrom=*
connection_host=ip

[script]
interval = 60.0
start_by_shell = false

[SSL]

SSL settings

The following provides modern TLS configuration that guarantees forward-

secrecy and efficiency. This configuration drops support for old Splunk

versions (Splunk 5.x and earlier).

To add support for Splunk 5.x set sslVersions to tls and add this to the

end of cipherSuite:

DHE-RSA-AES256-SHA:AES256-SHA:DHE-RSA-AES128-SHA:AES128-SHA

and this, in case Diffie Hellman is not configured:

AES256-SHA:AES128-SHA

sslVersions = tls1.2
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ecdhCurves = prime256v1, secp384r1, secp521r1

allowSslRenegotiation = true
sslQuietShutdown = false

[script://$SPLUNK_HOME\bin\scripts\splunk-wmi.path]
disabled = 0
interval = 10000000
source = wmi
sourcetype = wmi
queue = winparsing
persistentQueueSize=200MB

default single instance modular input restarts

[admon]
interval=60
baseline=0

[MonitorNoHandle]
interval=60

[WinEventLog]
interval=60
evt_resolve_ad_obj = 0
evt_dc_name=
evt_dns_name=

[WinNetMon]
interval=60

[WinPrintMon]
interval=60

[WinRegMon]
interval=60
baseline=0

[perfmon]
interval=300

[powershell]
interval=60

[powershell2]
interval=60

[druvakumar]

I've all the ports opened (8000, 8089, 9997). I've installed UF on the VM where I installed Splunk Server, there I'm able to see that Forwarder Instance but not able to see the other VM instance on Splunk.

[druvakumar]

even telnet also checked it is connecting

[robgora_deloitt]

if you check the _internal index are you able to see the UF Host in the logs at all?

[druvakumar]

I've checked all the files looks fine to me. but not able to get the data

[richgalloway]

Did you check splunkd.log on the Universal Forwarder VM?
Did you use curl (or a similar tool) to verify you can access Splunk's port 9997 from the UF VM?