ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

[AzmathShaik]

Hello

my splunkd.log shows the the following error
ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

i am pushing my outputs using my deployment server . the directory structure looks like
/apps/splunkforwarder/etc/deployment-apps/outputs/local/outputs.conf

and my outputs.conf looks like

[indexer_discovery:env-masternode]
master_uri = https://masternode:8089
pass4SymmKey = XXXXXXXXX
[tcpout]
defaultGroup = primary_indexers
forceTimebasedAutoLB = true
maxQueueSize = 7MB
useACK = true
[tcpout:primary_indexers]
autoLB = true
indexerDiscovery = env-masternode

and i verified that the outputs is downloaded from deployment server to my universal forwarder an it is under

/apps/splunkforwarder/etc/apps/outputs/local/outputs.conf

all configs looks fine. but why am getting this error??

[AzmathShaik]

./splunk cmd btool outputs list

this is the command i used

[hsesterhenn]

You output is hard to read...

Here an example from my test env:

/etc/system/default/outputs.conf                        [syslog]
 /etc/system/default/outputs.conf                        dropEventsOnQueueFull = -1
 /etc/system/default/outputs.conf                        maxEventSize = 1024
 /etc/system/default/outputs.conf                        priority = <13>
 /etc/system/default/outputs.conf                        type = udp
 /etc/apps/fwd_sendtoindexer/local/outputs.conf          [tcpout]
 /etc/system/default/outputs.conf                        ackTimeoutOnShutdown = 30
 /etc/system/default/outputs.conf                        autoLBFrequency = 30
 /etc/system/default/outputs.conf                        blockOnCloning = true
 /etc/system/default/outputs.conf                        blockWarnThreshold = 100
 /etc/system/default/outputs.conf                        compressed = false
 /etc/system/default/outputs.conf                        connectionTimeout = 20
 /etc/apps/fwd_sendtoindexer/local/outputs.conf          defaultGroup = default-autolb-group
 /etc/system/default/outputs.conf                        disabled = false
 /etc/system/default/outputs.conf                        dropClonedEventsOnQueueFull = 5
 /etc/system/default/outputs.conf                        dropEventsOnQueueFull = -1
 /etc/system/default/outputs.conf                        forceTimebasedAutoLB = false
 /etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.0.whitelist = .*
 /etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
 /etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.2.whitelist = (_audit|_introspection)
 /etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.filter.disable = false
 /etc/system/default/outputs.conf                        heartbeatFrequency = 30
 /etc/system/default/outputs.conf                        indexAndForward = false
 /etc/system/default/outputs.conf                        maxConnectionsPerIndexer = 2
 /etc/system/default/outputs.conf                        maxFailuresPerInterval = 2
 /etc/system/default/outputs.conf                        maxQueueSize = auto
 /etc/system/default/outputs.conf                        readTimeout = 300
 /etc/system/default/outputs.conf                        secsInFailureInterval = 1
 /etc/system/default/outputs.conf                        sendCookedData = true
 /etc/system/default/outputs.conf                        sslQuietShutdown = false
 /etc/system/default/outputs.conf                        tcpSendBufSz = 0
 /etc/system/default/outputs.conf                        useACK = false
 /etc/system/default/outputs.conf                        writeTimeout = 300
 /etc/apps/fwd_sendtoindexer/local/outputs.conf          [tcpout-server://10.204.240.180:9997]
 /etc/apps/fwd_sendtoindexer/local/outputs.conf          [tcpout:default-autolb-group]
 /etc/apps/fwd_sendtoindexer/local/outputs.conf          server = 10.204.240.180:9997


splunk list forward-server
Active forwards:
    None
Configured but inactive forwards:
    10.204.240.180:9997

I would switch of idxAck and IndexerDiscovery until you have everything up and running...

Are you sure your input is active?

Maybe keep it simple?

My outputs.conf...

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 10.204.240.180:9997

[tcpout-server://10.204.240.180:9997]

[hsesterhenn]

Hi,

please run

/apps/splunkforwarder/bin/splunk btool outputs list --debug | less 

on your forwarder and check whether the outputs.conf is really active.

Did the forwarder do a restart?

HTH,

Holger

[AzmathShaik]

if really i have problem with my outputs, i even should not see my internal logs. but all my internal logs are being forwarded and i can query them.

[hsesterhenn]

Please send the output of btool and search for TcpOutput error messages in index=_internal.

is your forwarder included in the search results if you do the following search?

| tstats count where index=* OR index=_* by host,index

HTH,

Holger

[AzmathShaik]

OUTPUT OF BTOOL:

[indexer_discovery:env-masternode]
master_uri = https://masternode:8089
pass4SymmKey = XXXXX
[syslog]
dropEventsOnQueueFull = -1
maxEventSize = 1024
priority = <13>
type = udp
[tcpout]
ackTimeoutOnShutdown = 30
autoLBFrequency = 30
blockOnCloning = true
blockWarnThreshold = 100
compressed = false
connectionTimeout = 20
defaultGroup = primary_indexers
disabled = false
dropClonedEventsOnQueueFull = 5
dropEventsOnQueueFull = -1
forceTimebasedAutoLB = true
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_introspection|_internal)
forwardedindex.filter.disable = false
heartbeatFrequency = 30
indexAndForward = false
maxConnectionsPerIndexer = 2
maxFailuresPerInterval = 2
maxQueueSize = 7MB
readTimeout = 300
secsInFailureInterval = 1
sendCookedData = true
sslQuietShutdown = false
tcpSendBufSz = 0
useACK = true
writeTimeout = 300
[tcpout:primary_indexers]
autoLB = true
indexerDiscovery = env-masternode

1. No results found for TcpOutput error messages in index=_internal

2. output of | tstats count where index= OR index=_ by host,index
   sqa01-ins01-scc51-dbs01 _internal 873298

[AzmathShaik]

there are no errors in outputs.
and the output of my ./splunk list forward-server

Active forwards:
None
Configured but inactive forwards:
None