Monitoring Splunk

ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

AzmathShaik
Path Finder

Hello

my splunkd.log shows the the following error
ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

i am pushing my outputs using my deployment server . the directory structure looks like
/apps/splunkforwarder/etc/deployment-apps/outputs/local/outputs.conf

and my outputs.conf looks like

[indexer_discovery:env-masternode]
master_uri = https://masternode:8089
pass4SymmKey = XXXXXXXXX
[tcpout]
defaultGroup = primary_indexers
forceTimebasedAutoLB = true
maxQueueSize = 7MB
useACK = true
[tcpout:primary_indexers]
autoLB = true
indexerDiscovery = env-masternode

and i verified that the outputs is downloaded from deployment server to my universal forwarder an it is under

/apps/splunkforwarder/etc/apps/outputs/local/outputs.conf

all configs looks fine. but why am getting this error??

0 Karma

AzmathShaik
Path Finder

./splunk cmd btool outputs list

this is the command i used

0 Karma

hsesterhenn
Path Finder

You output is hard to read...

Here an example from my test env:

/etc/system/default/outputs.conf                        [syslog]
 /etc/system/default/outputs.conf                        dropEventsOnQueueFull = -1
 /etc/system/default/outputs.conf                        maxEventSize = 1024
 /etc/system/default/outputs.conf                        priority = <13>
 /etc/system/default/outputs.conf                        type = udp
 /etc/apps/fwd_sendtoindexer/local/outputs.conf          [tcpout]
 /etc/system/default/outputs.conf                        ackTimeoutOnShutdown = 30
 /etc/system/default/outputs.conf                        autoLBFrequency = 30
 /etc/system/default/outputs.conf                        blockOnCloning = true
 /etc/system/default/outputs.conf                        blockWarnThreshold = 100
 /etc/system/default/outputs.conf                        compressed = false
 /etc/system/default/outputs.conf                        connectionTimeout = 20
 /etc/apps/fwd_sendtoindexer/local/outputs.conf          defaultGroup = default-autolb-group
 /etc/system/default/outputs.conf                        disabled = false
 /etc/system/default/outputs.conf                        dropClonedEventsOnQueueFull = 5
 /etc/system/default/outputs.conf                        dropEventsOnQueueFull = -1
 /etc/system/default/outputs.conf                        forceTimebasedAutoLB = false
 /etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.0.whitelist = .*
 /etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
 /etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.2.whitelist = (_audit|_introspection)
 /etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.filter.disable = false
 /etc/system/default/outputs.conf                        heartbeatFrequency = 30
 /etc/system/default/outputs.conf                        indexAndForward = false
 /etc/system/default/outputs.conf                        maxConnectionsPerIndexer = 2
 /etc/system/default/outputs.conf                        maxFailuresPerInterval = 2
 /etc/system/default/outputs.conf                        maxQueueSize = auto
 /etc/system/default/outputs.conf                        readTimeout = 300
 /etc/system/default/outputs.conf                        secsInFailureInterval = 1
 /etc/system/default/outputs.conf                        sendCookedData = true
 /etc/system/default/outputs.conf                        sslQuietShutdown = false
 /etc/system/default/outputs.conf                        tcpSendBufSz = 0
 /etc/system/default/outputs.conf                        useACK = false
 /etc/system/default/outputs.conf                        writeTimeout = 300
 /etc/apps/fwd_sendtoindexer/local/outputs.conf          [tcpout-server://10.204.240.180:9997]
 /etc/apps/fwd_sendtoindexer/local/outputs.conf          [tcpout:default-autolb-group]
 /etc/apps/fwd_sendtoindexer/local/outputs.conf          server = 10.204.240.180:9997


splunk list forward-server
Active forwards:
    None
Configured but inactive forwards:
    10.204.240.180:9997

I would switch of idxAck and IndexerDiscovery until you have everything up and running...

Are you sure your input is active?

Maybe keep it simple?

My outputs.conf...

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 10.204.240.180:9997

[tcpout-server://10.204.240.180:9997]

0 Karma

hsesterhenn
Path Finder

Hi,

please run

/apps/splunkforwarder/bin/splunk btool outputs list --debug | less 

on your forwarder and check whether the outputs.conf is really active.

Did the forwarder do a restart?

HTH,

Holger

0 Karma

AzmathShaik
Path Finder

if really i have problem with my outputs, i even should not see my internal logs. but all my internal logs are being forwarded and i can query them.

0 Karma

hsesterhenn
Path Finder

Please send the output of btool and search for TcpOutput error messages in index=_internal.

is your forwarder included in the search results if you do the following search?

| tstats count where index=* OR index=_* by host,index

HTH,

Holger

0 Karma

AzmathShaik
Path Finder

OUTPUT OF BTOOL:

[indexer_discovery:env-masternode]
master_uri = https://masternode:8089
pass4SymmKey = XXXXX
[syslog]
dropEventsOnQueueFull = -1
maxEventSize = 1024
priority = <13>
type = udp
[tcpout]
ackTimeoutOnShutdown = 30
autoLBFrequency = 30
blockOnCloning = true
blockWarnThreshold = 100
compressed = false
connectionTimeout = 20
defaultGroup = primary_indexers
disabled = false
dropClonedEventsOnQueueFull = 5
dropEventsOnQueueFull = -1
forceTimebasedAutoLB = true
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_introspection|_internal)
forwardedindex.filter.disable = false
heartbeatFrequency = 30
indexAndForward = false
maxConnectionsPerIndexer = 2
maxFailuresPerInterval = 2
maxQueueSize = 7MB
readTimeout = 300
secsInFailureInterval = 1
sendCookedData = true
sslQuietShutdown = false
tcpSendBufSz = 0
useACK = true
writeTimeout = 300
[tcpout:primary_indexers]
autoLB = true
indexerDiscovery = env-masternode

  1. No results found for TcpOutput error messages in index=_internal

  2. output of | tstats count where index= OR index=_ by host,index
    sqa01-ins01-scc51-dbs01 _internal 873298

0 Karma

AzmathShaik
Path Finder

there are no errors in outputs.
and the output of my ./splunk list forward-server

Active forwards:
None
Configured but inactive forwards:
None

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...