ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

[suman_july07]

Team,

In one of the Unix servers where SplunkForwarder is running, I have the below log in the splunkd.log file. Our web service is down due to this and Splunk is not working. Please see the log below and let us know, what should be done to make it running. Meanwhile, I will try to raise a case too.

11-24-2015 01:51:58.428 -0600 INFO  LicenseMgr - Initing LicenseMgr
11-24-2015 01:51:58.437 -0600 INFO  ServerConfig - Setting HTTP server compression state=on
11-24-2015 01:51:58.437 -0600 INFO  ServerConfig - Setting HTTP client compression state=0 (false)
11-24-2015 01:51:58.437 -0600 INFO  ServerConfig - Default output queue for file-based input: parsingQueue.
11-24-2015 01:51:58.437 -0600 INFO  LMConfig - connection_timeout=30
11-24-2015 01:51:58.437 -0600 INFO  LMConfig - send_timeout=30
11-24-2015 01:51:58.437 -0600 INFO  LMConfig - receive_timeout=30
11-24-2015 01:51:58.437 -0600 INFO  LMConfig - squash_threshold=1000
11-24-2015 01:51:58.437 -0600 INFO  LicenseMgr - Initing LicenseMgr runContext_splunkd=false
11-24-2015 01:51:58.437 -0600 INFO  LMStackMgr - closing stack mgr
11-24-2015 01:51:58.437 -0600 INFO  LMSlaveInfo - all slaves cleared
11-24-2015 01:51:58.437 -0600 INFO  LMStackMgr - init completed [9BB7F417-234F-4DAB-8056-A45D91F543D9,Forwarder,runContext_splunkd=false]
11-24-2015 01:51:58.437 -0600 INFO  LicenseMgr - StackMgr init complete...
11-24-2015 01:51:58.437 -0600 INFO  LMTracker - this is not splunkd, will perform partial init
11-24-2015 01:51:58.437 -0600 INFO  LMTracker - setting feature=Auth state=ENABLED (fs=1)
11-24-2015 01:51:58.437 -0600 INFO  LMTracker - setting feature=FwdData state=ENABLED (fs=1)
11-24-2015 01:51:58.437 -0600 INFO  LMTracker - setting feature=RcvData state=ENABLED (fs=1)
11-24-2015 01:51:58.437 -0600 INFO  LMTracker - setting feature=DistSearch state=ENABLED (fs=1)
11-24-2015 01:51:58.437 -0600 INFO  LMTracker - setting feature=RcvSearch state=ENABLED (fs=1)
11-24-2015 01:51:58.437 -0600 INFO  LMTracker - setting feature=ScheduledSearch state=ENABLED (fs=1)
11-24-2015 01:51:58.437 -0600 INFO  LMTracker - setting feature=Alerting state=ENABLED (fs=1)
11-24-2015 01:51:58.437 -0600 INFO  LMTracker - setting feature=DeployClient state=ENABLED (fs=1)
11-24-2015 01:51:58.437 -0600 INFO  LMTracker - setting feature=DeployServer state=ENABLED (fs=1)
11-24-2015 01:51:58.437 -0600 INFO  LMTracker - setting feature=SplunkWeb state=ENABLED (fs=1)
11-24-2015 01:51:58.437 -0600 INFO  LMTracker - setting feature=SyslogOutputProcessor state=ENABLED (fs=1)
11-24-2015 01:51:58.437 -0600 INFO  LMTracker - setting feature=SigningProcessor state=ENABLED (fs=1)
11-24-2015 01:51:58.437 -0600 INFO  LMTracker - setting feature=LocalSearch state=ENABLED (fs=1)
11-24-2015 01:51:58.437 -0600 INFO  LicenseMgr - Tracker init complete...
11-24-2015 01:51:58.522 -0600 INFO  loader - Splunkd starting (build 163460).
11-24-2015 01:51:58.522 -0600 INFO  loader - Detected 24 (virtual) CPUs and 48256MB RAM
11-24-2015 01:51:58.522 -0600 INFO  loader - Arguments are: "/opt/splunk/splunkforwarder/bin/splunkd" "check-transforms-keys"
11-24-2015 01:51:58.522 -0600 INFO  loader - Getting configuration data from: /opt/splunk/splunkforwarder/etc/myinstall/splunkd.xml
11-24-2015 01:51:58.522 -0600 INFO  loader - SPLUNK_MODULE_PATH environment variable not found - defaulting to /opt/splunk/splunkforwarder/etc/modules
11-24-2015 01:51:58.522 -0600 INFO  loader - loading modules from /opt/splunk/splunkforwarder/etc/modules
11-24-2015 01:51:58.523 -0600 INFO  loader - Writing out composite configuration file: /opt/splunk/splunkforwarder/var/run/splunk/composite.xml
11-24-2015 01:51:58.556 -0600 INFO  ServerConfig - Will generate GUID, as one does not exist on this server.
11-24-2015 01:51:58.556 -0600 INFO  ServerConfig - My newly generated GUID is 393F6BC8-BC55-4569-A747-407B7998167D
11-24-2015 01:51:58.558 -0600 INFO  ServerConfig - Setting HTTP server compression state=on
11-24-2015 01:51:58.558 -0600 INFO  ServerConfig - Setting HTTP client compression state=0 (false)
11-24-2015 01:51:58.558 -0600 INFO  ServerConfig - Default output queue for file-based input: parsingQueue.
11-24-2015 01:51:58.569 -0600 INFO  ulimit - Limit: virtual address space size: unlimited
11-24-2015 01:51:58.569 -0600 INFO  ulimit - Limit: data segment size: unlimited
11-24-2015 01:51:58.569 -0600 INFO  ulimit - Limit: resident memory size: unlimited
11-24-2015 01:51:58.569 -0600 INFO  ulimit - Limit: stack size: unlimited
11-24-2015 01:51:58.569 -0600 INFO  ulimit - Limit: core file size: 1024 bytes [hard maximum: unlimited]
11-24-2015 01:51:58.569 -0600 INFO  ulimit - Limit: data file size: unlimited
11-24-2015 01:51:58.569 -0600 INFO  ulimit - Limit: open files: 1048576 files
11-24-2015 01:51:58.569 -0600 INFO  ulimit - Limit: user processes: 1549513 processes
11-24-2015 01:51:58.569 -0600 INFO  ulimit - Limit: cpu time: unlimited
11-24-2015 01:51:58.570 -0600 INFO  loader - Splunkd starting (build 163460).
11-24-2015 01:51:58.571 -0600 INFO  loader - Detected 24 (virtual) CPUs and 48256MB RAM
11-24-2015 01:51:58.571 -0600 INFO  loader - Arguments are: "splunkd" "-p" "8089" "start"
11-24-2015 01:51:58.571 -0600 INFO  loader - Getting configuration data from: /opt/splunk/splunkforwarder/etc/myinstall/splunkd.xml
11-24-2015 01:51:58.571 -0600 INFO  loader - SPLUNK_MODULE_PATH environment variable not found - defaulting to /opt/splunk/splunkforwarder/etc/modules
11-24-2015 01:51:58.571 -0600 INFO  loader - loading modules from /opt/splunk/splunkforwarder/etc/modules
11-24-2015 01:51:58.572 -0600 INFO  loader - Writing out composite configuration file: /opt/splunk/splunkforwarder/var/run/splunk/composite.xml
11-24-2015 01:51:58.575 -0600 INFO  BundlesSetup - Setup stats for /opt/splunk/splunkforwarder/etc: wallclock_elapsed_msec=4, cpu_time_used=0.003, shared_services_generation=1, shared_services_population=1
11-24-2015 01:51:58.587 -0600 INFO  ClusteringMgr - initing clustering with: ht=60 rf=3 sf=2 ct=60 st=60 rt=60 rct=60 rst=60 rrt=60 rmst=600 rmrt=600 pe=1 im=0 is=0 mob=2 mor=5 pb=5 rep_port: 
11-24-2015 01:51:58.596 -0600 INFO  ClusteringMgr - clustering disabled
11-24-2015 01:51:58.601 -0600 WARN  DeploymentClient - Property targetUri not found. DeploymentClient is disabled.
11-24-2015 01:51:58.601 -0600 INFO  LicenseMgr - Initing LicenseMgr
11-24-2015 01:51:58.601 -0600 INFO  LMConfig - connection_timeout=30
11-24-2015 01:51:58.601 -0600 INFO  LMConfig - send_timeout=30
11-24-2015 01:51:58.601 -0600 INFO  LMConfig - receive_timeout=30
11-24-2015 01:51:58.601 -0600 INFO  LMConfig - squash_threshold=1000
11-24-2015 01:51:58.601 -0600 INFO  LicenseMgr - Initing LicenseMgr runContext_splunkd=true
11-24-2015 01:51:58.601 -0600 INFO  LMStackMgr - closing stack mgr
11-24-2015 01:51:58.601 -0600 INFO  LMSlaveInfo - all slaves cleared
11-24-2015 01:51:58.603 -0600 INFO  LMConfig - created default pool=auto_generated_pool_forwarder for stack=forwarder
11-24-2015 01:51:58.603 -0600 INFO  LMStackMgr - added default pool=auto_generated_pool_forwarder for stack=forwarder
11-24-2015 01:51:58.604 -0600 INFO  LMConfig - created default pool=auto_generated_pool_free for stack=free
11-24-2015 01:51:58.604 -0600 INFO  LMStackMgr - added default pool=auto_generated_pool_free for stack=free
11-24-2015 01:51:58.604 -0600 INFO  LMStackMgr - init completed [393F6BC8-BC55-4569-A747-407B7998167D,Forwarder,runContext_splunkd=true]
11-24-2015 01:51:58.604 -0600 INFO  LicenseMgr - StackMgr init complete...
11-24-2015 01:51:58.605 -0600 INFO  LMTracker - attempting to ping master=self from slave=393F6BC8-BC55-4569-A747-407B7998167D
11-24-2015 01:51:58.605 -0600 INFO  LMSlaveInfo - new slave='393F6BC8-BC55-4569-A747-407B7998167D' created
11-24-2015 01:51:58.605 -0600 INFO  LMSlaveInfo - Detected that masterTimeFromSlave(ZERO_TIME) < lastRolloverTime(Tue Nov 24 00:00:00 2015), meaning that the master has already rolled over. Ignore slave persisted usage.
11-24-2015 01:51:58.605 -0600 INFO  LMTracker - setting feature=Alerting state=DISABLED_DUE_TO_LICENSE (fs=2)
11-24-2015 01:51:58.605 -0600 INFO  LMTracker - setting feature=AllowDuplicateKeys state=DISABLED_DUE_TO_LICENSE (fs=2)
11-24-2015 01:51:58.605 -0600 INFO  LMTracker - setting feature=Auth state=ENABLED (fs=1)
11-24-2015 01:51:58.605 -0600 INFO  LMTracker - setting feature=CanBeRemoteMaster state=DISABLED_DUE_TO_LICENSE (fs=2)
11-24-2015 01:51:58.605 -0600 INFO  LMTracker - setting feature=DeployClient state=ENABLED (fs=1)
11-24-2015 01:51:58.605 -0600 INFO  LMTracker - setting feature=DeployServer state=DISABLED_DUE_TO_LICENSE (fs=2)
11-24-2015 01:51:58.605 -0600 INFO  LMTracker - setting feature=DistSearch state=DISABLED_DUE_TO_LICENSE (fs=2)
11-24-2015 01:51:58.605 -0600 INFO  LMTracker - setting feature=FwdData state=ENABLED (fs=1)
11-24-2015 01:51:58.605 -0600 INFO  LMTracker - setting feature=LocalSearch state=DISABLED_DUE_TO_LICENSE (fs=2)
11-24-2015 01:51:58.605 -0600 INFO  LMTracker - setting feature=RcvData state=ENABLED (fs=1)
11-24-2015 01:51:58.605 -0600 INFO  LMTracker - setting feature=RcvSearch state=DISABLED_DUE_TO_LICENSE (fs=2)
11-24-2015 01:51:58.605 -0600 INFO  LMTracker - setting feature=ResetWarnings state=DISABLED_DUE_TO_LICENSE (fs=2)
11-24-2015 01:51:58.605 -0600 INFO  LMTracker - setting feature=ScheduledSearch state=DISABLED_DUE_TO_LICENSE (fs=2)
11-24-2015 01:51:58.605 -0600 INFO  LMTracker - setting feature=SigningProcessor state=ENABLED (fs=1)
11-24-2015 01:51:58.605 -0600 INFO  LMTracker - setting feature=SplunkWeb state=ENABLED (fs=1)
11-24-2015 01:51:58.605 -0600 INFO  LMTracker - setting feature=SyslogOutputProcessor state=ENABLED (fs=1)
11-24-2015 01:51:58.605 -0600 INFO  LMTracker - setting masterGuid='393F6BC8-BC55-4569-A747-407B7998167D'
11-24-2015 01:51:58.605 -0600 INFO  LMTracker - attempting to ping master=self from slave=393F6BC8-BC55-4569-A747-407B7998167D success
11-24-2015 01:51:58.605 -0600 INFO  LicenseMgr - Tracker init complete...
11-24-2015 01:51:58.606 -0600 WARN  DeploymentProcessor - License feature=DeployServer not enabled, cannot bring up Deployment Server
11-24-2015 01:51:58.607 -0600 INFO  IndexProcessor - running splunkd specific init
11-24-2015 01:51:58.607 -0600 WARN  DistributedPeerManager - feature=DistSearch not enabled for your license level
11-24-2015 01:51:58.607 -0600 INFO  loader - Initializing from configuration
11-24-2015 01:51:58.609 -0600 INFO  TcpOutputProc - Initializing with fwdtype=lwf
11-24-2015 01:51:58.612 -0600 INFO  TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
11-24-2015 01:51:58.612 -0600 INFO  TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
11-24-2015 01:51:58.612 -0600 INFO  TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
11-24-2015 01:51:58.612 -0600 INFO  PipelineComponent - Pipeline merging disabled in default-mode.conf file
11-24-2015 01:51:58.612 -0600 INFO  PipelineComponent - Pipeline typing disabled in default-mode.conf file
11-24-2015 01:51:58.613 -0600 INFO  TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available
11-24-2015 01:51:58.613 -0600 INFO  TcpInputProc - Registering metrics callback for: tcpin_connections
11-24-2015 01:51:58.832 -0600 INFO  PipelineComponent - Pipeline fifo disabled in default-mode.conf file
11-24-2015 01:51:58.833 -0600 INFO  UDPInputProcessor - Registering metrics callback for: udpin_connections
11-24-2015 01:51:58.834 -0600 INFO  PipelineComponent - Launching the pipelines.
11-24-2015 01:51:58.835 -0600 INFO  loader - Server supporting SSL v2/v3
11-24-2015 01:51:58.835 -0600 INFO  loader - Using cipher suite ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
11-24-2015 01:51:58.902 -0600 INFO  TailingProcessor - TailWatcher initializing...
11-24-2015 01:51:58.902 -0600 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
11-24-2015 01:51:58.902 -0600 INFO  TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
11-24-2015 01:51:58.902 -0600 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
11-24-2015 01:51:58.902 -0600 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
11-24-2015 01:51:58.902 -0600 INFO  TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
11-24-2015 01:51:58.902 -0600 INFO  TailingProcessor - Adding watch on path: /opt/splunk/splunkforwarder/etc/splunk.version.
11-24-2015 01:51:58.902 -0600 INFO  TailingProcessor - Adding watch on path: /opt/splunk/splunkforwarder/var/log/splunk.
11-24-2015 01:51:58.902 -0600 INFO  TailingProcessor - Adding watch on path: /opt/splunk/splunkforwarder/var/spool/splunk.
11-24-2015 01:51:58.902 -0600 INFO  BatchReader - State transitioning from 2 to 0 (initOrResume).
11-24-2015 01:51:58.903 -0600 ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.
11-24-2015 03:13:16.070 -0600 INFO  TailingProcessor - Could not send data to output queue (parsingQueue), retrying...

[a548506]

Hello,

I am having the same issue as well. Here is my outputs.conf:

[tcpout]
server=myserver.com:9997

Thanks.

[mikelanghorst]

From the log, I don't see any indication that you have an outputs.conf configured. The only TcpOutputProc are the default whitelists. I'm not sure what you mean by your web service being down due to this. Could you provide your outputs.conf file?