topic Re: ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf. in Deployment Architecture

ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

ahmemohs03 — Tue, 29 Sep 2020 20:33:13 GMT

07-18-2018 21:20:40.725 +0000 WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see:
07-18-2018 21:20:40.736 +0000 INFO WatchedFile - Will begin reading at offset=392049 for file='/welldata/splunk/var/log/introspection/disk_objects.log'.
07-18-2018 21:20:40.740 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/welldata/splunk/var/log/introspection/http_event_collector_metrics.log'.
07-18-2018 21:20:40.799 +0000 ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.
07-18-2018 21:20:40.967 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/welldata/splunk/var/log/splunk/btool.log'.
07-18-2018 21:20:40.970 +0000 INFO WatchedFile - Will begin reading at offset=3894 for file='/welldata/splunk/var/log/splunk/splunkd-utility.log'.
07-18-2018 21:20:40.977 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/welldata/splunk/var/log/splunk/searchhistory.log'.
07-18-2018 21:20:40.981 +0000 INFO WatchedFile - Will begin reading at offset=238867 for file='/welldata/splunk/var/log/splunk/splunkd_access.log'.
07-18-2018 21:20:40.998 +0000 INFO WatchedFile - Will begin reading at offset=3141787 for file='/welldata/splunk/var/log/splunk/audit.log'.
07-18-2018 21:20:41.001 +0000 INFO WatchedFile - Will begin reading at offset=933 for file='/welldata/splunk/var/log/splunk/conf.log'.
07-18-2018 21:20:41.020 +0000 INFO WatchedFile - Will begin reading at offset=2076287 for file='/welldata/splunk/var/log/splunk/health.log'.
07-18-2018 21:20:43.337 +0000 INFO IntrospectionGenerator:resource_usage - RU_main - I-data gathering (Resource Usage) starting; period=10s
07-18-2018 21:20:43.349 +0000 INFO IntrospectionGenerator:resource_usage - RU_main - I-data gathering (IO Statistics) starting; interval=60s
07-18-2018 21:20:46.023 +0000 WARN TailReader - Could not send data to output queue (parsingQueue), retrying...
07-18-2018 21:20:48.590 +0000 INFO ExecProcessor - message from "python /welldata/splunk/etc/apps/splunk_monitoring_console/bin/dmc_config.py" Cannot detect SHC status because of License Restriction. Will not disable DMC.
07-18-2018 21:21:10.392 +0000 INFO ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views
07-18-2018 21:52:08.785 +0000 WARN TcpInputProc - Stopping all listening ports. Queues blocked for more than 300 seconds
07-18-2018 21:52:08.785 +0000 INFO TcpInputProc - Stopping IPv4 port 9997
07-18-2018 21:59:59.999 +0000 INFO ExecProcessor - setting reschedule_ms=3600002, for command=python /welldata/splunk/etc/apps/splunk_instrumentation/bin/instrumentation.py
07-18-2018 23:00:00.003 +0000 INFO ExecProcessor - setting reschedule_ms=3599997, for command=python /welldata/splunk/etc/apps/splunk_instrumentation/bin/instrumentation.py
07-19-2018 00:00:00.000 +0000 INFO ExecProcessor - setting reschedule_ms=3600000, for command=python /welldata/splunk/etc/apps/splunk_instrumentation/bin/instrumentation.py
07-19-2018 00:00:00.000 +0000 INFO ExecProcessor - setting reschedule_ms=86400000, for command=python /welldata/splunk/etc/apps/splunk_instrumentation/bin/schedule_delete.py
07-19-2018 00:00:00.945 +0000 INFO LMStackMgr - should rollover=true because _lastRolloverTime=1531872000 lastRolloverDay=1531872000 snappedNow=1531958400
07-19-2018 00:00:00.945 +0000 INFO LMStackMgr - finished rollover, new lastRolloverTime=1531958400
07-19-2018 00:00:28.945 +0000 INFO LMSlaveInfo - Detected that masterTimeFromSlave(Wed Jul 18 23:59:27 2018) < lastRolloverTime(Thu Jul 19 00:00:00 2018), meaning that the master has already rolled over. Ignore slave persisted usage.
07-19-2018 01:59:59.999 +0000 INFO ExecProcessor - setting reschedule_ms=3600002, for command=python /welldata/splunk/etc/apps/splunk_instrumentation/bin/instrumentation.py
07-19-2018 03:00:00.001 +0000 INFO ExecProcessor - setting reschedule_ms=3599999, for command=python /welldata/splunk/etc/apps/splunk_instrumentation/bin/instrumentation.py
07-19-2018 03:01:00.599 +0000 WARN TelemetryHandler - 1531872000.000000

Re: ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

patilsonali1729 — Thu, 19 Jul 2018 16:53:56 GMT

can you paste the content of your outputs.conf here

Re: ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

ahmemohs03 — Thu, 19 Jul 2018 17:02:31 GMT

[root@psdlepkl4 local]# cat outputs.conf
[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 12.34.342.87:9997

[tcpout-server://12.34.342.87:9997]

This on server where universal forwarder installed

Re: ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

auraria1 — Thu, 19 Jul 2018 17:54:59 GMT

Is this your first time starting this splunk forwarder?
I've add issues after running accept license where I need to stop the splunk instance and re-start it on versions 7.1 and above.

Re: ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

ahmemohs03 — Thu, 19 Jul 2018 18:21:48 GMT

Hi Auraria

Yes, first time after installating universal splunk forwarder..its 7.1.1 version.
splunk on A Linux server where splunk installed, Universal forwarder is on B Linux server, were the logs of Linux B will be seen on Linux A.
Do I need to start splunkforwarder also, along with splunk?

Re: ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

auraria1 — Thu, 19 Jul 2018 18:25:21 GMT

Both need to be running for logs to be indexed.

I'd try restarting the forwarder:

/directory to splunk/splunkforwarder/bin/splunk stop
/directory to splunk/splunkforwarder/bin/splunk start

Then check the splunkd logs, let me know if that fixes the issue.

Quick tip on troubleshooting and splunkd logs, rename the log to splunkd1 before starting the splunk forwarder and increase the number everytime so it creates a fresh log and you can compare the two.

Re: ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

ahmemohs03 — Tue, 29 Sep 2020 20:31:31 GMT

Thanks for quick reply...

Restarted splunk forwarder, getting this error on splunkd.logs
Weburl is not coming up. http://hostanme:8000

WARN X509Verify - X509 certificate (O=SplunkUser,CN=SplunkServerDefaultCert) should not be used, as it is issued by Splunk's own default Certificate Authority (CA). This puts your Splunk instance at very high-risk of the MITM attack. Either commercial-CA-signed or self-CA-signed certificates must be used; see:
07-19-2018 18:31:56.698 +0000 INFO WatchedFile - Will begin reading at offset=508858 for file='/welldata/splunk/var/log/introspection/disk_objects.log'.
07-19-2018 18:31:56.702 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/welldata/splunk/var/log/introspection/http_event_collector_metrics.log'.
07-19-2018 18:31:56.750 +0000 ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

Please advice

Re: ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

auraria1 — Thu, 19 Jul 2018 18:45:51 GMT

When you created your outputs.conf where did you place the file?

Should be:

/directory to splunk/splunkforwarder/etc/system/local/outputs.conf

Be sure it says outputs.conf and not output.conf.

Re: ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

patilsonali1729 — Thu, 19 Jul 2018 18:48:48 GMT

1.Splunk universal forwarder does not have web UI.
2.Where is your outputs.conf file located? Please run ./splunk btool --debug outputs list and paste the response here.

Re: ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

ahmemohs03 — Thu, 19 Jul 2018 18:52:40 GMT

Thanks .

location : /opt/splunkforwarder/etc/system/local

yes its outputs.conf

inside outputs.conf

root@psdlepkl4 local]# cat outputs.conf

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = 12.34.342.87:9997

[tcpout-server://12.34.342.87:9997

Re: ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

auraria1 — Thu, 19 Jul 2018 18:57:30 GMT

as @patilsonali1729 please post the btool debug output.

Under [tcpout:default-autolb-group]
add:

disabled = 0

This shouldn't be an issue however.

Also add the close bracket ( ] )to
so this
[tcpout-server://12.34.342.87:9997

to this

[tcpout-server://12.34.342.87:9997]

^This is also not needed as you specified the server and the output in the default autolb group.

Re: ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

ahmemohs03 — Tue, 29 Sep 2020 20:31:37 GMT

Hi Patil,

Yes, Splunk universal forwarder does not have web UI. I am saying about splunk enterprises this installed on Linux A machine and splunk forwarder on Linux B machine. Splunk B logs need to see forwarder on Linux A(Splunk web url).

output conf file of universal forwarder is located at : /opt/splunkforwarder/etc/system/local

after running run ./splunk btool --debug outputs list

/opt/splunkforwarder/etc/system/default/outputs.conf [syslog]
/opt/splunkforwarder/etc/system/default/outputs.conf maxEventSize = 1024
/opt/splunkforwarder/etc/system/default/outputs.conf priority = <13>
/opt/splunkforwarder/etc/system/default/outputs.conf type = udp
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf [tcpout]
/opt/splunkforwarder/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30
/opt/splunkforwarder/etc/system/default/outputs.conf autoLBFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf autoLBVolume = 0
/opt/splunkforwarder/etc/system/default/outputs.conf blockOnCloning = true
/opt/splunkforwarder/etc/system/default/outputs.conf blockWarnThreshold = 100
/opt/splunkforwarder/etc/system/default/outputs.conf cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256
/opt/splunkforwarder/etc/system/default/outputs.conf compressed = false
/opt/splunkforwarder/etc/system/default/outputs.conf connectionTimeout = 20
/opt/splunkforwarder/etc/system/local/outputs.conf defaultGroup = default-autolb-group
/opt/splunkforwarder/etc/system/default/outputs.conf disabled = false
/opt/splunkforwarder/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5
/opt/splunkforwarder/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/opt/splunkforwarder/etc/system/default/outputs.conf ecdhCurves = prime256v1, secp384r1, secp521r1
/opt/splunkforwarder/etc/system/default/outputs.conf forceTimebasedAutoLB = false
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.0.whitelist = .*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.2.whitelist = (_audit|_introspection|_internal|_telemetry)
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.filter.disable = false
/opt/splunkforwarder/etc/system/default/outputs.conf heartbeatFrequency = 30
/opt/splunkforwarder/etc/system/default/outputs.conf indexAndForward = false
/opt/splunkforwarder/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2
/opt/splunkforwarder/etc/system/default/outputs.conf maxFailuresPerInterval = 2
/opt/splunkforwarder/etc/system/default/outputs.conf maxQueueSize = auto
/opt/splunkforwarder/etc/system/default/outputs.conf readTimeout = 300
/opt/splunkforwarder/etc/system/default/outputs.conf secsInFailureInterval = 1
/opt/splunkforwarder/etc/system/default/outputs.conf sendCookedData = true
/opt/splunkforwarder/etc/system/default/outputs.conf sslQuietShutdown = false
/opt/splunkforwarder/etc/system/default/outputs.conf sslVersions = tls1.2
/opt/splunkforwarder/etc/system/default/outputs.conf tcpSendBufSz = 0
/opt/splunkforwarder/etc/system/default/outputs.conf useACK = false
/opt/splunkforwarder/etc/system/default/outputs.conf writeTimeout = 300
/opt/splunkforwarder/etc/system/local/outputs.conf [tcpout-server://10.46.249.41:9997]
/opt/splunkforwarder/etc/system/local/outputs.conf [tcpout:default-autolb-group]
/opt/splunkforwarder/etc/system/local/outputs.conf server = 10.22.139.99:9997

Re: ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

ahmemohs03 — Thu, 19 Jul 2018 19:16:39 GMT

Thanks for response.
Also add the close bracket ( ] )to already there may be my mistake

Do I need to add (disabled=0)?

Re: ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

patilsonali1729 — Thu, 19 Jul 2018 19:20:31 GMT

Also, verify if the user running Splunk has read permission to outputs.conf (the user should have access to all the conf files, in short to /opt/splunkforwarder)

Re: ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

auraria1 — Thu, 19 Jul 2018 19:20:32 GMT

It doesn't hurt.

After that try starting the forwarder again and let me know.

Re: ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

auraria1 — Thu, 19 Jul 2018 19:22:54 GMT

Good call,
@ahmemohs03 I'd run a chown on the entire directory for the dedicated splunk user,

chown splunk:splunk -R /opt/splunkforwarder/

Swap out splunk for the user running splunk.

Re: ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

ahmemohs03 — Thu, 19 Jul 2018 19:30:51 GMT

Do you mean -rw------- 1 root root 140 Jul 17 18:39 outputs.conf?

Re: ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

auraria1 — Thu, 19 Jul 2018 19:33:21 GMT

I would just change ownership to all the files and folders as it can cause issues in the future in the splunkforwarder directory.

Did the bracket and the disabled = 0 fix the issue or is it still persisting?

Re: ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

ahmemohs03 — Thu, 19 Jul 2018 19:48:28 GMT

after bracket and the disabled = 0 fix , I did a splunkforwarder restart, but see the error again.
ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

Re: ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.

ahmemohs03 — Thu, 19 Jul 2018 19:49:27 GMT

Do I need to restart splunk also on another server?