topic Re: SplunkForwarder monitoring issue for /opt/log/<file name> in Splunk Enterprise

SplunkForwarder monitoring issue for /opt/log/

Namdev — Tue, 04 Mar 2025 17:34:53 GMT

Hello team,

In my distributed Splunk lab created on VMware client virtual machine, facing the below issues. Distributed environment consists of below components with Splunk free licences

- 4 Indexers (part of an Indexer Cluster)

- 1 Cluster Manager (for managing the indexer cluster)

- 2 Universal Forwarders (UFs) sending data

- 1 DS/LM/MC (Deployment Server + License Manager + Monitoring Console combined on one server)

- 1 Search Head (for searching and dashboards)

I am facing an issue to enable Splunk monitoring for /opt/log directory.

I have checked that /var/log can be monitored successfully whereas Splunk forwarder is failed to monitor /opt/log directory. I have checked permission issue other things but no luck

Re: SplunkForwarder monitoring issue for /opt/log/

marnall — Tue, 04 Mar 2025 18:39:07 GMT

I recommend checking the internal logs for the forwarder. It may contain error messages that indicate why /opt/log/ is not logging. You can use various keywords:

index=_internal host=<forwardername> log_level=ERROR /opt/log/

Re: SplunkForwarder monitoring issue for /opt/log/

livehybrid — Tue, 04 Mar 2025 22:36:30 GMT

Hi @Namdev

Please could you confirm which user the Splunk Forwarder is running as? Is it splunkfwd, splunk or something else?

Please could you show a screenshot of the permissions on your /opt/log files in question.

Did you run anything like this against the directory to give splunk access?

setfacl -R -m u:splunkfwd:r-x /opt/log

Are there any logs in splunkd.log relating to these files?

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

Re: SplunkForwarder monitoring issue for /opt/log/

Namdev — Wed, 05 Mar 2025 07:54:56 GMT

Hey Buddy ,

No luck with your command, kindly find logs below :

root@hf2:/opt# ps aux | grep /opt/log/
root 3152 0.0 0.0 9276 2304 pts/2 S+ 13:17 0:00 grep --color=auto /opt/log/

root@hf2:/opt# ls -l /opt/log/
total 204
-rw-r-xr--+ 1 root root 207575 Feb 19 11:12 cisco_ironport_web.log
root@hf2:/opt#

SplunkD Logs for your refernecne :
03-04-2025 22:23:55.770 +0530 INFO TailingProcessor [32908 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/.
03-04-2025 22:29:34.873 +0530 INFO TailingProcessor [33197 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/.
03-04-2025 22:39:22.449 +0530 INFO TailingProcessor [33712 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/.
03-04-2025 22:44:59.341 +0530 INFO TailingProcessor [33979 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/cisco_ironport_web.log.
03-04-2025 22:44:59.341 +0530 INFO TailingProcessor [33979 MainTailingThread] - Adding watch on path: /opt/log/cisco_ironport_web.log.
03-04-2025 22:54:52.366 +0530 INFO TailingProcessor [34246 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/cisco_ironport_web.log.
03-04-2025 22:54:52.366 +0530 INFO TailingProcessor [34246 MainTailingThread] - Adding watch on path: /opt/log/cisco_ironport_web.log.
03-05-2025 12:35:53.768 +0530 INFO TailingProcessor [2117 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/cisco_ironport_web.log.
03-05-2025 12:35:53.768 +0530 INFO TailingProcessor [2117 MainTailingThread] - Adding watch on path: /opt/log/cisco_ironport_web.log.
03-05-2025 13:07:00.440 +0530 INFO TailingProcessor [2920 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/.
03-05-2025 13:16:28.483 +0530 INFO TailingProcessor [3132 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/.
03-05-2025 13:18:26.876 +0530 INFO TailingProcessor [3339 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/.
root@hf2:/opt#

Re: SplunkForwarder monitoring issue for /opt/log/

isoutamo — Wed, 05 Mar 2025 10:38:40 GMT

I’m expecting that you have Splunk trial not free license? Free license doesn’t contain most of those features which you are trying to use!

The easiest way to check why those files are not accessible is just sudo/su to your Splunk UF user and check if it can access those or not. If not the add permissions as @livehybrid already told. If it can access those, then start to debug with logs and e.g. with

splunk list inputstatus

etc.

You could find quite many posts here where this issue is already discussed and solved.

r. Ismo

Re: SplunkForwarder monitoring issue for /opt/log/

Namdev — Wed, 05 Mar 2025 10:39:20 GMT

NO logs on Search head

Re: SplunkForwarder monitoring issue for /opt/log/

Namdev — Wed, 05 Mar 2025 11:22:39 GMT

I am using Splunk trial license, I have checked permissions and it is not a permission issue

Re: SplunkForwarder monitoring issue for /opt/log/

livehybrid — Wed, 05 Mar 2025 12:30:00 GMT

Hi @Namdev

How did you get on with looking into the below?

@livehybrid wrote:
Hi @Namdev
Please could you confirm which user the Splunk Forwarder is running as? Is it splunkfwd, splunk or something else?
Please could you show a screenshot of the permissions on your /opt/log files in question.
Did you run anything like this against the directory to give splunk access?
setfacl -R -m u:splunkfwd:r-x /opt/log
Are there any logs in splunkd.log relating to these files?
Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards
Will

Re: SplunkForwarder monitoring issue for /opt/log/

Namdev — Wed, 05 Mar 2025 13:48:32 GMT

I checked by using this command but no luck , kindly find my logs

root@hf2:/opt# ps aux | grep /opt/log/
root 3152 0.0 0.0 9276 2304 pts/2 S+ 13:17 0:00 grep --color=auto /opt/log/

root@hf2:/opt# ls -l /opt/log/
total 204
-rw-r-xr--+ 1 root root 207575 Feb 19 11:12 cisco_ironport_web.log
root@hf2:/opt#

Re: SplunkForwarder monitoring issue for /opt/log/

isoutamo — Wed, 05 Mar 2025 15:36:42 GMT

It good to know that. Then this (on UF)

splunk list inputstatus

Shows to you what inputs your UF sees and what it has read.

Re: SplunkForwarder monitoring issue for /opt/log/

Namdev — Wed, 05 Mar 2025 16:57:52 GMT

Cooked:tcp :
tcp

Raw:tcp :
tcp

TailingProcessor:FileStatus :
$SPLUNK_HOME/etc/apps/sample_app/logs
type = missing

$SPLUNK_HOME/etc/splunk.version
file position = 70
file size = 70
percent = 100.00
type = finished reading

$SPLUNK_HOME/var/log/splunk
type = directory

$SPLUNK_HOME/var/log/splunk/configuration_change.log
type = directory

$SPLUNK_HOME/var/log/splunk/license_usage_summary.log
type = directory

$SPLUNK_HOME/var/log/splunk/metrics.log
type = directory

$SPLUNK_HOME/var/log/splunk/splunk_instrumentation_cloud.log*
type = directory

$SPLUNK_HOME/var/log/splunk/splunkd.log
type = directory

$SPLUNK_HOME/var/log/watchdog/watchdog.log*
type = directory

$SPLUNK_HOME/var/run/splunk/search_telemetry/*search_telemetry.json
type = directory

$SPLUNK_HOME/var/spool/splunk/tracker.log*
type = directory

/opt/log/
type = directory

/opt/log/cisco_ironport_web.log
file position = 207575
file size = 207575
parent = /opt/log/
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/audit.log
file position = 159471
file size = 159471
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = open file

/opt/splunkforwarder/var/log/splunk/btool.log
file position = 192268
file size = 192268
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/conf.log
file position = 9044
file size = 9044
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/configuration_change.log
file position = 3353479
file size = 3353479
parent = $SPLUNK_HOME/var/log/splunk/configuration_change.log
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/first_install.log
file position = 70
file size = 70
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/health.log
file position = 785728
file size = 785728
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/license_usage.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/license_usage_summary.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk/license_usage_summary.log
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/mergebuckets.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/metrics.log
file position = 21630761
file size = 21630761
parent = $SPLUNK_HOME/var/log/splunk/metrics.log
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/metrics.log.1
file position = 25000026
file size = 25000026
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/metrics.log.2
file position = 25000081
file size = 25000081
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/mongod.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/remote_searches.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/scheduler.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/search_messages.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/searchhistory.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/splunk_instrumentation_cloud.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk/splunk_instrumentation_cloud.log*
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/splunkd-utility.log
file position = 69012
file size = 69012
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/splunkd.log
file position = 12378562
file size = 12378562
parent = $SPLUNK_HOME/var/log/splunk/splunkd.log
percent = 100.00
type = open file

/opt/splunkforwarder/var/log/splunk/splunkd_access.log
file position = 44571
file size = 44571
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = open file

/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log
file position = 200
file size = 200
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/splunkd_stdout.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/splunkd_ui_access.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/wlm_monitor.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/watchdog/watchdog.log
file position = 12202
file size = 12202
parent = $SPLUNK_HOME/var/log/watchdog/watchdog.log*
percent = 100.00
type = finished reading

tcp_cooked:listenerports :
8089

Re: SplunkForwarder monitoring issue for /opt/log/

isoutamo — Wed, 05 Mar 2025 18:01:48 GMT

This shows

/opt/log/ type = directory /opt/log/cisco_ironport_web.log file position = 207575 file size = 207575 parent = /opt/log/ percent = 100.00 type = finished reading

that splunk has read this one log file 100%. This means that it had sent it to indexers (I suppose that this has defined in your inputs.conf).
Why you don’t see those? There could be several reasons for that

wrong timestamp recognition
wrong index definition
you have some transformations for drop those
something else

To tell the real reason you should try to query those e.g.

index=* earliest=1 latest=+5y

that shows if those have wrong time or those have gone to wrong index.

You should also check all conf files from UF to indexers and SH to see if there is something weird.