SplunkForwarder monitoring issue for /opt/log/<file name>

[Namdev]

Hello team,

In my distributed Splunk lab created on VMware client virtual machine, facing the below issues.  Distributed environment consists of below components with Splunk free  licences

- 4 Indexers (part of an Indexer Cluster)

- 1 Cluster Manager (for managing the indexer cluster)

- 2 Universal Forwarders (UFs) sending data

- 1 DS/LM/MC (Deployment Server + License Manager + Monitoring Console combined on one server)

- 1 Search Head (for searching and dashboards) 

I am facing an issue to enable Splunk monitoring for /opt/log directory.

I have checked that /var/log can be monitored successfully whereas Splunk forwarder is failed to monitor /opt/log directory. I have checked permission issue other things but no luck 

[isoutamo]

Hi

I’m expecting that you have Splunk trial not free license? Free license doesn’t contain most of those features which you are trying to use!

The easiest way to check why those files are not accessible is just sudo/su to your Splunk UF user and check if it can access those or not. If not the add permissions as @livehybrid already told. If it can access those, then start to debug with logs and e.g. with 

splunk list inputstatus 

etc.

You could find quite many posts here where this issue is already discussed and solved.

r. Ismo 

[Namdev]

I am using Splunk trial license, I have checked permissions and it is not a permission issue  

[isoutamo]

It good to know that. Then this (on UF)

splunk list inputstatus

Shows to you what inputs your UF sees and what it has read.

[Namdev]

Cooked:tcp :
tcp

Raw:tcp :
tcp

TailingProcessor:FileStatus :
$SPLUNK_HOME/etc/apps/sample_app/logs
type = missing

$SPLUNK_HOME/etc/splunk.version
file position = 70
file size = 70
percent = 100.00
type = finished reading

$SPLUNK_HOME/var/log/splunk
type = directory

$SPLUNK_HOME/var/log/splunk/configuration_change.log
type = directory

$SPLUNK_HOME/var/log/splunk/license_usage_summary.log
type = directory

$SPLUNK_HOME/var/log/splunk/metrics.log
type = directory

$SPLUNK_HOME/var/log/splunk/splunk_instrumentation_cloud.log*
type = directory

$SPLUNK_HOME/var/log/splunk/splunkd.log
type = directory

$SPLUNK_HOME/var/log/watchdog/watchdog.log*
type = directory

$SPLUNK_HOME/var/run/splunk/search_telemetry/*search_telemetry.json
type = directory

$SPLUNK_HOME/var/spool/splunk/tracker.log*
type = directory

/opt/log/
type = directory

/opt/log/cisco_ironport_web.log
file position = 207575
file size = 207575
parent = /opt/log/
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/audit.log
file position = 159471
file size = 159471
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = open file

/opt/splunkforwarder/var/log/splunk/btool.log
file position = 192268
file size = 192268
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/conf.log
file position = 9044
file size = 9044
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/configuration_change.log
file position = 3353479
file size = 3353479
parent = $SPLUNK_HOME/var/log/splunk/configuration_change.log
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/first_install.log
file position = 70
file size = 70
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/health.log
file position = 785728
file size = 785728
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/license_usage.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/license_usage_summary.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk/license_usage_summary.log
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/mergebuckets.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/metrics.log
file position = 21630761
file size = 21630761
parent = $SPLUNK_HOME/var/log/splunk/metrics.log
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/metrics.log.1
file position = 25000026
file size = 25000026
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/metrics.log.2
file position = 25000081
file size = 25000081
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/mongod.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/remote_searches.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/scheduler.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/search_messages.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/searchhistory.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/splunk_instrumentation_cloud.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk/splunk_instrumentation_cloud.log*
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/splunkd-utility.log
file position = 69012
file size = 69012
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/splunkd.log
file position = 12378562
file size = 12378562
parent = $SPLUNK_HOME/var/log/splunk/splunkd.log
percent = 100.00
type = open file

/opt/splunkforwarder/var/log/splunk/splunkd_access.log
file position = 44571
file size = 44571
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = open file

/opt/splunkforwarder/var/log/splunk/splunkd_stderr.log
file position = 200
file size = 200
parent = $SPLUNK_HOME/var/log/splunk
percent = 100.00
type = finished reading

/opt/splunkforwarder/var/log/splunk/splunkd_stdout.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/splunkd_ui_access.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/splunk/wlm_monitor.log
file position = 0
file size = 0
parent = $SPLUNK_HOME/var/log/splunk
percent = 100
type = finished reading

/opt/splunkforwarder/var/log/watchdog/watchdog.log
file position = 12202
file size = 12202
parent = $SPLUNK_HOME/var/log/watchdog/watchdog.log*
percent = 100.00
type = finished reading

tcp_cooked:listenerports :
8089

[isoutamo]

This shows 

/opt/log/
type = directory

/opt/log/cisco_ironport_web.log
file position = 207575
file size = 207575
parent = /opt/log/
percent = 100.00
type = finished reading

that splunk has read this one log file 100%. This means that it had sent it to indexers (I suppose that this has defined in your inputs.conf). 
Why you don’t see those? There could be several reasons for that

• wrong timestamp recognition 
• wrong index definition 
• you have some transformations for drop those
• something else

To tell the real reason you should try to query those e.g. 

index=* earliest=1 latest=+5y

that shows if those have wrong time or those have gone to wrong index.

You should also check all conf files from UF to indexers and SH to see if there is something weird.

[livehybrid]

Hi @Namdev 

Please could you confirm which user the Splunk Forwarder is running as? Is it splunkfwd, splunk or something else?

Please could you show a screenshot of the permissions on your /opt/log files in question. 

Did you run anything like this against the directory to give splunk access?

setfacl -R -m u:splunkfwd:r-x /opt/log

 Are there any logs in splunkd.log relating to these files? 

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[livehybrid]

Hi @Namdev 

How did you get on with looking into the below?

---

@livehybrid wrote:

Hi @Namdev 

Please could you confirm which user the Splunk Forwarder is running as? Is it splunkfwd, splunk or something else?

Please could you show a screenshot of the permissions on your /opt/log files in question. 

Did you run anything like this against the directory to give splunk access?

setfacl -R -m u:splunkfwd:r-x /opt/log

 Are there any logs in splunkd.log relating to these files? 

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

---

[Namdev]

I checked by using this command but no luck , kindly find my logs 

root@hf2:/opt# ps aux | grep /opt/log/
root 3152 0.0 0.0 9276 2304 pts/2 S+ 13:17 0:00 grep --color=auto /opt/log/

root@hf2:/opt# ls -l /opt/log/
total 204
-rw-r-xr--+ 1 root root 207575 Feb 19 11:12 cisco_ironport_web.log
root@hf2:/opt#

SplunkD Logs for your refernecne :
03-04-2025 22:23:55.770 +0530 INFO TailingProcessor [32908 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/.
03-04-2025 22:29:34.873 +0530 INFO TailingProcessor [33197 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/.
03-04-2025 22:39:22.449 +0530 INFO TailingProcessor [33712 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/.
03-04-2025 22:44:59.341 +0530 INFO TailingProcessor [33979 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/cisco_ironport_web.log.
03-04-2025 22:44:59.341 +0530 INFO TailingProcessor [33979 MainTailingThread] - Adding watch on path: /opt/log/cisco_ironport_web.log.
03-04-2025 22:54:52.366 +0530 INFO TailingProcessor [34246 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/cisco_ironport_web.log.
03-04-2025 22:54:52.366 +0530 INFO TailingProcessor [34246 MainTailingThread] - Adding watch on path: /opt/log/cisco_ironport_web.log.
03-05-2025 12:35:53.768 +0530 INFO TailingProcessor [2117 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/cisco_ironport_web.log.
03-05-2025 12:35:53.768 +0530 INFO TailingProcessor [2117 MainTailingThread] - Adding watch on path: /opt/log/cisco_ironport_web.log.
03-05-2025 13:07:00.440 +0530 INFO TailingProcessor [2920 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/.
03-05-2025 13:16:28.483 +0530 INFO TailingProcessor [3132 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/.
03-05-2025 13:18:26.876 +0530 INFO TailingProcessor [3339 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/.
root@hf2:/opt#

[Namdev]

Hey Buddy ,

No luck with your command, kindly find logs below : 

root@hf2:/opt# ps aux | grep /opt/log/
root 3152 0.0 0.0 9276 2304 pts/2 S+ 13:17 0:00 grep --color=auto /opt/log/

root@hf2:/opt# ls -l /opt/log/
total 204
-rw-r-xr--+ 1 root root 207575 Feb 19 11:12 cisco_ironport_web.log
root@hf2:/opt#

SplunkD Logs for your refernecne :
03-04-2025 22:23:55.770 +0530 INFO TailingProcessor [32908 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/.
03-04-2025 22:29:34.873 +0530 INFO TailingProcessor [33197 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/.
03-04-2025 22:39:22.449 +0530 INFO TailingProcessor [33712 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/.
03-04-2025 22:44:59.341 +0530 INFO TailingProcessor [33979 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/cisco_ironport_web.log.
03-04-2025 22:44:59.341 +0530 INFO TailingProcessor [33979 MainTailingThread] - Adding watch on path: /opt/log/cisco_ironport_web.log.
03-04-2025 22:54:52.366 +0530 INFO TailingProcessor [34246 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/cisco_ironport_web.log.
03-04-2025 22:54:52.366 +0530 INFO TailingProcessor [34246 MainTailingThread] - Adding watch on path: /opt/log/cisco_ironport_web.log.
03-05-2025 12:35:53.768 +0530 INFO TailingProcessor [2117 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/cisco_ironport_web.log.
03-05-2025 12:35:53.768 +0530 INFO TailingProcessor [2117 MainTailingThread] - Adding watch on path: /opt/log/cisco_ironport_web.log.
03-05-2025 13:07:00.440 +0530 INFO TailingProcessor [2920 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/.
03-05-2025 13:16:28.483 +0530 INFO TailingProcessor [3132 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/.
03-05-2025 13:18:26.876 +0530 INFO TailingProcessor [3339 MainTailingThread] - Parsing configuration stanza: monitor:///opt/log/.
root@hf2:/opt#

[marnall]

I recommend checking the internal logs for the forwarder. It may contain error messages that indicate why /opt/log/ is not logging. You can use various keywords:

index=_internal host=<forwardername> log_level=ERROR /opt/log/

[Namdev]

NO logs on Search head