When I run command
bin/splunk add monitor '/var/mqm/qmgrs/*/errors/AMQERR01.LOG' -follow-only True
to monitor specific log file under several directories
e.g.
$ ls /var/mqm/qmgrs/*/errors/AMQERR01.LOG
/var/mqm/qmgrs/QM1/errors/AMQERR01.LOG
/var/mqm/qmgrs/QM2/errors/AMQERR01.LOG
/var/mqm/qmgrs/QM3/errors/AMQERR01.LOG
bin/splunk list monitor
returns the following
/var/mqm/qmgrs/*/errors/AMQERR01.LOG
/var/mqm/qmgrs/@SYSTEM
/var/mqm/qmgrs/QM1
/var/mqm/qmgrs/QM1/@app
/var/mqm/qmgrs/QM1/@app/esem
/var/mqm/qmgrs/QM1/@app/esem/uateai07
/var/mqm/qmgrs/QM1/@app/isem
/var/mqm/qmgrs/QM1/@app/isem/uateai07
/var/mqm/qmgrs/QM1/@app/msem
/var/mqm/qmgrs/QM1/@app/msem/uateai07
/var/mqm/qmgrs/QM1/@app/shmem
.. another 4000 subdirectories
However despite what list monitor says - I do see the error files I am looking for in Splunk
Is this a bug?
Yes, the list monitor
command does have some enhancements coming down the pipeline, right now it doesn't give complete information on every file & directory being monitored, and it lists files that are not being monitored due to whitelist/blacklist settings. Also, it's possible that it could take a long time to complete if you are monitoring hundreds of thousands of files & directories.
There is a way you can get this information, but it's in the early stages of development and not as pretty as it could be. If you hit this endpoint with your browser - https://<servername>:8089/services/admin/inputstatus/TailingProcessor%3AFileStatus
- you will get a lot of XML back detailing the current status of your monitored files. Again, this is not fully developed yet so it just gives you a lot of bare-bones info. Remember to insert your servername and splunkd management port into the URL.
Hi,
Is this still current or can we rely on the output of list monitor?
Thanks,
I believe that btool
is the right way here - /opt/splunk/bin/splunk btool serverclass list --debug
Thanks for that. I did find that using the wild card wasn't the smartest thing as the splunkd process used a bunch of CPU - I assume scanning the file system for file matches repeatedly. It would be nice to separate the polling for files to monitor (say only poll for new file every 5,10,60 etc. minutes) versus polling the files themselves.
splunk add monitor '/var/mqm/qmgrs/*/errors/AMQERR01.LOG' -follow-only True
Anyway I switched and ran this instead
ls /var/mqm/qmgrs/*/errors/AMQERR01.LOG | xargs -i -t bin/splunk add monitor {} -follow-only True
which adds each found file separately. Cpu dropped dramatically.
I will just script to it to run that daily to pick up any new files.