Troubleshooting a file monitor on Universal Forwarder

[mike_k]

I currently have a Universal Forwarder running on a linux syslog server with a bunch of file monitors in place such as:

[monitor:///var/log/10.10.10.99/syslog.log]index=hphost_segment=3disabled=0

The index that i'm using for my new file monitor stanzas is a newly created index, that i haven't used previously.

I've created a couple of new deployment apps with the new file monitors and pushed them out to the UF on my syslog server. I can see other monitored files on the syslog server being forwarded into Splunk, however i'm not seeing my new files being monitored.

I've reloaded the deploy-server to ensure that the configs are being pushed out. I have also run a "./splunk btool inputs list" command and I can see that it is listing my new configuration as a part of the aggregated inputs.conf. However i'm not seeing any events for these new file monitors being forwarded into Splunk. The new index is showing 0 events received.

Is there a way to list events being outputted by the Universal Forwarder? Also is there a way to list events from my Universal Forwarder that are hitting the input queue on my Splunk indexer?

Thanks,

[mike_k · Accepted solution]

@gcusello, @isoutamo Thanks for those comments. They got me looking in the right direction. I looked through the UF logs and found a whole bunch of messages as follows:

WARN FilesystemChangeWatcher - error reading directory "/var/log/<ip_address_of_server>" Permission denied.

Had to go through and change the permissions for the syslog directories/files so that the "splunk" user could access these directories/files. Used Linux command "setfacl -R -m u:splunk:r-x /var/log/<ip_address_of_server" to do this

Once i had done this, the logs started populating in Splunk correctly.

Now when i look through the logs, i'm not seeing any further permission denied statements.

I am seeing some warning messages:

"ThruputProcessor - Current Data throughput (258kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf"

So i suspect that i may need to increase my maxKBps (which is still set at default 256kbps) to take into account the increased logging, however will wait to see whether it settles down once it has finished doing the initial file ingestion.

[gcusello]

Hi @mike_k,

the default for maxKBps is 256k for Universal Forwarders, if you have bandwidth availability, you can set this parameter and improve the quantity of logs sent to Indexers.

Ciao.

Giuseppe

[mike_k]

Thanks for those replies @gcusello and @isoutamo . Sorry i should have included that info in my original post :-/.

I did make sure that both "Enable App" and "Restart Splunkd" were enabled on the app before i pushed it out to the Universal Forwarder. To make doubly sure, i did as you suggested and added a comment line to my inputs.conf file and then reloaded the deployment server. However still don't seem to be getting any data in from those new file monitors i added.

For some reason I am unable to run the "splunk show config inputs" command. It prompts me for username/password and then gives me a login failed .. i'll have to look into that.

I also tied running the following search on my Search Head:

"index=_internal host="syslog_server_ip" group=per_source_thruput | stats count by series" which i think should be showing me what monitored files on the syslog server are coming through to my indexer, however it is not listing the new files that i am trying to monitor.

[gcusello]

Hi @mike_k,

you surely checked that you're receiving other data from that UF (index=_internal host=<your_host>).

If yes, please check if the new version was deployed and manually restart Splunk on Forwarder.

Ciao.

Giuseppe

[isoutamo]

Locally you found those logs under directory /opt/splunkforwarder/var/log/splunk

[isoutamo]

You probably need to create account on UF's splunk to run this command if you don't know it's admin account. Just look how to update admin pass from google (use user-seed.conf).

Have you look UF's internal log files to see if there is anything interesting?

[mike_k]

Thanks. Where abouts does the UF store it’s log files locally?

[gcusello]

Hi @mike_k,

Splunk logs are (in all Splunk installation) at $SPLUNK_HOME/var/log/splunk.

You can also see the Forwarder's logs in Splunk search running

index=_internal host=<your_host>

If you haven't them, probably there's a problem in log forwarding, the one I hinted to search.

Ciao.

Giuseppe

[gcusello]

i @mike_k,

only one question: did you configured a restart in ServerClass when App is updated?

by default, in a deployment server, local Splunk restart is disabled and you have to manually enable in the ServerClass.

You can check this on the Deployment Server.

Then you can quicly check if this is the problem on the target server, checking if the new input was deployed and manually restarting  Splunk on the target server.

Ciao.

Giuseppe

[isoutamo]

Hi

this is just like @gcusello said. For some reason, default is not to restart splunkd on target server. If you log to target and use btool, then you are seeing what configuration you have on disk, not what is currently in use. To see this you must use command “splunk show config”. If/when the missing restart was the reason, then just update that package e.g. add empty line in config + update version information. Then edit configuration on DS’s gui to ensure that you have checked restart and then redeploy it to all needed targets. 
r. Ismo