Troubleshoot - Linux Universal Forwarder is not forwarding all files

[daddyoh]

We have a UF on RHEL that forwards some files fine but one that is not being forwarded. I recently added a file to forward and it is not being forwarded. We are using splunk light 6.4 and UF 6.4.

I can log into the splunk account for that UF and cat the file. I can see the contents of the file. This is also a file type that is being forwarded on other servers fine. I have restarted the UF several times but no records are being forwarded. The volume of records in the file is low. Yesterday when I added it there were maybe 200 records. Today, after rotation. there are two records.

The records look like:

[26-Jul-2016 08:35:56 America/New_York] PHP Notice:  Trying to get property of non-object in /WWW/repos/kp4/includes/kp4/php/Artemis/Slideshow/Instagram.php on line 70
[26-Jul-2016 08:35:56 America/New_York] PHP Notice:  Trying to get property of non-object in /WWW/repos/kp4/includes/kp4/php/Artemis/Slideshow/Instagram.php on line 79

I'm very new to splunk. We have 5 servers successfully forwarding records from 16 files and folders. We forward about 500MB of records a day.

How can I diagnose this problem? We added this file to splunk via the Data Input menu item on the search head. We run a single search, index, deployment server. Very simple set up.

Thanks in advance for your help.

[ddrillic · Accepted solution]

The place to start is I can't find my data!

[daddyoh]

@ddrillic

The site won't let me post an answer because I don't have enough reputation points yet.

Thanks for the link. That is the first place I went to.

I did get it to work:

I ran this on the splunk search instance

http://webserlog:8000/en-US/debug/refresh

and restarted the UF instance. The contents of the file is now showing up.

[daddyoh]

I restarted splunk UF and looked at splunkd.log and could not see any references to the file in the log file. No progress.