Universal Forwarder not able to read all logs

[j_thomas]

Here is my input.conf:

[monitor:///var/log]
crcSalt =
disabled = false
index = main

From this it should recursively search all logs under /var/log/...

The issue I am seeing is that splunk user is not able to read apache2, audit, and some other logs. Permissions on all those logs that are 0640. Also to note, I have added the splunk user to groups: root, adm and syslog while trying to troubleshoot.

The forwarder should be able to read these files as its apart of the groups, but at this point I am at a loss.

Any help would be greatly appreciated!

UPDATE 11/3/14:

I know this is a little delayed, but the files are still not being read.

The addition of the "..." did help with recursive (and is a partial answer), but the files are still not being ingested/sent. I have performed some testing and maybe you guys can help me see something I don't see.

Current file permissions are 640 and owned by "root:adm". Splunk Forwarder is still apart root, adm and syslog groups. With these file permissions the logs are not being sent, but If i switch to the splunk user I can read the logs without issue. This tells me that the permissions on the logs should be fine and the UF should be able to sent them, correct? But, if i change the ownership to "root:root" the logs start being sent.

Any thoughts on this?

[frmaasdam]

So permission rights?
Files root:adm 640 right?
splunk member of adm group
Start Splunk with su -l splunk -c etc in initfile
Tried 650 rights on dir and files?

[frmaasdam]

Some ideas
If you are unable to read the files there must be some lines in the splunkd.log of the forwarder saying permission denied.
If not the files are read.
Are you sure of the inputs syntax?
Do you something in the output configuration?
Check the props and transforms
Are they indexed in a different index and you have not given the rights to view this index?
Are they in the index and you are searching with an incorrect date or time?

[ltrand]

When troubleshooting UF, the first three things I like to kick off first are as follows:

1. splunk cmd btool limits list thruput --debug
2. splunk list monitor
3. https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus

This gives me a nice list of the current state. THis will give you a list of what it is monitoring. From there I'd do a stop/start of the service to catch any read errors when it reaches the files you want. Perhaps it's not going deep enough in the recursion. Getting a list of what it does have will show you if it's a recursion depth issue.

[frmaasdam · Accepted solution]

I had more or less the same issues.
Logfiles owned by root:adm
Splunk running under user splunk
User splunk member of adm group
Login and su - splunk make it possible to cat the logfiles
BUT
Splunkforwarder running under user splunk was not able to read and forward the logfiles
2 options:
1. Start the Splunkforwarder using su -l splunk -c
This is what I have done
2. Setfacl on the logfiles to make USER splunk able to execute and read the files
I have done this in an other situation

,

[j_thomas]

Thank You frmaasdam!

After waiting half a day, the machine with these changes started pumping in all logs that were missing. Also, after adding this change to other machines it takes about 1hour before the unread logs start pumping in.

Thanks Again for all your suggestions guys!

[j_thomas]

Thanks frmaasdam! I have tried option 1 and 2 and it still does not send certain log files like apache's error log.

[lmyrefelt]

hehe, don't forget to have patience 😉
If the other data / logs are coming, there is really no reason for the rest not to come?

[j_thomas]

I 100% agree, but i have had this issue since the beginning of October. My patience is is running thin haha. Its just killing me on how other files of the same permissions have been coming in this entire time, but certain logs (apache, audit, etc...) are not.

[lmyrefelt]

aaaaah Sorry .. it seems you either need to set the "correct permission" on the log files and or change the user under which you are running Splunkforwarder.

You could for instance have two installed in some cases .. one for catching OS-logs and one for catching your App logs.

[j_thomas]

This would make it simple right? The thing is, the splunk user has access and can read both root:root and root:adm, but will only send root:root. So, in terms of file permissions and file access everything is proper. Is there a setting for the forwarder that only looks at certain permissions, such as root:root?

[lmyrefelt]

Well you also have except for the ulimit (OS-level) that you might want to increase but on high profil servers, hosting massive amounts of apps and or log files you might have to tweak limits.conf (on forwarder) for it to be able to coupe with the amount of data / number of files needed to monitor.

[inputproc]

max_fd =
* Maximum number of file descriptors that Splunk will keep open, to capture any trailing data from
files that are written to very slowly.
* Defaults to 100.

and (but not in your case .. you should see something regarding "block / blocked" in your log-files on the forwarder)

[thruput]

maxKBps =
* If specified and not zero, this limits the speed through the thruput processor to the specified
rate in kilobytes per second.
* To control the CPU load while indexing, use this to throttle the number of events this indexer
processes to the rate (in KBps) you specify.

[jeremiahc4]

I'm not sure if I missed it, but what version of Linux?

[j_thomas]

CentOS 6.5 and Ubuntu 12.04.4

[peter_krammer]

We had a similar problem on a server with many logfiles, but it was not related to permissions.

Have you had a look at your splunk forwarder logs? (index=_internal host=...)

First we increased the ulimit for the forwarder, because we had a lot of the following message:
"File descriptor cache is full (1024), trimming..."
See http://answers.splunk.com/answers/13313/how-to-tune-ulimit-on-my-server.html

Second we changed the stanze so splunk would only monitors recent files in the target directory.
I recommend this only if you have a setup which rotates files, because otherwise splunk won't monitor.

ignoreOlderThan=2d

See http://docs.splunk.com/Documentation/Splunk/6.1.3/Data/Editinputs.conf#Monitor_syntax_and_examples

[j_thomas]

Hey peter_krammer,

Thanks for the fast replies, but I have not seen this type of msg in the logs and to be honest the logs seem fine, no complaints.

If i change the logs ownership to root:root there is no problem and they ingest. The issue is the logs are created as root:adm and these are not being read, even though the splunk user can read both sets of permissions.

[peter_krammer]

Sorry I could not help, but I think you have to create a support case with splunk since this looks like bug.
One thing I would like to suggest you could try out.
1. Update your forwarder to 6.1.4
2. Set "SPLUNK_OS_USER = splunk" in splunk-launch.conf (if not already set)
See http://docs.splunk.com/Documentation/Splunk/6.1.4/Admin/Splunk-launchconf
3. Create a new /etc/init.d/splunk file by issuing the following command as root:
/opt/splunkforwarder/bin/splunk enable boot-start
This will recreate a new init config and hopefully the bug is fixed in the new version.

[j_thomas]

thanks peter, all but the 6.1.4 is my default config. Even re-checked.

[peter_krammer]

Can you please provide which Version of the Splunk UF you use?

[j_thomas]

Splunk Universal Forwarder 6.1.3 (build 220630)

[frmaasdam]

Please notice my remark on the GID bug.