topic Re: Universal Forwarder not able to read all logs in Getting Data In

Universal Forwarder not able to read all logs

j_thomas — Thu, 09 Oct 2014 15:38:19 GMT

Here is my input.conf:

[monitor:///var/log]
crcSalt =
disabled = false
index = main

From this it should recursively search all logs under /var/log/...

The issue I am seeing is that splunk user is not able to read apache2, audit, and some other logs. Permissions on all those logs that are 0640. Also to note, I have added the splunk user to groups: root, adm and syslog while trying to troubleshoot.

The forwarder should be able to read these files as its apart of the groups, but at this point I am at a loss.

Any help would be greatly appreciated!

UPDATE 11/3/14:

I know this is a little delayed, but the files are still not being read.

The addition of the "..." did help with recursive (and is a partial answer), but the files are still not being ingested/sent. I have performed some testing and maybe you guys can help me see something I don't see.

Current file permissions are 640 and owned by "root:adm". Splunk Forwarder is still apart root, adm and syslog groups. With these file permissions the logs are not being sent, but If i switch to the splunk user I can read the logs without issue. This tells me that the permissions on the logs should be fine and the UF should be able to sent them, correct? But, if i change the ownership to "root:root" the logs start being sent.

Any thoughts on this?

Re: Universal Forwarder not able to read all logs

peter_krammer — Thu, 09 Oct 2014 16:14:54 GMT

I think you need the wildcard for recursive monitoring in your config:

[monitor:///var/log/...]
disabled = false
index = main

Please refer to http://docs.splunk.com/Documentation/Splunk/6.1.4/admin/Inputsconf section "Note concerning wildcards and monitor"

Re: Universal Forwarder not able to read all logs

j_thomas — Thu, 09 Oct 2014 17:03:48 GMT

Thanks Peter!!! This seems to have done it.

Re: Universal Forwarder not able to read all logs

frmaasdam — Fri, 10 Oct 2014 07:28:17 GMT

There is a bug with GID in Splunk when you autostart splunk. I use su -l splunk -c in the init file.

Re: Universal Forwarder not able to read all logs

frmaasdam — Mon, 03 Nov 2014 15:53:07 GMT

Please notice my remark on the GID bug.

Re: Universal Forwarder not able to read all logs

j_thomas — Mon, 03 Nov 2014 15:54:07 GMT

That did not resolve the issue either

Re: Universal Forwarder not able to read all logs

peter_krammer — Mon, 03 Nov 2014 15:56:25 GMT

Can you please provide which Version of the Splunk UF you use?

Re: Universal Forwarder not able to read all logs

j_thomas — Mon, 03 Nov 2014 15:58:28 GMT

Splunk Universal Forwarder 6.1.3 (build 220630)

Re: Universal Forwarder not able to read all logs

peter_krammer — Mon, 03 Nov 2014 16:14:48 GMT

We had a similar problem on a server with many logfiles, but it was not related to permissions.

Have you had a look at your splunk forwarder logs? (index=_internal host=...)

First we increased the ulimit for the forwarder, because we had a lot of the following message:
"File descriptor cache is full (1024), trimming..."
See http://answers.splunk.com/answers/13313/how-to-tune-ulimit-on-my-server.html

Second we changed the stanze so splunk would only monitors recent files in the target directory.
I recommend this only if you have a setup which rotates files, because otherwise splunk won't monitor.

ignoreOlderThan=2d

See http://docs.splunk.com/Documentation/Splunk/6.1.3/Data/Editinputs.conf#Monitor_syntax_and_examples

Re: Universal Forwarder not able to read all logs

j_thomas — Mon, 03 Nov 2014 16:32:30 GMT

Hey peter_krammer,

Thanks for the fast replies, but I have not seen this type of msg in the logs and to be honest the logs seem fine, no complaints.

If i change the logs ownership to root:root there is no problem and they ingest. The issue is the logs are created as root:adm and these are not being read, even though the splunk user can read both sets of permissions.

Re: Universal Forwarder not able to read all logs

peter_krammer — Mon, 28 Sep 2020 18:04:14 GMT

Sorry I could not help, but I think you have to create a support case with splunk since this looks like bug.
One thing I would like to suggest you could try out.
1. Update your forwarder to 6.1.4
2. Set "SPLUNK_OS_USER = splunk" in splunk-launch.conf (if not already set)
See http://docs.splunk.com/Documentation/Splunk/6.1.4/Admin/Splunk-launchconf
3. Create a new /etc/init.d/splunk file by issuing the following command as root:
/opt/splunkforwarder/bin/splunk enable boot-start
This will recreate a new init config and hopefully the bug is fixed in the new version.

Re: Universal Forwarder not able to read all logs

jeremiahc4 — Mon, 03 Nov 2014 18:06:59 GMT

I'm not sure if I missed it, but what version of Linux?

Re: Universal Forwarder not able to read all logs

lmyrefelt — Mon, 03 Nov 2014 18:12:41 GMT

Well you also have except for the ulimit (OS-level) that you might want to increase but on high profil servers, hosting massive amounts of apps and or log files you might have to tweak limits.conf (on forwarder) for it to be able to coupe with the amount of data / number of files needed to monitor.

[inputproc]

max_fd =
* Maximum number of file descriptors that Splunk will keep open, to capture any trailing data from
files that are written to very slowly.
* Defaults to 100.

and (but not in your case .. you should see something regarding "block / blocked" in your log-files on the forwarder)

[thruput]

maxKBps =
* If specified and not zero, this limits the speed through the thruput processor to the specified
rate in kilobytes per second.
* To control the CPU load while indexing, use this to throttle the number of events this indexer
processes to the rate (in KBps) you specify.

Re: Universal Forwarder not able to read all logs

lmyrefelt — Mon, 03 Nov 2014 18:15:11 GMT

aaaaah Sorry .. it seems you either need to set the "correct permission" on the log files and or change the user under which you are running Splunkforwarder.

You could for instance have two installed in some cases .. one for catching OS-logs and one for catching your App logs.

Re: Universal Forwarder not able to read all logs

j_thomas — Mon, 03 Nov 2014 18:19:10 GMT

CentOS 6.5 and Ubuntu 12.04.4

Re: Universal Forwarder not able to read all logs

j_thomas — Mon, 03 Nov 2014 18:30:49 GMT

thanks peter, all but the 6.1.4 is my default config. Even re-checked.

Re: Universal Forwarder not able to read all logs

j_thomas — Mon, 03 Nov 2014 18:45:00 GMT

This would make it simple right? The thing is, the splunk user has access and can read both root:root and root:adm, but will only send root:root. So, in terms of file permissions and file access everything is proper. Is there a setting for the forwarder that only looks at certain permissions, such as root:root?

Re: Universal Forwarder not able to read all logs

frmaasdam — Mon, 03 Nov 2014 19:15:57 GMT

I had more or less the same issues.
Logfiles owned by root:adm
Splunk running under user splunk
User splunk member of adm group
Login and su - splunk make it possible to cat the logfiles
BUT
Splunkforwarder running under user splunk was not able to read and forward the logfiles
2 options:
1. Start the Splunkforwarder using su -l splunk -c
This is what I have done
2. Setfacl on the logfiles to make USER splunk able to execute and read the files
I have done this in an other situation

Re: Universal Forwarder not able to read all logs

j_thomas — Mon, 03 Nov 2014 19:53:02 GMT

Thanks frmaasdam! I have tried option 1 and 2 and it still does not send certain log files like apache's error log.

Re: Universal Forwarder not able to read all logs

lmyrefelt — Mon, 03 Nov 2014 19:55:22 GMT

hehe, don't forget to have patience 😉
If the other data / logs are coming, there is really no reason for the rest not to come?