topic Why is /etc/shadow being indexed? Does fschange index files? in Getting Data In

Why is /etc/shadow being indexed? Does fschange index files?

oreoshake — Wed, 05 May 2010 07:09:33 GMT

unix       [monitor:///etc]
unix       _blacklist = (/etc/shadow)
system     _rcvbuf = 1572864
unix       _whitelist = (\.conf|\.cfg|config$|\.ini|\.init|\.cf|\.cnf|shrc$|^ifcfg|\.profile|\.rc|\.rules|\.tab|tab$|\.login|policy$)
system     host = blah.blah.com
unix       index = os

/etc/shadow is being indexed and I can't figure out why. This is the only stanza related to /etc other than the fschange stanza. Does fschange index the files it monitors? We don't want /etc/shadow indexed 🙂 I added the _blacklist entry but that didn't do anything.

Re: Why is /etc/shadow being indexed? Does fschange index files?

dwaddle — Wed, 05 May 2010 07:47:36 GMT

"monitor://" is not fschange, but is file tailing/indexing. From http://www.splunk.com/base/Documentation/latest/Admin/Inputsconf you will want to use a "fschange:" stanza instead.

Re: Why is /etc/shadow being indexed? Does fschange index files?

jrodman — Wed, 05 May 2010 12:34:22 GMT

Agree with dwaddle.

However, assuming the events you get are not fschange events, it's unclear to me why your blacklist does not exclude the indexing of shadow. How does the event appear in index?

Independently, we have a config that tries to prevent this, but it's goofed. Look in etc/apps/unix/default/props.conf, you'll see:

[source::///etc/passwd]
sourcetype = ignored_type

[source::///etc/shadow*]
sourcetype = ignored_type

This wants to be:

[source::/etc/passwd]
sourcetype = ignored_type

[source::/etc/shadow*]
sourcetype = ignored_type

You can drop the lower set of lines in etc/apps/unix/local/props.conf (or another local directory) until we ship the next maintenance release. Investigating the blacklist fail is worthwhile though, since it also should have worked. I think you've a ticket in?

Re: Why is /etc/shadow being indexed? Does fschange index files?

Lowell — Wed, 05 May 2010 21:00:22 GMT

Small world. I just reported this as a bug to Splunk a few days back. (Splunk Case #42678, SPL-31254)

Also note that you probably want the entry for passwd to be:

[source::/etc/passwd*]
sourcetype = ignored_type

This should protect against backup files being indexed too, which can be just as dangerous. (Many different variations of Linux have different suffixes, so it seems easier and safer just to do passwd*)

Also note that /etc/ is in inputs.conf both as a monitor and via fschange which the docs say should should do. See the answer on Are there any reasons to setup both monitor and fschange on the same path?

Re: Why is /etc/shadow being indexed? Does fschange index files?

jrodman — Wed, 05 May 2010 21:52:05 GMT

Yeah, I probably had that answer ready after reading your bug. I read some bug that mentioned it, in any event.

Re: Why is /etc/shadow being indexed? Does fschange index files?

oreoshake — Thu, 06 May 2010 00:35:45 GMT

Thanks, this certainly makes sense. I do see the fschange events but I was only worried about the text being indexed. It actually came up when I was demoing the *nix app!!!

The weird thing is that it is very sporadic. My indexers and forwarders pretty much share the same config for the *nix app, but it was only seeing /etc/shadows from 2 of my 3 indexers and none from my forwarders. Odd. I will accept the answer if I don't see any events for a while. I created a saved search to automatically delete these events.

Re: Why is /etc/shadow being indexed? Does fschange index files?

oreoshake — Wed, 12 May 2010 08:48:13 GMT

This did not help, /etc/shadow is still showing up 😞

Re: Why is /etc/shadow being indexed? Does fschange index files?

Lowell — Tue, 18 May 2010 03:13:11 GMT

Oreoshake,

I'm surprised splunk is still reading your shadow file with the config entries given here. Whether your /etc/ files are being pulled in via monitor or fschange input, the given config really should be blocking these files.

Here is one theory about why this may not be working for you:

You may want to consider the possibility that some other [source::] stanza is matching before the ignored_type rules. Keep in mind that that using wildcards (such as *), will lower the priority of a matching stanza, where as a literal path will be given a very high (100) priority. This can be compensated for with the following:

[source::/etc/passwd*]
sourcetype = ignored_type
priority = 100

[source::/etc/shadow*]
sourcetype = ignored_type
priority = 100

I suggest that you open a shell on a machine where you are having this problem. Log in as, or switch to the user that splunkd runs as. Then issue the following command to see what rules are being used for this file:

splunk test sourcetype /etc/shadow

You should either see "cannot_read" (if OS permissions prevent access), or the sourcetype should be shown as ignored_type. Otherwise, your file will get indexed. This should help you track down either a config issue due to priority (which I confirmed that I had this problem on my system too, so that's for asking about this again!!!), or if your config change wasn't deployed properly, or some other config issue.

Unix Admin side note:

BTW, I assume your running splunk as root and not as the default splunk user. Changing that may be the best solution to this. From an unix admin perspective, I think that /etc/passwd needs to be readable by everyone, where as /etc/shadow should be pretty much only readable by the root user. So one suggestion would be to setup splunk to run as a non-privileged user.

I setup all my log files in /var/log/ to be readable by the adm user and then made splunk a member of that group, and it's worked quite well. (I do have one system with an older Linux install, and the syslog daemon on that system doesn't let me set user/group permissions on new log files. So that's a pain. I have an hourly job that runs a bunch of chown and chmod commands to get the permissions set properly.) This does require more initial setup, but it's probably a safer solution, especially if there are ever any security bugs that would allow a malicious user to take control of splunkd and run commands as that user.

Re: Why is /etc/shadow being indexed? Does fschange index files?

Lowell — Wed, 30 Jun 2010 22:37:27 GMT

For anyone following along wit this issue... If you do not set priority value here, than this solution will not work, as noted in my second answer. BTW, Splunk support said this would be fixed in 4.1.3, but now it's scheduled to be fixed in 4.1.4.