We are having trouble with our file server. I currently monitor disk usage and have alerts based on the amount of disk space left.
However, we are having large chunks disappear. Unfortunately, its not a single huge file or set of huge files causing this but I need to find a way to track what is getting copied. Just the other day, we managed to lose 10 GB overnight.
Is there anything Splunk can do to help me identify what files (which will lead me to WHO) are being copied over in a time period? I don't want changed files as this server is heavily accessed. I just want to see new files.
Thanks for any help. I am not sure if this is something I can accomplish with Splunk but if not, please feel free to recommend another tool that may be able to help.
Not sure if this is a silly answer... but comparing the output of 'tree' would probably show you what you wanted? Just schedule it and compare differences between the outputs.
EDIT: bah... what about the 'find' tool? use the 'ctime' option?
Or are you on windows...?
Splunk has a "fschange" input that will monitor for file system changes, but you probably don't want to start there. This could create tons of events and it may not even point to who is causing your issue, but it will give you lots of details to work with. The docs are here:
If you do go down this road, make sure that (1) you have
hashMaxSize=-1 to disable hashing, (2) disable
fullEvent=false so that you don't index the new/changed files that splunk finds. (3) Consider using a separate/temporary index for these events, and (4), look carefully at the polling interval and delay options.
A simpiler approach would be to start sampling (or increase your sampling rate) of disk space usage in splunk. (e.g. the "df" sourcetype if your on Unix, or "WMI:FreeDiskSpace" if your on windows.) Then simply chart the disk usage change over time and see if you can pinpoint when the extra space is being used. If you know when, this may lead you to other log messages that indicate who is logged in during that time. Or, you could be able to use that information to search your file system for all files modified during the time window in question; which may lead you to who.