fschange creating event for each line of file.

[jbidinger]

I am trying to implement file integrity monitoring. I have configured fschange as follows:

[fschange:/opt/bea/10_sp0_ltf/production/properties]
sourcetype = wls_monitor
index = fileint
disabled = false
_whitelist = \.xml$
recurse = false
pollPeriod = 600
fullEvent = true
sendEventMaxSize = -1

I'm having two problems. The first is that the sourcetype is not being set to "wls_monitor" but is instead showing up in the index as "xml-5".

The second issue is that each line of the monitored file is showing up as an individual event instead of the modified file being it's own event.

In addition, the events aren't showing up as having a source of fschangemonitor.

[jbidinger]

I think I found it. I had source: instead of source::

[jbidinger]

Also, this page is where I got the bad info on _whitelist.

http://www.splunk.com/base/Documentation/4.1.1/AppManagement/Configurationmonitoring

[jbidinger]

I'm getting closer. I'm now seeing events with a source of fschangemonitor. This is good. However, when the file is changed it's now being indexed as both single lines and as one large file. In both cases they still have a sourcetype xml-5.

I do have the following in local/props.conf

[source:/opt/bea/10_sp0_ltf/production/properties/*.xml]
sourcetype = wls_xml

[wls_xml]
LINE_BREAKER = ^()$
TRUNCATE = 1000000
SHOULD_LINEMERGE = false
DATETIME_CONFIG = NONE
CHECK_METHOD = modtime
KV_MODE = none
LEARN_MODEL = false

Thanks for the help!

[Lowell · Accepted solution]

There are a couple problems here.

You should be seeing events with the wls_monitor sourcetype, but that will only contain the actual audit events (notifying you of a new file, removed file, or modified file),

Here is an example of what these audit events will look like:

Wed Apr 28 21:52:42 2010 action=add, path="/path/to/myfile.cab", isdir=0, size=1401528, gid=65534, uid=1002, modtime="Fri Sep 19 13:57:19 2008", mode="r--r--r--", hash=d7l6zIDYsY7GyoC23M6QX5xQNaci3ulHyuqKn3kb0YM=

fschange also has the option of letting you store a copy of the file when a new file is created, or when an existing file is modified. I'm assuming that you want to do this based on your question. So you should have fullEvent=true as part of your config. Now, when the content of your event is fed into splunk, it uses the same source indentify rules as a "normal" input (for example, inputs using [monitor://]). So you will need to setup a matching entry in props.conf to establish your indexing options (such as sourcetype). I'm assuming that your looking at XML config files, so you will want these files indexed in one big chunk.

Also, _whitelist is only applicable for [monitor://] entries, not [fschange:] entries.

Try something like this:

inputs.conf:

[fschange:/opt/bea/10_sp0_ltf/production/properties]
sourcetype = wls_monitor
index = fileint
filters = xml_files, terminal-blacklist
recurse = false
pollPeriod = 600
fullEvent = true

[filter:whitelist:xml_files]
regex1 = \.xml$

[filter:blacklist:terminal-blacklist]
regex1 = .?

props.conf:

[source:/opt/bea/.../properties/*.xml]
sourcetype = wls_xml

[wls_xml]
LINE_BREAKER = ^()$
TRUNCATE = 1000000
SHOULD_LINEMERGE = false
DATETIME_CONFIG = NONE
CHECK_METHOD = modtime
KV_MODE = none
LEARN_MODEL = false

You could also use the config_file sourcetype which splunk provides in the unix app, instead of using the wls_xml sourcetype that I made up.

Docs:

• http://www.splunk.com/base/Documentation/latest/Admin/Monitorchangestoyourfilesystem

[Lowell]

Yeah. I have no officialness, but I can tell that's how I've seen it behave. (In fact I just fixed an issue caused by this on one of my systems yesterday.) I do agree that you would normally want to index the file as one big event, but you don't have to. For example, say your watching a directory *.csv config files. In that case you would probably want to index the file on a row-by-row basis instead of just one big event.

[jrodman]

Are you sure? I haven't spent much time with fschange, but I thought it always generated one event per file update.