How would I monitor file content history change?

[indeed_2000]

Hi I have two config files that need to monitor them, to answer these questions: Who?what?when? Change that file. Need content monitoring like git show different between versions, and history of file. Any idea? Thanks

[wyfwa4]

There are lots of unknowns about your requirement such as data format or size, but here are some suggestions that may help. They are based on the assumption that the file is small and you can read the entire file contents into Splunk on a regular basis. Once the data is in Splunk, you can use standard tools to compare lines, but it would only really be a good use case for Splunk if the data structure is simple and changes easy to identify.

Firstly the simplest option for a small file, is set up a script to read the contents on a regular basis - for example every hour, then once you have more than one copy of the file in Splunk, you can run a query to compare both versions. This will tell you what is different, but not what time it changed (only between the two collections) . Another script could just run a dir / ls command to show the file timestamp and collect that data if required.

If you want to detect the exact time the file content changed and trigger something at that specific time, initCrcLength & crcSalt may help to read the contents on change, but it can be hit and miss depending on the type of change. One other option may be to use indexed_extrations. if the file is a structured format (xml / json / csv etc) - you can  monitor the whole file and use the INDEXED_EXTRACTIONS and "CHECK_METHOD = modtime" options in a props.conf file on the collecting system. Every time the file mod time changes, it will re-read the whole file

You now potentially have two copies in Splunk that are indexed with the timestamp of the file change (if the contents have timestamps, you may need to disable this on indexing). In addtion, you would want to separate each line of the file into a separate event in Splunk. You then search for all events from both versions of the file, you tag events from each version so you can determine which version it was from - For example we create a variable called "version" and set this to "old" if it is from the previous version and set the value to "new" if from the latest version. Then you merge all the events (i.e. lines from the file) into one table using stats and include all the values of "version" for each line. If the value of "version" for a line contains both "old" and "new", then you know that line is present in both versions and so has not changed. If it only contains "old" the line has been removed and if it only contains "new", you know it has been added.

For "who" changed the file - you need to look at OS level auditing for that level of detail.

[PickleRick]

It's not a question about splunk itself but rather about yout ability to monitor changes to files. If you can detect those changes, you can send the events describing such change to splunk but first you have to detect it.

You have to look for file monitoring tools on the source system's side.

[richgalloway]

It depends on which config files you need to monitor.  If they're Splunk .conf files then Splunk 9.0 includes the ability to monitor them natively.  Search the _configtracker index to see what changes were made to the file and when.  The logs do not say who made the change, however.

For other config files, consider using the deprecated fschange input.  It will let you know when a monitored file has changed.  There's an option to include the added or changed record in the event.  Again, the user who made the change is not known.

[indeed_2000]

@richgalloway in my case, there are some xml configuration that related to another app, does it store full versions of file?

what is the appropriate configuration for this issue on splunk?

 Thanks 

[richgalloway]

So _configtracker won't help you in this case.

fschange does not store full versions of files it monitors.  It will only provide added or modified records, if configured to do so.  You can use monitor to get full versions of files, but it won't detect all changes since it's designed to look for new data on the end of a file.

Splunk is not meant to be a source code management system like git.  Consider using a third-party monitoring tool like SolarWinds and logging the tool's output in Splunk.