Find changed files and diff the first and last versions

[skooby]

I am writing app and searches that will find changed files and diff the first and last version of the file:

(This is in place of fschange which I understand is now deprecated).


inputs.conf:

#############################################################

# xml

#############################################################

[monitor:///.../*.xml]

sourcetype=my_xml

index = Main

crcSalt=< SOURCE >TEST1

props.conf:

[my_xml]

CHECK_METHOD = entire_md5

TRUNCATE = 0

LINE_BREAKER = (?!)

I have composed some searches that get the oldest and newest file....


index="main" source="/*xml"| top limit=1 host, source | table host,source | sort -_time asc| head 1

and

index="main" source="/*xml"| top limit=1 host, source | table host,source | sort -_time asc| tail 1


These pull back the desired info BUT but display the desired file details in the "Statistics" tab (in the Splunk UI).

If I try and feed this into a subsearch, e.g.


[search index="main" source="/*xml"| top limit=1 host, source | table host,source | sort -_time asc| head 1]



then Splunk disregards the "head -1" part and spits out all 8 file versions to the "Events" tab.

Perhaps because of this, my diff also fails (having said that, I'm not sure if diff is meant to be used like this, feel free to put me straight here!)


diff pos1=[search index="main" source="/*xml"| top limit=1 host, source | table host,source | sort -_time asc| head 1] pos2=[search index="main" source="/*xml"| top limit=1 host, source | table host,source | sort -_time asc| tail 1]

gives the error:


Error in 'search' command: Unable to parse the search: Comparator '=' has an invalid term on the right hand side.

The search job has failed due to an error. You may be able view the job in the Job Inspector.


Can anyone give me any pointers, please?

[dwaddle · Accepted solution]

In this case, fschange is safe to continue to use for some time. It may have been deprecated as of Splunk 5.0, but as of Splunk 6.1 it is still supported. I would think it continues to be "safe to use as-is" even though it is definitely not being improved.

Looking for things that do fschange-like-stuff, an obvious contender is OSSEC (http://ossec-docs.readthedocs.org/en/latest/manual/non-technical-overview.html) which fits nicely into splunk.

But, more specific to your question -- an approach like below might be more reliable for getting your first and last events for diffing.

index="main" source="/*xml" 
| streamstats count 
| eventstats max(count) as high, min(count) as low
| where ( (count=high) OR (count=low) )

With this, streamstats attaches a count to each event seen. First event is '1', second '2', etc. Then eventstats tacks on the highest and lowest values of count seen to the original events, giving you the ability to filter.

The diff command expects TWO events (and only two events).

[skooby]

Yay! that works!

My final search was:

[search index="m*" source="/*xml"| top limit=1 host, source | table host,source | sort -_time asc ] | streamstats count
| eventstats max(count) as high, min(count) as low
| where ( (count=high) OR (count=low) ) | diff pos1=1 pos2=2


Thank you dwaddle!