How to create a report in Splunk as a configuration management tool to compare all fields for the same object from different dates and display changes made?

[gnorud]

I am testing on using Splunk as a configuration management tool so I can compare if there were any configuration changes in my server (in this case, it is netapp filer add, modify, delete). I managed to import all configuration items as multiple objects with attributes into Splunk.

Does anyone know how I can compare ALL fields between dates on some object (essentially 2 events with same KV pair from 2 different dates.) and output results (equal and non-equal ones)? The idea is to track any unauthorized changed, without having to define which fields to compare, and basically compare the whole table.

For Example:

Day - 1
vserver="vs_test"
read_grants_exec="disabled"
smb2_enabled="true"
smb3_enabled="true"

Day - 2
vserver="vs_test"
read_grants_exec="disabled"
smb2_enabled="false"
smb3_enabled="false"

Day-3

I want my reports is able to tell me, without having to define field names, to compare all fields and generate report on: smb2_enabled/smb3_enabled attributes are changed from true to false.

[woodcock · Accepted solution]

Your base search is like this:

... | stats latest(*) AS latest_* list(*) AS list_* count dc(*) AS dc_* by vserver | eval fieldsChanged=" " | eval fieldsUnchanged=" " | foreach dc_* [eval fieldsChanged = if((<<FIELD>> > 1), fieldsChanged . "<<MATCHSTR>>,", fieldsChanged) | eval fieldsUnchanged = if((<<FIELD>> > 1), fieldsUnchanged, fieldsUnchanged . "<<MATCHSTR>>,") ] 

This example demonstrates everything that you say you need. Adjust it based on your needs from here.

[gnorud]

I've got another issue on the same question..
Some of attributes will take more than one value.
for instance, each vserver will have multiple NICs, the result from the query seems like only take one ip address out of many. Then it will compare the latest result (single value)and result in fieldsChanges on NIC IPs. which consists of array and they are not changed.
I tried to change latest() to values() and sort it by descending on value(). however it still doesn't give me matching results of 2 arrays values for the single fields.

Is there any workaround for it?

[woodcock]

I am sure that I am misunderstanding you but let met try to summarize:
The vserver field actually contains an IP Address.
Any server can have multiple NICs (e.g. multiple IP Addresses, or multiple vserver values).
The solution works but you desire that it break out by server, not vserver (IP Address).

If this is correct then the way to handle this is to normalize all IPAddreses to a new field called server and use that field name instead of vserver in the solution. Typically this is done by a lookup that is created from another system such as a CMDB.

[gnorud]

its working great. I ve got what I need. Thank you

Is there any reference kb for me to understand this eval ?

if((<<FIELD>> > 1), fieldsChanged . "<<MATCHSTR>>,", fieldsChanged)

[woodcock]

It is just foreach:

http://docs.splunk.com/Documentation/Splunk/6.3.0/SearchReference/Foreach

This step is stacking up the fields; if the number of values of the field is more than 1, add the field name as a value to the fieldsChanged field.