Need the ability to perform a hygiene check on certain data types by comparing fields & field quantities every time a new source is ingested for a particular sourcetype. This would be an extension of fieldsummary in a way, by showing fieldsummary over time and highlighting, for example if a field changes from FieldX to fieldX.
Set this up as an alert to run right after the new data gets injested everyday
base search earliest=@d | table * | untable _time today_fields data | table today_fields | append [ search base search earliest=-1d@d latest=@d | table * | untable _time yesterday_fields data | fields yesterday_fields ] | stats values(*) AS * | eval z=mvzip(today_fields, yesterday_fields) | mvexpand z | rex field=z "(?<today>[^\,]+)\,(?<yesterday>.*)" | eval diff=if(match(yesterday, today), "match", "no") | table yesterday today diff
maybe, we can manually create an alert and if the hourly or daily change is more than a particular amount, we can send an alert.
sourcetype=ActiveDirectory earliest=-1h@h latest=@h | stats count as Today | appendcols [search index=main earliest=-25h@h latest=-24h@h | stats count as Yesterday ] | eval Change = Today - Yesterday | table Change
Could you give more information?
it seems as you have to search in an index for a sourcetype showing al the fields values in a table, inserting some filters
I have a source that updates once a day. The fields are as follows:
Field1, Field2, Field3, Field4, Field5
The next day, the source is slightly changed, and now the fields are coming in as follows:
field1, Field2, Field3, Field4, Field5
How can I easily detect that the first field changed from Field1 to field1? When this happens, it breaks the dashboards that refer to Field1 (uppercase).