I have a search that compares an expanded multi value field against a lookup table and returns those events where at least one of the field values was not found. My thinking is: If a singleColumns
value is not found, I'll have at least two events with a shared _cd
value in my results, which I then dedup
to ensure my counts are correct.
base search | eval UID = _cd | eval singleColumns=split(column_name, " ") | mvexpand
singleColumns | search NOT [|inputlookup Known_Bad_Columns | rename bad_columns as
singleColumns ] | dedup UID | stats count by field1, field2 | sort by count desc
I ran this against some known events (roughly 7 million prior to the expand) and some (not all) of my event counts were lower than expected. I then reran this search filtering to those specific event values (500 thousand prior to expand) and my counts came back correct. Can someone explain my loss of precision and possibly suggest a correction?
This issue actually came down to the use of _cd as my unique identifier. I opted to use the tuple detailed here and it is returning all events: https://answers.splunk.com/answers/49/does-each-splunk-event-have-a-unique-identifier.html
This issue actually came down to the use of _cd as my unique identifier. I opted to use the tuple detailed here and it is returning all events: https://answers.splunk.com/answers/49/does-each-splunk-event-have-a-unique-identifier.html
Hi,
If you have no warning message when using the mvexpand
function (like memory problem), it might be because of the sort
limit.
Can you try that :
base search
| eval UID = _cd
| eval singleColumns=split(column_name, " ")
| mvexpand singleColumns
| search NOT [|inputlookup Known_Bad_Columns | rename bad_columns as singleColumns ]
| dedup UID
| stats count by field1, field2
| sort 0 -count
You can see in the documentation of sort
http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Sort that there is a default limit to 10000.
I hope this will solve the problem.
Kail
Appreciate the help but it turned out to be my UID. I'll post the answer shortly.