Is it possible to use lookups to validate data coming in, using metadata?

[a212830]

HI,

Is it possible to create get entries in a serverclass (or a lookup), and then validate that data has been received from each host by comparing it againsts metadata?

[gcusello · Accepted solution]

I usually verify that all the hosts in my perimeter are sending data using a lookup and a search like this:

|inputlookup perimeter.csv | eval count=0 | append [ search _internal | stats count by host ] | stats sum(count) AS Total by host | where Total=0

in this way the hosts with Total!=0 are sending logs and hosts with Total=0 don't do it.
Bye.
Giuseppe

[a212830]

Thanks. I try this, and get an error: [subsearch]: Could not find an index named "_blocksignature". My recollection is that index is no longer part of 6.4.

[gcusello]

where I used index=_internal, you have to use your index in which you want to verify the host list,
could you send the search to verify with the host list in perimeter?
Bye.
Giuseppe

[a212830]

here's my search, which returns counts for servers that are collecting, but doesn't show for servers that aren't....

|inputlookup testagain.csv | eval count=0 | append [ search index=dh_clocksync | stats count by host ] | stats sum(count) AS Total by host | where Total=0

My lookup has 195 entries. When I remove the where column at the end, I get counts for 161 servers. I need to see which of those 195 aren't showing.

[gcusello]

what is the name of the lookup's field in which there are the host's names?
if it isn't "host" you have to modify the search in this way:
|inputlookup testagain.csv | rename lookup_field AS host | eval count=0 | append [ search index=dh_clocksync | stats count by host ] | stats sum(count) AS Total by host | where Total=0

Bye.
Giuseppe

[a212830]

weird. i updated my lookup, and the total count is greater than the number of entries in my lookup file.

|inputlookup clocksync_lookup | eval count=0 | append [ search index=dh_clocksync | stats count by host ] | stats sum(count) AS Total by host

IF I do |inputlookup clocksync_lookup, i get a total of 195 entries. When I use this complete search, I get 366 entries.

[gcusello]

this means that you have different host names between lookup and index.
Verify the case, or insert transformation in uppercase in both the searches like this

|inputlookup clocksync_lookup | eval count=0 | eval host=upper(host) | append [ search index=dh_clocksync | eval host=upper(host) | stats count by host ] | stats sum(count) AS Total by host

In this way you'll have 195 items.
Bye.
Giuseppe

[a212830]

Fantastic. Didn't realize Splunk was case sensitive when comparing fields...

This works great and is a good prototype for future work. Thanks!

[sundareshr]

You could try something like this, for a list of host NOT in metadata

|inputlookup lookupfilewithhosts.csv | search NOT [| metadata type=hosts | fields host ]