Search for disappeared and new hosts

[chris]

Hi

I would like to have a way to find out whether hosts have stopped logging to our central log infrastructure or if new hosts have appeared.

I tried using the "set diff" command with two identical searches with different timeranges like so:

| set diff [search index=* * earliest=03/16/2010:08:30:0 latest=03/16/2010:08:35:0 | dedup host | fields host ] [search index=* * earliest=03/17/2010:08:30:0 latest=03/17/2010:08:35:0 | dedup host | fields host ]

I think the diff should list the hosts that I am looking for, but I seem to get the union of the 2 searches as the result. (The individual searches return 761 and 773 results, the search listed above returns 1534 results)

If intersect is used instead of diff in the above query it returns 0 results. If union is used in the query returns 1534 results.

What is wrong in the query or is there another (better) way to do this?

[gkanapathy · Accepted solution]

First of all, if you're just looking for hosts, it's much more efficient to get the information from a metadata search. This is extremely costly:

index=*

while

| metadata type=hosts index=* | search firstTime > blah lastTime < xxx | fields host

is about as cheap as it gets, where blah and xxx are epoch times (you can use ...| convert mktime(blah) timeformat="%m/%d/%Y:%H:%M:%S"... if you like to enter them the other way). This will also probably solve your set problem, because I think you might need a | fields - _* to get rid of hidden fields when you try to generate the host list from event data. metadata doesn't have hidden fields, so not an issue there.

[gkanapathy]

I'm not sure if the need to use | fields - _* is a bug or not. Some commands will ignore hidden fields after | fields - *, others (like set) apparently do not. This might be intentional but I don't know.

[chris]

You were right about the fields - _* and the metadata search is way faster.