Different results same search

[reswob4]

So we have both Snort and Sourcefire in our environment. I'm using a simple search to create a table of the top hits but the results are inconsistent. (all searches below are for the same time parameters and performed within seconds of each other)

sourcetype="SNORTIDS" OR sourcetype="eStreamer" | rename Message AS msg | top 25 msg

gives me 20 different messages. BUT reviewing the results, we noticed that some messages from eStreamer were missing. A quick check revealed the following:

sourcetype="SNORTIDS" | rename Message AS msg | top 25 msg

yields 20 unique messages.

sourcetype="eStreamer" | rename Message AS msg | top 25 msg

yields 10 unique messages.

If I search specifically for the messages that we know are missing, the search is successful, so I know those events are in the index.

But then why doesn't

sourcetype="SNORTIDS" OR sourcetype="eStreamer" | rename Message AS msg | top 25 msg

yield all 30 messages? (and yes, the 20 found in the first are completely different from the 10 found in the second)

Is this a mistake where I don't understand how the search syntax works or is there something deeper in the Splunk configuration that is wrong?

[somesoni2 · Accepted solution]

Can you test something like this

sourcetype="SNORTIDS" OR sourcetype="eStreamer" | eval NewMsg=coalesce(Message,msg) | top 35 NewMsg

[reswob4]

Hah! That looks like it did it. Ran the two individually again and compared against the results from your suggestion with the eval NewMsg=coalesce(Message,msg) statement and everything is there.

Can you explain why this works and the others don't?

[somesoni2]

When you're renaming, Where Message field was present, it will get renamed to msg, but where Message is not present, it will create a non-existent field Massage (will be null) and will overwrite msg field. The coalesce ensures that it takes the first available value between Message and msg.

[reswob4]

Ah. Thanks! Answer accepted!

[reswob4]

@woodcock results didn't change, results still not consistent.
@somesoni2 one sourcetype has the msg field, the other has a Message field, so yes, after I renamed it, they both have the common field msg.
@sfatnass Sorry, I don't feel comfortable sharing our top IDS hits on a public forum... Since that is exactly what is in the results of these searches.

So, some more testing and information. I again ran the three searches above and exported the results to a CSV. This time there were 21 results with the first search, 11 with the second and 21 with the third. Directly comparing the results, I found that (as the numbers suggested) the 21 results from the third search exactly matched the 21 results from the first search and that NONE of the 11 results from the second search included in the first search.

So then I ran the first search and tried to isolate a specific msg found in the second search (name of msg is changed fyi).

sourcetype="SNORTIDS" OR sourcetype="eStreamer" | rename Message AS msg | WHERE msg="BADPACKET*"

and that yielded NO RESULTS.

I then tried to run the third search and append the second search:

sourcetype="SNORTIDS" | rename Message AS msg | append [search sourcetype="eStreamer" | rename Message as msg] | top 35 msg

27 results! Yeah, progress! But I was still missing at least three msg's. So I tried to isolate one of the missing msg's:

sourcetype="SNORTIDS" | rename Message AS msg | append [search sourcetype="eStreamer" | rename Message as msg] | WHERE msg="BADPACKET*"

But that yielded NO results. I tried then to search for something that I knew showed up in the 27 results:

sourcetype="SNORTIDS" | rename Message AS msg | append [search sourcetype="eStreamer" | rename Message as msg] | WHERE msg="PACKET*"

But that ALSO yielded no results.

Can someone please explain what is going on here?

[woodcock]

I know this sounds silly but what do you get when you change top 25 to top 30?

[somesoni2]

Does the both sourcetypes have common field msg, after the Rename??

[sfatnass]

the rename is not in cause, can you show the result of the three searchs, with the count obtained.

i think it's normal, look at the count value.