Missing events from several indexes / sourcetypes

[ride76]

I am not sure if anyone else has seen this issue, but at least 3 times lately I have done a broad search on an IP, in our Splunk instance of 4.3.1, and have gotten at least 3 sourcetypes - this particular one being our Cisco ASA, DHCP, and web filter. However, when re-running the search 4 or 5 or 6 hours later the Cisco ASA sourcetype no longer shows up in the results.

Is anyone aware of this specific issue? Or where can I start to troubleshoot this? Within the SOS app, the Cisco ASA index is showing its receiving events and is current. And I can do a search on the Cisco ASA sourcetype.

Our Splunk instance is made up of 4 servers: a search head and 3 indexers. Would it make sense to login to the indexer receiver the Cisco events and check there?

[kristian_kolb]

Well, you could have some problems with your peers not returning results, if you by 'same timeframe' mean something like 'April 4th, 1AM-3PM' and not 'last 24 hours'.

When the events DO NOT turn up, do you get search results from all indexers? This can be seen in the splunk_server field, which is automatically extracted. Check the field picker on the left.

/k

[ride76]

I was definitely more alarmed when they did not show up. The events were packets being blocked at the firewall. UDP packets going from an internal network to an internet IP. Has anyone ever experienced this?

[carmackd]

By broad search do you mean over "All time"?

If you're searching 6 hours later, its very possible there simply are no cisco asa events for the "new" time period you are searching over. For example, searching over the last 24 hours and doing it again 6 hours later will exclude 6 hours of results on the back end of your original search results, while adding 6 new hours of results on the front end.

[carmackd]

When the behavior you're seeing occurs, can you return results by searching?

sourcetype=

I know you said you could by searching the sourcetype alone, but if you include the IP you're looking for, can you return results?

Do you ever experience problems with your search peers dropping off?

In your query are you searching the IP using a key valued pair i.e. field=, or simply searching for the IP as a string in the raw event data?

[ride76]

I ran the search for the same timeframe as the original search. he timeframes were the same, and the events from the other sourcetypes were there, just not from the Cisco ASA. Does this help?

[kristian_kolb]

Well, what did the Cisco events look like ? Was there no obvious reason why they showed up? Or were you more alarmed when they didn't turn up?

/k