- Splunk Answers
- :
- Splunk Administration
- :
- Getting Data In
- :
- Re: Missing events from several indexes / sourcety...
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Missing events from several indexes / sourcetypes
I am not sure if anyone else has seen this issue, but at least 3 times lately I have done a broad search on an IP, in our Splunk instance of 4.3.1, and have gotten at least 3 sourcetypes - this particular one being our Cisco ASA, DHCP, and web filter. However, when re-running the search 4 or 5 or 6 hours later the Cisco ASA sourcetype no longer shows up in the results.
Is anyone aware of this specific issue? Or where can I start to troubleshoot this? Within the SOS app, the Cisco ASA index is showing its receiving events and is current. And I can do a search on the Cisco ASA sourcetype.
Our Splunk instance is made up of 4 servers: a search head and 3 indexers. Would it make sense to login to the indexer receiver the Cisco events and check there?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Well, you could have some problems with your peers not returning results, if you by 'same timeframe' mean something like 'April 4th, 1AM-3PM' and not 'last 24 hours'.
When the events DO NOT turn up, do you get search results from all indexers? This can be seen in the splunk_server field, which is automatically extracted. Check the field picker on the left.
/k
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I was definitely more alarmed when they did not show up. The events were packets being blocked at the firewall. UDP packets going from an internal network to an internet IP. Has anyone ever experienced this?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
By broad search do you mean over "All time"?
If you're searching 6 hours later, its very possible there simply are no cisco asa events for the "new" time period you are searching over. For example, searching over the last 24 hours and doing it again 6 hours later will exclude 6 hours of results on the back end of your original search results, while adding 6 new hours of results on the front end.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
When the behavior you're seeing occurs, can you return results by searching?
sourcetype=
I know you said you could by searching the sourcetype alone, but if you include the IP you're looking for, can you return results?
Do you ever experience problems with your search peers dropping off?
In your query are you searching the IP using a key valued pair i.e. field=
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I ran the search for the same timeframe as the original search. he timeframes were the same, and the events from the other sourcetypes were there, just not from the Cisco ASA. Does this help?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Well, what did the Cisco events look like ? Was there no obvious reason why they showed up? Or were you more alarmed when they didn't turn up?
/k
Extending Observability Content to Splunk Cloud
More Control Over Your Monitoring Costs with Archived Metrics!
New in Observability Cloud - Explicit Bucket Histograms
