Events now Missing from Regular/Notable Index

[stranjer]

We have an alert that we had setup to create a notable event and email a notification when a particular Windows Event occurs.

A few weeks ago, we received an email of the event, and originally saw the event in the Splunk environment, and verified the event occurred on the Windows host referenced. However, when we went to review the event later, we can no longer find that event or the Notable Event that should have been generated when the activity occurred.

All search peers are up and search and replication factors are being met
Reran the original alert search logic to confirm, still not seeing the exact event that triggered it, but nothing in search logic exclude it
Tried to search for just the artifacts I know occurred, Event ID and host involved.
Checked Index retention settings, set for 1 year and I'm seeing plenty of space in the index and similar events from earlier, so shouldn't have rolled.
Looked through Notable Index around the time for the specific alert that triggered, and did not see the event.
I did searches for the event over multiple days range thinking there might be time difference issues, did not find events that matched and when we looked for the event on the host with Event Viewer, it doesn't look like there was much of a delay from when it occurred locally to when Splunk indexed it and sent an email.

I could understand if a search peer was down, there were replication issues, index got full and was rolling to frozen, or the event just never came into the indexers, but none of those appear to be the case and I'm not sure what other explanations could be.

Any suggestions on what to look for would be welcome.

[stranjer]

Thanks for the responses lakshman239 and tiagofbmm. Here is some additional information.

I see notable events using the macro, and just searching the index. I see them from before and after the missing notable event, both in general and for the specific alert.

The KV store is running and shows no error in status. The notable index and Windows index have longer retention policies setup, are not full, and have events prior to the ones that are missing. Indexer cluster shows no issues with search or replication issues.

I don't seen the original events that would have triggered the alert in Splunk, which is part of the issue I'm trying to understand. The alert should trigger off of Windows events for group additions. I can see the following info:
- In Splunk, I see Windows events from the specific host, where other users were added to the specific group, before and after.
- In Splunk, I see Windows events from the specific host, where the specific user was added to other groups, before and after.
- The notification email sent does not have details on the event configured, it just sent in the alert name. However, the person originally looking described a specific user being added to a specific group on a specific host at a specific time based on what they were seeing in Splunk. They did not have access to the Windows host.
- On the local host itself, we are able to confirm that the specific event happened with the details the other person originally saw in Splunk, at the time they mentioned.
- The day after making those notes, neither the original user nor anyone else could find the notable event or the Window event that generated it.

The information appears to have been in Splunk, but is now missing. Original person described details they couldn't have known without it being available in Splunk, and those details were verified so its not like they were just mistaken.

[tiagofbmm]

Good work narrowing this down. If the info was there and is not visible anymore:

• retention policy (you already mentioned it is not the cause)
• someone changed your searcheable indexes for your role?
  • someone changed the searcheable terms for the role, avoiding you to find those?
  • has someone deleted those events? (check _audit for delete commands both on the origin index and notable)
  • could that event have had a weird timestamp and timezone and be in a different time from what you are searching now?

[stranjer]

Thanks for the response.

I double checked to confirm, but the retention policy/searchable indexes/searchable terms changes all don't fit as I can see the search terms returning similar events from both indexes prior to the missing one.

I have tried a few searches for deleted events (index=_audit command="delete", and some others I found in:https://answers.splunk.com/answers/526983/how-can-i-find-what-users-deleted-specific-events.html) and I'm not seeing any hits that look like anything was deleted from either index. If there was a specific query you'd suggest or have had success with, I'm willing to try it. I'm not that familiar with what an event deleted by the system or a user would look like or what fields it'd have.

For timestamps, I checked the specific host+user+group combo for +/- 2 days, and the notable index for the saved search name for +/- 2 days for anything that matched username, with no luck.

[lakshman239]

Do you see any notables in the notable ( its a macro)?

if you run index=wineventlog host=yourhost are you seeing the original events that triggered alerts?

Is your kvstore up and running fine?

[stranjer]

Thanks for your response -
When I run notable I am seeing notable events. If I look for the particular search, I am seeing events from that search, on days before and days after the missing notable event.

If search for the original Windows events that caused the alert, I do not see them in the Windows Index related to the host.

KV store status comes back good, no issues seen there.

To make things clearer, the alert is for when users are added to particular groups. A user was added to a group that matches the criteria on the host, and an alert triggered. I see:
* Splunk showing group add events from the specific host, before and after the event in question
* Splunk showing the specific user being added to other groups, before and after
* Splunk showing other users being added to the specific group, on the specific host, both before and after
* Local Windows event on the host showing the specific user being added to the specific group
* Notable events for other instances where the alert was triggered for other cases of the activity, before and after.
* Email notification sent and reviewed. Person working noted the specific user being added to the specific group on the specific host, but none of this was in the email notification (the email info is pretty bland, just the alert name, just to generate ticket).

What I am not seeing as of now
* Splunk showing the Windows Event showing the specific user being added to the specific group
* Splunk showing the alert that triggered in the Notable Events

The person who worked the alert had covered details that were not in the notification email about the event that wouldn't have been possible to know unless the event was in Splunk, but later neither them nor anyone else could find the event again in Splunk. So it looks like the event was there, then vanished.

[tiagofbmm]

Can you check the retention policy in your notable index ?