topic Re: How can we find the missing events in splunk in Monitoring Splunk

How can we find the missing events in splunk

smith_ — Mon, 08 Jan 2024 08:25:19 GMT

Hi,

Splunk hasn't captured the 4743 events, indicating computer account deletions that occurred yesterday at 2 pm. Where should we investigate to determine the root cause?

Thanks

Re: How can we find the missing events in splunk

gcusello — Mon, 08 Jan 2024 09:04:42 GMT

Hi @smith_,

you should run a searh like the following:

index=your_index EventCode=4743

if you haven't results, you have to perform two checks:

at first on the Splunk_TA Windows that you're using to ingest logs, to see if this EventCode is ingested or not. maybe there's a white list or a blacklist the filters this EventCode.

if there isn't any filter, see in your Domain Controller if this EventCode is loged on Windows: not al events are logged by default, about this, I cannot help you: you need a Windows specialist.

Ciao.

Giuseppe

Re: How can we find the missing events in splunk

PickleRick — Mon, 08 Jan 2024 09:10:40 GMT

There are several possible causes for that. Starting from wrong permissions on the source side (we don't even know if these are the only events that are not ingested or if you're ingesting any events from the Security journal at all), through input black/whitelisting, to active filtering on HFs/indexers.

Don't get me wrong, but from this thread and other similar ones it looks as if your employer bought Splunk license but didn't invest in either trainings for the staff or maintenance services from your friendly local Splunk partner. And you seem to need it.

Re: How can we find the missing events in splunk

smith_ — Mon, 08 Jan 2024 09:19:07 GMT

@gcusello ,

we're ingesting logs with these event code, but occasionally, we're not receiving all the logs from the DCs into Splunk.

Re: How can we find the missing events in splunk

gcusello — Mon, 08 Jan 2024 09:22:49 GMT

Hi @smith_ ,

are you saying the usualy events with this EventCode are ingested, but sometimes you lose events?

In this case, analyze if there was some downtime of the Forwarder or of the connection, starting from the period where you're sure that you loosed some events.

If you're sure that there wasn't any downtime, open a case to Splunk Support.

Ciao.

Giuseppe

Re: How can we find the missing events in splunk

PickleRick — Mon, 08 Jan 2024 10:58:32 GMT

OK. So you have a different problem.

Are you missing any other events?

Are you having connection problems?

Are you getting any errors in _internal?

Are you hitting thruput limits?

Do you ingest all events from the beginning or just current ones?

Re: How can we find the missing events in splunk

smith_ — Mon, 08 Jan 2024 11:23:35 GMT

@PickleRick ,

Are you missing any other events?
Nope only 4743

Are you having connection problems?
I dnt think so / How to check

Are you getting any errors in _internal?
How to check ?

Are you hitting thruput limits?
yes

Do you ingest all events from the beginning or just current ones?
Yes we are ingesting all events from beginning.

Re: How can we find the missing events in splunk

inventsekar — Mon, 08 Jan 2024 11:48:37 GMT

Hi @smith_

As seen on previous reply, to troubleshoot this issue, lot more details are required from your side.

Any changes recently done on those DC systems inputs.conf / apps / addons etc

Lets say you were expecting the 4743 at 5pm yesterday. Pls check if you have events around that time from that particular windows box (search for 4pm to 6pm events from that windows box)

As said in other posts, the good questions will receive good answers. the more details you provide, the more better answers/suggestions we can help you with. Thanks.

Re: How can we find the missing events in splunk

smith_ — Thu, 11 Jan 2024 16:30:39 GMT

Hello,

Today, we made modifications to Domain Admin groups, for which we had previously enabled Notables. The issue is that I haven't received any alerts related to it, and the events have not been collected in Splunk yet.

Here is the services snapshot for the Universal Forwarder from that domain controller:

Do we need to make any changes pls let me know

Thanks

Re: How can we find the missing events in splunk

PickleRick — Thu, 11 Jan 2024 21:21:55 GMT

OK. So modifications to Domain Admins group is reflected with different events that 4743. So you seem to have a different problem than just losing one particular eventid.

Re: How can we find the missing events in splunk

mshakeb — Tue, 15 Apr 2025 05:00:57 GMT

Hi @AL3Z

did you fix the issue, i am also facing the same issue.

Re: How can we find the missing events in splunk

gcusello — Tue, 15 Apr 2025 06:40:19 GMT

Hi @mshakeb ,

please, don't attach your request to another one, even if on the same topic, open a new question.

In this way, you'll have more choices to describe your requirements and to receive an answer.

ciao.

Giuseppe