topic Re: Why are some expected events missing when applying a filter in our search? in Splunk Search

Why are some expected events missing when applying a filter in our search?

mdelwaide — Tue, 20 Dec 2016 19:59:04 GMT

We recently onboarded some applications' logs, and at our client request, we had to put a custom field to have the application name in the fields. Since those weren’t in the log file, we hard-coded the application name in the log with a “_meta” field in the inputs.conf stanza. Since each application has its own stanza, it was easy.

One of the analysts discovered this week that some events were discarded when a filter was applied for a certain application. We were able to reproduce the problem at a smaller scale.

Here the picture shows that there are 11 events for IAM, if we apply the filter for that application...

This is only a 5 minutes window, at a "All Time" time range it's more than 10K events that goes missing

Here is our Inputs.conf for that application

[monitor://D:\APPLogs\IAM\IAM.log]
index = grc
sourcetype = GRC:APP
alwaysOpenFile = 1
disabled = false
_meta = application::IAM

Re: Why are some expected events missing when applying a filter in our search?

somesoni2 — Tue, 20 Dec 2016 20:21:27 GMT

Do you get a correct count when you use either of these searches?

index=grc application=*IAM

index=grc | regex application="IAM"

index=grc application!=IAM:IWSS

Re: Why are some expected events missing when applying a filter in our search?

mdelwaide — Tue, 20 Dec 2016 20:34:43 GMT

index=grc application=*IAM

yes (Only IAM)

index=grc | regex application="IAM"

We get all events (IAM and IAM:IWSS)

index=grc application!=IAM:IWSS

Re: Why are some expected events missing when applying a filter in our search?

somesoni2 — Tue, 20 Dec 2016 20:38:29 GMT

Ok.. how about this?

index=grc application="IAM"

Re: Why are some expected events missing when applying a filter in our search?

mdelwaide — Tue, 20 Dec 2016 20:41:36 GMT

No, showing only 10 Events

Re: Why are some expected events missing when applying a filter in our search?

somesoni2 — Tue, 20 Dec 2016 20:44:03 GMT

Ok lets check if there are additional character coming in the application name.

index=grc application=*IAM | eval applength=len(application) | table applength

Re: Why are some expected events missing when applying a filter in our search?

mdelwaide — Tue, 20 Dec 2016 20:49:05 GMT

I've got only 3's

Re: Why are some expected events missing when applying a filter in our search?

somesoni2 — Tue, 20 Dec 2016 21:13:05 GMT

Try this fix. On your search head and Indexers, add following to fields.conf (preferred to be kept under some app). Should work (Reference: https://answers.splunk.com/answers/389567/why-is-a-search-for-fields-added-with-meta-in-inpu.html)

[application]
 INDEXED = true

Re: Why are some expected events missing when applying a filter in our search?

mdelwaide — Mon, 06 Feb 2017 22:16:54 GMT

Hi, sorry for the late reply. It fixed our problem. I'll add an answer to the question with your fix. Thanks for your help

Re: Why are some expected events missing when applying a filter in our search?

mdelwaide — Mon, 06 Feb 2017 22:19:24 GMT

Thanks somesoni2 for your help, your last fix helped us out with our problem.

Steps:

On our SH and Indexers we deployed a TA with a fields.conf file with this stanza :

[application]
INDEXED = true

(Reference: https://answers.splunk.com/answers/389567/why-is-a-search-for-fields-added-with-meta-in-inpu.html)

Restarted and the problem was gone.

Thanks again