Why are some expected events missing when applying a filter in our search?

[mdelwaide]

We recently onboarded some applications' logs, and at our client request, we had to put a custom field to have the application name in the fields. Since those weren’t in the log file, we hard-coded the application name in the log with a “_meta” field in the inputs.conf stanza. Since each application has its own stanza, it was easy.

One of the analysts discovered this week that some events were discarded when a filter was applied for a certain application. We were able to reproduce the problem at a smaller scale.

Here the picture shows that there are 11 events for IAM, if we apply the filter for that application...

This is only a 5 minutes window, at a "All Time" time range it's more than 10K events that goes missing

Here is our Inputs.conf for that application

[monitor://D:\APPLogs\IAM\IAM.log]
index = grc
sourcetype = GRC:APP
alwaysOpenFile = 1
disabled = false
_meta = application::IAM

[mdelwaide · Accepted solution]

Thanks somesoni2 for your help, your last fix helped us out with our problem.

Steps:

On our SH and Indexers we deployed a TA with a fields.conf file with this stanza :

[application]
INDEXED = true

(Reference: https://answers.splunk.com/answers/389567/why-is-a-search-for-fields-added-with-meta-in-inpu.html)

Restarted and the problem was gone.

Thanks again

M.

[somesoni2]

Do you get a correct count when you use either of these searches?

index=grc application=*IAM

index=grc | regex application="IAM"

index=grc application!=IAM:IWSS

[mdelwaide]

index=grc application=*IAM

• yes (Only IAM)

index=grc | regex application="IAM"

• We get all events (IAM and IAM:IWSS)

index=grc application!=IAM:IWSS

• yes

[somesoni2]

Ok.. how about this?

index=grc application="IAM"

[mdelwaide]

No, showing only 10 Events

[somesoni2]

Ok lets check if there are additional character coming in the application name.

index=grc application=*IAM | eval applength=len(application) | table applength

[mdelwaide]

I've got only 3's

[somesoni2]

Try this fix. On your search head and Indexers, add following to fields.conf (preferred to be kept under some app). Should work (Reference: https://answers.splunk.com/answers/389567/why-is-a-search-for-fields-added-with-meta-in-inpu.html)

[application]
 INDEXED = true

[mdelwaide]

Hi, sorry for the late reply. It fixed our problem. I'll add an answer to the question with your fix. Thanks for your help