Splunk Search

Why are some expected events missing when applying a filter in our search?

mdelwaide
Path Finder

We recently onboarded some applications' logs, and at our client request, we had to put a custom field to have the application name in the fields. Since those weren’t in the log file, we hard-coded the application name in the log with a “_meta” field in the inputs.conf stanza. Since each application has its own stanza, it was easy.

One of the analysts discovered this week that some events were discarded when a filter was applied for a certain application. We were able to reproduce the problem at a smaller scale.

alt text

Here the picture shows that there are 11 events for IAM, if we apply the filter for that application...

alt text

This is only a 5 minutes window, at a "All Time" time range it's more than 10K events that goes missing

Here is our Inputs.conf for that application

[monitor://D:\APPLogs\IAM\IAM.log]
index = grc
sourcetype = GRC:APP
alwaysOpenFile = 1
disabled = false
_meta = application::IAM
0 Karma
1 Solution

mdelwaide
Path Finder

Thanks somesoni2 for your help, your last fix helped us out with our problem.

Steps:

On our SH and Indexers we deployed a TA with a fields.conf file with this stanza :

[application]
INDEXED = true

(Reference: https://answers.splunk.com/answers/389567/why-is-a-search-for-fields-added-with-meta-in-inpu.html)

Restarted and the problem was gone.

Thanks again

M.

View solution in original post

0 Karma

mdelwaide
Path Finder

Thanks somesoni2 for your help, your last fix helped us out with our problem.

Steps:

On our SH and Indexers we deployed a TA with a fields.conf file with this stanza :

[application]
INDEXED = true

(Reference: https://answers.splunk.com/answers/389567/why-is-a-search-for-fields-added-with-meta-in-inpu.html)

Restarted and the problem was gone.

Thanks again

M.

0 Karma

somesoni2
Revered Legend

Do you get a correct count when you use either of these searches?

index=grc application=*IAM

index=grc | regex application="IAM"

index=grc application!=IAM:IWSS
0 Karma

mdelwaide
Path Finder

index=grc application=*IAM

  • yes (Only IAM)

index=grc | regex application="IAM"

  • We get all events (IAM and IAM:IWSS)

index=grc application!=IAM:IWSS

  • yes
0 Karma

somesoni2
Revered Legend

Ok.. how about this?

index=grc application="IAM"
0 Karma

mdelwaide
Path Finder

No, showing only 10 Events

0 Karma

somesoni2
Revered Legend

Ok lets check if there are additional character coming in the application name.

index=grc application=*IAM | eval applength=len(application) | table applength
0 Karma

mdelwaide
Path Finder

I've got only 3's

0 Karma

somesoni2
Revered Legend

Try this fix. On your search head and Indexers, add following to fields.conf (preferred to be kept under some app). Should work (Reference: https://answers.splunk.com/answers/389567/why-is-a-search-for-fields-added-with-meta-in-inpu.html)

[application]
 INDEXED = true

mdelwaide
Path Finder

Hi, sorry for the late reply. It fixed our problem. I'll add an answer to the question with your fix. Thanks for your help

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...