We recently onboarded some applications' logs, and at our client request, we had to put a custom field to have the application name in the fields. Since those weren’t in the log file, we hard-coded the application name in the log with a “_meta” field in the inputs.conf stanza. Since each application has its own stanza, it was easy.
One of the analysts discovered this week that some events were discarded when a filter was applied for a certain application. We were able to reproduce the problem at a smaller scale.
Here the picture shows that there are 11 events for IAM, if we apply the filter for that application...
This is only a 5 minutes window, at a "All Time" time range it's more than 10K events that goes missing
Here is our Inputs.conf for that application
[monitor://D:\APPLogs\IAM\IAM.log] index = grc sourcetype = GRC:APP alwaysOpenFile = 1 disabled = false _meta = application::IAM
Do you get a correct count when you use either of these searches?
index=grc application=*IAM index=grc | regex application="IAM" index=grc application!=IAM:IWSS
index=grc | regex application="IAM"
Ok lets check if there are additional character coming in the application name.
index=grc application=*IAM | eval applength=len(application) | table applength
Try this fix. On your search head and Indexers, add following to fields.conf (preferred to be kept under some app). Should work (Reference: https://answers.splunk.com/answers/389567/why-is-a-search-for-fields-added-with-meta-in-inpu.html)
[application] INDEXED = true
Hi, sorry for the late reply. It fixed our problem. I'll add an answer to the question with your fix. Thanks for your help
Thanks somesoni2 for your help, your last fix helped us out with our problem.
On our SH and Indexers we deployed a TA with a fields.conf file with this stanza :
INDEXED = true
Restarted and the problem was gone.