Odd search results where csv values do not seem to be getting indexed correctly

[Cuyose]

I had a user bring up an issue he was noticing. There were search results that technically should be returning the exact same results. One very inefficient and the other, the proper way to search.

Basically I am able to see returns for a query
index=aws "application=agent-softphone-app" NOT application=agent-softphone-app

These events are coming from a heavy forwarder to a clustered index tier. It is possible that the indexers are unable to parse all events, so some are being indexed as freeform text? I am still digging into some ideas regarding the indexers, and only 2 of the indexers are showing event counts inconstantly when running the above query as 2 different searches.

This environment is not being worked very hard at all.

[damiensurat]

what happenes if you search for just the terms and not the full string:
index=aws "application" "agent-softphone-app"

[Cuyose]

index=aws application="agent-softphone-app" = 1661 events
index=aws "application" "agent-softphone-app" = 3221 events
index=aws "application=agent-softphone-app" =3221 events

index=aws application=agent-softphone* = 13 events
index=aws application=agent-softphone-* = 0 events

seems something messed up with the -

All same timeframe

[damiensurat]

actually, if you put it in quotes ....
index=aws application="agent-softphone*"
or
index=aws application="agent-softphone-*"

I would expect 1661 results

[damiensurat]

so is the aws app addon installed on the heavy forwarder? also, since this is a custom log, have you configure the sourcetype properly to parse the logs as expected?

it would be helpful if you posted the actual log as well as the sourcetype and if you are using props.conf and/or transforms.conf

as for the sourcetype, that should be defined on the heavyforwarder as well...

[Sukisen1981]

interesting...how are your cvss getting indexed? Are you monitoring a file or is it a manual process? I had some issues like this when i was monitoring a notepad, the notepad was getting changed (data rows added/data rows modified) and I could see something similar...

[Cuyose]

This is coming from an AWS kinesis stream into a heavy forwarder then out to the clustered indexers. The sourcetype is the same on all indexers, and the 2 indexers seeming unable to find the events using the kv pair, miss about 90% of the time, but can return a few events by only searching vie the kv pair.

Super wierd