Why does this generic search not return results that are returned in a more specific search?

[scaparelli]

I am developing a query that shows stats for events with the same orderId. There is a flaw though. When I run the query, I get results with only one event for an orderId, but when I take the orderId associated to only one event and put it in the original query, the result comes up with 2 events. Here are my queries and results:

(index=k8s_main LogType="KafkaMessageProcessedSuccess" message="OrderLineDestinationChangeRequested" Environment="PROD") OR (index=k8s_main container_name=fraud-single-proxy-listener message="Sending a message to kafka topic=order-events-avro*OrderLineDestinationChangeRequested*")
| rename contextMap.orderId AS nefiOrderId OrderNumber AS omsOrderId
| rename contextMap.requestId AS nefiRequestId NordRequestId AS omsRequestId
| rename OrderLineId as omsOrderLineId
| rex field=message "\"orderLineId\": \"(?<nefiOrderLineId>.*?)\", " 
| eval orderLineId = coalesce(nefiOrderLineId, omsOrderLineId)
| eval requestId = mvappend(nefiRequestId, omsRequestId) 
| eval orderId = coalesce(nefiOrderId, omsOrderId)
| stats dc(_time) AS eventCount values(_time) AS eventTime values(orderLineId) AS orderLineId values(requestId) AS requestId BY orderId
| where eventCount = 1

Second query with the orderId in the initial search: 

(index=k8s_main LogType="KafkaMessageProcessedSuccess" message="OrderLineDestinationChangeRequested" Environment="PROD" 381263531) OR (index=k8s_main container_name=fraud-single-proxy-listener message="Sending a message to kafka topic=order-events-avro*OrderLineDestinationChangeRequested*" 381263531)
| rename contextMap.orderId AS nefiOrderId OrderNumber AS omsOrderId
| rename contextMap.requestId AS nefiRequestId NordRequestId AS omsRequestId
| rename OrderLineId as omsOrderLineId
| rex field=message "\"orderLineId\": \"(?<nefiOrderLineId>.*?)\", " 
| eval orderLineId = coalesce(nefiOrderLineId, omsOrderLineId)
| eval requestId = mvappend(nefiRequestId, omsRequestId) 
| eval orderId = coalesce(nefiOrderId, omsOrderId)
| stats dc(_time) AS eventCount values(_time) AS eventTime values(orderLineId) AS orderLineId values(requestId) AS requestId BY orderId

 

[ITWhisperer]

Can you share the two events in code blocks </> as I suspect it is something to do with the values extracted from them?

[scaparelli]

{"instant":{"epochSecond":1660665846,"nanoOfSecond":651267000},"thread":"inbound-listener-2","level":"INFO","loggerName":"com.nordstrom.fraud.fsp.listener.kafka.producer.KafkaProducer","message":"Sending a message to kafka topic=order-events-avro, messageKey=381263531, headers={AppId=APP02253, EventTime=1660665846651, Geolocation=47.613040, -122.334092, HEADER_GROUPING_ITEM_COUNT_KEY=1, HEADER_GROUPING_ITEM_INDEX=1, Id=OsOrsa3tjPJpBFwVQp6k56, Nord-Country-Code=US, Nord-Request-Id=16c555d0-a4d6-4059-9aec-9c015d9ce935, OmsModernStack=true, SchemaId=, SystemTime=1660665846651, Type=OrderLineChangeFraudApproved}, payload={\"orderNumber\": \"381263531\", \"orderLineId\": \"6c262fdae7bded09652eb32cf56546cb42d7e6cbc4f35625985a05b8ed2cda88\", \"serviceTicketId\": \"16c555d0-a4d6-4059-9aec-9c015d9ce935\", \"approvalDetails\": \"FRAUD_APPROVED\", \"eventTime\": 2022-08-16T16:04:06.651Z, \"source\": {\"channelCountry\": \"US\", \"channel\": \"OMNI\", \"platform\": \"CSR_PHONE\", \"feature\": \"OrderLineDestinationChangeRequested\", \"serviceName\": null, \"store\": null, \"register\": null}}","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","contextMap":{"aggregationGroupId":"16c555d0-a4d6-4059-9aec-9c015d9ce935","countryCode":"US","orderId":"381263531","requestId":"16c555d0-a4d6-4059-9aec-9c015d9ce935"},"threadId":334,"threadPriority":5}

NordClientId="APP03176"|LogCategory="Information"|LogType="KafkaMessageProcessedSuccess"|message="OrderLineDestinationChangeRequested"|ServiceTicketId="16c555d0-a4d6-4059-9aec-9c015d9ce935"|OrderNumber="381263531"|OrderLineId="6c262fdae7bded09652eb32cf56546cb42d7e6cbc4f35625985a05b8ed2cda88"|EventTime="08/16/2022 16:03:14"|KafkaGroupId="Care-CustomerOrderModificationRequestConsumerDynamo-prod"|NordRequestId="16c555d0-a4d6-4059-9aec-9c015d9ce935"|ServerTimestamp="2022-08-16T16:03:15.8035047Z"|NordCountryCode="US"|Environment="PROD"|AppName="customerordermodificationrequestconsumerdynamo-prod"|Pod="release-branch-customerordermodificationrequestconsumerdynamo-prod"|KafkaEventId="16c555d0-a4d6-4059-9aec-9c015d9ce935_1"|KafkaEventType="OrderLineDestinationChangeRequested"|KafkaEventSystemTime="1660665795625"

[yuanliu]

---

@scaparelli wrote:

{"instant":{"epochSecond":1660665846,"nanoOfSecond":651267000},"thread":"inbound-listener-2","level":"INFO","loggerName":"com.nordstrom.fraud.fsp.listener.kafka.producer.KafkaProducer","message":"Sending a message to kafka topic=order-events-avro, messageKey=381263531, headers={AppId=APP02253, EventTime=1660665846651, Geolocation=47.613040, -122.334092, HEADER_GROUPING_ITEM_COUNT_KEY=1, HEADER_GROUPING_ITEM_INDEX=1, Id=OsOrsa3tjPJpBFwVQp6k56, Nord-Country-Code=US, Nord-Request-Id=16c555d0-a4d6-4059-9aec-9c015d9ce935, OmsModernStack=true, SchemaId=, SystemTime=1660665846651, Type=OrderLineChangeFraudApproved}, payload={\"orderNumber\": \"381263531\", \"orderLineId\": \"6c262fdae7bded09652eb32cf56546cb42d7e6cbc4f35625985a05b8ed2cda88\", \"serviceTicketId\": \"16c555d0-a4d6-4059-9aec-9c015d9ce935\", \"approvalDetails\": \"FRAUD_APPROVED\", \"eventTime\": 2022-08-16T16:04:06.651Z, \"source\": {\"channelCountry\": \"US\", \"channel\": \"OMNI\", \"platform\": \"CSR_PHONE\", \"feature\": \"OrderLineDestinationChangeRequested\", \"serviceName\": null, \"store\": null, \"register\": null}}","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger","contextMap":{"aggregationGroupId":"16c555d0-a4d6-4059-9aec-9c015d9ce935","countryCode":"US","orderId":"381263531","requestId":"16c555d0-a4d6-4059-9aec-9c015d9ce935"},"threadId":334,"threadPriority":5}

---

This event should be impossible to be picked in either search.  It doesn't match LogType="KafkaMessageProcessedSuccess" because there is no LogType field, it also doesn't match container_name=fraud-single-proxy-listener because there is no container_name field.  These two terms exist in both searches.

[richgalloway]

The last line of the first query limits the results to those with a single event for each orderId.  The second query does not have that where command so orderIds with two or more events are shown.

[scaparelli]

So if you look closely at the queries, I am taking the orderId from the first query that has the `where` statement, and using it the exact same query without the `where` parameters but in the base search.

The second query comes up with 2 events for the orderId whereas the first comes up with 1 event for the orderId.

My question is why?

The logs for that orderId should not exist in the first query result