topic Re: Need search assist for a query that is not bringing back the expected result in Splunk Search

Need search assist for a query that is not bringing back the expected result

nls7010 — Thu, 16 Sep 2021 14:26:19 GMT

A client of mine is asking: I’m hoping you can help me with something. I am trying to analyze the volume to a particular Apigee endpoint so I have wrote the following query where I am grouping the stats by my calculated date value to see a day-wise view for a service that hit a particular backend.

index="apigee-prod-cne" sourcetype="apigee_metrics" (apiproxy="cc-cust-profile-01-v1") target_host = "sapisugw-prd.duke-energy.com" proxy_pathsuffix = "/email/bp/retrieve" environment="prod" | dedup gateway_flow_id | spath request_verb | search request_verb != "OPTIONS"| eval yourdate = strftime(_time,"%D")| eval yourhour = strftime(_time, "%H")| eval yourmin=strftime(_time,"%M") | stats count(x-apigee.edge.execution.stats.request_flow_start_timestamp) as hits by yourdate

What I’m not understanding is that when I add an additional parameter to group by (and changing none of my other conditions) that I suddenly see a spike in calls rather than a segmented number from the total?

Can you help me to understand what I may be missing to properly evaluate traffic through our proxies?

Re: Need search assist for a query that is not bringing back the expected result

ITWhisperer — Thu, 16 Sep 2021 14:47:43 GMT

Is developer_app a multi-value field because stats will be counted against each value in the mv fields?

Re: Need search assist for a query that is not bringing back the expected result

nls7010 — Thu, 16 Sep 2021 15:07:14 GMT

I will ask the client what he set up.

Re: Need search assist for a query that is not bringing back the expected result

nls7010 — Thu, 16 Sep 2021 15:39:48 GMT

This is what the customer responded:

To make sure I understand, you are asking if developer_app can be different values in a unique combination of hits per mydate+developer_app combinations, right (i.e. if I look in the event, can I have two developer apps in the event details)? They cannot be; for a single request, there can only be 1 developer app.

I’m trying to get to a result set where I can say:

I have 50 total calls to a service. 25 were from CC-R5-SITECORE-App and 25 were from CC-R5-IMP-SUPERUSER-App
If I add developer_app as a group by parameter, I was thinking I would see the result above where I get a unique number of hits per combination of date+developer+app

Re: Need search assist for a query that is not bringing back the expected result

ITWhisperer — Thu, 16 Sep 2021 16:08:16 GMT

Can you share some sample events from the search up to this point

Re: Need search assist for a query that is not bringing back the expected result

nls7010 — Thu, 16 Sep 2021 17:03:36 GMT

Re: Need search assist for a query that is not bringing back the expected result

ITWhisperer — Thu, 16 Sep 2021 17:22:31 GMT

Can you expand the fields for that event

and show the developer_app field contents

Also, can you share the _raw field?

Re: Need search assist for a query that is not bringing back the expected result

nls7010 — Thu, 16 Sep 2021 18:12:57 GMT

There does not appear to be a specific field, so I'm thinking he is making this field at search:

Re: Need search assist for a query that is not bringing back the expected result

ITWhisperer — Thu, 16 Sep 2021 18:21:26 GMT

Can you get the raw event and paste it into a code block </>

It is difficult to know what's going on without seeing the real data.

Re: Need search assist for a query that is not bringing back the expected result

nls7010 — Thu, 16 Sep 2021 18:31:01 GMT

sure, here is the raw data of one of the events:

{"proxy_basepath":"/v1/customer/profile","responsecache_l1_count":null,"x-apigee.edge.execution.stats.request_flow_endtimestamp":1631811648373,"responsecache_executed":null,"apiproxy":"cc-cust-profile-01-v1","x-apigee.edge.is_policy_error":0,"client_id":"RfmkbZ3EI161T1MHfGaIlHKvDsgeXJZ2","responsecache_name":null,"client_sent_start_timestamp":1631811649200,"target_sent_start_timestamp":1631811648374,"x-apigee.edge.is_target_error":0,"client_received_start_timestamp":1631811648364,"x-apigee.intelligence.client_ip_postal":null,"target_sent_end_timestamp":1631811648374,"client_ip":"139.46.106.177","is_error":false,"x-apigee.edge.stats.steps":"{\"qtAPIQuota.0\":3,\"o2VerifyAccessToken.0\":2,\"repRegExProtect.0\":0}","id":"fq51wz403hod2kqjww3lqd0y","request_size":235,"developer_app":"CC-R5-SITECORE-App","x-apigee.intelligence.client_ip_header":"104.209.233.38","target_response_code":201,"apigee.edge.execution.fault_code":null,"apigee_sdk_client_sessionid":null,"virtual_host":"secure","x-apigee.edge.mp_host":"lcltcust1edgp06.duke-energy.com","apigee_sdk_client_requestid":null,"x-apigee.edge.execution.fault_flow_name":null,"sla":false,"responsecache_key":null,"x-apigee.intelligence.service":"{}","target_received_end_timestamp":1631811649199,"client_sent_end_timestamp":1631811649200,"target_host":"sapisugw-prd.duke-energy.com","request_uri":"/sap/opu/odata/sap/ZDIG_SEARCH_BP_EMAIL_SRV/RequestSet","access_token":"okA4MFkAIEcVPaE4091YYvQEeRGX","proxy":"ProfilesProxyEndpoint","x-apigee.intelligence.client_ip_subdivision":null,"target_received_start_timestamp":1631811649198,"api_product":"CC-R5-BE-SC-Services-PROD","proxy_client_ip":"xxxxxxxxxxxx,"x-apigee.edge.dn.region":"dc-1","apigee_sdk_client_appname":null,"apigee_sdk_client_deviceid":null,"apigee.edge.execution.is_apigee_fault":0,"x-apigee.edge.target.latency.stats":"{\"targetList\":[{\"targetId\"😕"target://sapisugw-prd.duke-energy.com\",\"responseStatus\":201,\"latency\":824,\"isTLS\":true}]}","x-apigee.edge.stats.policy.execution":null,"target_url":null,"x-apigee.edge.execution.fault_flow_state":null,"useragent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36","proxy_pathsuffix":"/email/bp/retrieve","x-apigee.intelligence.client_ip_asn":null,"x-apigee.edge.execution.stats.request_flow_start_timestamp":1631811648371,"x_forwarded_for_ip":null,"response_size":1045,"cache_hit":null,"x-apigee.edge.execution.sense.action":null,"developer_email":"CC_APIGEE_REPORTING@duke-energy.com","x-apigee.edge.execution.fault_policy_name":null,"responsecache_source":null,"response_status_code":201,"request_verb":"POST","x-apigee.edge.true_client_ip":"104.209.233.38","apigee_sdk_client_orgname":null,"x-apigee.edge.execution.stats.response_flow_end_timestamp":1631811649199,"gateway_source":"message_processor","target":"Retrieve Primary BP from Email TargetEndpoint","x-apigee.intelligence.client_ip_country":null,"environment":"prod","client_received_end_timestamp":1631811648364,"target_ip":"xxxxxxxxx","organization":"cust","x-apigee.edge.execution.stats.response_flow_start_timestamp":1631811649199,"request_path":"/sap/opu/odata/sap/ZDIG_SEARCH_BP_EMAIL_SRV/RequestSet","developer":"cust@@@BJeQzJGwIGzHXAop","target_basepath":"/sap/opu/odata/sap/ZDIG_SEARCH_BP_EMAIL_SRV/RequestSet","flow_resource":null,"x-apigee.intelligence.client_ip_city":null,"gateway_flow_id":"lcltcust1edgp06-123559-840635-44","apiproxy_revision":"12"}

Re: Need search assist for a query that is not bringing back the expected result

nls7010 — Thu, 23 Sep 2021 11:28:11 GMT

This is an update from my client:

I’m trying to get to a result set where I can say:

I have 50 total calls to a service. 25 were from CC-R5-SITECORE-App and 25 were from CC-R5-IMP-SUPERUSER-App

If I add developer_app as a group by parameter, I was thinking I would see the result above where I get a unique number of hits per combination of date+developer+app