Sender_ID is present in logging: as example:  2024-02-16 09:55:41:829 EST| INFO |InterfaceName=USCUSTOMERPO POCanonical_JSONHttpDataProcess=END JSON data successfully processed to Order Processor application for TxID=20240216095535623-0EEu Sender_ID=hC Bioscience Inc Receiver_ID=ThermoFisher Scientific TxnType=USCustomer_PO Format=cXML Direction=Inbound PO_Num=2550 Status=Success   please help to form the query : i tried this but still the issue persist it is taking only 1st word from log line   

Before I invest too much time working on a regex, please can you share your events in a code block </>. Also, do your events really have "Part" in them? (Regex matches in patterns and unless the patterns are accurate, the match will not be found.)

Using join is not a Splunk way of doing things, generally you would use stats. I'm not entirely clear on what fields exist in what indexes in your example. Does InstanceId exist in index=main data - that is what you are joining on From your description it sounds like all you want are those InstanceIds that come from the subsearch, so maybe I'm missing something If you are looking to find only those ResourceId where ResourceId=InstanceId from your current subsearch, but are also looking for other information, then  index=main ResourceId=* OR (index=other type=instance earliest=-2h) | eval InstanceId=coalesce(ResourceId, InstanceId) | stats values(*) as * values(index) as indexes count by InstanceId | where mvcount(indexes)=2  

Hi, i was facing the same issue. I have changed under transforms.conf the following: [kv_cp_log_format] REGEX = ([a-zA-Z0-9_-]+)[:=]+([^|]+) [kv_cp_syslog_log_format] REGEX = ([a-zA-Z0-9_-]+)[:=]+"((?:[^"\\]|\\.)+)"  

To simplify things, I will just follow your initial clue and assume that ID and Name are also part of event.ResourceAttributes.   index=test field1=* field2=* | spath input=field3 | foreach "event.ResourceAttributes.Name", "event.ResourceAttributes.Resource Name", "event.ResourceAttributes.ID" [ | eval type=mvappend(type, if(isnotnull('<<FIELD>>'), '<<FIELD>>', null())) ] | stats values(type) as "Additional Details" by event.AccountId event.CloudPlatform event.CloudService   If they are in some other nodes, just rewrite the foreach list. Here is a fuller emulation that I made up based on your singular mock data point.   | makeresults | eval field3 = mvappend("{\"event\": { \"AccountId\": \"xxxxxxxxxx2\", \"CloudPlatform\": \"CloudProvider\", \"CloudService\": \"Service\", \"ResourceAttributes\": {\"key1\": \"value1\", \"ID\": \"value2\", \"key3\": \"value3\", \"key4\": [{\"key\": \"value\", \"key\": \"value\"}], \"Resource Name\": \"name-resource-121sg6fe\", \"etc\": \"etc\"} } }" ``` has ID, Resource Name, no Name ```, "{\"event\": { \"AccountId\": \"xxxxxxxxxx1\", \"CloudPlatform\": \"CloudProvider\", \"CloudService\": \"Service\", \"ResourceAttributes\": {\"key1\": \"value1\", \"key2\": \"value2\", \"key3\": \"value3\", \"key4\": [{\"key\": \"value\", \"key\": \"value\"}], \"Resource Name\": \"name-resource-121sg6fe\", \"etc\": \"etc\"} } }" ``` has Resource Name, no others ```, "{\"event\": { \"AccountId\": \"xxxxxxxxxx2\", \"CloudPlatform\": \"CloudProvider\", \"CloudService\": \"Service\", \"ResourceAttributes\": {\"Name\": \"value1\", \"key2\": \"value2\", \"ID\": \"value3\", \"key4\": [{\"key\": \"value\", \"key\": \"value\"}], \"etc\": \"etc\"} } }" ``` has ID, Name, no Resource Name ```, "{\"event\": { \"AccountId\": \"xxxxxxxxxx1\", \"CloudPlatform\": \"CloudProvider\", \"CloudService\": \"Service\", \"ResourceAttributes\": {\"key1\": \"value1\", \"key2\": \"value2\", \"key3\": \"value3\", \"key4\": [{\"key\": \"value\", \"key\": \"value\"}], \"etc\": \"etc\"} } }" ``` has none of the three ```) | mvexpand field3 ``` the above sort of emulates index=test field1=* field2=* ``` | eval type = json_object() | spath input=field3 | foreach "event.ResourceAttributes.Name", "event.ResourceAttributes.Resource Name", "event.ResourceAttributes.ID" [ | eval type=mvappend(type, if(isnotnull('<<FIELD>>'), '<<FIELD>>', null())) ] | stats values(type) as "Additional Details" by event.AccountId event.CloudPlatform event.CloudService   What this does is to add variations to which of "Name", "Resource Name", and "ID" do or do not appear in each event.  You can play with it and compare with real data.  The output is event.AccountId event.CloudPlatform event.CloudService Additional Details xxxxxxxxxx1 CloudProvider Service name-resource-121sg6fe {} xxxxxxxxxx2 CloudProvider Service name-resource-121sg6fe value1 value2 value3 {}   One more suggestion: @bowesmana's idea is just to use foreach.  The above format does not group the present or missing attributes in a very distinguishable manner.  An alternative to using mvappend inside the foreach subsearch is to also carry the input keys in addition to values in "Additional Details".  Using a JSON structure is one such method.   index=test field1=* field2=* | eval type = json_object() | spath input=field3 | foreach "event.ResourceAttributes.Name", "event.ResourceAttributes.Resource Name", "event.ResourceAttributes.ID" [ | eval type=json_set(type, replace("<<FIELD>>", "event.ResourceAttributes.", ""), '<<FIELD>>') ] | stats values(type) as "Additional Details" by event.AccountId event.CloudPlatform event.CloudService   This is a full emulation:   | makeresults | eval field3 = mvappend("{\"event\": { \"AccountId\": \"xxxxxxxxxx2\", \"CloudPlatform\": \"CloudProvider\", \"CloudService\": \"Service\", \"ResourceAttributes\": {\"key1\": \"value1\", \"ID\": \"value2\", \"key3\": \"value3\", \"key4\": [{\"key\": \"value\", \"key\": \"value\"}], \"Resource Name\": \"name-resource-121sg6fe\", \"etc\": \"etc\"} } }" ``` has ID, Resource Name, no Name ```, "{\"event\": { \"AccountId\": \"xxxxxxxxxx1\", \"CloudPlatform\": \"CloudProvider\", \"CloudService\": \"Service\", \"ResourceAttributes\": {\"key1\": \"value1\", \"key2\": \"value2\", \"key3\": \"value3\", \"key4\": [{\"key\": \"value\", \"key\": \"value\"}], \"Resource Name\": \"name-resource-121sg6fe\", \"etc\": \"etc\"} } }" ``` has Resource Name, no others ```, "{\"event\": { \"AccountId\": \"xxxxxxxxxx2\", \"CloudPlatform\": \"CloudProvider\", \"CloudService\": \"Service\", \"ResourceAttributes\": {\"Name\": \"value1\", \"key2\": \"value2\", \"ID\": \"value3\", \"key4\": [{\"key\": \"value\", \"key\": \"value\"}], \"etc\": \"etc\"} } }" ``` has ID, Name, no Resource Name ```, "{\"event\": { \"AccountId\": \"xxxxxxxxxx1\", \"CloudPlatform\": \"CloudProvider\", \"CloudService\": \"Service\", \"ResourceAttributes\": {\"key1\": \"value1\", \"key2\": \"value2\", \"key3\": \"value3\", \"key4\": [{\"key\": \"value\", \"key\": \"value\"}], \"etc\": \"etc\"} } }" ``` has none of the three ```) | mvexpand field3 ``` the above sort of emulates index=test field1=* field2=* ``` | eval type = json_object() | spath input=field3 | foreach "event.ResourceAttributes.Name", "event.ResourceAttributes.Resource Name", "event.ResourceAttributes.ID" [ | eval type=json_set(type, replace("<<FIELD>>", "event.ResourceAttributes.", ""), '<<FIELD>>') ] | stats values(type) as "Additional Details" by event.AccountId event.CloudPlatform event.CloudService   And output from this emulation. event.AccountId event.CloudPlatform event.CloudService Additional Details xxxxxxxxxx1 CloudProvider Service {"Name":null,"Resource Name":"name-resource-121sg6fe","ID":null} {"Name":null,"Resource Name":null,"ID":null} xxxxxxxxxx2 CloudProvider Service {"Name":"value1","Resource Name":null,"ID":"value3"} {"Name":null,"Resource Name":"name-resource-121sg6fe","ID":"value2"}

Hi @BTB, let me understand: you want to check if you received in the last three days less than 30% of the logs that you received in the previous 3 days for each sourcetype, is it correct? If this is your requirement, you could try something like this: | tstats count latest(_time) AS _time WHERE index=* earliest=-6d BY sourcetype | eval period=if(_time>now()-86400*3,"Last","Previous" | stats sum(eval(if(period="Last",count,0))) AS Last sum(eval(if(period="Previous",count,0))) AS Previous dc(period) AS period_count values(period) AS period BY sourcetype | eval diff_perc=(Last-Previous)/Previous*100 | where diff_perc<30 if you want a different algorithm you can implement using my approach. Ciao. Giuseppe