Dashboards & Visualizations

Rex table format

nithys
Path Finder


Screenshot 2023-11-04 at 4.22.55 PM.pngThere are three different events "input param" ,"sqs sent count", "total message published to SQS successfully"
with the first event "input param"- I am trying to fetch different entity say material/supplied material
with the 2nd event "sqs sent count"-getting the total sqs sent count for that particular material or supplied material
with the 3rd event "total message published to SQS successfully"-getting the count of total message published.
Now i want then to publish in a single row for all those count displayed in table for a single objectype to get in one dashboard panel
Then do a total counts for each columns and displays as a single row which will display in other panel of dashboard

Labels (1)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust

You can't have multiple columns with the same name so try this

index= source IN ("") "uniqObjectIds" OR "data retrieved for Ids" | eval PST=_time-28800 | eval PST_TIME=strftime(PST, "%Y-%d-%m %H:%M:%S") | spath output=uniqObjectIds path=uniqObjectIds{} |where isnotnull(uniqObjectIds) | spath output=uniqueRetrievedIds path=uniqueRetrievedIds{}| stats values(*) as * by _raw | table uniqObjectIds,uniqObjectIdsCount,uniqObjectIds{},PST_TIME | sort- PST_TIME 
| appendcols [search index= source IN ("") "data retrieved for Ids"| eval PST=_time-28800 | eval PST_TIME2=strftime(PST, "%Y-%d-%m %H:%M:%S") | spath output=uniqueRetrievedIds path=uniqueRetrievedIds{} |where isnotnull(uniqueRetrievedIds)| stats values(*) as * by _raw | table uniqueRetrievedIds{},uniqueRetrievedIds, PST_TIME2 | sort- PST_TIME2 ]
| appendcols [search index= source IN ("") "data not found for Ids"| eval PST=_time-28800 | eval PST_TIME3=strftime(PST, "%Y-%d-%m %H:%M:%S") | spath output=dataNotFoundIds path=dataNotFoundIds{}|where isnotnull(dataNotFoundIds) | stats values(*) as * by _raw | table dataNotFoundIds{},dataNotFoundIdsCount, PST_TIME3 | sort- PST_TIME3 ]

View solution in original post

ITWhisperer
SplunkTrust
SplunkTrust

You can use addtotals to sum the numbers. Is this what you are after?

If not, how do you correlate different events to generate separate rows with their respective counts?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @nithys,

could you share some samples of your logs and your search (please both in text format, not screenshot)?

Ciao.

Giuseppe

0 Karma

nithys
Path Finder

Hi @gcusello 
Below is the query.....Firstly I want them to get aligned all rows with each values in a single row.Currently there are 3 different rows for 1 particular result.....then apply sum of columns
index=""     source IN "" "input params" OR "sqs sent count" OR "Total messages published to SQS successfully"
 | rex "\"objectType\":\"(?<objectType>[^\"]+)"
  | rex "\"objectIdsCount\":\"(?<objectIdsCount>[^\"]+)"
  | rex "\"sqsSentCount\":\"(?<sqsSentCount>[^\"]+)"
    | rex "\"totalMessagesPublishedToSQS\":\"(?<totalMessagesPublishedToSQS>[^\"]+)"
  | table objectType,objectIdsCount,sqsSentCount,totalMessagesPublishedToSQS

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @nithys ,

without a sample of your logs I cannot check yur regexes and I don't understand what't the key to correlate values because it seems that there isn't any common key to use in a stats command.

Ciao.

Giuseppe

0 Karma

nithys
Path Finder

Hi @gcusello @ITWhisperer 
I have same source and index for below two events
first event:
{
 [-]
   awsRequestId
   hostname
   level30
   msgdata retrieved for Ids
   name
   pid8
   time
   uniqueRetrievedIds: [ [-

   275649
   ]

   v0

}
second event:
[-]
   awsRequestId
   hostname
   level30
   msgunique objectIds
   name
   pid8
   time
   uniqObjectIds: [ [-]
     275649
   ]
   uniqObjectIdsCount1
   v0
}
There is no common key in these two events,but want to have in table view
1.Currently uniqObjectIds,uniqueRetrievedIds are displayed in two rows in a table view,wanted as a single row
2.How to combine multiple event in a single query if there is no common key
.

 

index= ""    source IN ("")  "uniqObjectIds"  OR "data retrieved for Ids"
 | spath output=uniqObjectIds path=uniqObjectIds{}  | spath output=uniqueRetrievedIds path=uniqueRetrievedIds{} |  eval PST=_time-28800 | eval PST_TIME=strftime(PST, "%Y-%d-%m %H:%M:%S") | eval split_field= split(_raw, "Z\"}") | mvexpand split_field | rex field=split_field "objectIdsCount=(?<objectIdsCount>[^,]+)" | rex field=split_field "uniqObjectIdsCount=(?<uniqObjectIdsCount>[^,]+)" | rex field=split_field "recordsCount=(?<recordsCount>[^,]+)" | rex field=split_field "sqsSentCount=(?<sqsSentCount>[^,]+)" | table_time,PST_TIME,objectType,objectIdsCount,uniqObjectIdsCount,recordsCount,sqsSentCount,uniqObjectIds,uniqueRetrievedIds | sort _time desc

 

  Screenshot 2023-11-08 at 6.33.28 AM.png

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Please share the raw unformatted sample event in a code block </> to preserve the original formatting.

0 Karma

nithys
Path Finder

Hi @ITWhisperer 
Below are the raw events  which are need to be displayed in table format in a single row for below events with no common key value

{"name":"","awsRequestId":"","hostname":"","pid":8,"level":30,"uniqObjectIds":["275649"],"uniqObjectIdsCount":1,"msg":"unique objectIds","time":"2023-11-03T19:26:43.672Z","v":0}
{"name":"","awsRequestId":"","hostname":"","pid":8,"level":30,"uniqueRetrievedIds":["275649"],"msg":"data retrieved for Ids","time":"2023-11-06T22:48:03.594Z","v":0}
{"name":"","awsRequestId":"","hostname":"","pid":8,"level":30,"eventBody":{"objectType":"material","objectIds":["275649","108795","1234567","1234568","99999999","888888888"],"version":"all"},"msg":"request body","time":"2023-11-03T05:25:33.508Z","v":0}
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

OK so how do you want them combined e.g. do you want the times from the "unique objectIds" message and "data retrieved for Ids" message to be in the same row by object id?

| spath uniqObjectIds{} output=uniqObjectIds
| spath uniqueRetrievedIds{} output=uniqueRetrievedIds
| spath eventBody.objectIds{} output=eventBodyObjectIds
| eval eventBodyObjectIds=if(eventBodyObjectIds=="",null(),eventBodyObjectIds)
| eval objectId=coalesce(eventBodyObjectIds,coalesce(uniqueRetrievedIds,uniqObjectIds))
| eval retrievedTime=if(msg=="data retrieved for Ids",time,null())
| eval uniqueTime=if(msg=="unique objectIds",time,null())
| stats values(retrievedTime) as retrievedTime values(uniqueTime) as uniqueTime by objectId
0 Karma

nithys
Path Finder

Hi @ITWhisperer 
Thank for your reply and suggestions.Really appreciate !
Now I have added a log in all the below  raw event which will have "entity":"suppliedMaterial"

 

 

{"name":"","awsRequestId":"","hostname":"","pid":8,"level":30,"supportType":"reprocess","entity":"suppliedMaterial","uniqObjectIds":["122598","268817","88888888888","99999999999999999","abc"],"uniqObjectIdsCount":5,"msg":"unique objectIds","time":"2023-11-09T13:56:36.559Z","v":0}
2nd event

{"name":"","awsRequestId":"","hostname":"","pid":8,"level":30,"supportType":"reprocess","entity":"suppliedMaterial","uniqueRetrievedIds":[],"msg":"data retrieved for Ids","time":"2023-11-09T13:56:36.749Z","v":0}

3rd event

{"name":"","awsRequestId":"","hostname":"","pid":8,"level":30,"supportType":"reprocess","entity":"suppliedMaterial","dataNotFoundIds":["122598","268817","88888888888","99999999999999999","abc"],"dataNotFoundIdsCount":5,"msg":"data not found for Ids","time":"2023-11-09T13:56:36.749Z","v":0}

 

 

Currently i have three query which fetch the result from _raw in below table format 
Expected.I want all three different table combined and show their result   based on common key value entity: "suppliedMaterial" from raw events .

 

uniqObjectIds

uniqObjectIdsCount

uniqObjectIds{}

PST_TIME

122598

268817

88888888888

99999999999999999

abc

5

122598

268817

88888888888

99999999999999999

abc

2023-08-11 06:38:01

122598

268817

88888888888

99999999999999999

abcdefg

5

122598

268817

88888888888

99999999999999999

abcdefg

2023-08-11 06:37:44

122598

268817

88888888888

99999999999999999

abcdefg

5

122598

268817

88888888888

99999999999999999

abcdefg

2023-08-11 06:35:40

122598

268817

88888888888

99999999999999999

abcdefg

5

122598

268817

88888888888

99999999999999999

abcdefg

2023-08-11 06:33:14

uniqueRetrievedIds{}

uniqueRetrievedIds

PST_TIME

122598

268817

122598

268817

2023-08-11 06:38:01

122598

268817

122598

268817

2023-08-11 06:37:44

122598

268817

122598

268817

2023-08-11 06:35:40

122598

268817

122598

268817

2023-08-11 06:33:14

122598

268817

122598

268817

2023-08-11 03:28:56

275649

275649

2023-06-11 06:49:10

275649

275649

2023-03-11 04:28:36

275649

275649

2023-03-11 03:13:07

108795

108795

2023-03-11 03:09:07

108795

275649

108795

275649

2023-02-11 14:44:04

dataNotFoundIds{}

dataNotFoundIdsCount

PST_TIME

88888888888

99999999999999999

abc

3

2023-08-11 06:38:01

88888888888

99999999999999999

abcdefg

3

2023-08-11 06:37:44

88888888888

99999999999999999

abcdefg

3

2023-08-11 06:35:40

88888888888

99999999999999999

abcdefg

3

 

 

0 Karma

nithys
Path Finder

continue...

Hi @ITWhisperer The query used to get the above three table format

 

 

 

1st query:
index= source IN ("") "uniqObjectIds" OR "data retrieved for Ids" | eval PST=_time-28800 | eval PST_TIME=strftime(PST, "%Y-%d-%m %H:%M:%S") | spath output=uniqObjectIds path=uniqObjectIds{} |where isnotnull(uniqObjectIds) | spath output=uniqueRetrievedIds path=uniqueRetrievedIds{}| stats values(*) as * by _raw | table uniqObjectIds,uniqObjectIdsCount,uniqObjectIds{},PST_TIME | sort- PST_TIME 
2nd query"

index= source IN ("") "data retrieved for Ids"| eval PST=_time-28800 | eval PST_TIME=strftime(PST, "%Y-%d-%m %H:%M:%S") | spath output=uniqueRetrievedIds path=uniqueRetrievedIds{} |where isnotnull(uniqueRetrievedIds)| stats values(*) as * by _raw | table uniqueRetrievedIds{},uniqueRetrievedIds, PST_TIME | sort- PST_TIME 

3rd query:
index= source IN ("") "data not found for Ids"| eval PST=_time-28800 | eval PST_TIME=strftime(PST, "%Y-%d-%m %H:%M:%S") | spath output=dataNotFoundIds path=dataNotFoundIds{}|where isnotnull(dataNotFoundIds) | stats values(*) as * by _raw | table dataNotFoundIds{},dataNotFoundIdsCount, PST_TIME | sort- PST_TIME

 

 

 

Could you help

0 Karma

nithys
Path Finder

HI @ITWhisperer 

 

I want all three query combined and display their results in one table based on entity:suppliedMaterial

 

uniqObjectIds

 

uniqObjectIdsCountPST_TIME

uniqueRetrievedIds{}

uniqueRetrievedIds

PST_TIME

dataNotFoundIdsdataNotFoundIdsCountPST_TIME   
122598
268817
88888888888
99999999999999999
abc
5
122598
268817
88888888888
99999999999999999
abc
5
122598
268817
88888888888
99999999999999999
abc
2023-08-11 06:38:01
122598
268817
122598
268817
2023-08-11 06:38:01   
88888888888
99999999999999999
abc
3
122598
268817
88888888888
99999999999999999
abc
5
122598
268817
88888888888
99999999999999999
abc
2023-08-11 06:38:01
122598
268817
122598
268817
2023-08-11 06:37:44   
similar to abovesimilar to abovesimilar to abovesimilar to above ssimilar to above similar to abovesimilar to abovesimilar to abovesimilar to above  similar to above
  ....         

 

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

You can't have multiple columns with the same name so try this

index= source IN ("") "uniqObjectIds" OR "data retrieved for Ids" | eval PST=_time-28800 | eval PST_TIME=strftime(PST, "%Y-%d-%m %H:%M:%S") | spath output=uniqObjectIds path=uniqObjectIds{} |where isnotnull(uniqObjectIds) | spath output=uniqueRetrievedIds path=uniqueRetrievedIds{}| stats values(*) as * by _raw | table uniqObjectIds,uniqObjectIdsCount,uniqObjectIds{},PST_TIME | sort- PST_TIME 
| appendcols [search index= source IN ("") "data retrieved for Ids"| eval PST=_time-28800 | eval PST_TIME2=strftime(PST, "%Y-%d-%m %H:%M:%S") | spath output=uniqueRetrievedIds path=uniqueRetrievedIds{} |where isnotnull(uniqueRetrievedIds)| stats values(*) as * by _raw | table uniqueRetrievedIds{},uniqueRetrievedIds, PST_TIME2 | sort- PST_TIME2 ]
| appendcols [search index= source IN ("") "data not found for Ids"| eval PST=_time-28800 | eval PST_TIME3=strftime(PST, "%Y-%d-%m %H:%M:%S") | spath output=dataNotFoundIds path=dataNotFoundIds{}|where isnotnull(dataNotFoundIds) | stats values(*) as * by _raw | table dataNotFoundIds{},dataNotFoundIdsCount, PST_TIME3 | sort- PST_TIME3 ]

nithys
Path Finder

Thanks much @ITWhisperer It really worked

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

What would your expected result look like?

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In December, the Splunk Threat Research Team had 1 release of new security content via the Enterprise Security ...

Why am I not seeing the finding in Splunk Enterprise Security Analyst Queue?

(This is the first of a series of 2 blogs). Splunk Enterprise Security is a fantastic tool that offers robust ...

Index This | What are the 12 Days of Splunk-mas?

December 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...