Splunk Search

Help with field extraction?

sekhar463
Path Finder

Hi All,

we have events like below and in these need to extracts below id"s example d1c35370-1522-498c-8a79-ab07909a1c4a  as new fields with in the status is running  

we have muliple ID"S like this in the event 

status is like running and Collector is running in field 

it will also show if value other than running

 

 

2023-03-03T08:19:31,693 [INFO] [prod] [2f78061f-5f51-4636-8da1-3c9644b9e7a1] [34d3d64e-01c8-428e-a7b1-8b414dbd5478] [agent-AgentDataSourceStateManagerActor] - All collector health status has been updated- stateMap: [Map(d55c495c-52da-4e57-bc83-2ee02e92d978 -> running, 8194d562-beb4-4a44-a7f3-ec92ed549b3c -> running, e6f1b795-bf44-4640-880f-8b32f69586b7 -> running, 08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad -> running, 4925c2fc-7f47-46e5-9a78-63e596bb469a -> running, d1c35370-1522-498c-8a79-ab07909a1c4a -> running, 8e7f28fa-26e9-445a-a5b3-50e5746ca8ca -> running, db52b5b0-31b2-43dc-8887-9f2859762a62 -> running)], statusMap: [Map(d55c495c-52da-4e57-bc83-2ee02e92d978 -> Collector is running., 8194d562-beb4-4a44-a7f3-ec92ed549b3c -> Collector is running., e6f1b795-bf44-4640-880f-8b32f69586b7 -> Collector is running., 08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad -> Collector is running., 4925c2fc-7f47-46e5-9a78-63e596bb469a -> Collector is running., d1c35370-1522-498c-8a79-ab07909a1c4a -> Collector is running., 8e7f28fa-26e9-445a-a5b3-50e5746ca8ca -> Collector is running., db52b5b0-31b2-43dc-8887-9f2859762a62 -> Collector is running.)]

Labels (3)
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
| rex max_match=0 "((\[Map\()|(,\s+))(?<id>\w*-\w*-\w*-\w*-\w*)\s+-\>\s+(?<status>[^,\)]*)"
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @sekhar463,

please try this regex:

| rex "((\[Map\()|(,\s+))(?<id>\w*-\w*-\w*-\w*-\w*)\s+-\>\s+(?<status>[^,]*)"

that you can test at https://regex101.com/r/YLxfkD/1 

Ciao.

Giuseppe

0 Karma

sekhar463
Path Finder

Hai All,

thanks but when using regex and search with 

sourcetype = netapp:cloudsecure:agentlog | rex max_match=0 "((\[Map\()|(,\s+))(?<id>\w*-\w*-\w*-\w*-\w*)\s+-\>\s+(?<status>[^,\)]*)" | search id="08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad" status="Collector is running."

it was showing events data with other ID"s as well as below.

how can i divide this data

 

2023-03-08T08:17:33,625 [INFO] [prod] [2f78061f-5f51-4636-8da1-3c9644b9e7a1] [34d3d64e-01c8-428e-a7b1-8b414dbd5478] [agent-AgentDataSourceStateManagerActor] - All collector health status has been updated- stateMap: [Map(d55c495c-52da-4e57-bc83-2ee02e92d978 -> running, 8194d562-beb4-4a44-a7f3-ec92ed549b3c -> running, e6f1b795-bf44-4640-880f-8b32f69586b7 -> running, 08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad -> running, 4925c2fc-7f47-46e5-9a78-63e596bb469a -> running, d1c35370-1522-498c-8a79-ab07909a1c4a -> running, 8e7f28fa-26e9-445a-a5b3-50e5746ca8ca -> running, db52b5b0-31b2-43dc-8887-9f2859762a62 -> running)], statusMap: [Map(d55c495c-52da-4e57-bc83-2ee02e92d978 -> Collector is running., 8194d562-beb4-4a44-a7f3-ec92ed549b3c -> Collector is running., e6f1b795-bf44-4640-880f-8b32f69586b7 -> Collector is running., 08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad -> Collector is running., 4925c2fc-7f47-46e5-9a78-63e596bb469a -> Collector is running., d1c35370-1522-498c-8a79-ab07909a1c4a -> Collector is running., 8e7f28fa-26e9-445a-a5b3-50e5746ca8ca -> Collector is running., db52b5b0-31b2-43dc-8887-9f2859762a62 -> Collector is running.)]

 

 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

This id exists in this event which is why it is showing. Perhaps if you show what it is you are expecting to be able to show, we might be able to point you in the right direction.

0 Karma

sekhar463
Path Finder

hai i need to deivide the data based on the ID"S so if filter with id filed it will shows the data about only those ID"S not other ID"S data 

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
| rex max_match=0 "((\[Map\()|(,\s+))(?<id_status>\w*-\w*-\w*-\w*-\w*\s+-\>\s+[^,\)]*)"
| mvexpand id_status
| rex field=id_status "(?<id>\w*-\w*-\w*-\w*-\w*)\s+-\>\s+(?<status>[^,\)]*)"
0 Karma

sekhar463
Path Finder

its working 

but i am getting error as 

command.mvexpand: output will be truncated at 67200 results due to excessive memory usage. Memory threshold of 500MB as configured in limits.conf / [mvexpand] / max_mem_usage_mb has been reached.

how can i overcome this

0 Karma

yuanliu
SplunkTrust
SplunkTrust

As @ITWhisperer said, how you handle memory limit very much depends on what want to do with the extraction.  You can also extend memory limit.

If your goal is to return the status of a specific id, say "08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad", you do not have to mvexpand.  But search command is inadequate for the job.  Do this instead,

| eval mystatus = mvindex(status, mvfind(id, "^08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad$"))

Unfortunately, this will only return mystatus = "running" becaue mvfind only returns the first matching index.  Now, your desired action is to filter by status="Collector is running."  This tells me that you are not interested in stateMap, but statusMap.  In other words, you expect status of stateMap and statusMap to be distinct, even though both may contain the same id's. (You could have saved volunteers a lot of time by explaining these nuances clearly.)  

To make this distinction, I'll offer two paths, one also uses regex, the other semantical.  First using regex.

| rex max_match=0 "stateMap: \[Map\((?<stateMap>[^\)]+)" ``` not being used ```
| rex field=stateMap max_match=0 "\s*(?<id>\S+) -> (?<state>[^,]+)" ``` not being used ```
| rex max_match=0 "statusMap: \[Map\((?<statusMap>[^\)]+)"
| rex field=statusMap max_match=0 "\s*(?<id>\S+) -> (?<status>[^,]+)"
| eval mystatus = mvindex(status, mvfind(id, "^08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad$"))
| where mystatus == "Collector is running."

Second, a semantic extraction

| rex max_match=0 "stateMap: \[Map\((?<stateMap>[^\)]+)" ``` not being used ```
| rex max_match=0 "statusMap: \[Map\((?<statusMap>[^\)]+)"
| eval statusMap = mvmap(statusMap, split(statusMap, ", "))
| eval id = mvmap(statusMap, mvindex(split(statusMap, " -> "), 0))
| eval status = mvmap(statusMap, mvindex(split(statusMap, " -> "), 1))
| eval mystatus = mvindex(status, mvfind(id, "^08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad$"))
| where mystatus == "Collector is running."

Hope this helps.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Reduce your data set - you could try splitting the search into chunks which are appended, but it depends on your data and what you are trying to do. You could also try storing the chunks in a summary index (for example) to offload some of the processing.

0 Karma

sekhar463
Path Finder

any alternate search to change to avoid the error.

to get exact status as above 

| rex max_match=0 "((\[Map\()|(,\s+))(?<id_status>\w*-\w*-\w*-\w*-\w*\s+-\>\s+[^,\)]*)"
| mvexpand id_status
| rex field=id_status "(?<id>\w*-\w*-\w*-\w*-\w*)\s+-\>\s+(?<status>[^,\)]*)"

 

0 Karma

yuanliu
SplunkTrust
SplunkTrust

If you cannot reduce dataset as @ITWhisperer suggested, mvexpand is not suitable.  Alternatives see my answer above.

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...