Help with field extraction?

[sekhar463]

Hi All,

we have events like below and in these need to extracts below id"s example d1c35370-1522-498c-8a79-ab07909a1c4a  as new fields with in the status is running  

we have muliple ID"S like this in the event 

status is like running and Collector is running in field 

it will also show if value other than running

2023-03-03T08:19:31,693 [INFO] [prod] [2f78061f-5f51-4636-8da1-3c9644b9e7a1] [34d3d64e-01c8-428e-a7b1-8b414dbd5478] [agent-AgentDataSourceStateManagerActor] - All collector health status has been updated- stateMap: [Map(d55c495c-52da-4e57-bc83-2ee02e92d978 -> running, 8194d562-beb4-4a44-a7f3-ec92ed549b3c -> running, e6f1b795-bf44-4640-880f-8b32f69586b7 -> running, 08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad -> running, 4925c2fc-7f47-46e5-9a78-63e596bb469a -> running, d1c35370-1522-498c-8a79-ab07909a1c4a -> running, 8e7f28fa-26e9-445a-a5b3-50e5746ca8ca -> running, db52b5b0-31b2-43dc-8887-9f2859762a62 -> running)], statusMap: [Map(d55c495c-52da-4e57-bc83-2ee02e92d978 -> Collector is running., 8194d562-beb4-4a44-a7f3-ec92ed549b3c -> Collector is running., e6f1b795-bf44-4640-880f-8b32f69586b7 -> Collector is running., 08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad -> Collector is running., 4925c2fc-7f47-46e5-9a78-63e596bb469a -> Collector is running., d1c35370-1522-498c-8a79-ab07909a1c4a -> Collector is running., 8e7f28fa-26e9-445a-a5b3-50e5746ca8ca -> Collector is running., db52b5b0-31b2-43dc-8887-9f2859762a62 -> Collector is running.)]

[ITWhisperer]

| rex max_match=0 "((\[Map\()|(,\s+))(?<id>\w*-\w*-\w*-\w*-\w*)\s+-\>\s+(?<status>[^,\)]*)"

[gcusello]

Hi @sekhar463,

please try this regex:

| rex "((\[Map\()|(,\s+))(?<id>\w*-\w*-\w*-\w*-\w*)\s+-\>\s+(?<status>[^,]*)"

that you can test at https://regex101.com/r/YLxfkD/1 

Ciao.

Giuseppe

[sekhar463]

Hai All,

thanks but when using regex and search with 

sourcetype = netapp:cloudsecure:agentlog | rex max_match=0 "((\[Map\()|(,\s+))(?<id>\w*-\w*-\w*-\w*-\w*)\s+-\>\s+(?<status>[^,\)]*)" | search id="08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad" status="Collector is running."

it was showing events data with other ID"s as well as below.

how can i divide this data

2023-03-08T08:17:33,625 [INFO] [prod] [2f78061f-5f51-4636-8da1-3c9644b9e7a1] [34d3d64e-01c8-428e-a7b1-8b414dbd5478] [agent-AgentDataSourceStateManagerActor] - All collector health status has been updated- stateMap: [Map(d55c495c-52da-4e57-bc83-2ee02e92d978 -> running, 8194d562-beb4-4a44-a7f3-ec92ed549b3c -> running, e6f1b795-bf44-4640-880f-8b32f69586b7 -> running, 08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad -> running, 4925c2fc-7f47-46e5-9a78-63e596bb469a -> running, d1c35370-1522-498c-8a79-ab07909a1c4a -> running, 8e7f28fa-26e9-445a-a5b3-50e5746ca8ca -> running, db52b5b0-31b2-43dc-8887-9f2859762a62 -> running)], statusMap: [Map(d55c495c-52da-4e57-bc83-2ee02e92d978 -> Collector is running., 8194d562-beb4-4a44-a7f3-ec92ed549b3c -> Collector is running., e6f1b795-bf44-4640-880f-8b32f69586b7 -> Collector is running., 08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad -> Collector is running., 4925c2fc-7f47-46e5-9a78-63e596bb469a -> Collector is running., d1c35370-1522-498c-8a79-ab07909a1c4a -> Collector is running., 8e7f28fa-26e9-445a-a5b3-50e5746ca8ca -> Collector is running., db52b5b0-31b2-43dc-8887-9f2859762a62 -> Collector is running.)]

[ITWhisperer]

This id exists in this event which is why it is showing. Perhaps if you show what it is you are expecting to be able to show, we might be able to point you in the right direction.

[sekhar463]

hai i need to deivide the data based on the ID"S so if filter with id filed it will shows the data about only those ID"S not other ID"S data 

[ITWhisperer]

| rex max_match=0 "((\[Map\()|(,\s+))(?<id_status>\w*-\w*-\w*-\w*-\w*\s+-\>\s+[^,\)]*)"
| mvexpand id_status
| rex field=id_status "(?<id>\w*-\w*-\w*-\w*-\w*)\s+-\>\s+(?<status>[^,\)]*)"

[sekhar463]

its working 

but i am getting error as 

command.mvexpand: output will be truncated at 67200 results due to excessive memory usage. Memory threshold of 500MB as configured in limits.conf / [mvexpand] / max_mem_usage_mb has been reached.

how can i overcome this

[yuanliu]

As @ITWhisperer said, how you handle memory limit very much depends on what want to do with the extraction.  You can also extend memory limit.

If your goal is to return the status of a specific id, say "08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad", you do not have to mvexpand.  But search command is inadequate for the job.  Do this instead,

| eval mystatus = mvindex(status, mvfind(id, "^08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad$"))

Unfortunately, this will only return mystatus = "running" becaue mvfind only returns the first matching index.  Now, your desired action is to filter by status="Collector is running."  This tells me that you are not interested in stateMap, but statusMap.  In other words, you expect status of stateMap and statusMap to be distinct, even though both may contain the same id's. (You could have saved volunteers a lot of time by explaining these nuances clearly.)  

To make this distinction, I'll offer two paths, one also uses regex, the other semantical.  First using regex.

| rex max_match=0 "stateMap: \[Map\((?<stateMap>[^\)]+)" ``` not being used ```
| rex field=stateMap max_match=0 "\s*(?<id>\S+) -> (?<state>[^,]+)" ``` not being used ```
| rex max_match=0 "statusMap: \[Map\((?<statusMap>[^\)]+)"
| rex field=statusMap max_match=0 "\s*(?<id>\S+) -> (?<status>[^,]+)"
| eval mystatus = mvindex(status, mvfind(id, "^08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad$"))
| where mystatus == "Collector is running."

Second, a semantic extraction

| rex max_match=0 "stateMap: \[Map\((?<stateMap>[^\)]+)" ``` not being used ```
| rex max_match=0 "statusMap: \[Map\((?<statusMap>[^\)]+)"
| eval statusMap = mvmap(statusMap, split(statusMap, ", "))
| eval id = mvmap(statusMap, mvindex(split(statusMap, " -> "), 0))
| eval status = mvmap(statusMap, mvindex(split(statusMap, " -> "), 1))
| eval mystatus = mvindex(status, mvfind(id, "^08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad$"))
| where mystatus == "Collector is running."

Hope this helps.

[ITWhisperer]

Reduce your data set - you could try splitting the search into chunks which are appended, but it depends on your data and what you are trying to do. You could also try storing the chunks in a summary index (for example) to offload some of the processing.

[sekhar463]

any alternate search to change to avoid the error.

to get exact status as above 

| rex max_match=0 "((\[Map\()|(,\s+))(?<id_status>\w*-\w*-\w*-\w*-\w*\s+-\>\s+[^,\)]*)"
| mvexpand id_status
| rex field=id_status "(?<id>\w*-\w*-\w*-\w*-\w*)\s+-\>\s+(?<status>[^,\)]*)"

[yuanliu]

If you cannot reduce dataset as @ITWhisperer suggested, mvexpand is not suitable.  Alternatives see my answer above.