topic Re: Field extraction in Splunk Search

Help with field extraction?

sekhar463 — Mon, 06 Mar 2023 14:22:37 GMT

Hi All,

we have events like below and in these need to extracts below id"s example d1c35370-1522-498c-8a79-ab07909a1c4a as new fields with in the status is running

we have muliple ID"S like this in the event

status is like running and Collector is running in field

it will also show if value other than running

2023-03-03T08:19:31,693 [INFO] [prod] [2f78061f-5f51-4636-8da1-3c9644b9e7a1] [34d3d64e-01c8-428e-a7b1-8b414dbd5478] [agent-AgentDataSourceStateManagerActor] - All collector health status has been updated- stateMap: [Map(d55c495c-52da-4e57-bc83-2ee02e92d978 -> running, 8194d562-beb4-4a44-a7f3-ec92ed549b3c -> running, e6f1b795-bf44-4640-880f-8b32f69586b7 -> running, 08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad -> running, 4925c2fc-7f47-46e5-9a78-63e596bb469a -> running, d1c35370-1522-498c-8a79-ab07909a1c4a -> running, 8e7f28fa-26e9-445a-a5b3-50e5746ca8ca -> running, db52b5b0-31b2-43dc-8887-9f2859762a62 -> running)], statusMap: [Map(d55c495c-52da-4e57-bc83-2ee02e92d978 -> Collector is running., 8194d562-beb4-4a44-a7f3-ec92ed549b3c -> Collector is running., e6f1b795-bf44-4640-880f-8b32f69586b7 -> Collector is running., 08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad -> Collector is running., 4925c2fc-7f47-46e5-9a78-63e596bb469a -> Collector is running., d1c35370-1522-498c-8a79-ab07909a1c4a -> Collector is running., 8e7f28fa-26e9-445a-a5b3-50e5746ca8ca -> Collector is running., db52b5b0-31b2-43dc-8887-9f2859762a62 -> Collector is running.)]

Re: Field extraction

gcusello — Fri, 03 Mar 2023 14:59:31 GMT

Hi @sekhar463,

please try this regex:

| rex "((\[Map\()|(,\s+))(?<id>\w*-\w*-\w*-\w*-\w*)\s+-\>\s+(?<status>[^,]*)"

that you can test at https://regex101.com/r/YLxfkD/1

Ciao.

Giuseppe

Re: Field extraction

ITWhisperer — Fri, 03 Mar 2023 15:29:23 GMT

| rex max_match=0 "((\[Map$)|(,\s+))(?<id>\w*-\w*-\w*-\w*-\w*)\s+-\>\s+(?<status>[^,$]*)"

Re: Field extraction

sekhar463 — Wed, 08 Mar 2023 14:21:29 GMT

Hai All,

thanks but when using regex and search with

sourcetype = netapp:cloudsecure:agentlog | rex max_match=0 "((\[Map$)|(,\s+))(?<id>\w*-\w*-\w*-\w*-\w*)\s+-\>\s+(?<status>[^,$]*)" | search id="08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad" status="Collector is running."

it was showing events data with other ID"s as well as below.

how can i divide this data

2023-03-08T08:17:33,625 [INFO] [prod] [2f78061f-5f51-4636-8da1-3c9644b9e7a1] [34d3d64e-01c8-428e-a7b1-8b414dbd5478] [agent-AgentDataSourceStateManagerActor] - All collector health status has been updated- stateMap: [Map(d55c495c-52da-4e57-bc83-2ee02e92d978 -> running, 8194d562-beb4-4a44-a7f3-ec92ed549b3c -> running, e6f1b795-bf44-4640-880f-8b32f69586b7 -> running, 08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad -> running, 4925c2fc-7f47-46e5-9a78-63e596bb469a -> running, d1c35370-1522-498c-8a79-ab07909a1c4a -> running, 8e7f28fa-26e9-445a-a5b3-50e5746ca8ca -> running, db52b5b0-31b2-43dc-8887-9f2859762a62 -> running)], statusMap: [Map(d55c495c-52da-4e57-bc83-2ee02e92d978 -> Collector is running., 8194d562-beb4-4a44-a7f3-ec92ed549b3c -> Collector is running., e6f1b795-bf44-4640-880f-8b32f69586b7 -> Collector is running., 08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad -> Collector is running., 4925c2fc-7f47-46e5-9a78-63e596bb469a -> Collector is running., d1c35370-1522-498c-8a79-ab07909a1c4a -> Collector is running., 8e7f28fa-26e9-445a-a5b3-50e5746ca8ca -> Collector is running., db52b5b0-31b2-43dc-8887-9f2859762a62 -> Collector is running.)]

Re: Field extraction

ITWhisperer — Wed, 08 Mar 2023 15:03:24 GMT

This id exists in this event which is why it is showing. Perhaps if you show what it is you are expecting to be able to show, we might be able to point you in the right direction.

Re: Field extraction

sekhar463 — Fri, 10 Mar 2023 10:38:23 GMT

hai i need to deivide the data based on the ID"S so if filter with id filed it will shows the data about only those ID"S not other ID"S data

Re: Field extraction

ITWhisperer — Fri, 10 Mar 2023 11:08:39 GMT

| rex max_match=0 "((\[Map$)|(,\s+))(?<id_status>\w*-\w*-\w*-\w*-\w*\s+-\>\s+[^,$]*)" | mvexpand id_status | rex field=id_status "(?<id>\w*-\w*-\w*-\w*-\w*)\s+-\>\s+(?<status>[^,\)]*)"

Re: Field extraction

sekhar463 — Tue, 28 Mar 2023 08:58:11 GMT

its working

but i am getting error as

command.mvexpand: output will be truncated at 67200 results due to excessive memory usage. Memory threshold of 500MB as configured in limits.conf / [mvexpand] / max_mem_usage_mb has been reached.

how can i overcome this

Re: Field extraction

ITWhisperer — Tue, 28 Mar 2023 10:08:47 GMT

Reduce your data set - you could try splitting the search into chunks which are appended, but it depends on your data and what you are trying to do. You could also try storing the chunks in a summary index (for example) to offload some of the processing.

Re: Field extraction

yuanliu — Tue, 28 Mar 2023 11:46:45 GMT

As @ITWhisperer said, how you handle memory limit very much depends on what want to do with the extraction. You can also extend memory limit.

If your goal is to return the status of a specific id, say "08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad", you do not have to mvexpand. But search command is inadequate for the job. Do this instead,

| eval mystatus = mvindex(status, mvfind(id, "^08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad$"))

Unfortunately, this will only return mystatus = "running" becaue mvfind only returns the first matching index. Now, your desired action is to filter by status="Collector is running." This tells me that you are not interested in stateMap, but statusMap. In other words, you expect status of stateMap and statusMap to be distinct, even though both may contain the same id's. (You could have saved volunteers a lot of time by explaining these nuances clearly.)

To make this distinction, I'll offer two paths, one also uses regex, the other semantical. First using regex.

| rex max_match=0 "stateMap: \[Map$(?<stateMap>[^$]+)" ``` not being used ``` | rex field=stateMap max_match=0 "\s*(?<id>\S+) -> (?<state>[^,]+)" ``` not being used ``` | rex max_match=0 "statusMap: \[Map$(?<statusMap>[^$]+)" | rex field=statusMap max_match=0 "\s*(?<id>\S+) -> (?<status>[^,]+)" | eval mystatus = mvindex(status, mvfind(id, "^08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad$")) | where mystatus == "Collector is running."

Second, a semantic extraction

| rex max_match=0 "stateMap: \[Map$(?<stateMap>[^$]+)" ``` not being used ``` | rex max_match=0 "statusMap: \[Map$(?<statusMap>[^$]+)" | eval statusMap = mvmap(statusMap, split(statusMap, ", ")) | eval id = mvmap(statusMap, mvindex(split(statusMap, " -> "), 0)) | eval status = mvmap(statusMap, mvindex(split(statusMap, " -> "), 1)) | eval mystatus = mvindex(status, mvfind(id, "^08ff35ad-f7b8-4ef2-bf29-1ccf5e50caad$")) | where mystatus == "Collector is running."

Hope this helps.

Re: Field extraction

sekhar463 — Wed, 29 Mar 2023 14:22:33 GMT

any alternate search to change to avoid the error.

to get exact status as above

| rex max_match=0 "((\[Map\()|(,\s+))(?<id_status>\w*-\w*-\w*-\w*-\w*\s+-\>\s+[^,\)]*)"
| mvexpand id_status
| rex field=id_status "(?<id>\w*-\w*-\w*-\w*-\w*)\s+-\>\s+(?<status>[^,\)]*)"

Re: Field extraction

yuanliu — Wed, 29 Mar 2023 17:40:03 GMT

If you cannot reduce dataset as @ITWhisperer suggested, mvexpand is not suitable. Alternatives see my answer above.