topic Re: specific field extraction from _raw event data/message in Splunk Search

specific field extraction from _raw event data/message

ssamant007 — Thu, 02 Dec 2021 08:47:25 GMT

I have event data from the search result in format as shown in the image, now I want to extract the following fields with their corresponding values excluding the remaining fields or data from the event data/string:

id = b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09

start_date_time = 1638433382 (value always required)

end_date_time = null or 1638433491 (if value not present)

current = <value> (only if the field exist) (6 in the example)

total = <value> (6 in the example)

status_type = COMPLETED

bot_uri = repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot

I tried using <search query> | rex field=_raw "(?msi)(?<ev_field>\{.+\}$)"
| spath input=ev_field to extract all the fields in the Event data, but did not change the search results. Any suggestion or help highly appreciated I am newbie to Splunk...

TIA

12/2/21
7:24:52.106 PM

2021-Dec-02 Thu 19:24:52.106 INFO [pool-12-thread-1] - com.automationanywhere.nodemanager.service.impl.NodeMessagingServiceImpl - {} - writeSuccess(NodeMessagingServiceImpl.java:395) - Message eventData { id: "b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09" bot_execution { start_date_time { seconds: 1638433382 nanos: 210329300 } end_date_time { seconds: 1638433491 nanos: 993822800 } progress { current: 6 total: 6 percentage: 100 } status_type: COMPLETED bot_uri: "repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot?fileId=1098948&workspace=PRIVATE" }} sent to CR successfully.

Re: specific field extraction from _raw event data/message

isoutamo — Thu, 02 Dec 2021 11:35:05 GMT

Hi
it seems that your event's is not correct json format (e.g. missing : and , characters). Is it possible that logging system will fix those (the best option) or should you fix those (if many events with many formats, this will be quite hard task)?
r. Ismo

Re: specific field extraction from _raw event data/message

isoutamo — Thu, 02 Dec 2021 11:45:05 GMT

Without fixing the source event you can try this with normal rex extractions like

| makeresults | eval _raw = "INFO [pool-12-thread-1] - com.automationanywhere.nodemanager.service.impl.NodeMessagingServiceImpl - {} - writeSuccess(NodeMessagingServiceImpl.java:395) - Message eventData { id: \"b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09\" bot_execution { start_date_time { seconds: 1638433382 nanos: 210329300 } end_date_time { seconds: 1638433491 nanos: 993822800 } progress { current: 6 total: 6 percentage: 100 } status_type: COMPLETED bot_uri: \"repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot?fileId=1098948&workspace=PRIVATE\" }} sent to CR successfully." ``` Above create test event``` | rex "id: \"(?<id>[^\"]+)" | rex "start_date_time { seconds: (?<start_date_time>\d+)" | rex "end_date_time { seconds: (?<end_date_time>\d+)" | rex "current: (?<current>\d+)" | rex "total: (?<total>\d+)" | rex "status_type: (?<status_type>\w+)" | rex "bot_uri: \"(?<bot_uri>[^\"]+)" | table id start_date_time end_date_time current total status_type bot_uri

If your events have fixed format you probably could combine some rex together, but then you should look from job inspector which mode is more efficient.

r. Ismo

Re: specific field extraction from _raw event data/message

Gr0und_Z3r0 — Thu, 02 Dec 2021 12:01:13 GMT

This is one way of doing it....
I've currently set end_date_time & Current to default to null if there is no value.
If you want you can set Current to 0 if it doesn't exist by adding one more line
| fillnull value=0 Current

| makeresults | eval _raw= "2021-Dec-02 Thu 19:24:52.106 INFO [pool-12-thread-1] - com.automationanywhere.nodemanager.service.impl.NodeMessagingServiceImpl - {} - writeSuccess(NodeMessagingServiceImpl.java:395) - Message eventData { id: \"b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09\" bot_execution { start_date_time { seconds: 1638433382 nanos: 210329300 } end_date_time { seconds: 1638433491 nanos: 993822800 } progress { current: 6 total: 6 percentage: 100 } status_type: COMPLETED bot_uri: \"repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot?fileId=1098948&workspace=PRIVATE\" }} sent to CR successfully." | rex field=_raw "id\:\s\"(?<ID>[a-z0-9\-\_]+)\"\s" | rex field=_raw "start\_date\_time\s\{\sseconds\:\s(?<start_date_time>[\d]+)\s" | rex field=_raw "end\_date\_time\s\{\sseconds\:\s(?<end_date_time>[\d]+)\s" | rex field=_raw "\{\scurrent\:\s(?<Current>[\d]+)\stotal" | rex field=_raw "\stotal\:\s(?<Total>[\d]+)\s" | rex field=_raw "status\_type\:\s(?<Status>[\w]+)\s" | rex field=_raw "bot_uri\:\s\"(?<bot_uri>.*)\?" | table _time _raw ID start_date_time end_date_time Current Total Status bot_uri | fillnull value=null end_date_time Current

If it helps, an upvote would be appreciated.

Re: specific field extraction from _raw event data/message

PickleRick — Thu, 02 Dec 2021 12:12:05 GMT

Adding to all other good answers - what do you mean by excluding remaining fields/data?

If you want to only operate further in the search on those extracted fields and will definitely not need the raw event, you might just

| fields - _raw

to tell splunk not to bother with the original event.

Re: specific field extraction from _raw event data/message

ssamant007 — Fri, 03 Dec 2021 04:16:38 GMT

Thanks @Gr0und_Z3r0 . this is what I was looking for.