specific field extraction from _raw event data/message

[ssamant007]

I have event data from the search result in format as shown in the image, now I want to extract the following fields with their corresponding values excluding the remaining fields or data from the event data/string:

id = b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09

start_date_time = 1638433382 (value always required)

end_date_time = null or 1638433491  (if value not present)

current = <value> (only if the field exist) (6 in the example)

total = <value> (6 in the example)

status_type = COMPLETED

bot_uri = repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot

I tried using <search query> | rex field=_raw "(?msi)(?<ev_field>\{.+\}$)"
| spath input=ev_field  to extract all the fields in the Event data, but did not change the search results. Any suggestion or help highly appreciated I am newbie to Splunk...

TIA

12/2/21 7:24:52.106 PM

2021-Dec-02 Thu 19:24:52.106 INFO [pool-12-thread-1] - com.automationanywhere.nodemanager.service.impl.NodeMessagingServiceImpl - {} - writeSuccess(NodeMessagingServiceImpl.java:395) - Message eventData { id: "b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09" bot_execution { start_date_time { seconds: 1638433382 nanos: 210329300 } end_date_time { seconds: 1638433491 nanos: 993822800 } progress { current: 6 total: 6 percentage: 100 } status_type: COMPLETED bot_uri: "repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot?fileId=1098948&workspace=PRIVATE" }} sent to CR successfully.

[PickleRick]

Adding to all other good answers - what do you mean by excluding remaining fields/data?

 If you want to only operate further in the search on those extracted fields and will definitely not need the raw event, you might just

| fields - _raw

to tell splunk not to bother with the original event.

[Gr0und_Z3r0 · Accepted solution]

This is one way of doing it....
I've currently set end_date_time & Current to default to null if there is no value.
If you want you can set Current to 0 if it doesn't exist by adding one more line
| fillnull value=0 Current

| makeresults 
| eval _raw= "2021-Dec-02 Thu 19:24:52.106 INFO [pool-12-thread-1] - com.automationanywhere.nodemanager.service.impl.NodeMessagingServiceImpl - {} - writeSuccess(NodeMessagingServiceImpl.java:395) - Message eventData { id: \"b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09\" bot_execution { start_date_time { seconds: 1638433382 nanos: 210329300 } end_date_time { seconds: 1638433491 nanos: 993822800 } progress { current: 6 total: 6 percentage: 100 } status_type: COMPLETED bot_uri: \"repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot?fileId=1098948&workspace=PRIVATE\" }} sent to CR successfully." 
| rex field=_raw "id\:\s\"(?<ID>[a-z0-9\-\_]+)\"\s" 
| rex field=_raw "start\_date\_time\s\{\sseconds\:\s(?<start_date_time>[\d]+)\s" 
| rex field=_raw "end\_date\_time\s\{\sseconds\:\s(?<end_date_time>[\d]+)\s" 
| rex field=_raw "\{\scurrent\:\s(?<Current>[\d]+)\stotal" 
| rex field=_raw "\stotal\:\s(?<Total>[\d]+)\s" 
| rex field=_raw "status\_type\:\s(?<Status>[\w]+)\s" 
| rex field=_raw "bot_uri\:\s\"(?<bot_uri>.*)\?" 
| table _time _raw ID start_date_time end_date_time Current Total Status bot_uri 
| fillnull value=null end_date_time Current

If it helps, an upvote would be appreciated.

[ssamant007]

Thanks @Gr0und_Z3r0 . this is what I was looking for.

[isoutamo]

Hi
it seems that your event's is not correct json format (e.g. missing : and , characters). Is it possible that logging system will fix those (the best option) or should you fix those (if many events with many formats, this will be quite hard task)?
r. Ismo

[isoutamo]

Without fixing the source event you can try this with normal rex extractions like

| makeresults
| eval _raw = "INFO [pool-12-thread-1] - com.automationanywhere.nodemanager.service.impl.NodeMessagingServiceImpl - {} - writeSuccess(NodeMessagingServiceImpl.java:395) - Message eventData { id: \"b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09\" bot_execution { start_date_time { seconds: 1638433382 nanos: 210329300 } end_date_time { seconds: 1638433491 nanos: 993822800 } progress { current: 6 total: 6 percentage: 100 } status_type: COMPLETED bot_uri: \"repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot?fileId=1098948&workspace=PRIVATE\" }} sent to CR successfully."
``` Above create test event```

| rex "id: \"(?<id>[^\"]+)"
| rex "start_date_time { seconds: (?<start_date_time>\d+)"
| rex "end_date_time { seconds: (?<end_date_time>\d+)"
| rex "current: (?<current>\d+)"
| rex "total: (?<total>\d+)"
| rex "status_type: (?<status_type>\w+)"
| rex "bot_uri: \"(?<bot_uri>[^\"]+)"
| table id start_date_time end_date_time current total status_type bot_uri

If your events have fixed format you probably could combine some rex together, but then you should look from job inspector which mode is more efficient.

r. Ismo