I have event data from the search result in format as shown in the image, now I want to extract the following fields with their corresponding values excluding the remaining fields or data from the event data/string:
id = b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09
start_date_time = 1638433382 (value always required)
end_date_time = null or 1638433491 (if value not present)
current = <value> (only if the field exist) (6 in the example)
total = <value> (6 in the example)
status_type = COMPLETED
bot_uri = repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot
I tried using <search query> | rex field=_raw "(?msi)(?<ev_field>\{.+\}$)"
| spath input=ev_field to extract all the fields in the Event data, but did not change the search results. Any suggestion or help highly appreciated I am newbie to Splunk...
TIA
12/2/21 7:24:52.106 PM | 2021-Dec-02 Thu 19:24:52.106 INFO [pool-12-thread-1] - com.automationanywhere.nodemanager.service.impl.NodeMessagingServiceImpl - {} - writeSuccess(NodeMessagingServiceImpl.java:395) - Message eventData { id: "b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09" bot_execution { start_date_time { seconds: 1638433382 nanos: 210329300 } end_date_time { seconds: 1638433491 nanos: 993822800 } progress { current: 6 total: 6 percentage: 100 } status_type: COMPLETED bot_uri: "repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot?fileId=1098948&workspace=PRIVATE" }} sent to CR successfully. |
This is one way of doing it....
I've currently set end_date_time & Current to default to null if there is no value.
If you want you can set Current to 0 if it doesn't exist by adding one more line
| fillnull value=0 Current
| makeresults
| eval _raw= "2021-Dec-02 Thu 19:24:52.106 INFO [pool-12-thread-1] - com.automationanywhere.nodemanager.service.impl.NodeMessagingServiceImpl - {} - writeSuccess(NodeMessagingServiceImpl.java:395) - Message eventData { id: \"b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09\" bot_execution { start_date_time { seconds: 1638433382 nanos: 210329300 } end_date_time { seconds: 1638433491 nanos: 993822800 } progress { current: 6 total: 6 percentage: 100 } status_type: COMPLETED bot_uri: \"repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot?fileId=1098948&workspace=PRIVATE\" }} sent to CR successfully."
| rex field=_raw "id\:\s\"(?<ID>[a-z0-9\-\_]+)\"\s"
| rex field=_raw "start\_date\_time\s\{\sseconds\:\s(?<start_date_time>[\d]+)\s"
| rex field=_raw "end\_date\_time\s\{\sseconds\:\s(?<end_date_time>[\d]+)\s"
| rex field=_raw "\{\scurrent\:\s(?<Current>[\d]+)\stotal"
| rex field=_raw "\stotal\:\s(?<Total>[\d]+)\s"
| rex field=_raw "status\_type\:\s(?<Status>[\w]+)\s"
| rex field=_raw "bot_uri\:\s\"(?<bot_uri>.*)\?"
| table _time _raw ID start_date_time end_date_time Current Total Status bot_uri
| fillnull value=null end_date_time Current
If it helps, an upvote would be appreciated.
Adding to all other good answers - what do you mean by excluding remaining fields/data?
If you want to only operate further in the search on those extracted fields and will definitely not need the raw event, you might just
| fields - _raw
to tell splunk not to bother with the original event.
This is one way of doing it....
I've currently set end_date_time & Current to default to null if there is no value.
If you want you can set Current to 0 if it doesn't exist by adding one more line
| fillnull value=0 Current
| makeresults
| eval _raw= "2021-Dec-02 Thu 19:24:52.106 INFO [pool-12-thread-1] - com.automationanywhere.nodemanager.service.impl.NodeMessagingServiceImpl - {} - writeSuccess(NodeMessagingServiceImpl.java:395) - Message eventData { id: \"b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09\" bot_execution { start_date_time { seconds: 1638433382 nanos: 210329300 } end_date_time { seconds: 1638433491 nanos: 993822800 } progress { current: 6 total: 6 percentage: 100 } status_type: COMPLETED bot_uri: \"repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot?fileId=1098948&workspace=PRIVATE\" }} sent to CR successfully."
| rex field=_raw "id\:\s\"(?<ID>[a-z0-9\-\_]+)\"\s"
| rex field=_raw "start\_date\_time\s\{\sseconds\:\s(?<start_date_time>[\d]+)\s"
| rex field=_raw "end\_date\_time\s\{\sseconds\:\s(?<end_date_time>[\d]+)\s"
| rex field=_raw "\{\scurrent\:\s(?<Current>[\d]+)\stotal"
| rex field=_raw "\stotal\:\s(?<Total>[\d]+)\s"
| rex field=_raw "status\_type\:\s(?<Status>[\w]+)\s"
| rex field=_raw "bot_uri\:\s\"(?<bot_uri>.*)\?"
| table _time _raw ID start_date_time end_date_time Current Total Status bot_uri
| fillnull value=null end_date_time Current
If it helps, an upvote would be appreciated.
Thanks @Gr0und_Z3r0 . this is what I was looking for.
Without fixing the source event you can try this with normal rex extractions like
| makeresults
| eval _raw = "INFO [pool-12-thread-1] - com.automationanywhere.nodemanager.service.impl.NodeMessagingServiceImpl - {} - writeSuccess(NodeMessagingServiceImpl.java:395) - Message eventData { id: \"b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09\" bot_execution { start_date_time { seconds: 1638433382 nanos: 210329300 } end_date_time { seconds: 1638433491 nanos: 993822800 } progress { current: 6 total: 6 percentage: 100 } status_type: COMPLETED bot_uri: \"repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot?fileId=1098948&workspace=PRIVATE\" }} sent to CR successfully."
``` Above create test event```
| rex "id: \"(?<id>[^\"]+)"
| rex "start_date_time { seconds: (?<start_date_time>\d+)"
| rex "end_date_time { seconds: (?<end_date_time>\d+)"
| rex "current: (?<current>\d+)"
| rex "total: (?<total>\d+)"
| rex "status_type: (?<status_type>\w+)"
| rex "bot_uri: \"(?<bot_uri>[^\"]+)"
| table id start_date_time end_date_time current total status_type bot_uri
If your events have fixed format you probably could combine some rex together, but then you should look from job inspector which mode is more efficient.
r. Ismo