Splunk Search

specific field extraction from _raw event data/message

ssamant007
Explorer

I have event data from the search result in format as shown in the image, now I want to extract the following fields with their corresponding values excluding the remaining fields or data from the event data/string:

id = b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09

start_date_time = 1638433382 (value always required)

end_date_time = null or 1638433491  (if value not present)

current = <value> (only if the field exist) (6 in the example)

total = <value> (6 in the example)

status_type = COMPLETED

bot_uri = repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot

I tried using <search query> | rex field=_raw "(?msi)(?<ev_field>\{.+\}$)"
| spath input=ev_field  to extract all the fields in the Event data, but did not change the search results. Any suggestion or help highly appreciated I am newbie to Splunk...

TIA

 

ssamant007_0-1638434187803.png

12/2/21
7:24:52.106 PM
 
2021-Dec-02 Thu 19:24:52.106 INFO [pool-12-thread-1] - com.automationanywhere.nodemanager.service.impl.NodeMessagingServiceImpl - {} - writeSuccess(NodeMessagingServiceImpl.java:395) - Message eventData { id: "b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09" bot_execution { start_date_time { seconds: 1638433382 nanos: 210329300 } end_date_time { seconds: 1638433491 nanos: 993822800 } progress { current: 6 total: 6 percentage: 100 } status_type: COMPLETED bot_uri: "repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot?fileId=1098948&workspace=PRIVATE" }} sent to CR successfully.

 

Labels (5)
0 Karma
1 Solution

Gr0und_Z3r0
Path Finder

This is one way of doing it....
I've currently set end_date_time & Current to default to null if there is no value.
If you want you can set Current to 0 if it doesn't exist by adding one more line
| fillnull value=0 Current

 

| makeresults 
| eval _raw= "2021-Dec-02 Thu 19:24:52.106 INFO [pool-12-thread-1] - com.automationanywhere.nodemanager.service.impl.NodeMessagingServiceImpl - {} - writeSuccess(NodeMessagingServiceImpl.java:395) - Message eventData { id: \"b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09\" bot_execution { start_date_time { seconds: 1638433382 nanos: 210329300 } end_date_time { seconds: 1638433491 nanos: 993822800 } progress { current: 6 total: 6 percentage: 100 } status_type: COMPLETED bot_uri: \"repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot?fileId=1098948&workspace=PRIVATE\" }} sent to CR successfully." 
| rex field=_raw "id\:\s\"(?<ID>[a-z0-9\-\_]+)\"\s" 
| rex field=_raw "start\_date\_time\s\{\sseconds\:\s(?<start_date_time>[\d]+)\s" 
| rex field=_raw "end\_date\_time\s\{\sseconds\:\s(?<end_date_time>[\d]+)\s" 
| rex field=_raw "\{\scurrent\:\s(?<Current>[\d]+)\stotal" 
| rex field=_raw "\stotal\:\s(?<Total>[\d]+)\s" 
| rex field=_raw "status\_type\:\s(?<Status>[\w]+)\s" 
| rex field=_raw "bot_uri\:\s\"(?<bot_uri>.*)\?" 
| table _time _raw ID start_date_time end_date_time Current Total Status bot_uri 
| fillnull value=null end_date_time Current

 

Gr0und_Z3r0_0-1638446324206.png



If it helps, an upvote would be appreciated.

 

View solution in original post

PickleRick
Champion

Adding to all other good answers - what do you mean by excluding remaining fields/data?

 If you want to only operate further in the search on those extracted fields and will definitely not need the raw event, you might just

| fields - _raw

to tell splunk not to bother with the original event.

0 Karma

Gr0und_Z3r0
Path Finder

This is one way of doing it....
I've currently set end_date_time & Current to default to null if there is no value.
If you want you can set Current to 0 if it doesn't exist by adding one more line
| fillnull value=0 Current

 

| makeresults 
| eval _raw= "2021-Dec-02 Thu 19:24:52.106 INFO [pool-12-thread-1] - com.automationanywhere.nodemanager.service.impl.NodeMessagingServiceImpl - {} - writeSuccess(NodeMessagingServiceImpl.java:395) - Message eventData { id: \"b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09\" bot_execution { start_date_time { seconds: 1638433382 nanos: 210329300 } end_date_time { seconds: 1638433491 nanos: 993822800 } progress { current: 6 total: 6 percentage: 100 } status_type: COMPLETED bot_uri: \"repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot?fileId=1098948&workspace=PRIVATE\" }} sent to CR successfully." 
| rex field=_raw "id\:\s\"(?<ID>[a-z0-9\-\_]+)\"\s" 
| rex field=_raw "start\_date\_time\s\{\sseconds\:\s(?<start_date_time>[\d]+)\s" 
| rex field=_raw "end\_date\_time\s\{\sseconds\:\s(?<end_date_time>[\d]+)\s" 
| rex field=_raw "\{\scurrent\:\s(?<Current>[\d]+)\stotal" 
| rex field=_raw "\stotal\:\s(?<Total>[\d]+)\s" 
| rex field=_raw "status\_type\:\s(?<Status>[\w]+)\s" 
| rex field=_raw "bot_uri\:\s\"(?<bot_uri>.*)\?" 
| table _time _raw ID start_date_time end_date_time Current Total Status bot_uri 
| fillnull value=null end_date_time Current

 

Gr0und_Z3r0_0-1638446324206.png



If it helps, an upvote would be appreciated.

 

View solution in original post

ssamant007
Explorer

Thanks @Gr0und_Z3r0 . this is what I was looking for.

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Hi
it seems that your event's is not correct json format (e.g. missing : and , characters). Is it possible that logging system will fix those (the best option) or should you fix those (if many events with many formats, this will be quite hard task)?
r. Ismo
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Without fixing the source event you can try this with normal rex extractions like

| makeresults
| eval _raw = "INFO [pool-12-thread-1] - com.automationanywhere.nodemanager.service.impl.NodeMessagingServiceImpl - {} - writeSuccess(NodeMessagingServiceImpl.java:395) - Message eventData { id: \"b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09\" bot_execution { start_date_time { seconds: 1638433382 nanos: 210329300 } end_date_time { seconds: 1638433491 nanos: 993822800 } progress { current: 6 total: 6 percentage: 100 } status_type: COMPLETED bot_uri: \"repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot?fileId=1098948&workspace=PRIVATE\" }} sent to CR successfully."
``` Above create test event```

| rex "id: \"(?<id>[^\"]+)"
| rex "start_date_time { seconds: (?<start_date_time>\d+)"
| rex "end_date_time { seconds: (?<end_date_time>\d+)"
| rex "current: (?<current>\d+)"
| rex "total: (?<total>\d+)"
| rex "status_type: (?<status_type>\w+)"
| rex "bot_uri: \"(?<bot_uri>[^\"]+)"
| table id start_date_time end_date_time current total status_type bot_uri

If your events have fixed format you probably could combine some rex together, but then you should look from job inspector which mode is more efficient.

r. Ismo

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!