Splunk Search

specific field extraction from _raw event data/message

ssamant007
Explorer

I have event data from the search result in format as shown in the image, now I want to extract the following fields with their corresponding values excluding the remaining fields or data from the event data/string:

id = b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09

start_date_time = 1638433382 (value always required)

end_date_time = null or 1638433491  (if value not present)

current = <value> (only if the field exist) (6 in the example)

total = <value> (6 in the example)

status_type = COMPLETED

bot_uri = repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot

I tried using <search query> | rex field=_raw "(?msi)(?<ev_field>\{.+\}$)"
| spath input=ev_field  to extract all the fields in the Event data, but did not change the search results. Any suggestion or help highly appreciated I am newbie to Splunk...

TIA

 

ssamant007_0-1638434187803.png

12/2/21
7:24:52.106 PM
 
2021-Dec-02 Thu 19:24:52.106 INFO [pool-12-thread-1] - com.automationanywhere.nodemanager.service.impl.NodeMessagingServiceImpl - {} - writeSuccess(NodeMessagingServiceImpl.java:395) - Message eventData { id: "b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09" bot_execution { start_date_time { seconds: 1638433382 nanos: 210329300 } end_date_time { seconds: 1638433491 nanos: 993822800 } progress { current: 6 total: 6 percentage: 100 } status_type: COMPLETED bot_uri: "repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot?fileId=1098948&workspace=PRIVATE" }} sent to CR successfully.

 

Labels (5)
0 Karma
1 Solution

Gr0und_Z3r0
Contributor

This is one way of doing it....
I've currently set end_date_time & Current to default to null if there is no value.
If you want you can set Current to 0 if it doesn't exist by adding one more line
| fillnull value=0 Current

 

| makeresults 
| eval _raw= "2021-Dec-02 Thu 19:24:52.106 INFO [pool-12-thread-1] - com.automationanywhere.nodemanager.service.impl.NodeMessagingServiceImpl - {} - writeSuccess(NodeMessagingServiceImpl.java:395) - Message eventData { id: \"b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09\" bot_execution { start_date_time { seconds: 1638433382 nanos: 210329300 } end_date_time { seconds: 1638433491 nanos: 993822800 } progress { current: 6 total: 6 percentage: 100 } status_type: COMPLETED bot_uri: \"repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot?fileId=1098948&workspace=PRIVATE\" }} sent to CR successfully." 
| rex field=_raw "id\:\s\"(?<ID>[a-z0-9\-\_]+)\"\s" 
| rex field=_raw "start\_date\_time\s\{\sseconds\:\s(?<start_date_time>[\d]+)\s" 
| rex field=_raw "end\_date\_time\s\{\sseconds\:\s(?<end_date_time>[\d]+)\s" 
| rex field=_raw "\{\scurrent\:\s(?<Current>[\d]+)\stotal" 
| rex field=_raw "\stotal\:\s(?<Total>[\d]+)\s" 
| rex field=_raw "status\_type\:\s(?<Status>[\w]+)\s" 
| rex field=_raw "bot_uri\:\s\"(?<bot_uri>.*)\?" 
| table _time _raw ID start_date_time end_date_time Current Total Status bot_uri 
| fillnull value=null end_date_time Current

 

Gr0und_Z3r0_0-1638446324206.png



If it helps, an upvote would be appreciated.

 

View solution in original post

PickleRick
SplunkTrust
SplunkTrust

Adding to all other good answers - what do you mean by excluding remaining fields/data?

 If you want to only operate further in the search on those extracted fields and will definitely not need the raw event, you might just

| fields - _raw

to tell splunk not to bother with the original event.

0 Karma

Gr0und_Z3r0
Contributor

This is one way of doing it....
I've currently set end_date_time & Current to default to null if there is no value.
If you want you can set Current to 0 if it doesn't exist by adding one more line
| fillnull value=0 Current

 

| makeresults 
| eval _raw= "2021-Dec-02 Thu 19:24:52.106 INFO [pool-12-thread-1] - com.automationanywhere.nodemanager.service.impl.NodeMessagingServiceImpl - {} - writeSuccess(NodeMessagingServiceImpl.java:395) - Message eventData { id: \"b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09\" bot_execution { start_date_time { seconds: 1638433382 nanos: 210329300 } end_date_time { seconds: 1638433491 nanos: 993822800 } progress { current: 6 total: 6 percentage: 100 } status_type: COMPLETED bot_uri: \"repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot?fileId=1098948&workspace=PRIVATE\" }} sent to CR successfully." 
| rex field=_raw "id\:\s\"(?<ID>[a-z0-9\-\_]+)\"\s" 
| rex field=_raw "start\_date\_time\s\{\sseconds\:\s(?<start_date_time>[\d]+)\s" 
| rex field=_raw "end\_date\_time\s\{\sseconds\:\s(?<end_date_time>[\d]+)\s" 
| rex field=_raw "\{\scurrent\:\s(?<Current>[\d]+)\stotal" 
| rex field=_raw "\stotal\:\s(?<Total>[\d]+)\s" 
| rex field=_raw "status\_type\:\s(?<Status>[\w]+)\s" 
| rex field=_raw "bot_uri\:\s\"(?<bot_uri>.*)\?" 
| table _time _raw ID start_date_time end_date_time Current Total Status bot_uri 
| fillnull value=null end_date_time Current

 

Gr0und_Z3r0_0-1638446324206.png



If it helps, an upvote would be appreciated.

 

ssamant007
Explorer

Thanks @Gr0und_Z3r0 . this is what I was looking for.

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Hi
it seems that your event's is not correct json format (e.g. missing : and , characters). Is it possible that logging system will fix those (the best option) or should you fix those (if many events with many formats, this will be quite hard task)?
r. Ismo
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Without fixing the source event you can try this with normal rex extractions like

| makeresults
| eval _raw = "INFO [pool-12-thread-1] - com.automationanywhere.nodemanager.service.impl.NodeMessagingServiceImpl - {} - writeSuccess(NodeMessagingServiceImpl.java:395) - Message eventData { id: \"b0ad6627-a6e1-4f5e-92f4-9c2deaa1ff2a_1cd4b06f83caac09\" bot_execution { start_date_time { seconds: 1638433382 nanos: 210329300 } end_date_time { seconds: 1638433491 nanos: 993822800 } progress { current: 6 total: 6 percentage: 100 } status_type: COMPLETED bot_uri: \"repository:///Automation%20Anywhere/Bots/Test%20A2019/AALogTestBot?fileId=1098948&workspace=PRIVATE\" }} sent to CR successfully."
``` Above create test event```

| rex "id: \"(?<id>[^\"]+)"
| rex "start_date_time { seconds: (?<start_date_time>\d+)"
| rex "end_date_time { seconds: (?<end_date_time>\d+)"
| rex "current: (?<current>\d+)"
| rex "total: (?<total>\d+)"
| rex "status_type: (?<status_type>\w+)"
| rex "bot_uri: \"(?<bot_uri>[^\"]+)"
| table id start_date_time end_date_time current total status_type bot_uri

If your events have fixed format you probably could combine some rex together, but then you should look from job inspector which mode is more efficient.

r. Ismo

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...