When forwarding alerts to Splunk via SQS, it automatically pushes the entire JSON document into one field, called "BodyJson".
How would I extract actual fields out of this, so that they are not embedded?
{
"Body": "{\n \"field_1\": \"test\", \n \"field_2\": \"10000\", \n \"field_3\": \"2017-04-20T00:00:00.000Z\", \n \"field_4\": \"10\"\n}",
"BodyJson": {
"field_1": "test",
"field_2": 10000,
"field_3": "2017-04-20T00:00:00.000Z",
"field_4": 10
}
}
| makeresults | eval _raw= "{
\"BodyJson\": {
\"field_1\": \"test\",
\"field_2\": 10000,
\"field_3\": \"2017-04-20T00:00:00.000Z\",
\"field_4\": 10
}
}"|spath
The above is working. Do you had that Body in the beginning of JSON?
Hi Ashif,
have you got any solution for access the fields using spath
Body is at the beginning and then there is a BodyJson. What I need is to move everything that is in BodyJson into Body and remove BodyJson. Also this should be applied to every new event. SO, I assume I will need to modification of props.conf file.
Can you assist with this?
Why you wanna move like this?May I know what you gonna achieve out of this movement?
It is needed to extract the actual fields out of this, so that they are not embedded, but top-level fields that can be more easily used for search / reporting.
Actual fields can be extracted easily with the above search. Moving to another attribute before extracting is an additional overhead. SPATH can easily extract fields from JSON.
The problems is that Splunk puts the overall document into "Body.json". I need to completely get rid of this field, so the data is not put into there.
hello....?
I didn't get you.What you mean by this "The problems is that Splunk puts the overall document into "Body.json". I need to completely get rid of this field, so the data is not put into there"
Please look at the example that I posted in my first post. There is a "Body" with the raw data and "BodyJson" with the exact same data, but structured. This is messy and I want to configure Splunk to not do that. Therefore, I need to get rid of the "BodyJson field" from all the events. I am not very well familiar with Splunk and after doing my research, I found out that it could be done by modifying props.conf file. Could you please assist in this?
Did you get this solved?
A) Splunk is already extracting fields in BodyJson for me (and yes, the field-names are long and undesirable because they are json tucked inside the json of the queue item). It seems to me that creating aliases of those fields is the easy solution to what you've asked. However...
B) What I'm needing to solve is, I think, related to what you were asking: The average size of the original event (i.e., the original size of "Body" field) is ~2k (pretty big). The result of the SQS payload is ~6k. This is largely due, of course, to the fact that the original event is getting sent both as a string (in Body) and as json (in BodyJson). And in my situation, we're talking millions of such events per day ... I need to trim off everything but Body or BodyJson -- whichever Splunk will most effectively parse effectively as json.
I'm wondering if you (or anyone) has already figured out how to do this sort of trimming-at-index-time for AWS SQS messages. (...to preserve and parse just the original message/body.)