topic Re: Remove the field from a JSON ingested by Splunk in Getting Data In

Remove the field from a JSON ingested by Splunk

kgz25 — Fri, 08 Jun 2018 20:35:43 GMT

When forwarding alerts to Splunk via SQS, it automatically pushes the entire JSON document into one field, called "BodyJson".

How would I extract actual fields out of this, so that they are not embedded?

{
     "Body": "{\n    \"field_1\": \"test\", \n    \"field_2\": \"10000\", \n    \"field_3\": \"2017-04-20T00:00:00.000Z\", \n    \"field_4\": \"10\"\n}",
     "BodyJson": {
          "field_1": "test",
          "field_2": 10000,
          "field_3": "2017-04-20T00:00:00.000Z",
          "field_4": 10
     }
}

Re: Remove the field from a JSON ingested by Splunk

ansif — Tue, 29 Sep 2020 19:57:41 GMT

| makeresults | eval _raw= "{
\"BodyJson\": {
\"field_1\": \"test\",
\"field_2\": 10000,
\"field_3\": \"2017-04-20T00:00:00.000Z\",
\"field_4\": 10
}
}"|spath

The above is working. Do you had that Body in the beginning of JSON?

Re: Remove the field from a JSON ingested by Splunk

kgz25 — Mon, 11 Jun 2018 15:33:41 GMT

Body is at the beginning and then there is a BodyJson. What I need is to move everything that is in BodyJson into Body and remove BodyJson. Also this should be applied to every new event. SO, I assume I will need to modification of props.conf file.

Can you assist with this?

Re: Remove the field from a JSON ingested by Splunk

ansif — Mon, 11 Jun 2018 15:38:24 GMT

Why you wanna move like this?May I know what you gonna achieve out of this movement?

Re: Remove the field from a JSON ingested by Splunk

kgz25 — Mon, 11 Jun 2018 18:34:27 GMT

It is needed to extract the actual fields out of this, so that they are not embedded, but top-level fields that can be more easily used for search / reporting.

Re: Remove the field from a JSON ingested by Splunk

ansif — Tue, 12 Jun 2018 06:51:56 GMT

Actual fields can be extracted easily with the above search. Moving to another attribute before extracting is an additional overhead. SPATH can easily extract fields from JSON.

Re: Remove the field from a JSON ingested by Splunk

kgz25 — Wed, 13 Jun 2018 21:44:15 GMT

The problems is that Splunk puts the overall document into "Body.json". I need to completely get rid of this field, so the data is not put into there.

Re: Remove the field from a JSON ingested by Splunk

kgz25 — Mon, 18 Jun 2018 13:42:57 GMT

hello....?

Re: Remove the field from a JSON ingested by Splunk

ansif — Mon, 18 Jun 2018 14:04:25 GMT

I didn't get you.What you mean by this "The problems is that Splunk puts the overall document into "Body.json". I need to completely get rid of this field, so the data is not put into there"

Re: Remove the field from a JSON ingested by Splunk

kgz25 — Mon, 18 Jun 2018 14:31:28 GMT

Please look at the example that I posted in my first post. There is a "Body" with the raw data and "BodyJson" with the exact same data, but structured. This is messy and I want to configure Splunk to not do that. Therefore, I need to get rid of the "BodyJson field" from all the events. I am not very well familiar with Splunk and after doing my research, I found out that it could be done by modifying props.conf file. Could you please assist in this?

Re: Remove the field from a JSON ingested by Splunk

dhirendra_singh — Fri, 05 Mar 2021 18:03:35 GMT

Hi Ashif,

have you got any solution for access the fields using spath

Re: Remove the field from a JSON ingested by Splunk

wryanthomas — Sat, 06 Mar 2021 17:30:43 GMT

Did you get this solved?

A) Splunk is already extracting fields in BodyJson for me (and yes, the field-names are long and undesirable because they are json tucked inside the json of the queue item). It seems to me that creating aliases of those fields is the easy solution to what you've asked. However...

B) What I'm needing to solve is, I think, related to what you were asking: The average size of the original event (i.e., the original size of "Body" field) is ~2k (pretty big). The result of the SQS payload is ~6k. This is largely due, of course, to the fact that the original event is getting sent both as a string (in Body) and as json (in BodyJson). And in my situation, we're talking millions of such events per day ... I need to trim off everything but Body or BodyJson -- whichever Splunk will most effectively parse effectively as json.

I'm wondering if you (or anyone) has already figured out how to do this sort of trimming-at-index-time for AWS SQS messages. (...to preserve and parse just the original message/body.)