Getting Data In

Remove the field from a JSON ingested by Splunk

kgz25
New Member

When forwarding alerts to Splunk via SQS, it automatically pushes the entire JSON document into one field, called "BodyJson".

How would I extract actual fields out of this, so that they are not embedded?

{
     "Body": "{\n    \"field_1\": \"test\", \n    \"field_2\": \"10000\", \n    \"field_3\": \"2017-04-20T00:00:00.000Z\", \n    \"field_4\": \"10\"\n}",
     "BodyJson": {
          "field_1": "test",
          "field_2": 10000,
          "field_3": "2017-04-20T00:00:00.000Z",
          "field_4": 10
     }
}
Tags (1)
0 Karma

ansif
Motivator

| makeresults | eval _raw= "{
\"BodyJson\": {
\"field_1\": \"test\",
\"field_2\": 10000,
\"field_3\": \"2017-04-20T00:00:00.000Z\",
\"field_4\": 10
}
}"|spath

The above is working. Do you had that Body in the beginning of JSON?

0 Karma

dhirendra_singh
Loves-to-Learn Lots

Hi Ashif,

 

have you got any solution for access the fields using spath

0 Karma

kgz25
New Member

Body is at the beginning and then there is a BodyJson. What I need is to move everything that is in BodyJson into Body and remove BodyJson. Also this should be applied to every new event. SO, I assume I will need to modification of props.conf file.

Can you assist with this?

0 Karma

ansif
Motivator

Why you wanna move like this?May I know what you gonna achieve out of this movement?

0 Karma

kgz25
New Member

It is needed to extract the actual fields out of this, so that they are not embedded, but top-level fields that can be more easily used for search / reporting.

0 Karma

ansif
Motivator

Actual fields can be extracted easily with the above search. Moving to another attribute before extracting is an additional overhead. SPATH can easily extract fields from JSON.

0 Karma

kgz25
New Member

The problems is that Splunk puts the overall document into "Body.json". I need to completely get rid of this field, so the data is not put into there.

0 Karma

kgz25
New Member

hello....?

0 Karma

ansif
Motivator

I didn't get you.What you mean by this "The problems is that Splunk puts the overall document into "Body.json". I need to completely get rid of this field, so the data is not put into there"

0 Karma

kgz25
New Member

Please look at the example that I posted in my first post. There is a "Body" with the raw data and "BodyJson" with the exact same data, but structured. This is messy and I want to configure Splunk to not do that. Therefore, I need to get rid of the "BodyJson field" from all the events. I am not very well familiar with Splunk and after doing my research, I found out that it could be done by modifying props.conf file. Could you please assist in this?

0 Karma

wryanthomas
Contributor

Did you get this solved?

A) Splunk is already extracting fields in BodyJson for me (and yes, the field-names are long and undesirable because they are json tucked inside the json of the queue item).  It seems to me that creating aliases of those fields is the easy solution to what you've asked.  However...

B) What I'm needing to solve is, I think, related to what you were asking: The average size of the original event (i.e., the original size of "Body" field) is ~2k (pretty big). The result of the SQS payload is ~6k. This is largely due, of course, to the fact that the original event is getting sent both as a string (in Body) and as json (in BodyJson). And in my situation, we're talking millions of such events per day ... I need to trim off everything but Body or BodyJson -- whichever Splunk will most effectively parse effectively as json.

I'm wondering if you (or anyone) has already figured out how to do this sort of trimming-at-index-time for AWS SQS messages. (...to preserve and parse just the original message/body.)

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...