Getting Data In

Remove the field from a JSON ingested by Splunk

kgz25
New Member

When forwarding alerts to Splunk via SQS, it automatically pushes the entire JSON document into one field, called "BodyJson".

How would I extract actual fields out of this, so that they are not embedded?

{
     "Body": "{\n    \"field_1\": \"test\", \n    \"field_2\": \"10000\", \n    \"field_3\": \"2017-04-20T00:00:00.000Z\", \n    \"field_4\": \"10\"\n}",
     "BodyJson": {
          "field_1": "test",
          "field_2": 10000,
          "field_3": "2017-04-20T00:00:00.000Z",
          "field_4": 10
     }
}
Tags (1)
0 Karma

ansif
Motivator

| makeresults | eval _raw= "{
\"BodyJson\": {
\"field_1\": \"test\",
\"field_2\": 10000,
\"field_3\": \"2017-04-20T00:00:00.000Z\",
\"field_4\": 10
}
}"|spath

The above is working. Do you had that Body in the beginning of JSON?

0 Karma

dhirendra_singh
Loves-to-Learn Lots

Hi Ashif,

 

have you got any solution for access the fields using spath

0 Karma

kgz25
New Member

Body is at the beginning and then there is a BodyJson. What I need is to move everything that is in BodyJson into Body and remove BodyJson. Also this should be applied to every new event. SO, I assume I will need to modification of props.conf file.

Can you assist with this?

0 Karma

ansif
Motivator

Why you wanna move like this?May I know what you gonna achieve out of this movement?

0 Karma

kgz25
New Member

It is needed to extract the actual fields out of this, so that they are not embedded, but top-level fields that can be more easily used for search / reporting.

0 Karma

ansif
Motivator

Actual fields can be extracted easily with the above search. Moving to another attribute before extracting is an additional overhead. SPATH can easily extract fields from JSON.

0 Karma

kgz25
New Member

The problems is that Splunk puts the overall document into "Body.json". I need to completely get rid of this field, so the data is not put into there.

0 Karma

kgz25
New Member

hello....?

0 Karma

ansif
Motivator

I didn't get you.What you mean by this "The problems is that Splunk puts the overall document into "Body.json". I need to completely get rid of this field, so the data is not put into there"

0 Karma

kgz25
New Member

Please look at the example that I posted in my first post. There is a "Body" with the raw data and "BodyJson" with the exact same data, but structured. This is messy and I want to configure Splunk to not do that. Therefore, I need to get rid of the "BodyJson field" from all the events. I am not very well familiar with Splunk and after doing my research, I found out that it could be done by modifying props.conf file. Could you please assist in this?

0 Karma

wryanthomas
Communicator

Did you get this solved?

A) Splunk is already extracting fields in BodyJson for me (and yes, the field-names are long and undesirable because they are json tucked inside the json of the queue item).  It seems to me that creating aliases of those fields is the easy solution to what you've asked.  However...

B) What I'm needing to solve is, I think, related to what you were asking: The average size of the original event (i.e., the original size of "Body" field) is ~2k (pretty big). The result of the SQS payload is ~6k. This is largely due, of course, to the fact that the original event is getting sent both as a string (in Body) and as json (in BodyJson). And in my situation, we're talking millions of such events per day ... I need to trim off everything but Body or BodyJson -- whichever Splunk will most effectively parse effectively as json.

I'm wondering if you (or anyone) has already figured out how to do this sort of trimming-at-index-time for AWS SQS messages. (...to preserve and parse just the original message/body.)

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...