topic Re: Why is Splunk not extracting all fields in JSON? in Getting Data In

Why is Splunk not extracting all fields in JSON?

LordSnooz — Wed, 17 Oct 2018 14:04:49 GMT

Hi,

I don't understand why Splunk show the field tag in List view and not in view Raw and Table.

Also, this field is not selectable... Why?

Line :

{"line":"[\u001b[37minfo\u001b[0m] k.m.a.c.BrokerViewCacheActor - Updating broker view...","source":"stdout","tag":"7b91119dbad4","attrs":{"appName":"kafka-manager","appType":"kafka-manager"}}

I have a screenshot of the problem I'm talking about :

https://www.dropbox.com/s/du5pkwqzwyi8miq/Capture%20d%E2%80%99%C3%A9cran%202018-10-17%20%C3%A0%2009.56.02.png?dl=0

Regards,

Re: Why is Splunk not extracting all fields in JSON?

MousumiChowdhur — Wed, 17 Oct 2018 15:02:38 GMT

Hey! Can you attach the screenshot? Also, if you could share a tad more information would be helpful to understand the problem.
Thanks.

Re: Why is Splunk not extracting all fields in JSON?

LordSnooz — Wed, 17 Oct 2018 15:42:51 GMT

Hey MousumiChowdhury,

Thx for your reply.

I don’t know why my link doesn’t work on my previous post, so the screenshot is there : https://www.dropbox.com/s/du5pkwqzwyi8miq/Capture%20d%E2%80%99%C3%A9cran%202018-10-17%20%C3%A0%2009.56.02.png?dl=0

My problem is very simple. I use Docker Container and I have configured Splunk logging drivers on my container to send logs through Splunk HTTP Event Collector. My Docker Log Driver is set up to send data in json format and Splunk seems to have received logs in good format. But, if I search for certain fields, like tag in my example, Splunk seem not extracted this field and no result resturn from search.

Why Splunk not extract all fields?

Regards

Re: Why is Splunk not extracting all fields in JSON?

kamlesh_vaghela — Wed, 17 Oct 2018 16:02:14 GMT

@LordSnooz,

Your provided JSON is proper and provided screen is confirming that tag must be extracted. You can try below search also for that.

| makeresults | eval _raw="{\"line\":\"[\u001b[37minfo\u001b[0m] k.m.a.c.BrokerViewCacheActor - Updating broker view...\",\"source\":\"stdout\",\"tag\":\"7b91119dbad4\",\"attrs\":{\"appName\":\"kafka-manager\",\"appType\":\"kafka-manager\"}}" | kv

I have a question regarding extraction. Specially if you had done any CIM related mapping. Can you please confirm that there are no extraction which can nullify the tag value? if any then you can search after removing such extraction. This is just for testing. 🙂

Re: Why is Splunk not extracting all fields in JSON?

LordSnooz — Wed, 17 Oct 2018 16:32:46 GMT

Thx @kamlesh_vaghela

You're right, Splunk is extracting all the fields with makeresults!

But, how I make simple search base on this field like this ? index=ecs-dev attrs.appName=ms-communicationservice tag=f47474ce8091

Re: Why is Splunk not extracting all fields in JSON?

LordSnooz — Wed, 17 Oct 2018 18:26:48 GMT

I found something, but I don't understand why... If I disable the SplunkApp Splunk Add-on for AWS, my field tag as automatically extracted...

Do you have a clue how to fix that?

Re: Why is Splunk not extracting all fields in JSON?

kamlesh_vaghela — Wed, 17 Oct 2018 20:13:07 GMT

@LordSnooz,

I think it is because tag is Splunk's Internal fields. I have did some workaround ad It works for me.

1) I have created temp sourcetype ad indexed your given sample into it using below search. You can skip this step if you have already these events.

| makeresults | eval _raw="{\"line\":\"[\u001b[37minfo\u001b[0m] k.m.a.c.BrokerViewCacheActor - Updating broker view...\",\"source\":\"stdout\",\"tag\":\"7b91119dbad4\",\"attrs\":{\"appName\":\"kafka-manager\",\"appType\":\"kafka-manager\"}}" | collect index=main sourcetype=temp

2) I have added eval to store original tag value in another field my_tag by adding EVAL under temp stanza in props.conf. Add eval in your existing sourcetype stanza.

[temp]
.
.
.
EVAL-my_tag = tag

3) execute search

index=main sourcetype=temp my_tag="7b91119dbad4"

Please try and let me know if it is working for you or not.

Re: Why is Splunk not extracting all fields in JSON?

LordSnooz — Thu, 18 Oct 2018 00:49:28 GMT

You rock! Your solution work perfectly if I create new source type.

I see two things.

1) If I use your solution, but with _json source type, it does not work. So it has a parameter that comes into conflict
2) This afternoon I discovered that by disabling the Splunk Add-on for AWS, Splunk extracts natively all fields of json input without problems.

I have two solutions, use your workaround or discover why this App change the behaviour of the _json source type.

I'm not a Splunk expert... I probably used your solution lol

Thank you for your time, I appreciate!

Re: Why is Splunk not extracting all fields in JSON?

marend — Wed, 30 Sep 2020 00:51:04 GMT

Hi @LordSnooz

For this purpose, I going to use the Splunk _json sourcetype default settings (It works in my case)
My sourcetype name for this example will be "test"

A workaround to do this would be the following:

1) Create a custom sourcetype
2) Configure your custom sourcetype (in opt/splunk/etc/system/local/props.conf) as:

[ test ]
SHOULD_LINEMERGE=true
LINE_BREAKER=([\r\n]+)
NO_BINARY_CHECK=true
CHARSET=AUTO
INDEXED_EXTRACTIONS=json
KV_MODE=none
category=Structured
description=JavaScript Object Notation format. For more information, visit http://json.org/
disabled=false
pulldown_type=true
EVAL-my_tag = tag

3) Configure your data input (Using the sourcetype created, [ test ] in my case )
4) Search your results

index=< your_index_name > sourcetype=test my_tag="7b91119dbad4"

Please try and let me know if it is working for you or not.