topic Re: Splunk - stop auto indexing JSON in Getting Data In

Splunk - stop auto indexing JSON

namrithadeepak — Wed, 27 Feb 2019 15:26:44 GMT

Hi,

I have a log that looks like the below,

2019-02-27 09:40:23,312 | INFO  | [myapp-metrics-publisher] | [myapp.core.instrument.CustomMetrics.CustomRegistry] | MY_APP | {payload={headers={content-type=application/json}, method=POST, body={userOrder={order=[{type={orderCd=ELECTRONICS, symbolCd=EL159, market={countryCd=MFD}}, orderInstruction={orderTypeCd=MARKET, deliveryCd=EXPEDITED, weekendInd=false, actionCd=BUY, returnPolicyInd=false, allOrNoneInd=false, dollarAmt=222.11, residencyCd=ON, payOnDeliveryInd=false}, account={accountNum=1111F}, traderInfo={behalfCd=AMA, traderRRId=AMAEL6, fxRate=11}, commission={deliveryGauranteedInd=true}, orderControl={proNonProInd=false, prospectusInd=true}}], alternativeUserId=ABC01, strategyCd=SPL, cashbackInd=true}, conditionRapid={acknowledgeActionCd=AUTO}}, url=http://100.100.100.100:8080/v1/userorders}, headers={sequenceNumber=1, orderExecutionId=1a2b3c4d-5678-91e0-11f2-1234567g890h, file_name=20190225-143730.trigger, sequenceSize=1, jms_destination=MY.SAMPLE.DESTINATION.00, JMSXUserID=AMAEL101145, Solace_JMS_Prop_IS_Reply_Message=false, priority=0, jms_timestamp=1551123465128, file_relativePath=20190225-143730.trigger, JMS_Solace_isXML=true, jms_redelivered=false, JMS_Solace_DeliverToOne=false, orderExecutionId=1a2b3c4d-5678-91e0-11f2-1234567g890h, JMS_Solace_ElidingEligible=false, orderId=1a2b3c4d-5678-91e0-11f2-1234567g890h, JMS_Solace_DeadMsgQueueEligible=false, traceId=1a2b3c4d-5678-9e0f1g2h-3456, firstTrade=true, id=1ab2345-67c8-90d1-23e4-ff5678901234, contentType=application/json, jms_messageId=ID:01.23.456.789abc24a0:0, timestamp=1551123465399}} | userId = [] | orderId = [] | process.files.open{} value=37 files

I have 2 questions:

Part of the log message is JSON and part of it is regular logging. Will Splunk automatically index the JSON and give me fields like orderCd, symbolCd etc. (although only part of the message is JSON)?
Assume that I convert the whole log line to JSON,I do not want my payload to be indexed into individual fields i.e. I want Splunk to give me the field payload = <the whole payload>, and no fields called orderCd, symbolCd etc.

Thanks,
Namritha

Re: Splunk - stop auto indexing JSON

lakshman239 — Wed, 27 Feb 2019 15:37:58 GMT

for 1 - no splunk will treat the whole line as an event.
for 2 - if you convert the log into json (including timestamp), splunk will use json sourcetype to parse the complete event. you can then use spath to extract the fields you want.

Re: Splunk - stop auto indexing JSON

namrithadeepak — Wed, 27 Feb 2019 18:14:30 GMT

Thankyou lakshman, just to clarify, if my log event is:

{
timestamp="2019-02-27 09:40:23,312" ,
level="INFO",
class="myClassName.class",
payload = {
"field1" : "value 1",
"field2" : "value 2",
.
.
.
"field100": "value100"
}
}
Since Splunk will automatically extract fields for JSON format, I want the extracted fields to be timestamp, level, class, payload.

I do not want nested fields field1, field2 . . . field100 to be extracted.

Will spath give me this behavior?

Re: Splunk - stop auto indexing JSON

lakshman239 — Wed, 27 Feb 2019 20:15:19 GMT

Yes, that will be the behaviour. You can your sample (like the format above) and upload to your dev instance with json sourcetype and you can see how it gets parsed/indexed/extracted.