Splunk - stop auto indexing JSON

[namrithadeepak]

Hi,

I have a log that looks like the below,

2019-02-27 09:40:23,312 | INFO  | [myapp-metrics-publisher] | [myapp.core.instrument.CustomMetrics.CustomRegistry] | MY_APP | {payload={headers={content-type=application/json}, method=POST, body={userOrder={order=[{type={orderCd=ELECTRONICS, symbolCd=EL159, market={countryCd=MFD}}, orderInstruction={orderTypeCd=MARKET, deliveryCd=EXPEDITED, weekendInd=false, actionCd=BUY, returnPolicyInd=false, allOrNoneInd=false, dollarAmt=222.11, residencyCd=ON, payOnDeliveryInd=false}, account={accountNum=1111F}, traderInfo={behalfCd=AMA, traderRRId=AMAEL6, fxRate=11}, commission={deliveryGauranteedInd=true}, orderControl={proNonProInd=false, prospectusInd=true}}], alternativeUserId=ABC01, strategyCd=SPL, cashbackInd=true}, conditionRapid={acknowledgeActionCd=AUTO}}, url=http://100.100.100.100:8080/v1/userorders}, headers={sequenceNumber=1, orderExecutionId=1a2b3c4d-5678-91e0-11f2-1234567g890h, file_name=20190225-143730.trigger, sequenceSize=1, jms_destination=MY.SAMPLE.DESTINATION.00, JMSXUserID=AMAEL101145, Solace_JMS_Prop_IS_Reply_Message=false, priority=0, jms_timestamp=1551123465128, file_relativePath=20190225-143730.trigger, JMS_Solace_isXML=true, jms_redelivered=false, JMS_Solace_DeliverToOne=false, orderExecutionId=1a2b3c4d-5678-91e0-11f2-1234567g890h, JMS_Solace_ElidingEligible=false, orderId=1a2b3c4d-5678-91e0-11f2-1234567g890h, JMS_Solace_DeadMsgQueueEligible=false, traceId=1a2b3c4d-5678-9e0f1g2h-3456, firstTrade=true, id=1ab2345-67c8-90d1-23e4-ff5678901234, contentType=application/json, jms_messageId=ID:01.23.456.789abc24a0:0, timestamp=1551123465399}} | userId = [] | orderId = [] | process.files.open{} value=37 files

I have 2 questions:

• Part of the log message is JSON and part of it is regular logging. Will Splunk automatically index the JSON and give me fields like orderCd, symbolCd etc. (although only part of the message is JSON)?
• Assume that I convert the whole log line to JSON,I do not want my payload to be indexed into individual fields i.e. I want Splunk to give me the field payload = <the whole payload>, and no fields called orderCd, symbolCd etc.

Thanks,
Namritha

[lakshman239]

for 1 - no splunk will treat the whole line as an event.
for 2 - if you convert the log into json (including timestamp), splunk will use json sourcetype to parse the complete event. you can then use spath to extract the fields you want.

[namrithadeepak]

Thankyou lakshman, just to clarify, if my log event is:

{
timestamp="2019-02-27 09:40:23,312" ,
level="INFO",
class="myClassName.class",
payload = {
"field1" : "value 1",
"field2" : "value 2",
.
.
.
"field100": "value100"
}
}
Since Splunk will automatically extract fields for JSON format, I want the extracted fields to be timestamp, level, class, payload.

I do not want nested fields field1, field2 . . . field100 to be extracted.

Will spath give me this behavior?

[lakshman239]

Yes, that will be the behaviour. You can your sample (like the format above) and upload to your dev instance with json sourcetype and you can see how it gets parsed/indexed/extracted.