I have a trade message sourcetype in JSON, which I properly set up in props.conf and can query fine.
To do a reconciliation with my trade DB, in order to ensure all trade messages are fed to Splunk, I ran below query to extract all tradeID for May 4th:
sourcetype=foo |TradeEvent=NEW TradeDate="2017-05-04"
Say from above table list, I found TradeID 123456 is missing. But if I search by:
sourcetype=foo TradeDate="2017-05-04" TradeID=123456
The event shows up!
I tried to check any setting was wrong. The sampling setting is set as "No Event Sampling"; time range is set as all time, etc. everything looks fine.
Could you help for my purpose of recon?
1 - Give this search a try :
sourcetype=foo TradeEvent=NEW TradeDate="2017-05-04" | search TradeID=* |table TradeID
If you see TradeID=123456 then to resolve the issue add those lines to your fields.conf :
[TradeID] INDEXED_VALUE= False
If this doesn't work, can you tell me if the value 123456 comes from the raw log or it's populated by an object knowledge (lookup, etc...) ?
Hi I found more detailed symptom now.
If instead I specify the TradeID field, but rather search like below
The event shows up!
I check the event on GUI, and found that the GUI displays the event text (the log is in JSON format) as raw text, instead of showing as "syntax highlighted", and only SOME, but not other fields like TradeEvent and TradeID in the JSON log are listed under the log text.
I double checked and pasted the log text into JSONLint, and it is a valid JSON message.
Why does Splunk not index this message like other JSON event messages in my sourcetype?
p.s. to your question, yes the TradeID is in _raw log, and not a lookup field. The full spath is TradeEventObject.TradeID
thanks for the reply. yes the "TradeEvent=NEW" was supposed to be in the 2nd search string. My bad I forgot to add it when I composed the dummy search string.
sourcetype=foo TradeEvent=NEW TradeDate="2017-05-04" TradeID=123456
try to run this search and see if you get the TradeID=123456 event
sourcetype = foo TradeEvent=NEW | fields TradeDate TradeID
also, which mode are you searching in? verbose, smart or fast?
just add to your search
TradeId=* and that will tell splunk you want that field from all events
verify your results are correct
read here more about search modes: