some records are missing when I list by table; but when I query that specific event, I can find it.

[leonjxtan]

I have a trade message sourcetype in JSON, which I properly set up in props.conf and can query fine.

To do a reconciliation with my trade DB, in order to ensure all trade messages are fed to Splunk, I ran below query to extract all tradeID for May 4th:
sourcetype=foo |TradeEvent=NEW TradeDate="2017-05-04"
|table TradeID

Say from above table list, I found TradeID 123456 is missing. But if I search by:
sourcetype=foo TradeDate="2017-05-04" TradeID=123456
The event shows up!

I tried to check any setting was wrong. The sampling setting is set as "No Event Sampling"; time range is set as all time, etc. everything looks fine.

Could you help for my purpose of recon?

[3no]

1 - Give this search a try :

 sourcetype=foo TradeEvent=NEW TradeDate="2017-05-04" | search TradeID=* |table TradeID

If you see TradeID=123456 then to resolve the issue add those lines to your fields.conf :

[TradeID] 
INDEXED_VALUE= False

If this doesn't work, can you tell me if the value 123456 comes from the raw log or it's populated by an object knowledge (lookup, etc...) ?

[leonjxtan]

Hi I found more detailed symptom now.
If instead I specify the TradeID field, but rather search like below

sourcetype=foo 123456

The event shows up!
I check the event on GUI, and found that the GUI displays the event text (the log is in JSON format) as raw text, instead of showing as "syntax highlighted", and only SOME, but not other fields like TradeEvent and TradeID in the JSON log are listed under the log text.

I double checked and pasted the log text into JSONLint, and it is a valid JSON message.

Why does Splunk not index this message like other JSON event messages in my sourcetype?

p.s. to your question, yes the TradeID is in _raw log, and not a lookup field. The full spath is TradeEventObject.TradeID

[leonjxtan]

to add, the data size is 5 million events for "all time"

[adonio]

is the pipe before TradeEvent=NEW is part of the search?

[leonjxtan]

thanks for the reply. yes the "TradeEvent=NEW" was supposed to be in the 2nd search string. My bad I forgot to add it when I composed the dummy search string.

sourcetype=foo TradeEvent=NEW TradeDate="2017-05-04" TradeID=123456

[adonio]

try to run this search and see if you get the TradeID=123456 event

  sourcetype = foo TradeEvent=NEW | fields TradeDate TradeID

also, which mode are you searching in? verbose, smart or fast?

[leonjxtan]

I'm on smart mode.

[adonio]

just add to your search TradeId=* and that will tell splunk you want that field from all events
verify your results are correct
read here more about search modes:
https://docs.splunk.com/Documentation/SplunkCloud/6.5.1612/Search/Changethesearchmode