topic Re: some records are missing when I list by table; but when I query that specific event, I can find it. in Splunk Search

some records are missing when I list by table; but when I query that specific event, I can find it.

leonjxtan — Tue, 09 May 2017 06:53:10 GMT

I have a trade message sourcetype in JSON, which I properly set up in props.conf and can query fine.

To do a reconciliation with my trade DB, in order to ensure all trade messages are fed to Splunk, I ran below query to extract all tradeID for May 4th:
sourcetype=foo |TradeEvent=NEW TradeDate="2017-05-04"
|table TradeID

Say from above table list, I found TradeID 123456 is missing. But if I search by:
sourcetype=foo TradeDate="2017-05-04" TradeID=123456
The event shows up!

I tried to check any setting was wrong. The sampling setting is set as "No Event Sampling"; time range is set as all time, etc. everything looks fine.

Could you help for my purpose of recon?

Re: some records are missing when I list by table; but when I query that specific event, I can find it.

leonjxtan — Tue, 09 May 2017 07:15:04 GMT

to add, the data size is 5 million events for "all time"

Re: some records are missing when I list by table; but when I query that specific event, I can find it.

adonio — Tue, 09 May 2017 12:37:13 GMT

is the pipe before TradeEvent=NEW is part of the search?

Re: some records are missing when I list by table; but when I query that specific event, I can find it.

leonjxtan — Tue, 09 May 2017 13:27:57 GMT

thanks for the reply. yes the "TradeEvent=NEW" was supposed to be in the 2nd search string. My bad I forgot to add it when I composed the dummy search string.

sourcetype=foo TradeEvent=NEW TradeDate="2017-05-04" TradeID=123456

Re: some records are missing when I list by table; but when I query that specific event, I can find it.

adonio — Tue, 09 May 2017 13:34:13 GMT

try to run this search and see if you get the TradeID=123456 event

  sourcetype = foo TradeEvent=NEW | fields TradeDate TradeID

also, which mode are you searching in? verbose, smart or fast?

Re: some records are missing when I list by table; but when I query that specific event, I can find it.

leonjxtan — Tue, 09 May 2017 13:49:11 GMT

I'm on smart mode.

Re: some records are missing when I list by table; but when I query that specific event, I can find it.

adonio — Tue, 09 May 2017 13:52:34 GMT

just add to your search TradeId=* and that will tell splunk you want that field from all events
verify your results are correct
read here more about search modes:
https://docs.splunk.com/Documentation/SplunkCloud/6.5.1612/Search/Changethesearchmode

Re: some records are missing when I list by table; but when I query that specific event, I can find it.

3no — Tue, 09 May 2017 14:30:23 GMT

1 - Give this search a try :

 sourcetype=foo TradeEvent=NEW TradeDate="2017-05-04" | search TradeID=* |table TradeID

If you see TradeID=123456 then to resolve the issue add those lines to your fields.conf :

[TradeID] 
INDEXED_VALUE= False

If this doesn't work, can you tell me if the value 123456 comes from the raw log or it's populated by an object knowledge (lookup, etc...) ?

Re: some records are missing when I list by table; but when I query that specific event, I can find it.

leonjxtan — Wed, 10 May 2017 07:10:57 GMT

Hi I found more detailed symptom now.
If instead I specify the TradeID field, but rather search like below

sourcetype=foo 123456

The event shows up!
I check the event on GUI, and found that the GUI displays the event text (the log is in JSON format) as raw text, instead of showing as "syntax highlighted", and only SOME, but not other fields like TradeEvent and TradeID in the JSON log are listed under the log text.

I double checked and pasted the log text into JSONLint, and it is a valid JSON message.

Why does Splunk not index this message like other JSON event messages in my sourcetype?

p.s. to your question, yes the TradeID is in _raw log, and not a lookup field. The full spath is TradeEventObject.TradeID