Archive

some records are missing when I list by table; but when I query that specific event, I can find it.

Path Finder

I have a trade message sourcetype in JSON, which I properly set up in props.conf and can query fine.

To do a reconciliation with my trade DB, in order to ensure all trade messages are fed to Splunk, I ran below query to extract all tradeID for May 4th:
sourcetype=foo |TradeEvent=NEW TradeDate="2017-05-04"
|table TradeID

Say from above table list, I found TradeID 123456 is missing. But if I search by:
sourcetype=foo TradeDate="2017-05-04" TradeID=123456
The event shows up!

I tried to check any setting was wrong. The sampling setting is set as "No Event Sampling"; time range is set as all time, etc. everything looks fine.

Could you help for my purpose of recon?

Tags (1)
0 Karma

Communicator

1 - Give this search a try :

 sourcetype=foo TradeEvent=NEW TradeDate="2017-05-04" | search TradeID=* |table TradeID

If you see TradeID=123456 then to resolve the issue add those lines to your fields.conf :

[TradeID] 
INDEXED_VALUE= False

If this doesn't work, can you tell me if the value 123456 comes from the raw log or it's populated by an object knowledge (lookup, etc...) ?

0 Karma

Path Finder

Hi I found more detailed symptom now.
If instead I specify the TradeID field, but rather search like below

sourcetype=foo 123456

The event shows up!
I check the event on GUI, and found that the GUI displays the event text (the log is in JSON format) as raw text, instead of showing as "syntax highlighted", and only SOME, but not other fields like TradeEvent and TradeID in the JSON log are listed under the log text.

I double checked and pasted the log text into JSONLint, and it is a valid JSON message.

Why does Splunk not index this message like other JSON event messages in my sourcetype?

p.s. to your question, yes the TradeID is in _raw log, and not a lookup field. The full spath is TradeEventObject.TradeID

0 Karma

Path Finder

to add, the data size is 5 million events for "all time"

0 Karma

SplunkTrust
SplunkTrust

is the pipe before TradeEvent=NEW is part of the search?

0 Karma

Path Finder

thanks for the reply. yes the "TradeEvent=NEW" was supposed to be in the 2nd search string. My bad I forgot to add it when I composed the dummy search string.

sourcetype=foo TradeEvent=NEW TradeDate="2017-05-04" TradeID=123456
0 Karma

SplunkTrust
SplunkTrust

try to run this search and see if you get the TradeID=123456 event

  sourcetype = foo TradeEvent=NEW | fields TradeDate TradeID

also, which mode are you searching in? verbose, smart or fast?

0 Karma

Path Finder

I'm on smart mode.

0 Karma

SplunkTrust
SplunkTrust

just add to your search TradeId=* and that will tell splunk you want that field from all events
verify your results are correct
read here more about search modes:
https://docs.splunk.com/Documentation/SplunkCloud/6.5.1612/Search/Changethesearchmode

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!