Splunk Search

some records are missing when I list by table; but when I query that specific event, I can find it.

leonjxtan
Path Finder

I have a trade message sourcetype in JSON, which I properly set up in props.conf and can query fine.

To do a reconciliation with my trade DB, in order to ensure all trade messages are fed to Splunk, I ran below query to extract all tradeID for May 4th:
sourcetype=foo |TradeEvent=NEW TradeDate="2017-05-04"
|table TradeID

Say from above table list, I found TradeID 123456 is missing. But if I search by:
sourcetype=foo TradeDate="2017-05-04" TradeID=123456
The event shows up!

I tried to check any setting was wrong. The sampling setting is set as "No Event Sampling"; time range is set as all time, etc. everything looks fine.

Could you help for my purpose of recon?

Tags (1)
0 Karma

3no
Communicator

1 - Give this search a try :

 sourcetype=foo TradeEvent=NEW TradeDate="2017-05-04" | search TradeID=* |table TradeID

If you see TradeID=123456 then to resolve the issue add those lines to your fields.conf :

[TradeID] 
INDEXED_VALUE= False

If this doesn't work, can you tell me if the value 123456 comes from the raw log or it's populated by an object knowledge (lookup, etc...) ?

0 Karma

leonjxtan
Path Finder

Hi I found more detailed symptom now.
If instead I specify the TradeID field, but rather search like below

sourcetype=foo 123456

The event shows up!
I check the event on GUI, and found that the GUI displays the event text (the log is in JSON format) as raw text, instead of showing as "syntax highlighted", and only SOME, but not other fields like TradeEvent and TradeID in the JSON log are listed under the log text.

I double checked and pasted the log text into JSONLint, and it is a valid JSON message.

Why does Splunk not index this message like other JSON event messages in my sourcetype?

p.s. to your question, yes the TradeID is in _raw log, and not a lookup field. The full spath is TradeEventObject.TradeID

0 Karma

leonjxtan
Path Finder

to add, the data size is 5 million events for "all time"

0 Karma

adonio
Ultra Champion

is the pipe before TradeEvent=NEW is part of the search?

0 Karma

leonjxtan
Path Finder

thanks for the reply. yes the "TradeEvent=NEW" was supposed to be in the 2nd search string. My bad I forgot to add it when I composed the dummy search string.

sourcetype=foo TradeEvent=NEW TradeDate="2017-05-04" TradeID=123456
0 Karma

adonio
Ultra Champion

try to run this search and see if you get the TradeID=123456 event

  sourcetype = foo TradeEvent=NEW | fields TradeDate TradeID

also, which mode are you searching in? verbose, smart or fast?

0 Karma

leonjxtan
Path Finder

I'm on smart mode.

0 Karma

adonio
Ultra Champion

just add to your search TradeId=* and that will tell splunk you want that field from all events
verify your results are correct
read here more about search modes:
https://docs.splunk.com/Documentation/SplunkCloud/6.5.1612/Search/Changethesearchmode

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...