We are having some odd search results. When the query is ran in Verbose mode all fields return as expected, however when the same query is ran in Smart or Fast mode only the default fields are returned and the two fields extracted at search time via the "| rex" command. Anyone run into this before?
EDIT
Query used:
(index=foo sourcetype=bar) OR (index=metrics sourcetype=stats
[ search index=foo sourcetype=bar
| mvexpand combine
| rex field=combine (?<id>\d+)=(?<hits>\d+)
| dedup id site
| fields id site] earliest=-1y)
| fillnull combine
| mvexpand combine
| rex field=combine (?<id>\d+)=(?<hits>\d+)
| eventstats latest(ip) as IP latest(info_id) as Info_ID by id, site
| where isnotnull(hits)
| table _time hits id site IP Info_ID
When my search runs in Verbose mode I see all fields "_time hits id site IP Info_ID" However when I run it in Fast mode I am missing IP and Info_ID.
If I remove the "| table" section altogether and run it in fast mode the query runs and all interesting fields are populated on the left side of the events.
Splunk JIRA SPL-153269
A configuration added somewhere between Splunk Enterprise versions 6.4.? and 7.0.2 introduced an issue where using a macro with several lookups against the same lookup table results in only a single match attempt with subsequent matches against the lookup table being skipped.
Make the following configuration change to limits.conf:
[search_optimization::projection_elimination] cmds_black_list = lookup
There should not be a significant performance hit since this is just reverting this configuration to that in a previous version of Splunk.
Fix has been tested and confirmed in my environment, under these specific test conditions. I know the problem didn't exist under some version of 6.x and started in some version of 7.x, I just don't recall which upgrade specifically broke the macro/lookups. I am not sure if it resolves other similar behavior observed under different conditions.
it would be interesting to see if testing shows this resolves the issue
it's possible that this specific config doesn't but another option in the stanza would
@hagjos43 ping me on linkedin if you want to try to test this or identify some other possible config that looks promising
also...you could ping support on this and reference the JIRA, they might be able to get a faster answer to you on your issue if they are at least related and this config doesn't help
Try adding info_id=* ip=*
to the appropriate place to force splunk to care about those fields early on, like this maybe:
(index=foo sourcetype=bar) OR (index=metrics sourcetype=stats info_id=* ip=*
[ search index=foo sourcetype=bar
| mvexpand combine
| rex field=combine (?<id>\d+)=(?<hits>\d+)
| dedup id site
| fields id site] earliest=-1y)
| fillnull combine
| mvexpand combine
| rex field=combine (?<id>\d+)=(?<hits>\d+)
| eventstats latest(ip) as IP latest(info_id) as Info_ID by id, site
| where isnotnull(hits)
| table _time hits id site IP Info_ID
Thanks, unfortunately this didn't do the trick.
Curious: Is any of your data summary-indexed?
The search is actually what is populating a summary index.
You should pass the fields
command to make your fields show up under "Interested Fields" while in fast mode
index=... sourcetype=... | fields + foo
See updated original question. I've included the query.
I added "|fields *" and it gave me the results i wanted.
Your fields are still there, but may not be displayed under the "Interesting Fields" column. For instance, if you have a field named "foo" in your search results, but it doesn't show up under "Interesting Fields" due to being in Fast mode, you can still do a search like this:
index=main foo=*
Fast mode prioritizes the performance of the search and does not return nonessential field or event data. This means that the search returns what is essential and required.
Verbose mode returns all of the field and event data it possibly can, even if it means the search takes longer to complete, and even if the search includes reporting commands.
Smart mode tries to make a determination based on your search (reporting vs. transforming vs. generating etc.)
More information can be found on search modes here -> https://docs.splunk.com/Documentation/Splunk/latest/Search/Changethesearchmode
Thanks for your reply, however that doesn't do the trick. I should elaborate....the interesting fields DO in-fact populate in fast mode. But when I pipe the output to a table it does not properly show the statistics table.