topic Re: Search in fast vs smart modes does not return same number of events in Splunk Search

Search in fast vs smart modes does not return same number of events

tomasmoser — Thu, 05 Dec 2019 09:40:09 GMT

We ran into a problem where a search in smart mode returns 6 events, while the same search in fast mode returns 2 events even if we enrich search in fast mode with some more fields we want to extract on top fields extracted by default. No effect. Searches are very simple – no transforming commands at the end.

We found out that in fast mode if we add

“| fields *”

it automagically returns all 6 events we expected. These searches run in dashboards/reports – so by default in FAST. What does adding |fields * have to do with the search? Isn’t it the same as Smart/Verbose after that? Does not make sense to me.

I read the docs detailing difference between fast vs. smart.
https://docs.splunk.com/Documentation/Splunk/8.0.0/Search/Changethesearchmode

Still I do not get it.

BTW: The additional fields in events we want to see in FAST are extracted using EXTACT and REPORT sin props.conf + corresponding stanzas in transforms.conf.

Why is this happening?

Tomas

Re: Search in fast vs smart modes does not return same number of events

kamlesh_vaghela — Thu, 05 Dec 2019 09:51:00 GMT

@tomasmoser

I will not suggest | fields * but you can use | fields by mentioning required fields.
Like, YOUR_SEARCH | fields F1 F2 F3 etc

Re: Search in fast vs smart modes does not return same number of events

gcusello — Thu, 05 Dec 2019 10:00:00 GMT

Hi @tomasmoser,
this was a bug in version 6.2, that was solved in the following releases!

Only one question: what's the time period of your search?
Please try non using earlies=now, in other words: use an old time period as yesterday or last hour and check if you have the same error.
At the same time the hint of @kamlesh_vaghela ·is fully correct: use only the fields you need!

If you continue to have the problem, open a ticket to Splunk Support: This helps all the Splunk users!

Ciao.
Giuseppe

Re: Search in fast vs smart modes does not return same number of events

Sukisen1981 — Thu, 05 Dec 2019 10:22:34 GMT

adding to what @gcusello says, I can confirm that he is correct. I am using the same code in 7.x vs 6.x.
I started to notice that if i ran the search in verbose mode (6.x) it was returning 1-2 events more thatn my dashboard results, now when i go to 7.x(at least for this particular instance) there is no difference.
This was a big bug in my opinion, since users would not even know that they are missing events in a dashboard but looks to have been fixed now

Re: Search in fast vs smart modes does not return same number of events

tomasmoser — Thu, 05 Dec 2019 11:43:54 GMT

One important fact. We are extracting fields using following "special" hack

[(?::){0}k8s*]
EXTRACT-uno_fields = ^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}.\d{3}Z\s+(?P<severity>\w+)\s+(?P<logger>[^\s]+)\s+(?:(?P<th
readId>[^\s]+)?\s+)?\[(?P<thread>[^\]]+)\]\s+-\s+(?P<message>.*)
REPORT-uno_thread_fields = uno_thread_fields

EVAL-severity = if(isnotnull(severity), severity, "n/a")
EVAL-traceId = if(isnotnull(traceId), traceId, "n/a")
EVAL-parentId = if(isnotnull(parentId), parentId, "n/a")
EVAL-spanId = if(isnotnull(spanId), spanId, "n/a")
EVAL-logger = if(isnotnull(logger), logger, "n/a")

# temporarily for backward compatibility
FIELDALIAS-uno_logLevel = severity as uno_logLevel
FIELDALIAS-uno_logger = logger as uno_logger
FIELDALIAS-uno_thread = thread as uno_thread
FIELDALIAS-uno_message = message as uno_message
FIELDALIAS-uno_time = _time as uno_time

Re: Search in fast vs smart modes does not return same number of events

gcusello — Thu, 05 Dec 2019 11:52:10 GMT

do you use one of these fields as a filter in your search?

Ciao.
Giuseppe

Re: Search in fast vs smart modes does not return same number of events

tomasmoser — Thu, 05 Dec 2019 13:26:00 GMT

Yes, we use simple searches like

`index=blabla sourcetype=blabla`.

This does not work.

But if I explicitely add a filter with another field to be extracted it DOES NOT work too.

index=blabla sourcetype=blabla spanId=*

Then if I add | fields * it immediately starts working.

index=blabla sourcetype=blabla spanId=*
| fields *

Re: Search in fast vs smart modes does not return same number of events

tomasmoser — Thu, 05 Dec 2019 13:27:48 GMT

We tried that. Does not work. Only "|fields *" works.

Re: Search in fast vs smart modes does not return same number of events

gcusello — Thu, 05 Dec 2019 14:01:58 GMT

Hi @tomasmoser,
the problem is that in Fast Mode you can use only the basic fields (index, sourcetype, source, host), you cannot use other fields.
For more details see at https://docs.splunk.com/Documentation/Splunk/8.0.0/Search/Changethesearchmode

Ciao.
Giuseppe

Re: Search in fast vs smart modes does not return same number of events

tomasmoser — Thu, 05 Dec 2019 14:49:20 GMT

We found a problem but we do not understand why it's happening.

There were some fields defined via EXTRACT- stanza and some fields defined via REPORT- stanza using transforms.conf. In FAST mode only fields using EXTRACT were really extracted when we enforced extraction in SPL using field=. So far so good. However, those fields defined via REPORT- stanza were not extracted when enforced with field= in SPL.

When we switched from REPORT- EXTRACT- it started working as expected.

This does not make sense! What should we do if we wanted some more complex transformations via REPORT in FAST mode as this one is defualt for all dashboards ???

Re: Search in fast vs smart modes does not return same number of events

tomasmoser — Thu, 05 Dec 2019 14:50:15 GMT

That is not true. See my answer above.

Re: Search in fast vs smart modes does not return same number of events

woodcock — Thu, 05 Dec 2019 15:33:29 GMT

Adding fields * is a clever way to FORCE splunk to run in Smart Mode even when Fast Mode is selected. It should be that both Dashbaords and Scheduled Searches run in Smart Mode. If you are sure that this is not the case for you, then DEFINITELY open a support case. You need to tell Splunk more about the fields that are creating/referencing by adding INDEXED and or INDEXED_VALUE settings in fields.conf (it is a tricky thing):
https://docs.splunk.com/Documentation/Splunk/latest/Admin/Fieldsconf

Re: Search in fast vs smart modes does not return same number of events

tomasmoser — Thu, 05 Dec 2019 15:40:11 GMT

It looks that default search mode for Splunk dashboards is FAST. I am finding contradictory answers to this question - some says FAST some says SMART. We seem to be running in FAST mode in dashboards and then do not get any results. In SMART wet get all events.

See my answer above. It looks like REPORT- stanza does not work in FAST mode while EXTRACT- does work. Weird.

Re: Search in fast vs smart modes does not return same number of events

woodcock — Thu, 05 Dec 2019 15:47:36 GMT

There should be no functional difference between REPORT- and EXTRACT- other than the order in which the 2 are done. Definitely open a support case.

Re: Search in fast vs smart modes does not return same number of events

woodcock — Thu, 05 Dec 2019 15:48:50 GMT

The sourcetype-wildcard-hack is fine and should have nothing to do with this.