topic Re: Why would a same search running on 2 different instances shows a huge difference in job size? in Splunk Search

Why would a same search running on 2 different instances shows a huge difference in job size?

meenu_2017 — Mon, 09 Jul 2018 18:29:45 GMT

Hello Fellow Splunkers,
Need help to understand a scenario that I came across in my org.
Why would the same search running on 2 different instances shows a huge difference in job size.

for eg ,
Instance 1 returns 13,647,640,178 results with job size 36.61 MB
Instance 2 returns 13,669,171,100 results with job size 84KB.

I don't care about the difference in event counts but wondering about the huge variation in size. Any guidance as to what i should look?

Re: Why would a same search running on 2 different instances shows a huge difference in job size?

pradeepkumarg — Mon, 09 Jul 2018 19:02:51 GMT

Can you check if the mode of the search is different between the instances? Verbose vs smart vs fast ?

Re: Why would a same search running on 2 different instances shows a huge difference in job size?

meenu_2017 — Mon, 09 Jul 2018 20:24:59 GMT

They both are running in fast mode.

Re: Why would a same search running on 2 different instances shows a huge difference in job size?

PowerPacked — Mon, 09 Jul 2018 21:18:44 GMT

Hi meenu_2017

are Instance 1 & Instance 2 - SH Clustered ?

if they are not clustered, can be many reasons - permissions, distributed search groups, time zones, etc.

if they are clustered, are you running the search from individual search head url or load balancer url.

Thanks

Re: Why would a same search running on 2 different instances shows a huge difference in job size?

woodcock — Tue, 10 Jul 2018 00:53:50 GMT

Here are some reasons:

Not peered to same indexers
Routing to some indexers is bad causing timeouts and partial results
User running search has different `Time zone` setting so searching across different times
The Knowledge Objects are not synchronized causing fields/tags to be different
Difference in RAM causing smaller Search Head to max out and return partial results.

Re: Why would a same search running on 2 different instances shows a huge difference in job size?

meenu_2017 — Tue, 10 Jul 2018 19:22:31 GMT

These search heads are not clustered. They are set separately for each of the instances.
As they both are returning approximately the same no. of records, i couldn't think of a permission issue .

Re: Why would a same search running on 2 different instances shows a huge difference in job size?

meenu_2017 — Tue, 10 Jul 2018 19:26:30 GMT

Thanks for the suggestions. I might have to work with the admin here for some of these.
But since they both returns almost the same no.of records, is it that Instance 2 is calculating the size wrongly or so?

Re: Why would a same search running on 2 different instances shows a huge difference in job size?

PowerPacked — Tue, 10 Jul 2018 20:09:54 GMT

Ok, check if one of the instance ( search head ) has more number of extracted fields and showing up in the results

& also inspect both the search jobs by going into -- job --- inspect job when the searches finishes.

from the inspect job you can actually compare between the both jobs to see, which part of the job is making it to occupy more space.

Thanks

Re: Why would a same search running on 2 different instances shows a huge difference in job size?

woodcock — Sun, 15 Jul 2018 13:34:07 GMT

So which was it?