Why would a same search running on 2 different instances shows a huge difference in job size?

[meenu_2017]

Hello Fellow Splunkers,
Need help to understand a scenario that I came across in my org.
Why would the same search running on 2 different instances shows a huge difference in job size.

for eg ,
Instance 1 returns 13,647,640,178 results with job size 36.61 MB
Instance 2 returns 13,669,171,100 results with job size 84KB.

I don't care about the difference in event counts but wondering about the huge variation in size. Any guidance as to what i should look?

[PowerPacked]

Ok, check if one of the instance ( search head ) has more number of extracted fields and showing up in the results

& also inspect both the search jobs by going into -- job --- inspect job when the searches finishes.

from the inspect job you can actually compare between the both jobs to see, which part of the job is making it to occupy more space.

Thanks

[woodcock · Accepted solution]

Here are some reasons:

Not peered to same indexers
Routing to some indexers is bad causing timeouts and partial results
User running search has different `Time zone` setting so searching across different times
The Knowledge Objects are not synchronized causing fields/tags to be different
Difference in RAM causing smaller Search Head to max out and return partial results.

[meenu_2017]

Thanks for the suggestions. I might have to work with the admin here for some of these.
But since they both returns almost the same no.of records, is it that Instance 2 is calculating the size wrongly or so?

[woodcock]

So which was it?

[pradeepkumarg]

Can you check if the mode of the search is different between the instances? Verbose vs smart vs fast ?

[meenu_2017]

They both are running in fast mode.

[PowerPacked]

Hi meenu_2017

are Instance 1 & Instance 2 - SH Clustered ?

if they are not clustered, can be many reasons - permissions, distributed search groups, time zones, etc.

if they are clustered, are you running the search from individual search head url or load balancer url.

Thanks

[meenu_2017]

These search heads are not clustered. They are set separately for each of the instances.
As they both are returning approximately the same no. of records, i couldn't think of a permission issue .