Archive

Why would a same search running on 2 different instances shows a huge difference in job size?

New Member

Hello Fellow Splunkers,
Need help to understand a scenario that I came across in my org.
Why would the same search running on 2 different instances shows a huge difference in job size.

for eg ,
Instance 1 returns 13,647,640,178 results with job size 36.61 MB
Instance 2 returns 13,669,171,100 results with job size 84KB.

I don't care about the difference in event counts but wondering about the huge variation in size. Any guidance as to what i should look?

Tags (1)
0 Karma
1 Solution

Esteemed Legend

Here are some reasons:

Not peered to same indexers
Routing to some indexers is bad causing timeouts and partial results
User running search has different `Time zone` setting so searching across different times
The Knowledge Objects are not synchronized causing fields/tags to be different
Difference in RAM causing smaller Search Head to max out and return partial results.

View solution in original post

0 Karma

Builder

Ok, check if one of the instance ( search head ) has more number of extracted fields and showing up in the results

& also inspect both the search jobs by going into -- job --- inspect job when the searches finishes.

from the inspect job you can actually compare between the both jobs to see, which part of the job is making it to occupy more space.

alt text

Thanks

0 Karma

Esteemed Legend

Here are some reasons:

Not peered to same indexers
Routing to some indexers is bad causing timeouts and partial results
User running search has different `Time zone` setting so searching across different times
The Knowledge Objects are not synchronized causing fields/tags to be different
Difference in RAM causing smaller Search Head to max out and return partial results.

View solution in original post

0 Karma

New Member

Thanks for the suggestions. I might have to work with the admin here for some of these.
But since they both returns almost the same no.of records, is it that Instance 2 is calculating the size wrongly or so?

0 Karma

Esteemed Legend

So which was it?

0 Karma

Influencer

Can you check if the mode of the search is different between the instances? Verbose vs smart vs fast ?

0 Karma

New Member

They both are running in fast mode.

0 Karma

Builder

Hi meenu_2017

are Instance 1 & Instance 2 - SH Clustered ?

if they are not clustered, can be many reasons - permissions, distributed search groups, time zones, etc.

if they are clustered, are you running the search from individual search head url or load balancer url.

Thanks

0 Karma

New Member

These search heads are not clustered. They are set separately for each of the instances.
As they both are returning approximately the same no. of records, i couldn't think of a permission issue .

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!