topic Re: Why are we receiving inconsistent data? in Deployment Architecture

Why are we receiving inconsistent data?

vxl65703 — Tue, 14 Feb 2017 15:00:30 GMT

I am a first time, I have a user who says his search heads are kicking different results on two different search heads.

We are using Splunk 6.3

Re: Why are we receiving inconsistent data?

pradeepkumarg — Tue, 14 Feb 2017 16:18:49 GMT

Are the search heads in cluster? or independent? You might want to verify the search peers for both of these search heads. You will have to ask more details to your user like what is the search? time range selected? Is he using any lookups that are available on one SH and not other? Are the results always consistently different ? Did he take a look at the job inspector to see if there were any errors from a search peer that had trouble sending data back ?

Re: Why are we receiving inconsistent data?

vxl65703 — Tue, 14 Feb 2017 16:45:46 GMT

the search heads are in a cluster, one instance pulls up data the other instance only pulls up a part of the data, would.

Would need to look at the indexers or forwarders to see if either is corrupt ?

Would I need to SSH into the servers to review the indexers/forwarders ?

Re: Why are we receiving inconsistent data?

jkat54 — Tue, 14 Feb 2017 17:19:19 GMT

To be sure, these search heads are part of a SEARCH HEAD CLUSTER? Yes/no?

Noting SEARCH HEAD CLUSTER is not the same as a SPLUNK CLUSTER (which is the general term used for a cluster of indexers, a cluster master, license master, and search head(s))

Re: Why are we receiving inconsistent data?

vxl65703 — Tue, 14 Feb 2017 18:07:27 GMT

Yes these are search heads apart of the cluster.

Re: Why are we receiving inconsistent data?

jkat54 — Tue, 14 Feb 2017 18:17:12 GMT

-hangs head-

Are they part of a search head cluster?

Re: Why are we receiving inconsistent data?

vxl65703 — Tue, 14 Feb 2017 19:52:25 GMT

Yes they are a part of a search head cluster

Re: Why are we receiving inconsistent data?

DalJeanis — Tue, 14 Feb 2017 22:34:13 GMT

Have them give you the exact search they are running, and the results.

First, make sure that the search they are kicking off is a consistent search. Make sure it has a fixed earliest and latest value, and so on. If they are kicking off "last 30 minutes" on one head then later on another had, then of course the answer will be different.

Second, run that query yourself on each head. See if your results are the same or different from his. Ideally, limit the search to as small a time range and amount of detail as possible, as long as he gets a different result on each head.

If there are lookups in the search, or joins, then run those subsearches independently and check whether they are consistent on the two heads. Perhaps a lookup isn't propagating fully, or whatever.