Splunk Search

Why do I get different results from save search

a212830
Champion

Hi,

I have a customer who is exporting data via the REST API, and getting different results from the same time period, when testing, and I can't determine why. The data is kept for 90 days, so it shouldn't be archving. Here's the search:

curl -k -u ${SPLUSR}:${SPLPWD} --url https://lrtp449:8089/services/search/jobs/export --data-urlencode search='search earliest=10/5/2017:11:00:00 latest=10/5/2017:11:10:00 index=main sourcetype="ms:o365:management" | table _raw,_time' -d output_mode=json  -o - testfile.$$.json

I run this via cron every 15 minutes, and get different results - sometimes as many as 500 lines or more.

0 Karma

sohailmohammed
Explorer

Hello there,

I am facing the same issue here.. I get different results when I run a rest call. 
For example I ran a rest command to bring all the dashboards on h1 search head it brings 300 to me and for my colleague it brings 305 on same h1 search head. What may be the problem ?

Also if I get 300 results on SH H1, I see different count on H2 with 310 results.. what is the issue here for this inconsistencies ? 

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Check the job inspector for the SID to see what's up.

Is it always run for Oct 5th? Are some indexers unavailable? Does that instance of the SHP have different search peers mapped? Does it change based on which user is running it (RBAC?)?

0 Karma

sloshburch
Splunk Employee
Splunk Employee

I think your best bet is to post the job inspector from two of the results that SHOULD be the same but ARE NOT. We believe you that they are different, we're trying to help see WHY. Cool?

0 Karma

a212830
Champion

Cool. I'll get to it later today.

0 Karma

a212830
Champion

Just want to be clear - this isn't only an issue of different search-heads giving different results - it's the same search-heads giving different results. Here's output from the same cronjob:

-rw-r--r-- 1 splunk splunk 113585795 Oct 24 11:31 lrtp450.9934.json
-rw-r--r-- 1 splunk splunk 115420582 Oct 24 11:46 lrtp450.17161.json
-rw-r--r-- 1 splunk splunk 122899855 Oct 24 12:01 lrtp450.24383.json
-rw-r--r-- 1 splunk splunk 106106522 Oct 24 12:16 lrtp450.31857.json
-rw-r--r-- 1 splunk splunk 113585795 Oct 24 12:31 lrtp450.7500.json
-rw-r--r-- 1 splunk splunk 113585795 Oct 24 12:46 lrtp450.14662.json
-rw-r--r-- 1 splunk splunk 113585795 Oct 24 13:01 lrtp450.21928.json
-rw-r--r-- 1 splunk splunk 115632478 Oct 24 13:16 lrtp450.29403.json
-rw-r--r-- 1 splunk splunk 113585795 Oct 24 13:31 lrtp450.5782.json
-rw-r--r-- 1 splunk splunk 113585795 Oct 24 13:46 lrtp450.13010.json
-rw-r--r-- 1 splunk splunk 115420582 Oct 24 14:01 lrtp450.20201.json
-rw-r--r-- 1 splunk splunk 115420582 Oct 24 14:16 lrtp450.27668.json
-rw-r--r-- 1 splunk splunk 122899855 Oct 24 14:31 lrtp450.3274.json
-rw-r--r-- 1 splunk splunk 115420582 Oct 24 14:46 lrtp450.10649.json
0 Karma

a212830
Champion

Meanwhile, the csv output from both servers is consistent:

rw-r--r-- 1 splunk splunk 76690414 Oct 24 11:30 lrtp449.9934.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 11:31 lrtp450.9934.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 11:45 lrtp449.17161.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 11:45 lrtp450.17161.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 12:00 lrtp449.24383.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 12:01 lrtp450.24383.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 12:15 lrtp449.31857.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 12:16 lrtp450.31857.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 12:30 lrtp449.7500.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 12:30 lrtp450.7500.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 12:45 lrtp449.14662.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 12:46 lrtp450.14662.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 13:00 lrtp449.21928.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 13:01 lrtp450.21928.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 13:15 lrtp449.29403.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 13:16 lrtp450.29403.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 13:30 lrtp449.5782.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 13:31 lrtp450.5782.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 13:45 lrtp449.13010.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 13:46 lrtp450.13010.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 14:00 lrtp449.20201.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 14:01 lrtp450.20201.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 14:15 lrtp449.27668.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 14:16 lrtp450.27668.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 14:30 lrtp449.3274.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 14:31 lrtp450.3274.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 14:45 lrtp449.10649.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 14:46 lrtp450.10649.csv
0 Karma

rphillips_splk
Splunk Employee
Splunk Employee

is there any difference if you run the curl command with the admin credentials or are you already doing that?

0 Karma

a212830
Champion

The indexers are available, and I'm focusing on specific dates, for testing. I am hard-coding a SH, as well, to remove that as a potential issue. How do I find the SID for a job that is run externally?

0 Karma

rphillips_splk
Splunk Employee
Splunk Employee

To find the job inspector:

    After the job finishes running go to the UI of the SH and select Activity >Jobs
    find your job before it expires and select Actions >Job> Inspect Job

The sid will be listed at the top
Compare the two job inspectors between running the same search , do they show the same number of search providers (indexers) ?

0 Karma

sloshburch
Splunk Employee
Splunk Employee

It may also be listed in the result's stanza.

0 Karma

rphillips_splk
Splunk Employee
Splunk Employee

also if required to open a case with Splunk Support it would be helpful to provide the following:

put the search process in DEBUG on the SH:
$SPLUNK_HOME/etc/
copy log-searchprocess.cfg and make a new file log-searchprocess-local.cfg

edit log-searchprocess-local.cfg
rootCategory=DEBUG,searchprocessAppender
(no restart of splunk required)
Re-run the searches to produce the issue and copy the job inspector and search.log into separate .txt
files and upload to the case

In addition capture a diag from the SH
$SPLUNK_HOME/bin
./splunk diag
Once finished, delete log-searchprocess-local.cfg

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...