Splunk Search

Why do I get different results from save search

Champion

Hi,

I have a customer who is exporting data via the REST API, and getting different results from the same time period, when testing, and I can't determine why. The data is kept for 90 days, so it shouldn't be archving. Here's the search:

curl -k -u ${SPLUSR}:${SPLPWD} --url https://lrtp449:8089/services/search/jobs/export --data-urlencode search='search earliest=10/5/2017:11:00:00 latest=10/5/2017:11:10:00 index=main sourcetype="ms:o365:management" | table _raw,_time' -d output_mode=json  -o - testfile.$$.json

I run this via cron every 15 minutes, and get different results - sometimes as many as 500 lines or more.

0 Karma

Ultra Champion

Check the job inspector for the SID to see what's up.

Is it always run for Oct 5th? Are some indexers unavailable? Does that instance of the SHP have different search peers mapped? Does it change based on which user is running it (RBAC?)?

0 Karma

Ultra Champion

I think your best bet is to post the job inspector from two of the results that SHOULD be the same but ARE NOT. We believe you that they are different, we're trying to help see WHY. Cool?

0 Karma

Champion

Cool. I'll get to it later today.

0 Karma

Champion

Just want to be clear - this isn't only an issue of different search-heads giving different results - it's the same search-heads giving different results. Here's output from the same cronjob:

-rw-r--r-- 1 splunk splunk 113585795 Oct 24 11:31 lrtp450.9934.json
-rw-r--r-- 1 splunk splunk 115420582 Oct 24 11:46 lrtp450.17161.json
-rw-r--r-- 1 splunk splunk 122899855 Oct 24 12:01 lrtp450.24383.json
-rw-r--r-- 1 splunk splunk 106106522 Oct 24 12:16 lrtp450.31857.json
-rw-r--r-- 1 splunk splunk 113585795 Oct 24 12:31 lrtp450.7500.json
-rw-r--r-- 1 splunk splunk 113585795 Oct 24 12:46 lrtp450.14662.json
-rw-r--r-- 1 splunk splunk 113585795 Oct 24 13:01 lrtp450.21928.json
-rw-r--r-- 1 splunk splunk 115632478 Oct 24 13:16 lrtp450.29403.json
-rw-r--r-- 1 splunk splunk 113585795 Oct 24 13:31 lrtp450.5782.json
-rw-r--r-- 1 splunk splunk 113585795 Oct 24 13:46 lrtp450.13010.json
-rw-r--r-- 1 splunk splunk 115420582 Oct 24 14:01 lrtp450.20201.json
-rw-r--r-- 1 splunk splunk 115420582 Oct 24 14:16 lrtp450.27668.json
-rw-r--r-- 1 splunk splunk 122899855 Oct 24 14:31 lrtp450.3274.json
-rw-r--r-- 1 splunk splunk 115420582 Oct 24 14:46 lrtp450.10649.json
0 Karma

Champion

Meanwhile, the csv output from both servers is consistent:

rw-r--r-- 1 splunk splunk 76690414 Oct 24 11:30 lrtp449.9934.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 11:31 lrtp450.9934.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 11:45 lrtp449.17161.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 11:45 lrtp450.17161.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 12:00 lrtp449.24383.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 12:01 lrtp450.24383.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 12:15 lrtp449.31857.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 12:16 lrtp450.31857.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 12:30 lrtp449.7500.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 12:30 lrtp450.7500.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 12:45 lrtp449.14662.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 12:46 lrtp450.14662.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 13:00 lrtp449.21928.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 13:01 lrtp450.21928.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 13:15 lrtp449.29403.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 13:16 lrtp450.29403.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 13:30 lrtp449.5782.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 13:31 lrtp450.5782.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 13:45 lrtp449.13010.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 13:46 lrtp450.13010.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 14:00 lrtp449.20201.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 14:01 lrtp450.20201.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 14:15 lrtp449.27668.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 14:16 lrtp450.27668.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 14:30 lrtp449.3274.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 14:31 lrtp450.3274.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 14:45 lrtp449.10649.csv
-rw-r--r-- 1 splunk splunk 76690414 Oct 24 14:46 lrtp450.10649.csv
0 Karma

Splunk Employee
Splunk Employee

is there any difference if you run the curl command with the admin credentials or are you already doing that?

0 Karma

Champion

The indexers are available, and I'm focusing on specific dates, for testing. I am hard-coding a SH, as well, to remove that as a potential issue. How do I find the SID for a job that is run externally?

0 Karma

Splunk Employee
Splunk Employee

To find the job inspector:

    After the job finishes running go to the UI of the SH and select Activity >Jobs
    find your job before it expires and select Actions >Job> Inspect Job

The sid will be listed at the top
Compare the two job inspectors between running the same search , do they show the same number of search providers (indexers) ?

0 Karma

Ultra Champion

It may also be listed in the result's stanza.

0 Karma

Splunk Employee
Splunk Employee

also if required to open a case with Splunk Support it would be helpful to provide the following:

put the search process in DEBUG on the SH:
$SPLUNK_HOME/etc/
copy log-searchprocess.cfg and make a new file log-searchprocess-local.cfg

edit log-searchprocess-local.cfg
rootCategory=DEBUG,searchprocessAppender
(no restart of splunk required)
Re-run the searches to produce the issue and copy the job inspector and search.log into separate .txt
files and upload to the case

In addition capture a diag from the SH
$SPLUNK_HOME/bin
./splunk diag
Once finished, delete log-searchprocess-local.cfg

0 Karma