topic Re: subsearch not running in Splunk Search

subsearch not running

gcusello — Mon, 11 Dec 2017 09:31:18 GMT

Hi at all,
I have a very strange question:
I have a search with a subsearch that's correctly running on a test environment (Splunk 7.0.0).

index=my_index1 sourcetype=my_sourcetype1 "Start time"="2017-12-04T11:00:01"
| rename NetBIOS as hostname , "Start time" as scandate
| eval date=substr(scandate,1,10)
| stats dc(hostname) as tot_host_disc by date
| appendcols [ search
     index=my_index2  sourcetype=my_sourcetype2 earliest=1511517602 latest=1512468002
     | eval shostname=substr(Hostname,0,1) 
     | WHERE NOT shostname="*" 
     | stats dc(Hostname) as tot_host
     ]

Now I copied it on a production environment (Splunk 7.0.0), but it doesn't run: the subsearch has always zero as result.

The strange thing is that both the searches run correctly by themselves, but when together the subsearch has always zero results.
In other words there a problem on the second search only when executed in subsearch.

The only difference between the two environments is that test environment is a standalone server, instead production environment is based on two indexer clustered servers.

All the servers are Linux

It there a know issue on permission in subsearches?

Thank you in advance.

Bye.
Giuseppe

Re: subsearch not running

kamlesh_vaghela — Mon, 11 Dec 2017 09:39:38 GMT

Hi @cusello,

Does user has appropriate rights for index=my_index2??

Re: subsearch not running

gcusello — Mon, 11 Dec 2017 09:44:26 GMT

An additional strange information:
if I add to the first search the subsearch index, the search gives the correct result, in other words:

 index=my_index1 OR index=my_index2 sourcetype=my_sourcetype1 "Start time"="2017-12-04T11:00:01"
 | rename NetBIOS as hostname , "Start time" as scandate
 | eval date=substr(scandate,1,10)
 | stats dc(hostname) as tot_host_disc by date
 | appendcols [ search
      index=my_index2  sourcetype=my_sourcetype2 earliest=1511517602 latest=1512468002
      | eval shostname=substr(Hostname,0,1) 
      | WHERE NOT shostname="*" 
      | stats dc(Hostname) as tot_host
      ]

And the number of events is the same with or without index2

Bye.
Giuseppe

Re: subsearch not running

gcusello — Mon, 11 Dec 2017 09:53:07 GMT

Another additional information:
there'a a difference between two environments:
In test environment index is created in the same app, instead in clustered production environment index is created in _cluster app.
Could this thing create a problem only in subsearches?
In other words, could an index created outside of the app give problems when used in subsearches?
The same problem is present also using _internal index.
Bye.
Giuseppe

Re: subsearch not running

gcusello — Mon, 11 Dec 2017 10:01:01 GMT

Yes, infact the subsearch correctly runs if executed by itself!
It seems that there's a limitation in subsearches.
But not present if I add the second index to the main search and only in clustered environment!
Bye.
Giuseppe

Re: subsearch not running

gcusello — Mon, 11 Dec 2017 10:18:35 GMT

Another additional information:
I created a local non clustered index and subsearch correctly runs, so the problem is in clustering.
Bye.
Giuseppe

Re: subsearch not running

gcusello — Mon, 11 Dec 2017 10:44:04 GMT

last information:
this problem is present in 7.0.0 version, I have the same architecture on 6.4.2 version and the problem isn't present.
Bye.
Giuseppe

Re: subsearch not running

gcusello — Tue, 19 Dec 2017 16:28:59 GMT

the problem was that an architecture with two clustered indexers used each one both as Indexer and as Search Head doesn'r run on 7.0.0!
In other words executing a search with subsearches on a clustered indexer it doesn't work, there must be a Search Head!

I have this architecture on 6.4.2 and it's still running, instead on 7.0.0. probably is changed somebody in search execution so subsearches don't run if I execute this search on the Indexer.

Bye.
Giuseppe