subsearch not running

[gcusello]

Hi at all,
I have a very strange question:
I have a search with a subsearch that's correctly running on a test environment (Splunk 7.0.0).

index=my_index1 sourcetype=my_sourcetype1 "Start time"="2017-12-04T11:00:01"
| rename NetBIOS as hostname , "Start time" as scandate
| eval date=substr(scandate,1,10)
| stats dc(hostname) as tot_host_disc by date
| appendcols [ search
     index=my_index2  sourcetype=my_sourcetype2 earliest=1511517602 latest=1512468002
     | eval shostname=substr(Hostname,0,1) 
     | WHERE NOT shostname="*" 
     | stats dc(Hostname) as tot_host
     ]

Now I copied it on a production environment (Splunk 7.0.0), but it doesn't run: the subsearch has always zero as result.

The strange thing is that both the searches run correctly by themselves, but when together the subsearch has always zero results.
In other words there a problem on the second search only when executed in subsearch.

The only difference between the two environments is that test environment is a standalone server, instead production environment is based on two indexer clustered servers.

All the servers are Linux

It there a know issue on permission in subsearches?

Thank you in advance.

Bye.
Giuseppe

[gcusello · Accepted solution]

the problem was that an architecture with two clustered indexers used each one both as Indexer and as Search Head doesn'r run on 7.0.0!
In other words executing a search with subsearches on a clustered indexer it doesn't work, there must be a Search Head!

I have this architecture on 6.4.2 and it's still running, instead on 7.0.0. probably is changed somebody in search execution so subsearches don't run if I execute this search on the Indexer.

Bye.
Giuseppe

[gcusello]

An additional strange information:
if I add to the first search the subsearch index, the search gives the correct result, in other words:

 index=my_index1 OR index=my_index2 sourcetype=my_sourcetype1 "Start time"="2017-12-04T11:00:01"
 | rename NetBIOS as hostname , "Start time" as scandate
 | eval date=substr(scandate,1,10)
 | stats dc(hostname) as tot_host_disc by date
 | appendcols [ search
      index=my_index2  sourcetype=my_sourcetype2 earliest=1511517602 latest=1512468002
      | eval shostname=substr(Hostname,0,1) 
      | WHERE NOT shostname="*" 
      | stats dc(Hostname) as tot_host
      ]

And the number of events is the same with or without index2

Bye.
Giuseppe

[gcusello]

Another additional information:
there'a a difference between two environments:
In test environment index is created in the same app, instead in clustered production environment index is created in _cluster app.
Could this thing create a problem only in subsearches?
In other words, could an index created outside of the app give problems when used in subsearches?
The same problem is present also using _internal index.
Bye.
Giuseppe

[gcusello]

Another additional information:
I created a local non clustered index and subsearch correctly runs, so the problem is in clustering.
Bye.
Giuseppe

[gcusello]

last information:
this problem is present in 7.0.0 version, I have the same architecture on 6.4.2 version and the problem isn't present.
Bye.
Giuseppe

[kamlesh_vaghela]

Hi @cusello,

Does user has appropriate rights for index=my_index2??

[gcusello]

Yes, infact the subsearch correctly runs if executed by itself!
It seems that there's a limitation in subsearches.
But not present if I add the second index to the main search and only in clustered environment!
Bye.
Giuseppe