Splunk Search

subsearch not running

gcusello
SplunkTrust
SplunkTrust

Hi at all,
I have a very strange question:
I have a search with a subsearch that's correctly running on a test environment (Splunk 7.0.0).

index=my_index1 sourcetype=my_sourcetype1 "Start time"="2017-12-04T11:00:01"
| rename NetBIOS as hostname , "Start time" as scandate
| eval date=substr(scandate,1,10)
| stats dc(hostname) as tot_host_disc by date
| appendcols [ search
     index=my_index2  sourcetype=my_sourcetype2 earliest=1511517602 latest=1512468002
     | eval shostname=substr(Hostname,0,1) 
     | WHERE NOT shostname="*" 
     | stats dc(Hostname) as tot_host
     ]

Now I copied it on a production environment (Splunk 7.0.0), but it doesn't run: the subsearch has always zero as result.

The strange thing is that both the searches run correctly by themselves, but when together the subsearch has always zero results.
In other words there a problem on the second search only when executed in subsearch.

The only difference between the two environments is that test environment is a standalone server, instead production environment is based on two indexer clustered servers.

All the servers are Linux

It there a know issue on permission in subsearches?

Thank you in advance.

Bye.
Giuseppe

0 Karma
1 Solution

gcusello
SplunkTrust
SplunkTrust

the problem was that an architecture with two clustered indexers used each one both as Indexer and as Search Head doesn'r run on 7.0.0!
In other words executing a search with subsearches on a clustered indexer it doesn't work, there must be a Search Head!

I have this architecture on 6.4.2 and it's still running, instead on 7.0.0. probably is changed somebody in search execution so subsearches don't run if I execute this search on the Indexer.

Bye.
Giuseppe

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

the problem was that an architecture with two clustered indexers used each one both as Indexer and as Search Head doesn'r run on 7.0.0!
In other words executing a search with subsearches on a clustered indexer it doesn't work, there must be a Search Head!

I have this architecture on 6.4.2 and it's still running, instead on 7.0.0. probably is changed somebody in search execution so subsearches don't run if I execute this search on the Indexer.

Bye.
Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

An additional strange information:
if I add to the first search the subsearch index, the search gives the correct result, in other words:

 index=my_index1 OR index=my_index2 sourcetype=my_sourcetype1 "Start time"="2017-12-04T11:00:01"
 | rename NetBIOS as hostname , "Start time" as scandate
 | eval date=substr(scandate,1,10)
 | stats dc(hostname) as tot_host_disc by date
 | appendcols [ search
      index=my_index2  sourcetype=my_sourcetype2 earliest=1511517602 latest=1512468002
      | eval shostname=substr(Hostname,0,1) 
      | WHERE NOT shostname="*" 
      | stats dc(Hostname) as tot_host
      ]

And the number of events is the same with or without index2

Bye.
Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Another additional information:
there'a a difference between two environments:
In test environment index is created in the same app, instead in clustered production environment index is created in _cluster app.
Could this thing create a problem only in subsearches?
In other words, could an index created outside of the app give problems when used in subsearches?
The same problem is present also using _internal index.
Bye.
Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Another additional information:
I created a local non clustered index and subsearch correctly runs, so the problem is in clustering.
Bye.
Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

last information:
this problem is present in 7.0.0 version, I have the same architecture on 6.4.2 version and the problem isn't present.
Bye.
Giuseppe

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

Hi @cusello,

Does user has appropriate rights for index=my_index2??

0 Karma

gcusello
SplunkTrust
SplunkTrust

Yes, infact the subsearch correctly runs if executed by itself!
It seems that there's a limitation in subsearches.
But not present if I add the second index to the main search and only in clustered environment!
Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...