What correlation is actually performed between a subsearch and the main search?

[tenorway]

Hi All!

I am trying to use the subsearch functionality to find a token which should be used in the main search. Pretty basic, I guess.
However, I don't think I quite understand the fundamentals. What happens with the time in the main search? What happens with the other properties? Do they influence the outer search?

If I don't return any values from the subsearch, the search results seems to be limited by the time from the hits in the subsearch. Might be expected behaviour. Search string:

index=klpi [search index=klpi 2345678910]

This results in many results over a period of time.
This search, however, only results in a a few results. And only the ones which has sessionIndex=

index=klpi [search index=klpi sessionIndex=2345678910 ]

The subsearch seems to influence the main search, but what correlation is actually performed?

Here is the actual search I've been trying. I have tried both with and without the time modification.

index=klpi sourcetype="websphere:system:out" [search index=klpi sessionIndex sourcetype="fedag:debug" | rex("sessionIndex=(?<sessionIndex>.+)") | eval earliest=_time-1000 | eval latest=_time+1000  | format "(" "(" " " ")" "OR" ")" | rename sessionIndex as search]

Although I know there are hits in the source type of the outer search in the period given by the subsearch, I receive no results.

Thank for any assistance, both on the basic understanding, and the last search.

Regards

Tor Erik

[woodcock]

Here is the best way to test out what subsearches do; look at the results of these searches:

index=klpi 2345678910 | format
index=klpi sessionIndex=2345678910 | format
index=klpi sessionIndex sourcetype="fedag:debug" | rex("sessionIndex=(?<sessionIndex>.+)") | eval earliest=_time-1000 | eval latest=_time+1000  | format "(" "(" " " ")" "OR" ")" | rename sessionIndex as search | format

Each should result in a single field called search that is the string that will be appended to your outer search.
It is unclear to me what you are trying to do but this does explicitly answer your question.

[Sebastian2]

I'm not really sure what you are trying to do; but the basic idea of a sub-search is to generate arguments for your outer search.

What happens from a technical view

This is pretty straight forward: The "innermost" search is performed first. When the search is complete, the outer search is started. They are serially executed. How ever, I'm not sure if two "parallel" subsearches could be executed parallel; example:

index=example [ ... subsearch1 ... ] [ ... subsearch2 ... ]

What happens from a logical view **
The reason why an outer search isn't started before an inner search is completed is, that the **result or output of the inner search is used as argument for the outer search. Let's have an example:

index=klpi [search index=klpi 2345678910]

First of all, the inner search search index=klpi 2345678901 is executed. The index is going to be searched for any field containing 2345678901 (not only the sessionIndex!). Let's say we have two matching events with following fields:

{field1=hello, field2=world, sessionIndex=2345678901, uri="myserver.com/index.html?sessionIndex=2345678901" },
{field1=another, field2=planet, sessionIndex=9876543210, uri="myserver.com/2345678901/list?sessionIndex=9876543210" }

Since you haven't any further argument, splunk is going to translate this into the following:

(
    (field1="hello" AND field2="world" AND sessionIndex="2345678901" AND uri="myserver.com/index.html?sessionIndex=2345678901")
    OR
    (field1="another" AND field2="planet" AND sessionIndex="9876543210" AND uri="myserver.com/2345678901/list?sessionIndex=9876543210")
)

Imagine this as a simple copy&paste operation to the outer search, which will become:

index=klpi
(
    (field1="hello" AND field2="world" AND sessionIndex="2345678901" AND uri="myserver.com/index.html?sessionIndex=2345678901")
    OR
    (field1="another" AND field2="planet" AND sessionIndex="9876543210" AND uri="myserver.com/2345678901/list")
)

The first line index=klpiis the outer search, the rest is the inner. Your second example how ever would have another result, since you know search with a projection first (The second event in our example will not match when using sessionIndex=2345678901 instead of just 2345678901 because the pattern 2345678901 matches a part of the uri and not the sessionIndex).
Now having pooled the fundamentals of subsearches; back to your problem:

1. Your subsearch is searching through another source-type with the argument sessionIndex - are you sure that the source-type fedag:debug actually has the field sessionIndex extracted? (Check if the subsearch alone returns any result)
2. Be aware that subsearches are limited by default to 10k events and 60s execution time - are you sure that your subsearch doesn't violates these rules? (Especially because you are using a rex, which is pretty slow)
3. Why are you using a rex? If your data is indexed correctly Splunk should parse key=value pairs on its own.
4. Again, check the rollout of subresults: Does the sourcetype websphere:system:outcontain all fields of fedag:debug and is it mandatory that all fields match? (Otherwise reduce the fields of the subsearch using | fields + sessionIndex or what ever field you want to join your data by

Sorry for the wall of text - I'm completely new to Splunk and I'm trying to learn it by looking for question like yours and answer them.

[tenorway]

Thanks for the answer. I think that by doing rename sessionIndex as search i do not search for sesssionIndex= in the outer search, I search for the value of sessionIndex from the inner search. These are used in the outer search.

The source type in the outer search does not contain sessionIndex=, and that's why I do a rename as search

[sundareshr]

Look at "normalizedSearch" in Job Instpector. That will spell out the actual search that is being executed. I found that to be the most helpful in understanding how subsearch influences main search.

[tenorway]

The normalized search only contains this:
litsearch NOT () | fields keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server" | remotetl nb=300 et=1447827180.000000 lt=1447829003.000000 remove=true max_count=1000 max_prefetch=100

[sundareshr]

What the NOT () means is that the subsearch is not returning any results back. It could the subsearch has reached its limit - either time or number of events. Does the job inspector have a warning icon?