Syntax/use of subsearches

[Mick_OBrien]

I have a query …

index=blah "BAD_REQUEST" | rex "(?i) requestId (?P<requestId>[^:]+)" | table requestId | dedup requestId

…that returns 7 records/fields…

92d246dd-7aac-41f7-a398-27586062e4fa

ba79c6f5-5452-4211-9b89-59d577adbc50

711b9bb4-b9f1-4a2b-ba56-f2b3a9cdf87c

e227202a-0b0a-4cdf-9b11-3080b0ce280f

6099d5a3-61fc-418b-87b4-ddc57c482dd6

348fb576-0c36-4de9-a55a-97157b00a304

c34b7b96-094d-45bb-b03d-f9c98a4efd5f

…that I then want to use as input for another search on the same index

I looked at manual and can see that subsearches are allowed [About subsearches - Splunk Documentation] but when I add my subsearch as input …

index=blah [search index=blah "BAD_REQUEST" | rex "(?i) requestId (?P<requestId>[^:]+)" | table requestId | dedup requestId]

..I would have expected at least 7 records to have been returned BUT I do not see any output.

There are no syntax issues so can someone explain to me what I’m not seeing/doing?

Any help appreciated.

[PickleRick]

Firstly, if your subsearch uses the same source index as the outer search, it's more often than not that the search can be written without using the subsearch.

Secondly, the subsearches have their limitations (for execution time and number of returned results). Their most confusing and annoying "feature" however is that if the subsearch hits such limit, it gets silently finalized and you're only getting partial (possibly empty) results from the subsearch _with no warning about that whatsoever_.

So if your subsearch run on its own produces proper results and your "outer search" with the results from the subsearch manually copy-pasted produces proper results as well it's highly probable that this is the issue you're hitting.

Check your job log to see what your main search is rendered into in the end (after the subsearch is run).

(Of course @ITWhisperer 's point of field extraction is still valid).

[Mick_OBrien]

@ITWhisperer 

Changed to match format as detailed...

index=blah [search index=blah "BAD_REQUEST" | rex "(?i) requestId (?P<search>[^:]+)" | table search | dedup search]

...but new format ONLY returned rows containing 92d246dd-7aac-41f7-a398-27586062e4fa [first row] and no other rows.  I removed 'dedup' but that did not help

How can I include all returned items from inner search as input to outer [main] search?

[ITWhisperer · Accepted solution]

Try with format (I thought this was no longer necessary but it looks like it is!)

index=blah [search index=blah "BAD_REQUEST" | rex "(?i) requestId (?P<search>[^:]+)" | table search | dedup search | format]

[Mick_OBrien]

@ITWhisperer 

With 'format' at the end worked - thank you very much

Just checked documentation which indicates [to me] that returned string have input search results separated by 'OR' command - do I understand correctly?

format - Splunk Documentation

This command is used implicitly by subsearches. This command takes the results of a subsearch, formats the results into a single result and places that result into a new field called search.

The format command performs similar functions as the return command.

.

.

.

mvsepSyntax: mvsep="<string>"Description: The separator to use for multivalue fields.Default: ORmvsepSyntax: mvsep="<string>"Description: The separator to use for multivalue fields.Default: OR

[ITWhisperer]

format can take up to 6 parameters - these default so that the values are put in quotes, there are ANDs between field/value pairs from the same row, rows are enclosed in brackets, there are ORs between rows, and the whole thing is enclosed in brackets. For example:

( ( a="11" AND b="21" AND c="31" ) OR ( a="12" AND b="22" AND c="32" ) OR ( a="13" AND b="23" AND c="33" ) )

These are how the parameter (positions) map to the formatted result

1 2 a="11" 3 b="21" 3 c="31" 4 5 2 a="12" 3 b="22" 3 c="32" 4 5 2 a="13" 3 b="23" 3 c="33" 4 6

You can test this with this runanywhere example

| makeresults count=3
| streamstats count as a
| eval a=a+10, b=a+10, c=a+20
| format 1 2 3 4 5 6

[Mick_OBrien]

@ITWhisperer 

I ran...

| makeresults count=3
| streamstats count as a
| eval a=a+10, b=a+10, c=a+20
| format

...which gave...

( ( a="11" AND b="21" AND c="31" ) OR ( a="12" AND b="22" AND c="32" ) OR ( a="13" AND b="23" AND c="33" ) )

...which tallies with fields/rows as 'AND'/'OR' but if don't understand output from your original example...

| makeresults count=3
| streamstats count as a
| eval a=a+10, b=a+10, c=a+20
| format 1 2 3 4 5 6

1 2 a="11" 3 b="21" 3 c="31" 4 5 2 a="12" 3 b="22" 3 c="32" 4 5 2 a="13" 3 b="23" 3 c="33" 4 6

??

[ITWhisperer]

Suppose you wanted a differently formatted return, e.g. & for AND | for OR, curly brackets for the inner group and square brackets for the outer group, you would substitute the corresponding numbers for the symbols you wanted, e.g. 

| makeresults count=3
| streamstats count as a
| eval a=a+10, b=a+10, c=a+20
| format "[" "{" "&" "}" "|" "]"

which gives

[ { a="11" & b="21" & c="31" } | { a="12" & b="22" & c="32" } | { a="13" & b="23" & c="33" } ]

Possibly not the most useful example, but I have used this sort of thing to, for example, remove the brackets, and change the ANDs to ORs etc.

[Mick_OBrien]

Thanks - now I get it

[ITWhisperer]

Your search assume that requestid has already been extracted into a field in the index. If you want to just do a string search based on the requestids, try something like this

index=blah [search index=blah "BAD_REQUEST" | rex "(?i) requestId (?P<search>[^:]+)" | table search | dedup search]

The field search (and query) are given special treatment for subsearches in that the field name is not return, just the contents of the field