APPEND give no records - even if first search result has 1 event

[JYTTEJ]

This search (search 1) returns 1 event:

host=psdkxt05 APP=TMA ORG=HPP PRJ=XX* SVC=x1 OR SVC="x2" OR SVC=x3 | JOIN F5I [ search APP=HPP PRJ=XX*] |EVAL MSECTOT=ELT*1000|WHERE MSECTOT>5000|EVAL PS_TIME=ELT-PRT|EVAL MSECTOT=ELT*1000|WHERE MSECTOT>5000|TABLE F5I,LID, PRC, PS_TIME, PRT, STM, ELT, RCD, BRC, SVC, APP|RENAME LID AS TrxID, PRT AS PRC_TIME, ELT AS TOTAL_TIME, STM AS TIMESTAMP

I want to append another search (search 2):

host="psdkxT05*" FMT="IOSTAT*" PRJ=XX* SVC=x1 OR SVC="x2" OR SVC=x3 ORG=OTHER |EVAL MSECTOT=ELT*1000|WHERE MSECTOT>5000|EVAL PS_TIME=ELT-PRT

Search 2 return 0 event (which is correct)

I have then created below search (search 3) which does NOT return any events. I had expected below search to return the 1 event from the search 1.

What have I done wrong?

host=psdkxt05 APP=TMA ORG=HPP PRJ=XX* SVC=x1 OR SVC="x2" OR SVC=x3 | JOIN F5I [ search APP=HPP PRJ=XX*] |EVAL MSECTOT=ELT*1000|WHERE MSECTOT>5000|EVAL PS_TIME=ELT-PRT|APPEND [SEARCH host="psdkxT05*" FMT="IOSTAT*" PRJ=XX* SVC=x1 OR SVC="x2" OR SVC=x3 ORG=OTHER |EVAL MSECTOT=ELT*1000|WHERE MSECTOT>5000|EVAL PS_TIME=ELT-PRT ]|TABLE F5I,LID, PRC, PS_TIME, PRT, STM, ELT, RCD, BRC, SVC, APP|RENAME LID AS TrxID, PRT AS PRC_TIME, ELT AS TOTAL_TIME, STM AS TIMESTAMP

[sideview]

If you want to combine the results of two searches, using append should be a last resort. Your first efforts should go towards combining them into a single search clause with some parentheses and OR's. The main reasons are;

1. performance. having splunkd run one search is better than 2
2. limits. searches run in square brackets have limits imposed on them: if they return more than some number of rows they can get quietly truncated. If they take more than N seconds to complete, they can get quietly finalized. Unless you know the search is nice and small I'd try and get it out of the square brackets. You can also test for yourself by putting searches inside square brackets that match huge amounts of events, and that return huge amounts of rows.

OK granted, you have a lot going on here already, and in the end you might prefer to use append cause it's more intuitive and it might make your head hurt less. But anyway, with a dash of eval, searchmatch() and a little diligence and testing, I think it can be made to work.

Here's my attempt below:

search 1:

host=psdkxt05 APP=TMA ORG=HPP PRJ=XX* SVC=x1 OR SVC="x2" OR SVC=x3     
| JOIN F5I [ search APP=HPP PRJ=XX*] 
| EVAL MSECTOT=ELT*1000 | WHERE MSECTOT>5000 | EVAL PS_TIME=ELT-PRT 
| EVAL MSECTOT=ELT*1000 | WHERE MSECTOT>5000 
| TABLE F5I,LID, PRC, PS_TIME, PRT, STM, ELT, RCD, BRC, SVC, APP
| RENAME LID AS TrxID, PRT AS PRC_TIME, ELT AS TOTAL_TIME, STM AS TIMESTAMP

plus search 2:

host="psdkxT05*" FMT="IOSTAT*" PRJ=XX* SVC=x1 OR SVC="x2" OR SVC=x3 ORG=OTHER
| EVAL MSECTOT=ELT*1000
| WHERE MSECTOT>5000 
| EVAL PS_TIME=ELT-PRT

can (I think) be combined into:

SVC=x1 OR SVC="x2" OR SVC=x3 PRJ=XX* 
( host=psdkxt05 APP=TMA ORG=HPP ) OR (host="psdkxT05*" FMT="IOSTAT*" ORG=OTHER)
| EVAL PS_TIME=if(searchmatch("host=psdkxt05 APP=TMA ORG=HPP"),ELT-PRT,PS_TIME)
| eval joinField=if(searchmatch("host=psdkxt05 APP=TMA ORG=HPP"),F5I,joinField) 
| join joinField [ search APP=HPP PRJ=XX* | eval joinField=F5I] 
| eval F5I=if(isnotnull(joinField),joinField,F5I
| fields - joinField
| eval MSECTOT=ELT*1000
| where MSECTOT>5000 
| table F5I,LID, PRC, PS_TIME, PRT, STM, ELT, RCD, BRC, SVC, APP
| rename LID AS TrxID, PRT AS PRC_TIME, ELT AS TOTAL_TIME, STM AS TIMESTAMP

of course it's a pretty tricky search to write without having the data to test it all and I might have made a mistake.

Note: eval someField=if(searchmatch("foo"),someOtherField, someField). This can be a very useful way to normalize and herd the data into the form you want.

http://www.splunk.com/base/Documentation/latest/SearchReference/CommonEvalFunctions

[zschmid]

I found a workaround for this.

At the end of your search instead of doing * | table field1, field2, field3 use the fields command |fields field1 fields2 fields3 from there you can add that search to a dashboard using the SimpleResultsTable module. Hope this helps!

[ziegfried]

This seems to be a bug.