Map command returning limited results

[ahogbin]

I am running a query that uses the map command to take the values of one search for use in another (in my case Policy Numbers. The query I am using is

index=aalalive "Policy Number allocated for Quote" | rex "O [^A-Z]*(?<ENV>[A-Z\-\d+\s]+) \[" | search ENV="PRD*" | eval PRODUCT=substr(Quote_Number, len(Quote_Number)-2,3) | search PRODUCT=COM | map search="search index=stps policyNo=$Policy_Number$ contractStage="NB"" | dedup quoteNo | table quoteNo policyNo contractStage upperAccountName insuredName transactionPremium

I know the query is working as it returns the results I am after but for some reason it only ever returns 10 results. I can not for the life of me figure out why. If the actual number is less than 10 then all well and good. There is also no set pattern for the results returned (eg - running against current month returns a couple for the beginning of the month then a few 2 weeks later).

If I enter a specific policy number (ie: one not returned in the list of 10 and that fall within the searched time period) then the correct data is returned (so I know the policy exists).

If anyone can shed any light on why the results are limited to 10 it would be a huge help (the dedup command has no bearing on the number returned).

One very puzzled user.

Cheers,

Alastair

[sideview]

Color me crazy but this doesn't sound like a map command use case at all. Map is supposed to be a corner-case command that you use only use in pretty extraordinary cases, whereas this sounds like a straight up subsearch use case.

This search below does a straight-up conversion from your map search to a more standard subsearch search.

index=stps contractStage="NB" [ search index=aalalive "Policy Number allocated for Quote" | rex "O [^A-Z]*(?<ENV>[A-Z\-\d+\s]+) \[" | search ENV="PRD*" | eval PRODUCT=substr(Quote_Number, len(Quote_Number)-2,3) | search PRODUCT=COM | rename Policy_Number as policyNo | fields policyNo ] | dedup quoteNo | table quoteNo policyNo contractStage upperAccountName insuredName transactionPremium

And this version tries to be a little more explicit at the end as to what we're doing with the bits and pieces. Possibly some idiosyncracy of how map was arranging the final rows was what drew you to it in the first place, but that can almost certainly be done with a little stats.

index=stps contractStage="NB" [ search index=aalalive "Policy Number allocated for Quote" | rex "O [^A-Z]*(?<ENV>[A-Z\-\d+\s]+) \[" | search ENV="PRD*" | eval PRODUCT=substr(Quote_Number, len(Quote_Number)-2,3) | search PRODUCT=COM | rename Policy_Number as policyNo | fields policyNo ] | stats values(policyNo) as policyNo values(contractStage) as contractStage values(upperAccountName) as upperAccountName values(insuredName) as insuredName  values(transactionPremium) as transactionPremium by quoteNo

What was said elsewhere here, about using map but setting the limit to an arbitrarily high idea, is really not a good idea. The best case scenario out of that is going to be an incredibly slow search that uses quite a lot of resources. By comparison I suspect a subsearch refactoring like the one I've laid out above will do the same job in a small fraction of the execution time.

[ahogbin]

Thank you.. your suggestion makes good sense and I was never particuarly keen on have to hard set a number for the search results...

However, the initial search has moved on a bit as I found the data was not giving me what i was after so now have a search with a whole lot of xpath commands (to get the data I need from the xml).

index=aalalive "Policy Number allocated for Quote" | rex "O [^A-Z]*(?<ENV>[A-Z\-\d+\s]+) \[" | search ENV="PRD*" | eval PRODUCT=substr(Quote_Number, len(Quote_Number)-2,3) | search PRODUCT=COM | map search="search index=stps NEVO policyNumber=$Policy_Number$" maxsearches=10000 | 
xpath outfield=Net_Premium "//*[local-name()='AALNet' and *[local-name()='AnnualAmount']]/*[local-name()='EndOfTermAmount']" | 
xpath outfield=Net_Returned_Premium "//*[local-name()='AALNet' and *[local-name()='AnnualAmount']]/*[local-name()='ComparisonAmount']" |
xpath outfield=Gross_Premium "//*[local-name()='Gross' and *[local-name()='FortnightlyAmount']]/*[local-name()='EndOfTermAmount']"  |
xpath outfield=Gross_Returned_Premium "//*[local-name()='Gross' and *[local-name()='FortnightlyAmount']]/*[local-name()='ComparisonAmount']"  |
xpath outfield=Stage "//*[local-name()='TransactionInformation']/*[local-name()='ContractStage']" | 
xpath outfield=Status "//*[local-name()='TransactionInformation']/*[local-name()='ContractStatus']" | 
xpath outfield=Account "//*[local-name()='OrganisationName' and *[local-name()='TypeCode']]/*[local-name()='FullName']" | 
where Status="Closed"  | 
eval Gross_Premium=if(match(Gross_Returned_Premium , "0.00") , Gross_Premium , Gross_Returned_Premium) | 
eval Net_Premium=if(match(Net_Returned_Premium ,"0.00") , Net_Premium , Net_Returned_Premium) |
fieldformat Gross_Premium = "$" + tostring(Gross_Premium, "commas") | 
fieldformat Net_Premium = "$" + tostring(Net_Premium, "commas") | 
table Account policyNumber Stage Status Gross_Premium  Net_Premium

I have tried to play around with your example above but cannot figure out where to insert the xpath commands so that the search returns results - each time I run it it simply returns 'No Results'

Any pointers will be a great help..

A million thanks as always

Alastair

[sideview]

Well, since this seems to be the descendant of the index=aalalive search, I think it would/should replace the entire index=aalalive search string that is contained in those square brackets. ie everything between [ search index=aalalive ... to ... | fields policyNo ].

However, from a subsearch you have to use a fields command to restrict down to just the fields that you intend to pass out to the outer search. I won't try and explain how subsearches work, but instead you should take some time to read the official Splunk docs and tutorial/examples/etc http://docs.splunk.com/Documentation/Splunk/6.4.3/SearchTutorial/Useasubsearch

[svenwendler · Accepted solution]

From the docs:

maxsearches
    Syntax: maxsearches=<int>
    Description: The maximum number of searches to run. A message is generated if there are more search results than the maximum number that you specify. 
    Default: 10

So I suggest setting maxsearches to something higher 🙂

[ahogbin]

Thank you.. that seems to work...Is there a way to specify the number of results to be the number of supplied values from the parent search using a stats commant (count) to then pass the value as a variable to the maxsearches command ?

It seem crazy to hard set a max number when you do not know the possible number of matches.

[svenwendler]

Set it to something arbitrarily large ...

[svenwendler]

contractStage="NB""

looks a bit suss - should that be :

contractStage=\"NB\""

[ahogbin]

Makes no difference - thanks for the suggestion though.
I should add that if I run the first part of the search

index=aalalive "Policy Number allocated for Quote" | rex "O [^A-Z]*(?<ENV>[A-Z\-\d+\s]+) \[" | search ENV="PRD*" | eval PRODUCT=substr(Quote_Number, len(Quote_Number)-2,3) | table Policy_Number

The correct number of results is returned (in this case 24)

I have played around with the query but it stubbornly refuses to go above 10