Optmise dedup Splunk search

[tomjb94]

Hi - 

I am looking to optimise this search by removing dedup, the idea of the search is to remove duplicate paymentId fields & create a table with the fields specified under the stats count.

The search currently takes 300+ seconds to run for 4hrs worth of data.

AND card_issuer_stats AND acqRespCode!="0" AND acqRespCode!=85 AND acqRespCode!=100 AND gwyCode="test123"
|dedup paymentId
|stats count by acqCode,acqRespCode,mainBrand
|sort -count

Any help would be greatly appreciated, thanks

[PickleRick]

Check the "inspect job" screen. Most probably dedup is _not_ your main culprit here.

You're using a lot of non-equality checks which means (assuming this is in your initial search) that Splunk has to parse each event containing string test123. If there's a lot of data there's a lot of parsing.

BTW, unless you're absolutely sure about your data, your process with dedup is flawed - dedup just filters out subsequent occurrences of given field in any event regardless of other fields so are you absolutely sure that deduping at this point does not influence your stats further down the pipeline?