I have a extremely slow search and I cannot understand why it is so. I'd appreciate any pointers.
Hardware is not a problem, nor is the data volume. The search runs to 100% (in the web gui) in 5-10 minutes then spends approximately 1 hour before it changes from 100% to the Finished state.
I suspect the culprit is the mvexpand command because when I check the dispatch folder for the search I find ~20 mvexpand_1, mvexpand_2, etc files of approx 100-200MB in size. From what I can tell Splunk seems to be reading/writing to these files the whole time. Longer the search duration the more of those files are present and the longer this stage takes.
Is this normal for mvexpand or am I doing things in an inefficient manner?
If it were me, I'd throw away the data I don't want before I start expanding things. I'm assuming you want a count over time of external emails sent by internal users. I don't think you need mvexpand, because if there is 1 external email in the recipient list, the email is counted as "external", and by expanding the results, you may end up with multiple counts of the same email (i.e. the same email goes to 2 or more external email addresses).
sourcetype=WHATEVER sender_address="*myCompany.com*" |makemv delim=";" recipient_address|eval recipient_msgtype=if(match(recipient_address, "@myCompany.com"),"internal","external")|bucket total_bytes span=1048576|timechart span=1d usenull=f useother=t count(eval(recipient_msg_type="external")) by total_bytes where total_bytes>0