More information also on troubleshooting your buckets at bottom of the page in http://docs.splunk.com/Documentation/Splunk/4.3.2/admin/HowSplunkstoresindexes ... View more

If it were me, I'd throw away the data I don't want before I start expanding things. I'm assuming you want a count over time of external emails sent by internal users. I don't think you need mvexpand, because if there is 1 external email in the recipient list, the email is counted as "external", and by expanding the results, you may end up with multiple counts of the same email (i.e. the same email goes to 2 or more external email addresses). sourcetype=WHATEVER sender_address="*myCompany.com*" |makemv delim=";" recipient_address|eval recipient_msgtype=if(match(recipient_address, "@myCompany.com"),"internal","external")|bucket total_bytes span=1048576|timechart span=1d usenull=f useother=t count(eval(recipient_msg_type="external")) by total_bytes where total_bytes>0 OR sourcetype=WHATEVER eval sender_msgtype=if(match(sender_address,"@myCompany.com$"), "internal","external")| search sender_msgtype="internal" |makemv delim=";" recipient_address|eval recipient_msgtype=if(match(recipient_address, "@myCompany.com"),"internal","external")|bucket total_bytes span=1048576|timechart span=1d usenull=f useother=t count(eval(recipient_msg_type="external")) by total_bytes where total_bytes>0 ... View more

I've just noticed another very interesting thing. I ran splunk-wmi.exe to get a better idea of what it does, which isn't that much--just passes wql to the WMI provider. What it does NOT do is return any kind of hostname. So where is that coming from? If this can be changed to something sensible for WMI inputs, all will be well. ... View more

I'll concur with piebob and erga00. I also have over 10B events indexed without issue. If you have a single server on older hardware with default network setting, you might be seeing issues during large searches. Things that will be helpful in understanding why you might be seeing slowness; number of Splunk servers: If more than 1 server, what roles do they operate: server OS: server hardware (memory, CPU, NIC (speed and #)): splunk version: splunk LFCs or UFCs reporting to indexers: index=* host=* earliest=-7d | table host | dedup host number of searches run (Expensive searches): Search > Search Activity > Search details ... View more

@fervin: Please post any more information you may receive on this. Thanks! ... View more

check this, http://wiki.splunk.com/Community:MoveIndexes ... View more

I was told this issue was fixed in Splunk 4.3 but I haven't tested as I'm still running an older release. ... View more

I've done some analysis of timing, pattern, etc of the errors and found nothing that stands out other than it's related to the volume of data being forwarded. I'd have expected forwarders relaying via a WAN link to have more errors but that's not the case. The worst offender is on the same LAN segment as the indexers. I'm going to start with the Networking team and see what if any network devices are between the indexers and forwarders. Thanks for the insight. ... View more

Updated icon sets now available in Visio 2010, Visio 2002, and Omnigraffle, plus an icon style guide! ... View more

Is it reasonable for you to have a parallel directory containing symlinks to the subdirs you wish the monitor? For example: logsyms/ a -> /logs/a b -> /logs/b c -> /logs/c d -> /logs/d This would do what you want, as you would be monitoring the logsyms directory, and DONTINDEX* wouldn't come into play. We could even make the 'source' field look like it's coming from /logs, if you need it. Whether or not this works for you depends more on the specifics of what those subdirectory names are... ... View more

Okay, What is not working? I have created a SOP that is specific to my environment, but I am happy to share it with you. There is a link I posted on how to setup a ARR reverse proxy through IIS. The key is to install the models in the sequence listed above. After that point IIS to the Splunk\bin folder, then ensure ISAPI_Filters have entries to the Helicon ISAPI models, then configure URL Re-write to turn on the proxy component of ARR. My helicon httpd.conf looks like the following: i. #Helicon ISAPI_Rewrite configuration file HTTPD.CONF ii. # Version 3.1.0.104 iii. # iv. RewriteEngine on v. RewriteLogLevel 0 vi. LogLevel info vii. RewriteCond %{REMOTE_USER} .\([^\]+) viii. RewriteHeader Remote-User: . %1 ix. RewriteBase / x. RewriteProxy ^(.*)$ [NC,P] Lastly there is some tweaking in the \etc\system\web.conf to enable SSO. HTH ... View more

Thanks. Is there an ETA on 4.1.4? ... View more

The expansion of the sourcetype into multiple sourcetypes allows the system to store the list of fields on a per-file basis, since there's no guarantee it doesn't change. Why the first one is -2, instead of nothing or -1, is some minor internal misbehavior that doesn't really change the situation, since you'll get multiple sourcetypes eventually. This behavior forces you to account for multiple sourcetypes in your searches manually, by eg folding them back together using eval and such. It's a pain. ... View more