I have a extremely slow search and I cannot understand why it is so. I'd appreciate any pointers. Hardware is not a problem, nor is the data volume. The search runs to 100% (in the web gui) in 5-10 minutes then spends approximately 1 hour before it changes from 100% to the Finished state. I suspect the culprit is the mvexpand command because when I check the dispatch folder for the search I find ~20 mvexpand_1, mvexpand_2, etc files of approx 100-200MB in size. From what I can tell Splunk seems to be reading/writing to these files the whole time. Longer the search duration the more of those files are present and the longer this stage takes. Is this normal for mvexpand or am I doing things in an inefficient manner? Sample data (all on a single line): <date> senderA@myCompany.com recipientA@myCompany.com;recipientB@myCompany.com;recipient@outside.com 1048576183 Search command: ... | makemv delim=";" recipient_address | mvexpand recipient_address | eval sender_msgtype=if(match(sender_address,"@myCompany.com$"),"internal","external") | search sender_msgtype="internal" | eval recipient_msgtype=if(match(recipient_address,"@myCompany.com$"),"internal","external") | eval msgtype=recipient_msgtype | bucket total_bytes span=1048576 | timechart span=1d usenull=f useother=t count(eval(msgtype="external")) by total_bytes where total_bytes>0 Running Splunk 4.1.6 on Windows 2008 R2 ... View more

Has anyone run into this? I've opened a case with Support but I thought I'd ask here as well. None of the Windows inputs (WMI, perfmon, registry) are setting the proper value in the hostname field. I've set the fully qualified domain name in the hostname parameter of the inputs.conf file (hostname=server.domain.com) however the events have what appears to be the contents of the COMPUTERNAME environment variable (hostname=SERVER). All other events have the expected hostname value (hostname=server.domain.com). There's a note in the Release Notes for Splunk 4.2 describing changes to how the default value of hostname is derived. However, the workaround described there is to set the hostname in the inputs.conf as I've done. I doubt it's something to do with my environment as I've reproduced the problem on a fresh install of Splunk without any custom configs. I'm using Splunk 4.2.2 on Windows 2008 R2 by the way. ... View more

I have some settings that I want to apply to several sourcetypes with similar names. Can I do something like this in props.conf? [examplesourcetype*] stuff... Instead of: [examplesourcetype1] [examplesourcetype2] ...and so on. The documentation doesn't say it's forbidden but it doesn't explicitly say its possible either. You can do wildcards with sources & hosts so it's not such a stretch to think sourcetypes should work too. Does anyone know for sure either way? ... View more

I'm getting thousands of instances of the following error on my indexers. 02-16-2011 02:20:02.921 ERROR TcpInputProc - Error encountered for connection from host=ftlpprint01, ip=10.13.98.68. Winsock error 10054 According to MSDN error 10054 means connection was forcibly closed by the remote host which in my case would be a forwarder or LWF. We have 6 indexers all running Splunk 4.1.6 on Windows 2008 R2 x64. There are ~150 LWF and 6 regular forwarders all running Windows & Splunk 4.1.6 and all of which send to the indexers using AutoLB. All generate that alert to some extent. The odd thing is that about 20% of the forwarders are connecting over the WAN but the the majority of the messages are for connections with a forwarder in the same site as the indexers. Is this normal (doesn't seem like it) and if not how would you suggest I start troubleshooting? ... View more

http://www.splunk.com/wiki/Community:Splunk_Visio_Stencil I'm not able to download the Visio stencils linked above using IE8 on Win7. Windows Explorer generates an error because the downloaded zip file is corrupt. Downloads properly when using Firefox though. Thanks to ftk for pointing out that the file wasn't corrupt & providing an alternate download link. ... View more

I have a folder containing logs as below. I want to exclude all directories not named DONTINDEX_* and index the contents of every other subfolder of 'logs'. logs\ a\ b\ DONTINDEX_a\ c\ DONTINDEX_b\ d\ Using _blacklist inside the [monitor:///logs] stanza works but Splunk still scans the DONTINDEX* folders. Problem is that those folders contain 100k+ small files which slows indexing & places a heavy load on the source server. How do I exclude the entire folder without specifying a separate monitor stanza for each folder I want to scan (a,b,c,d)? I'm running Splunk 4.1.2 on Windows 2008 R2. ... View more