I'm seeing the same thing with our VM indexers and Intermediate forwarders. I've load Wireshark some of my indexers and forwarders. What I've seen is that the intermediate and indexers are sending tcp resets immediately and tcp zero windowing. Digging in further I started to watch disk transfer rate, system processor queue, Network bytes in/out, thread counts, thread status, cpu % time. I recommend watch these stats to help diagnose your issue. Also try using a sniffer like wireshark. The TCP reset seem to occur in bulk when System processor queue and disk transfer rates are high. System processor queue usually mean you have threads waiting to be processed. I am assume that in my case during high disk transfer time I am not able perform network tasks fast enough and commit to disk causing TCP zero windowing and TCP resets. Adding more processors to a VM may cause more problems as all processors have to be schedule simultaneously or the VM waits to long to be schedule on the physical resources. If you have an antivirus application on my servers exclude all splunk process and directories. This seems to help. If you have a bunch of thread queuing/waiting try disabling no critical services. If you are on a windows server you can disable tcp auto tuning. Windows has three primary reg keys that can help: TcpTimedWaitDelay (Tcp Connection release after fin or reset), MaxUserPort (maxium number user ports for applications), TcpNumConnections. This settings done help too much as auto tuning is fairly good in windows server 2008 and 2012. For more information try reading; Windows TCP tuning ... View more

Valid response, but has anyone or Splunk performed any testing regarding this? ... View more

Consider using a drop box (file share) with an rsync process to move the files. Also have corporate IT create a network share of the deployment directory on the deployment server giving you or your group access. However this does not get around the issue of having to run **./splunk reload deploy-server. ... View more

@yannk, I thought the default behavior was that of a light forwarder? ... View more

I currently have the same problem in my test lab. ... View more

That depends if you have the run splunk disable local-index. Prior to 5.0.2 _internal index was not forwarded only the _audit was. In later versions all indexes are forwarded if the outputs.conf is defined. Additional Reading: - Routeandfilterdatad Hope this helps or gets you started. ... View more

I know Splunk does not tolerate cpu ready time. Do you know how to calculate percent CPU ready time? - (CPU summation value / ( * 1000)) * 100 = CPU ready % -(1000 / (20s * 1000)) * 100 = 5% CPU ready Realtime: 20 seconds Past Day: 5 minutes (300 seconds) Past Week: 30 minutes (1800 seconds) Past Month: 2 hours (7200 seconds) Past Year: 1 day (86400 seconds) ... View more

Is that search head acting as indexer? ... View more

Question are you using CPU reservation or affinity? Also have looked at your virtual network metrics and os network stats ( retrans, drop packets in/out, connects established/passive/active)? What vmguest stats have you viewed? ... View more

Welcome, In terms of performance sub searches consume two CPUs ( one for the main search and one for the subsearch). Subsearches have a significantly lower returned event limit. Performing using field extractions in your base search is costlier as Splunk has to apply extracts immediately. I recommend specifying index if possible. Consider using fields before performing any stats or eval this will improve search performance as Splunk only has to work with result necessary for final output. #if you are looking just for bob index=myindex sourcetype="WMI:UserAccounts" Bob | dedup host | fields host The easiest way without the expense of a sub search. #if you are trying to find where bob has not been index=myindex sourcetype="WMI:UserAccounts" NOT Bob | dedup host | fields host or index=myindex sourcetype="WMI:UserAccounts" | fields host, user | where NOT user="Bob" Take the time to Read Splunk "SEARCH PROCESSING LANGUAGE (SPL) PRIMER AND COOKBOOK" its a free PDF download. Hope this helps or gets you started. Dont forget to vote up and accept answers. Cheers, ... View more

I tried to create two capture groups . $1 = Console [hype . $2 =] log at 2013-08-15 00:00:00 PDT. 2013-08-15 14:25:48 Setting hostname hype362: [ OK ]. The two capture groups exclude the 33 value. using format = $1$1 to replace _raw the event should contain the whole event excluding 33. I know this works during indexing phase and should doing search. ... View more

You will need to use a transform.conf and props.conf. You you will do a capture and exclude the values you don't want and apply it at search time with REPORT. I didnt check my regex but this should give you some ideas. #transform.conf [data-anonymizer] REGEX = (?m)^(.*\[\w[^\d])\d+(\].*) FORMAT = $1$2 DEST_KEY = _raw #props.conf [yoursource] REPORT-anonymizer=data-anonymizer Hope this helps or gives you some ideas. Dont forget to vote and accept answers that help. Cheers ... View more

what index are you storing your DC Security Event Logs? Also do you know what the event id associated with the a log on event is? ... View more

Events are constantly being stream from your forwards including your script. You can create multiple apps with the deployment server and have each app have its own interval. No sure what you are trying to solve. ... View more

Question are you picking your syslog data from a flat file or UDP stream? Is every syslog entry being sent to one syslog? ... View more

Be careful with using real-time searches as you are grabbing events from the stream. If you have queue blocking on the forwarders or indexers where events are delayed, realtime charts may give an incomplete view of whats happening. ... View more

probably little more that you wanted. I cant see the XML since its blocked for me. <d:Message>\(\d+/\d+\)\d\d:\d\d\s(?<eventtype>[^\s]+)\son\s(?<location>[^\(]+)\s\((?<direction>[^\)]+)\)\safter\s(?<area>[^\.]+) If your useing IFX might looks something like this. <d:Message>\(\d+/\d+\)\d\d:\d\d\s(?<field1>[^\s]+)\son\s(?<field2>[^\(]+)\s\((?<field3>[^\)]+)\)\safter\s(?<field4>[^\.]+) The return fields: eventtype=Accident location=Aye direction=toward Tuas area=Jurong Port Rd Exit I've tested it with the sample give above. Hope this helps or gets you started. ... View more

I am not sure of the fields or the data format, but you could do some of that. Create search where fields dont contain posts or friends data. ... View more

It would help if you posted the stanzas in your .conf related to this. ... View more