I am at a site where we are using a Splunk Forwarder to mount a DFS share and read EVTX Archive Files placed there by another entity. The Splunk Forwarder is 4.1.5 x64 on Windows 2008. The Splunk Forwarder is also a Search Head federating search requests to the same indexer where it is sending these evtx logs (not that it should make a difference). There are no problems with the reads/parse of the EVTX files; however, we appear to have a problem with modifying the Splunk Metadata for these events. Ideally, we would customize the Index field. This appears to not work. In the inputs.conf I have tried setting the index and the sourcetype but no matter what I enter, here is where the events show up: index=main sourcetype=WinEventLog:Security I have also tried using props/transforms to set the Metatdata DEST_KEY on both the Forwarder and the Indexer (several ways). No matter, what I select or set, it appears that I can have no impact on the index or sourcetype. My thought is that since the evtx files are binary, that a separate process handles these that isn't accepting modification. Is this crazy or some limitation of the (evt|evtx) parser? Sean ... View more

At a site, we are doing 5000-20000 EPS (Events Per Second) all udp/514 at each of 10 locations. At this time, we are required to do this all with a single server/indexer in each location. This week we made syslog-ng as the listening service (instead of Splunk) since it seemed to work better regarding the other log recipients (see below). Syslog-ng listens on udp/514 and then: Forwards 100% via udp/514 to another direct-connected logging system Forwards a certain "match" of lt 1% to Netcool Writes 100% to a file Splunk reads the file and seems to be performing well; however, the system reports many UDP errors indicating syslog-ng is not receiving it all. Tests with a traffic-generator (loggen bundled with syslog-ng) show that a fraction of what is sent gets processed by syslog-ng. A tcpdump at the same time shows the EXACT packet count that was sent was received, about 1/3 dropped by the kernel and about 1/3-1/2 of what was sent is what makes it the syslog-ng file. Applying the tuning tweaks below have reduced the number of drops substantially; however, they have not been eliminated. Maybe it isn't possible in this situation, but I think it should be. Questions Besides a packet capture, is there anything in RHEL (built-in preferred) that provides a counter of UDP packets attempted and not just those that are received (see comments on netstat -su output below)? Is it possible to receive all packets sent yet still get UDP errors? What is a reasonable amount of syslog EPS to expect with this hardware? Are there any other tweaks to the Kernel or Syslog-ng to make? Environment Indexer: RHEL 5.4 2 6core processors – 12 cores 24GB Memory 5TB SAS 15k disk Data: Syslog (udp/514) from Palo Alto Firewall Data from several direct-connected firewalls (same switch) Messages are 250-350 bytes mostly, with some just over 400 and NONE over 500 bytes. ERRORS netstat -su shows output like Udp: 3436531 packets received 1553 packets to unknown port received. 8402 packet receive errors 109202 packets sent RcvbufErrors: 8402 The "packets received" is how many were accepted by the listening application and NOT how many were attempted The "packets to unknown port" is what came in for the application when the application was down or not listening (e.g. during a restart) The "packet receive errors" clearly indicates an error; HOWEVER, it is not a one-for-one packets-to-error. You can have many more errors than packets were sent, indicating it is possible for a single packet to generate multiple errors. Tuning Steps These are similar to what was done; however, multiple values were tried so the numbers below are not exactly what is in production now: We tried turning off the udp/514 forwarding to the other applications but we did not see a noticeable drop in errors kernel net.ipv4.udp_rmem_min = 131072 net.ipv4.udp_wmem_min = 131072 net.core.netdev_max_backlog=2000 net.core.rmem_max=67108864 syslog-ng options { sync (5000); time_reopen (10); time_reap(5); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); log_fifo_size (536870912); stats_freq(60); flush_lines(500); flush_timeout(10000); }; ... View more

Does anyone know of work that has been done to make Splunk talk to Sourcefire's eStreamer (log API) available on their Defense Center? Does eStreamer integration require a strategic partnership between vendors or do they post the API info so that anyone can play? Thanks, Sean ... View more