In that case you seem to have firewall or routing issues. ... View more

If you have to run as root that is a kludge not a proper fix. You should never have a process like Splunk as root. It is far too open to abuse. The correct response is what the user is trying to do - find the real fault and fix it. ... View more

Do you understand the script, or are you simply relying on a legacy job instated by someone else? You give no indication as to what the script is, or your underlying platform, so it is hard to answer the question. A number of generic possibilities: Have you tried running the script manually? Does it even run any more? Has someone screwed your source permissions (or indeed the run permissions of the script) in such a fashion that the two are now incompatible? Has the data the script works from become corrupted, or too large for whatever tool you are using to manage? Have there been any updates to the system? Perhaps an update of some tool critical to the success of the script has been broken by the update. Have you checked for disk space problems wherever it is that the script places any intermediate or output files? Have you checkpointed the script? If it is running on unix you could try "touch"ing semaphore files to track progress in the background. In short - have you done any debugging before assuming it is the scheduling which has broken? ... View more

The cron notation is just fine - the zeroth minute of every hour. And besides, if you didn't change it why would it be wrong now when it wasn't before? ... View more

I have since checked the Enterprise installation at work and found that exactly the same thing was happening there, but in comparison with the production logging the duplicated Splunk data were a drop in the ocean. ... View more

Did you ever get this resolved? Did you consider using $SPLUNK_HOME/bin/splunk btool outputs list to make sure your total config was what you expected it to be? ... View more

Not quite. Search head says to all indexers: I want everything you have for "search x"; indexers say "here are my results" but that return traffic will be very little if they return empty results. ... View more

Very simple: use the "stop" icon (white cross; red circle). ... View more

Is the new destination configured for Enterprise licencing, yet? Just a thought. ... View more

And you could do me a favour by clicking the "accepted answer". ... View more

I have made a correction to the blacklist regex. ... View more

I think you're missing the point. The regex is a pattern match to to target the format of the records within the text files, not the text file names. I am assuming that the text files follow some regular format. The regex matching means that yes the files get processed, but only the matching records will actually be indexed. ... View more

Right, well I've just had a look at the latest CentOS 5 netstat source, and at first glance it seems incapable of rendering buffer the errors, not that it is simply not compiled in. (More accurately it seems the snmp parser library is probably incapable of recognsing them.) It looks like you will have to awk /proc/net/snmp. (Yes, I know there are later versions of CentOS, and possibly more recent versions of the source, but that is the current version I have, and it is relevant to the discussion here seeing as we already agree it is the version we both have.) ... View more

In its simplest form you just need something like the following stanza in the inputs.conf on the rsyslog server. (I assume, from the mention of other logs already being pushed, you have installed a light forwarder instance at the very least.) [monitor:///logs/server-logs/] host_segment = 3 sourcetype = syslog index = syslog disabled = false As splunk try the following: bin/splunk btool inputs list --debug This should show you your complete inputs configuration. To interrogate what is or is not being consumed, point this at your indexer: https://{yoursplunkserver}:8089/services/admin/inputstatus/TailingProcessor:FileStatus It will show you what is and what is not being processed, and why. ... View more

In that case your first stop is to find out if their kernel(s) is(are) even counting UDP buffer errors, and if they are does the client even HAVE a version of netstat that returns the data? If not you could parse it out of /proc/net/snmp (assuming the fields are there, of course), but that's dirty. But if course, bottom line is you can't report what's not there. ... View more

I don't see specifically what this has to do with Splunk, but my netstat -su produces similar output to yours. The answer is quite simple: $ sudo cat /proc/net/snmp | grep -i udp Udp: InDatagrams NoPorts InErrors OutDatagrams Udp: 13451987 2317 0 14525121 Conclusion - the kernel is not recording the error counts. If it was, the first line of output would be: Udp: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors Hmmm - actually, let me amend that. I have one kernel here which is counting them, but netstat is not reporting them. Perhaps this is because they currently stand at zero. Or perhaps it is simply we neither of us have a recent enough version of netstat: $ netstat -V net-tools 1.60 netstat 1.42 (2001-04-15) Fred Baumgarten, Alan Cox, Bernd Eckenfels, Phil Blundell, Tuan Hoang and others +NEW_ADDRT +RTF_IRTT +RTF_REJECT +FW_MASQUERADE +I18N AF: (inet) +UNIX +INET +INET6 +IPX +AX25 +NETROM +X25 +ATALK +ECONET +ROSE HW: +ETHER +ARC +SLIP +PPP +TUNNEL +TR +AX25 +NETROM +X25 +FR +ROSE +ASH +SIT +FDDI +HIPPI +HDLC/LAPB +EUI64 I can't seem to find a definitive answer as to whether the support needs to be compiled into netstat, whether the version is simply not current enough, or whether it simply does not report zero values. What version to you have for an example that works? I presume you have seen a working example or you would not be asking the question. ... View more

First let me stress I have not done this, and I am not even completely confident of the file syntax, but I suspect the path you want is something along the lines of: $SPLUNK_HOME/etc/system/local/inputs.conf [monitor://C:\Users\angeliga\Filer\...] disabled = false whitelist=*.txt followTail = 0 sourcetype = my_type $SPLUNK_HOME/etc/system/local/props.conf [source:://C:\Users\angeliga\Filer\...] TRANSFORMS-set=droprecord,userecord $SPLUNK_HOME/etc/system/local/transforms.conf [droprecord] REGEX=. DEST_KEY=queue FORMAT=nullQueue [userecord] REGEX={targetmatch} DEST_KEY=queue FORMAT=indexQueue This assumes that rather than targetting the .txt files within the .zip file, you have a record structure you can target for the "userecord" regex. Certainly, if I were to investigate this is where I would begin, but I could be entirely and utterly wrong. It is at best an educated guess. I will be watching with interest to see if there is, in fact, a direct solution to what you want to do. ... View more

Sometimes it's the obvious that gets overlooked. I do it all the time. ... View more

The internal logs DO contribute toward your licence usage. At the very least the throughput is logged in the audits files, and contributes to the usage figure returned by a query I was given by Splunk support for the Enterprise installation at work. In an Enterprise installation the Splunk logs are just a drop in the ocean so go unnoticed. In the free installation they certainly seem to get recorded as contrbuting to licence consumption. ... View more

I posted the question and the answer together, because it took me a few months to track the "obvious" down, and thought it might be informative to other users. I am aware that the time conversion in the search is sloppy, but I first created it months ago, before I learned about the "convert" function. ... View more