I downvoted this post because that is not a solution. It requires that the pid file be removable. The process you link to will fail because the file-system has been set to read-only by the kernel, a situation which implies a significant system fault which needs to be addressed. The very problem being explicitly reported by the start-up is that the pid file cannot be removed. ... View more

This is a 3 year-old question. You should, really, if you want to garner a new answer, open a new question. That is largely irrelevant, though. As it is all of my previous responses stand. There's nothing new or different about your problem that isn't already addressed above. The file-system in read-only mode, when to have written the PID file it must have previously have been read-write suggests a file-system error was detected by the kernel. You need to check your logs and dmesg buffer for disk and file-system errors and remediate the problem. Until you do, the system is highly suspect and at risk. You may want to consider backing it up before you do, but you're going to need to reboot (with the understanding that if this is the first manifestation of a bigger fault, it may not come back). Read my answers above. This isn't a Splunk problem, it's a fundamental sysadmin problem. ... View more

Years on and this is - as far as I can tell - haunting us. The choice of 24 hour clock isn't regional. Nor is, say, the preference for ISO dates. There needs to be an easy way to set the base standard for the local search instance. ... View more

You should still be able to perform searches on Splunk's own internal data sources, even if general indexes/sources are not available. Try this to look at where your data sources are: index=_internal (source=*/metrics.log* OR source=*\\metrics.log*) group=per_sourcetype_thruput | timechart span=1d sum(kb) by series | addcoltotals | addtotals The resulting table should show you on a daily basis what is causing your excess. Also you need to be aware that access to the general data will remain blocked as long as 3 or more days of excess remain in a window of the last 30 days. With the Enterprise licence this 5 excess days, which could explain why things broke when you switched to the Free licence - if the licence was expired by 4 days you would have exceeded a cap of zero by 4 days, but the interface would still work. On switching to free you only have a leeway of 2 over-capacity days in 30, which would cause you to be excluded from general data. The only solution to that is to address the excessive data, and then wait until the window is no longer exceeded - worst case scenario, 28 days. ... View more

It's not true to say a paid licence does not expire. It IS possible to purchase a perpetual licence, but they don't come cheap (and they still have an expiry date, just one that's 20 years or so hence). Ordinarily paid licencing expires annually. ... View more

I'd be inclined to fire off the alert to a single, collective address, and have the mail server expand it. That way you don't have to maintain the mail list if recipient addresses change. It becomes part of the natural id management of users. It's also just more readily achievable. Keeping your roles in text files seems a strangely archaic way of doing things. Do you not manage authentication, roles, mail groups, etc. with some kind of centralised directory service like LDAP or AD? On a Linux Splunk server, you could conceivably have a cron job which recreates an app-packaged search/alert config, or simply use an address list as a recipient list in an alert-spawned script-generated e-mail. ... View more

Splunk is not designed to run multiple instances. That doesn't mean you can't. But it is unecessary hard work when there are so many ways to just virtualise. ... View more

Search and tabulate the entries by punctuation, and see of there's a spike in any particular pattern. Tabulate by hour, and look for the start of the increase. Find the source of the anomaly. The answer to your question is in the very data itself... One thing to look out for is that you're not doing some weekly rotation and ending up with all the renumbered rotated logs being reindexed. (If you're using log rotation you also need to implement appropriate black/whitelist rules.) ... View more

Well, funnily enough, the GeoIP database is exactly what is at fault here. Following up over 2 years on, but here's an article that references exactly this: http://fusion.net/story/287592/internet-mapping-glitch-kansas-farm/ ... View more

Indeed it should - duly corrected. ... View more

84600 seconds is actually 23:30. I think you mean 86400. ... View more

There are methods for removing indexed data from visibility, and then re-indexing. But that would eat into your daily consumption cap. There is no method that I know of for bulk manipulation of extant index data. ... View more

Well, again, you're assuming that the Heavy Forwarder is at fault in not forwarding. I'm still suspicious of the UDP service. It could be that the system is dropping it in favour of something else, with the service restart prompting a reset. ... View more

My first thought is have you guaranteed that at periods when you think it is not being forwarded it is actually getting there in the first place. UDP is not guaranteed delivery, and may end up dumped on the floor in times of congestion or failure. You're better off using a proper Syslog sub-service over TCP, and locally ingesting the log files generated. (As it happens, this was also Splunk's official advice last time I looked.) My faithful recommendation is to use syslog-ng for its flexibility of configuration. ... View more

That doesn't help you, at least not greatly. The search is still going to appear in the logs when it is executed. It only obscures it from direct view in the UI, so again, any administrator will be able to see it with ease if they choose to go looking. It still doesn't provide a total solution. ... View more

OK, well I understand your problem, but regardless of the intent or motivation the reality doesn't change. Regardless of the fact someone didn't like my original answer, the fact remains it can't be done. You can't do it with file permissions, because Splunk as an entirety runs as the same system user (more often than not with sysadmin rights which will override any permissions anyway), and at the application level a user account with administration privileges has total access to everything within the application. Short of setting up a dedicated Splunk search head administered by the right people, you simply cannot ring-fence the data. ... View more