I'm a little confused as IST appears to be 5:30 ahead of BST? but that is a bit more than that. Anyway, at a first guess have you gone into your user profile and changed your local timezone? Splunk will use this to adjust the timestamp to appear in your local time, so the event may be correct but the displayed time is different. If the event is now different then your data source has a timestamping issue ... View more

I was talking about the actual value, including spaces in and around the equals is fine. This was actually a network issue in the end. ... View more

Have you checked the firewall rules? By default the firewall is enabled and blocking on CentOS ... View more

It doesn't look like a sideview utils issue to me. Is this a single head/indexer setup and is the box under load? Splunkd not responding can often be because the box is under load, I believe there is a hard coded timeout between the Webui and splunkd. Otherwise there have been a few others posting recently about not responding issues ... View more

Its worth adding that if you install Splunk via the rpm on linux it will handle this for you ... View more

I think the problem here is that unlike a lot of other users you're moving from compressed files to more compressed files, thus you may be finding the size inflating. My suggestion would be that the metadata added to the rawdata is causing a lot of this inflation, Splunk will append the usual metadata, host,source, sourcetype and so on to each event in the .gz file which can cause it to inflate. Normally users would experience a reduction of around 40-50% in data size due to the compression Splunk applies but in your case, its decompressing your data, appending additional metadata and then storing it in a compressed archive. In fact thinking about it, each event is so small that the metadata is definitely longer than the original event.. well you would probably experience this kind of inflation. Each event is effectively more than doubling in length before compression. Could you perhaps change the event breaking and search over multi-line events instead? Probably not but I thought I'd throw it out there ... View more

http://docs.splunk.com/Documentation/Splunk/5.0.2/Knowledge/Usesummaryindexing Futhermore, you can then expand Splunk horizontally. Its biggest strength is in its distributed setup, the more indexers you have the more efficient the search is as it can start to utilise map/reduce functions to increase the speed of statistical searches. I've deployed Splunk in a telecoms environment where we are monitoring switches and routers across Europe on well spec'ed machines with no performance issues. ... View more

I wouldn't neccessarily trust the number of events per second, this will vary wildly depending on timestamps, bucket settings, structure of the data etc. Also its worth pointing out the that is the bare minimum for running Splunk, we wouldn't run it on anything like that in production. Now its true that running a statistical command will cause a hit on the system but then you need to think about how to architect your system. Firstly I would say that for a 30 day statistical analysis I would run an hourly or daily scheduled search to build summary data which you can then search quicker. ... View more

Splunk is perfect for your needs but I suspect you haven't look at the requirements? Splunk locks a core each time it runs a search, if you have a single CPU you have to allow for the fact that it will be handling the OS, other processes, any searches you are running and any possible scheduled searches you may have setup or have from other apps along with monitoring files. Add to this that the OS scheduler will be swapping it in and out and you will find it creak to a halt. Secondly IOPS are critical, try running bonnie++ to see how fast you can read/write to disk and this will dictate how quickly Splunk can run searches and process data. Reference Hardware - http://docs.splunk.com/Documentation/Splunk/latest/Installation/Referencehardware What you want to achieve is perfect for Splunk, the way it stores and searches data makes it considerably more agile than a normal relational database but you do have to give it some hardware to get it off the ground. ... View more

Out of interest, does blocked=true appear anywhere in the metrics.log on the indexer or forwarder? ... View more

I expect there are just some issues with their hosting atm. Normally its pretty good ... View more

After starting Splunk are the index sizes the same on both servers? Is it just defaultdb you transferred over? Also, have you checked the permissions on the buckets you've moved over? Just to be sure Splunk can read all of them. Final check might be to run FSCK on them to check their integrity; http://docs.splunk.com/Documentation/Splunk/5.0.2/Indexer/HowSplunkstoresindexes#Troubleshoot_your_buckets (Also, if an old answer doesn't help it might be worthwill posting a comment instead of voting it down when it answers the original question 🙂 ) ... View more

Could you perhaps post your xml? ... View more

Ok, there are a few pointers to go through here; Everytime you run a search it will spawn a new Splunkd process, each process will consume a CPU core. From what you've said you need to be sure that you can support up to 15+ instances. It is most likely caused by users running searches combined with scheduled searches you may have. The old adage that running Splunk on VM's is bad is a little out of date nowadays. If deployed correctly you won't experience too many issues, the key problem is ensuring that the indexer can get about 800 (Really I'd aim for 1200) IOPS on whatever storage it has, giving it native read/write access to a disk can help improve this. VM's are great for Splunk because they make the deployment of Search Heads quite simple, just ensure it has some IO and then load up the CPU's. Finally, are these all on the same host? How many network ports does the box have? If its a single port and you've got data flowing over it into the indexer, perhaps tcp acks coming back out, users connecting to run searches etc.. its quite possible you're running it near capacity. Have you checked any of the box performance metrics? ... View more

Well splunkweb is actually a python based web server called cherrypy so in your list of processes you should have a splunkd plus something like python -O /opt/splunk etc etc. Also its worth noting that it uses its own instance of Python, you don't need another independant install. In your case, if you've done a ./splunk start and the ./splunk status command shows everything running, you probably just need to add a firewall rule for port 8000 on the CentOS box. On a new install by default it should probably be blocking it. ... View more

This has been covered in a couple of threads, one more about the feature and another about options. http://splunk-base.splunk.com/answers/67574/fschange-deprecated-what-options-are-available http://splunk-base.splunk.com/answers/63874/why-is-fschange-a-deprecated-feature-in-splunk-50 ... View more

I'm not entirely sure what Splunk has done wrong here? On a free license you can index up to 500mb a day, if you blow this its alright. Splunk will let you blow this limit three times within a 30 day rolling window. That means that if on day 1 you use 600 meg you have a violation, if 29 days later you have a second violation then the window resets and you have 2 violations and need to have none for 30 more days to clear them. If you exceed 3 then you lose the ability to search. That seems somewhat fair to me? Its Splunk allowing room for people to sometimes blow the limit or run batch jobs, if you always go over then its an indication that you need a bigger license. Also you don't need to dump and reindex, this won't affect the license. Best fix is to backup your buckets and reinstall Splunk, move the buckets back over and carry on as usual. So in short, Splunk would have given you three warnings and as you've pasted above, you used 818% of your quota..... Also Splunk hasn't made anything more restrictive, its the same free license that was there a version or two ago. EDIT: Just double checked, the Splunk free model hasn't changed since 4.1 (Which is also when it was introduced) ... View more

Well played sir, well played ... View more

But why use Splunk? You'd be better using a dedicated monitoring solution or some other cronjob to monitor for it ... View more

Why use another instance? Why not just do it the usual way of a cronjob checking every X minutes and firing an email if its down/trying to restart it? ... View more

Within the Google maps app itself are a series of comprehensive examples that you can use to build this behaviour ... View more

The blue one ... View more