Once you turn your indexers into slaves, they do not write their information on license_usage.log locally. This information is all written to license_usage.log on the license master. In order to get visibility into this again you need to do one of the following: 1. install deployment monitor on your license master 2. just add the license master as a peer (via distributed search) to the box that is currently hosting the deployment monitor app. Either of the two should work, however, the second is somewhat simpler. Cheers! .gz ... View more

This can be achieved by the following search: source="set1.log" | JOIN type=left MAC [search source="set2.log" | eval x=1] | Where NOT x=1 Explanation: the subsearch will find events in set2.log and add a new field, x=1 to the event. (that is for D, E, F, G, the field x=1 will be associated to the events) Then, were doing a LEFT JOIN, on the field MAC (which will return A B C D) , but we are leaving out those events for which x=1 (hence we are leaving out D). The final result then becomes: A, B and C, that is, set1 - set2 ... View more

MarioM is right. The setup page makes a restAPI call to the enterprise license and grabs the correct value from it.Then it displays it for you to see and hit save. Moreover, changing that value manually, and making it different then the correct value is would be counter productive and would give you false information on when you are reaching your limit etc.. Is there a reason / use case for you wanting to change that value? Cheers, .gz ... View more

As the log itself mentions this is caused by an unclean shutdown. An unclean shutdown might happen when: - the splunkd process is killed - the whole server goes down - the splunkd process crashed We have received many notification of this happening also when actually issuing a ./splunk stop or a ./splunk restart One thing to mention though is that in most cases i have seen (if not in all of the cases i have seen) this is a by product of splunk also taking a very long time to shut down. (very long time = 6 minutes and change in seconds). This causes splunk to actually send a forced stop and kill the process. Hence, the next time you start the daemon it complains about the unclean shutdown. This is related to a known issue SPL-37407 with the fix expected in the next maintenance release, 4.2.1 of the product. This actually has been an issue for a while and not introduced in the 4.2 version, however it is now visible because the 4.2 release of the product actually checks for previous shutdowns as well as performs a check on all the indexes and databases. Hence why the user is able to see it only now. In order to have splunk start without human interaction then you might want to run the following: ./splunk start --answer-yes Check this answers post for more thorough instructions in how to add these attributes in your init.d/splunk start script. ... View more

this is an issue in 4.2 SPL-37407 which is fixed in the first maintenance release of 4.2.1 are you by any chance forwarding data from the search head to an indexer? is the tcpout queue full and blocked by any chance? Cheers, .gz ... View more

Miguel, There is currently a bug with installing Splunk 4.2 UF via the CLI. http://answers.splunk.com/questions/13073/installing-setting-up-unix-app-with-universal-forwarder/13078#13078 However, you can still easily install the app via the configuration files. Here is a quick installation guide: 1 - Download the Unix app from splunkbase 2 - untar the package in the /splunk/etc/apps directory so that it looks like: /splunk/etc/apps/unix 3 - Copy /splunk/etc/apps/unix/default/app.conf to /splunk/etc/apps/unix/local/app.conf 4 - Edit the app.conf in the local directory to say: app=enabled 5 - Copy /splunk/etc/apps/unix/default/inputs.conf to /splunk/etc/apps/unix/local/ 6 - Edit /splunk/etc/apps/unix/local/inputs.conf so that you ENABLE (set to 1) each and all inputs you would like to send to the indexer. 7 - Restart splunk (Assuming youve already set up forwarding/receiving) This should do it... ... View more

Actually, there are crashlogs that get created. [build 96430] 2011-03-24 13:09:43 Received fatal signal 11 (Segmentation fault). Cause: No memory mapped at address [0x0000000000000070]. Crashing thread: HTTPRequestHandlerThread Ill create a bug for this if there is not one already. .gz ... View more

This was asked by caffisimo here I thought it would be better having a totally new answer for it. The easiest way i have found is to edit flashtimeline.xml, find and change the following: <module name="Count" layoutPanel="pageControls"> <param name="options"> <list> <param name="text">10</param> <param name="value">10</param> </list> <list> <param name="text">20</param> <param name="value">20</param> </list> <list> <param name="text">50</param> <param name="selected">True</param> <param name="value">50</param> </list> </param> You can either add one more <list> with the necessary <param name> and have it go to 500 or modify one of the above ones. Something like: <list> <param name="text">500</param> <param name="selected">True</param> <param name="value">500</param> </list> Should do the trick. Hope this helps, .gz PS. make sure you make a backup of the flashtimeline.xml file before trying to edit the file. ... View more

When i upgraded an environment of 2 search heads and 10 indexers i started with the search heads first, then the 10 indexers. (did not need to upgrade forwarders). However, i believe the rule of thumb is, to never have forwarders be of a higher version then the indexers, and not having indexers be of a higher version then search heads. Hence it flows logically to have SearchHeads -> Indexers -> Forwarders. I believe it is possible to have search heads on 4.2 and indexers on 4.1, however doing it this way you will not be able to use the combined license pooling, and basically you will not be using the full capabilities/improvements of the 4.2 version. Moreover, 4.2 is much faster then 4.1 in regards to indexing, and a huge amount of bugs have been fixed, hence even though a hybrid state is possible, it is not imo recomendable. ... View more