thanks Guilhem, we are using 6.0.3 version so is there a way to get this information by executing splunk query. thanks in advance. ... View more

thanks Giuseppe, but we are using splunk 6.0.3 version and I did not find the DMC console, so is there a way to write a query to fetch this information (disk space) from all splunk servers. thanks in advance. ... View more

hey can you tell me where/how to check whether the saved search job is running or not in splunk. As I had set an saved search report to execute the query for every 2 hours but I had not set a email notification for the same. In this case how/where to check. thanks in advance. ... View more

Hi Sekar, thanks for your inputs on this. It worked, It triggered email notification alerts to our mail id, when an Splunk service went down. Yes I had already set the cron Job like what you had mentioned above. **/15 * * * ** ... View more

thanks all for throwing some lights on this issue. I am getting an email alert when I had set the following time range as we had some splunk service failure on last weekend. Time Range --> Earliest= -3d Latest=now. But when I had set the time range to -15m and latest=now, I am not getting any alert, so waiting for some failure to happen, so that I can validate the query. ... View more

yes, I am able to get the result while executing the query and even I was getting the email alerts, when Cron Expression alone was set to this value /15 * * * * but not getting the latest output. so when I include **Time Range --> Earliest= -15m Latest=now*, i am not getting the email alert. my requirement is to monitor the splunk services status for every 15 mins and should alert in case of failure of the service ( by capturing the splunkd.log) by executing the above query. But i am not whether the problem is with the splunk query or with the splunk configuration. Kindly guide me to fix this issue. thanks in advance. ... View more

thanks sekar, Initially I had set the trigger condition as number of alerts if it greater than 0 and it did not work. More over this condition will trigger an alert when there is an event generated. So for testing purpose I had set the condition like this "Trigger if number of results --> if less then 0" . So kindly let me know what is going wrong ? why its not triggering the alert. thanks in advance. ... View more

thanks Cusello for your quick response on my doubts. As guided we want to install three splunk instance separately in to three VM machine and configure the same apps that are running in the production environment and check their functionalities. But what about the shared search head pooling instance do we need to configure this in VM machine to replicate the prod environment is this is necessary for performing upgrade test. Kindly guide me on this thanks in advance. Below are the Shared search pooling setup Prod Environment details we have two search head, Search job scheduler instance, three indexer are communicating with shared search head pooling instance running separate machine. Note : All the splunk instance are running with 6.2.1 version and only two search head is running in 6.0.3 version, so first we need to make all our instance to run with same version. ... View more

thanks Cusello, Actually we are planning to upgrade our distributed shared search head pooling from 6.0.3 to 6.2.1 and we do not have any test environment to preform this activity, so our intension is to do this upgrade activity in test before implementing to prod environment. Kindly guide me whether we need to have license to do this. thanks in advance. ... View more

Hi Masa, can you guide me on below points as its being done in PRD Environment, want to make sure nothing goes wrong. Kindly guide me thanks in advance. ... View more

thanks Masa for providing the required details. Since I am going upgrade the version directly in production environment, want to be double sure about doing this. I have the following doubts on upgrading. 1) We are not using Splunk default root certificate for communication, so when I upgrade whether the splunk automatically fetch the required certificate information or do I need copy paste the same from back up. 2) I did not understand these two steps mentioned in the documents, Copy the apps and user preferences from the search head to the shared storage. Place the confirmed working apps in the search head pool shared storage area Copy the apps and user preferences from the search head to the shared storage 3) How to remove the search head from shared pooling ? As this is my first attempt on making this changes in splunk on production, want to get clarified all my doubts. Kindly guide me on this. thanks in advance. ... View more

Hi Jkat, could you guide me on this problem How to upgrade the Shared search head polling from 6.0.3 to 6.2.1 version? I have posted this in splunk.answers.com with steps but I did not get any response for this. it will be really helpful if you can throw some lights on this. ... View more

I have followed these Steps to rectify the issue as guide by jkat. steps: 1) Stop the splunk service running in the search02 server by executing the command /opt/splunk/bin ./splunk stop 2) Remove the partially created folder under the mount point /splunk_search_pool from search02 server by executing the command rm -rf /splunk_search_pool 3) Use the below command to mount back to the correct mount point as it is in search01 mount -t nfs splunfs01:/opt/splunk_shp /splunk_search_pool 4) Then restart the splunk service by executing the below command /opt/splunk/bin/ ./splunk start 5) Need to check whether both the URL is working by hitting the respective URLS https://search01.com:8443 https://search02.com:8443 Thanks in advance. ... View more

sure I will hit the answer option for this problem. ... View more

Hi All, can any one guide me on how to upgrade the Shared search head polling from 6.0.3 to 6.2.1 and let me know whether we can follow the steps mentioned in the comments. thanks in advance. ... View more

Hi jkat, thanks for your effort. Now I could see both the search head are up and running with the same port number. F5 Team confirmed that both the search head are showing up. One more thing I wanted to confirm whether both the search head are insycn with each other and how much of load is hitting both the search head . thanks in Advance. ... View more

thanks Jkat, since its in production Environment, I had to make double check before proceeding on this. let me update once done. ... View more

Hi Jkat, could you guide me on this problem ? thanks in advance. ... View more

Hi All, Can any one guide me on upgrading Shared search head polling from 6.0.3 to 6.2.1 as all other Splunk components are already running with the 6.2.1 splunk instances, except the two search heads. Kindly let me know whether the above mentioned steps in comments can be followed to upgrade as I am going to do this directly on Prod Environment. Important – We are not using the default root certificate for communication, so when we upgrade will to 6.2.1 version will it automatically fetch the necessary detail 0r we need to copy paste details from the backup. Thanks in Advance. ... View more

Hi Jkat, kindly guide me to know whether the below steps can be followed to fix this issue. Steps : 1) Stop the splunk service running in the search02 server by executing the command /opt/splunk/bin ./splunk stop 2) Remove the partially created folder under the mount point /splunk_search_pool from search02 server by executing the command rm -rf /splunk_search_pool 3) Use the below command to mount back to the correct mount point as it is in search01 mount -t nfs splunfs01:/opt/splunk_shp /splunk_search_pool 4) Then restart the splunk service by executing the below command /opt/splunk/bin/ ./splunk start 5) Need to check whether both the URL is working by hitting the respective URLS https://search01.com:8443 https://search02.com:8443 Thanks in advance. ... View more

thanks woodcock for providing the steps. But as I said currently we are planning to upgrade only two search head from Splunk version 6.0.3 to 6.2.1, as other splunk instance are already running Splunk with same Splunk 6.2.1version. Infrastructure Details : Search head 01 / 02 Splunk 6.0.3 (build 204106) Linux – Red Hat Enterprise Linux Server release 6.7 (Santiago) Virtual machine 2.6.32-573.22.1.el6.x86_64 HTTPS Port no: 8443 Splunk shared pooling server: Linux – Red Hat Enterprise Linux Server release 6.7 (Santiago) Virtual machine 2.6.32-573.22.1.el6.x86_64 Using splunk as sudo root user Steps to upgrade – Search head 01 1) Initial steps before doing the upgrade need to take the entire configuration step up backed up by executing ./splunk diag 2) Once the entire splunk folder is backed up, need to stop the splunk service by executing the below command /opt/splunk/bin/splunk stop 3) Remove the entire splunk folder by executing the rm –rf command rm –rf /opt/splunk 4) Download the splunk version Splunk 6.2.1 (build 245427) from this link [https://www.splunk.com/eula/splunk/6.2.1?redirecturl=https%3A%2F%2Fwww.splunk.com%2Fpage%2Fdownload_track%3Ffile%3D6.2.1%2Flinux%2Fsplunk-6.2.1-245427-Linux-x86_64.tgz%26platform%3DLinux%26architecture%3Dx86_64%26version%3D6.2.1%26product%3Dsplunk%26typed%3Drelease%26name%3Dlinux_installer][1] 5) Install the tar file in under the /opt directory by executing the below command. tar xvzf splunk-6.2.1-245427-Linux-x86_64.tgz -C /opt 6) Once installed then start the splunk service by executing the below command /opt/splunk/bin/ splunk start --accept-license 7) Follow the same steps for another search head 02 Note:- Should we need to do anything with the splunk shared pooling server/Deployment manager as both the search heads shares the apps/user specific details from this server? Since we are going to implement the steps directly to Prod environment, I need to make sure nothing goes wrong. so kindly guide me whether these are steps should be followed to upgrade from 6.0.3 to 6.2.1 and also about splunk shared pooling server. Thanks in Advance. ... View more