how to find out from which location it reading the configuration file of indexes.conf file. Currently we are facing a space crunch in the PCI search head, on investigating we found that splunk internal index like audit, access_summarydb is consuming more space almost 80GB of space is consumed by this two indexes. Path /opt/splunk/var/lib/splunk --> disk usage is almost 95% my question is how to find out which is the active indexes.conf file from where these configuration files are read in the search head. Kindly let me know whether we can use this btool command to get this input. ./splunk cmd btool indexes list --debug thanks in advance. ... View more

thanks Jkat for your inputs, anyway I agree with you to you and I know its very risk to run splunk as root, but need to get an approval to change this in prod. ... View more

Hey Jkat54, thanks for your input but I have restart the splunk with root user as other splunk instance is using the same user id and now I could see the is no permission denied error as I am using the root user id over splunk folder. But anyway these are the steps should be followed to change splunk from root user id to non root id, i.e. run splunk with splunk user id: 1) sudo su - 2) with root user id stop the splunk service 3) With root user id change the permission of the splunk folder to use splunk user id chown -R splunk:splunk /opt/splunk 4) Add this stanza " SPLUNK_OS_USER=splunk" in /opt/splunk/etc/splunk-launch.conf and save it. 5) switch to splunk user id using su splunk 6) Try to start the splunk using splunk user id. 7) check the splunk process using which id by using command ps -ef | grep splunkd Note: If you see any permission problem try to identify the folder which is having permission issue by going to the folder,check for the folder permission by using ls -ll command. If you find that particular folder is used by root change it splunk to that folder alone. ... View more

thanks Jkat54, but now I am little bit concern on executing commands as it is in Production environment. By doing it force fully will it fix the issue. thanks in advance. ... View more

HI Somesoni, I have gone through the link which you had mentioned in the comments, as per that when we go for aupgrade we need to test the apps which are working fine. But I could see there are 93 apps / 200 users present in the shared storage server under the path /opt/splunk_shp/etc/apps, so how to identify which apps are working and which are not ? You must test apps and configurations that the search heads use prior to upgrading the search head pool Kindly guide me on this . ... View more

Hi Somesoni, we are getting data when we keep the duration for 15min or even I could see data for last 7 days but when we set for more that then we are getting no result found. But I need to how to check whether the _internal indexes are full ? as we have 15 splunk instance running in our environment. Kindly guide me on this.. Wish you a merry Christmas and Happy New Year. thanks in advance ... View more

thanks somesoni for quick response, Yes I am able to get the data when we search with the query index=_internal sourcetype=splunkd source="/opt/splunk/var/log/splunk/splunkd.log. Retention period is set as 30 days but when we search with the time period for last 10 days, we are getting no result found. As per retention we should get the data right . Kindly guide me to trouble shoot this issue. ... View more

Hi Sierrax, I have found from how/where this events are getting into Splunk, I mean the exact inputs.conf location in the specific host by executing the splunk btool command and on finding the inputs.conf file, I added the disabled = I stanza and restarted the splunk services and this fixed my issue. Solution : 1) executed ./splunk cmd btool inputs list rest -- > Path /opt/splunk/bin 2) On finding the location of the inputs.conf file added the disabled =1 stanza path :/opt/splunk/etc/apps/search/local [rest://Solarwinds Nodes] auth_type = none disable = 1 endpoint = https://ws.xxxx.com/sw/getnodes http_method = GET index = main index_error_response_codes = 0 polling_interval = 300 response_type = json sourcetype = rest:solarwinds:nodes streaming_request = 0 3) Restarted the splunk service by executing ./splunk restart --> /opt/splunk/bin thanks for sharing your trouble shooting skills. ... View more

somesoni, is it possible to tell me how to blacklist the particular events based on multiple hostname or source type or source. Any assistance would be greatly appreciated. ... View more

thanks Sierrax, but not sure where I can find this configuration ? I mean whether we need to check the host machine where logs are getting generated ? Because when I run the below query we are getting events as show below host=host1* source = rest://Solarwinds Nodes sourcetype = rest:solarwinds:nodes Output {[-] results:[[+] ] } Show as raw text And the moment when we hit the Shows as raw text it displays list of server details "results":[{"solarwinds_node_id":2,"polling_engine_id":14,"polling_engine":"Test01","solarwinds_prefix":"N:","src_ip":"10.X.X.X","host":"","percent_memory_used":62,"cpu_load":11,"up_since":"2016-09-29T19:28:00","host_tier":null},{"solarwinds_node_id":3,"polling_engine_id":13,"polling_engine":"Test07","solarwinds_prefix":"N:","src_ip":"10..X.X.X","host":"","percent_memory_used":58,"cpu_load":11,"up_since":"2016-09-29T18:53:00","host_tier":null},{"solarwinds_node_id":4,"polling_engine_id":2,"polling_engine":"Test01","solarwinds_prefix":"N:","src_ip":"10.X.X.X","host":"","percent_memory_used":16,"cpu_load":6,"up_since":"2016-11-06T19:51:00","host_tier":null}, When checked with users they confirmed that the list of server are having Solar wind application running into it. Question : 1) From the list of host machine Test01, under opt/SplunkUniversalforwarder/etc/apps/TA-Solarwinds/local/inputs.conf file are already disabled and also the above events are getting into default index main. And there is no Props or transform .conf file present under this folder. Inputs.conf details [WinEventLog://Microsoft-IIS-Configuration/Operational] disabled = 1 ignoreOlderThan = 4d index = win_srv 2) Whereas in host=host1 server (splunk--> license/deployment instances are running). We could see apps called TA-Solarwinds --> default --> under this folder there are props,transform,eventype,workflow_action.conf and app.conf files. But not sure from how/where this events are getting into Splunk. so kindly guide me where exactly I need to trouble shoot this issue to disable the events getting into splunk. thanks in advance. ... View more

thanks somesoni2, above events are not getting into splunk after disabling and restarting the splunk service. In Portal when searched for this events we are not getting any result, after disabling and restarting the splunk services. solution : adding disable = 1 stanza into the inputs.conf and restarting the splunk service resolved the issue [rest://Solarwinds Nodes] auth_type = none disable = 1 endpoint = https://ws.xxxx.com/sw/getnodes http_method = GET index = main index_error_response_codes = 0 polling_interval = 300 response_type = json sourcetype = rest:solarwinds:nodes streaming_request = 0 ... View more

thanks Somesoni, for providing some inputs on this issue but not sure about the command which you had mentioned. cmd path /opt/splunk/bin ./splunk _internal call /configs/conf-inputs/_reload thanks in advance. ... View more

thanks Somesoni, this particular input configuration is present under this path. Inputs.conf details : path :/opt/splunk/etc/apps/search/local [rest://Solarwinds Nodes] auth_type = none endpoint = https://ws.xxxx.com/sw/getnodes http_method = GET index = main index_error_response_codes = 0 polling_interval = 300 response_type = json sourcetype = rest:solarwinds:nodes streaming_request = 0 But we do not have the GUI access to disable this setting , so we had directly edited the inputs.conf file but did not execute splunk restart or reload command after making the changes in configuration file. So kindly let me know whether we need to restart the service/reload. thanks in advance. ... View more

thanks Somesoni2 for throwing some lights on this but initially I had tried to do disabled = 1 but did not execute ./splunk reload deploy-server as this configuration itself present in the deployment manager instance. So kindly let me know whether should I need to execute the reload deploy-server command and also let us know how to blacklist the particular events based on host/source type/source. thanks in advance. ... View more

Hi Masa, I had tried to install the splunk 6.0.3 version in one of the test servers, but unfortunately after installing the splunk 6.0.3 and accepting the license agreement, when I tried to login via Splunk web I am getting notified with license expiry for this version only, Whereas when I tried to install different version of splunk like 6.2.1, 6.3.3 ,6.4.1 , 6.5.0 I am able to login and access the portal. not sure why I am getting this information only for 6.0.3. Since we have current prod environment is running with 6.0.3, I tired to install the same in test. Kindly advise me why I am getting message and how to over come this. thanks in advance. ... View more

Hi Sierrax, thanks for your effort on this, I got doubt whether I need to add this below stanza into the props.conf, along with the stanza that are already present in the props.conf as mentioned above comments. [rest:solarwinds:nodes] DEST_KEY = nullQueue thanks in advance. ... View more

Hi Sierrax, Could please guide on this issue, as these events are taking unwanted space and huge licenses. thanks in advance. ... View more

Hi All, Can any one guide me on the above problem as it consuming more licenses. thanks in advance. ... View more