thanks Jeremiah, though its enabled but when I tried to execute the below query to find out indexer and forwarder communication using SSL or not it showing no result found. index=_internal source=*metrics.log* group=tcpin_connections | dedup hostname | table hostname sourceIp fwdType version destPort ssl even tried to execute the index=_internal source=*metrics.log* it did not fetch any output. Time Frame set as last 7 days. Do guide me if there is any other option to figure out whether the indexer and forwarders are using default root SSL certificate or not. thanks in Advance ... View more

thanks Renjith , I am not sure why they had configured like this but we wanted to make all to run with same version, so let me check the details give in the links , but mean while can you guide me on how to reconfigure the search heads to behave as clustered. Is it possible to get me a step by step procedure to do this. thanks in advance. ... View more

okay, initially _internal indexes was disabled, but I had enabled it to test the below SPL query and again when tried to disable the index it was throwing the error. Query to find out indexer and forwarder communication using SSL or not index=_internal source=*metrics.log* group=tcpin_connections | dedup hostname | table hostname sourceIp fwdType version destPort ssl so is there a way to disable the _internal indexes from this search portal? thanks in advance ... View more

thank Renjith, We are using the 6.0.3 and 6.2.1 version in our environment. As per the architecture diagram, the two searched heads are clustered along with this file sharing server. I had ran the below command to identify whether the two search head are in clustered or not, but I did not get any output after executing the CLI. ./splunk cmd btool server list --debug shclustering I had even stopped one of the search head to test for clustering behavior, but it was not getting switched to another server. It was still pointing to the same server, I had verified this by going to setting --> system details --> General setttings --> splunk server name. Kindly guide me to fix this problem. thanks in advance ... View more

thanks Ranjith, after executing the above command in all the indexer, It returned only the default values from /etc/system/default/server.conf, from this it was clear that ours is Distributed Splunk Environment not a clustered. Similarly when I ran the command on the search heads (Four search heads / one Scheduled Job search severs) got the same default result. But as per the architecture diagram, I could see two search heads are connected to file sharing pools where it share the information about the search's. Kindly guide why its not showing the clustered results or whether its clustered one but currently its not working as clustered. thanks in Advance. ... View more

thank Woodcock, You are right, I have checked the path /opt/splunk/etc/users/username/search/local/savedsearches.conf and deleted the VSID = XXXX from the saved search. I have got this above information from file system pooling not in the search heads. After finding the correct search from the savedsearches.conf file, deleted the VSID =xxxx stanza from the search and restarted the splunk service in the search heads. There is no VSID = XXXX in viewstates.conf so did not change any settings. Now the user can edit and save the searches from the setting-->search-->reporting--searchAPP--CISCO. ... View more

thanks Woodcock, In the search head under the path /opt/splunk/etc/apps/search/local/savedsearches.conf, did not find any stanza related to this saved search. I had checked the same in the viewstates.conf but nothing was related to this search. So I had created the same search query and saved it in the savedsearches.conf and corresponding viewstates.conf, finally restarted the service but still its throwing the same error. [Cisco ] action.email.inline = 1 alert.digest_mode = True alert.suppress = 0 alert.track = 0 cron_schedule = * * * * * dispatch.latest_time = now displayview = flashtimeline request.ui_dispatch_view = flashtimeline search = source="/var/log/syslog_info" _raw=DUAL-3-SIA _raw!=INDIVIDUAL earliest=-30d@h | table _time, _raw | sort -_time vsid = xxxx Viewstates.conf - [flashtimeline:xxxx] - Same Vsid value given in the savedsearches.conf AxisScaleFormatter_0_18_0.default = "" ButtonSwitcher_0_8_0.selected = splIcon-results-table ChartTypeFormatter_0_13_0.default = column Count_0_7_1.default = 50 DataOverlay_0_13_0.dataOverlayMode = none DataOverlay_0_13_0.default = none FieldPicker_0_5_1.fields = host,sourcetype,source,User FieldPicker_0_5_1.sidebarDisplay = True FlashTimeline_0_4_1.height = 94px FlashTimeline_0_4_1.minimized = False JSChart_0_13_1.height = 300px LegendFormatter_0_19_0.default = right MaxLines_0_13_0.default = 10 MaxLines_0_13_0.maxLines = 10 NullValueFormatter_0_18_0.default = gaps RowNumbers_0_12_0.default = true RowNumbers_0_12_0.displayRowNumbers = true RowNumbers_1_12_0.default = true RowNumbers_1_12_0.displayRowNumbers = true Segmentation_0_14_0.default = full Segmentation_0_14_0.segmentation = full SoftWrap_0_11_0.enable = True SplitModeFormatter_0_17_0.default = false StackModeFormatter_0_16_0.default = default YAxisRangeMaximumFormatter_0_17_0.default = "" YAxisRangeMinimumFormatter_0_16_0.default = "" please do let us know is any other way we can resolve this problem. Even in the /opt/splunk/etc/apps/search/metadata/local.meta, I could not see the stanza related to this saved search. thanks in Advance. ... View more

Yes I am new to this environment and the person who had worked in the organization had left the company and there is no an documentation. In Architecture diagram I could see there are 5 indexer instance running and had assumed that its a cluster environment. I had executed the command in one of the indexer server and but there no result popped out. /opt/splunk/bin ./splunk btool server list clustering --debug | grep -v system ./splunk btool server list clustering --debug | grep -v default Currently we are using splunk 6.2 version and 6.0.3 version in our environment. As you aware coming 21 July 2016 the default root certificate will expiry and we had planned to stick with version by executing the renewcert.sh script provided by splunk. So I was checking for the details. One more question regarding the default certificate ? I could see 1098 UF agent running in our environment, to determine default certificate used by the forwarder, I had ran the below query and the result indicated that SSL is false, does this means there is no SSL configured in the forwarders? if not then should I need to execute the query individually to all the agents? Kindly clarify this index=_internal source=metrics.log group=tcpin_connections | dedup hostname | table hostname sourceIp fwdType version destPort ssl thanks in advance. ... View more

Thanks Renjith, I have checked the server. Conf file in the indexers under the following paths /opt/splunk/etc/apps/xxxx/default/server.conf and I could see only below stanza [httpServer] maxThreads = 1000 Similarly under this path /opt/splunk/etc/system/local - I could see this stanza [sslConfig] sslKeysfilePassword = xxxxxxxxxx [lmpool:auto_generated_pool_download-trial] description = auto_generated_pool_download-trial quota = MAX slaves = * stack_id = download-trial [lmpool:auto_generated_pool_forwarder] description = auto_generated_pool_forwarder quota = MAX slaves = * stack_id = forwarder [lmpool:auto_generated_pool_free] description = auto_generated_pool_free quota = MAX slaves = * stack_id = free [license] master_uri = https://:8089 --> [general] pass4SymmKey = xxxxxxxx serverName = xxxxxxxxxx incase of License /deployment master instance under this path /opt/splunk/etc/shcluster/apps/XXX-ADMIN-all_indexers/default/server.conf [httpServer] maxThreads = 1000 /opt/splunk/etc/shcluster/apps/XXX-ADMIN-hvy_forwarders/default/server.conf [queue] maxSize = 200MB /opt/splunk/etc/shcluster/apps/XXX-ADMIN-searchheadcluster/default/server.conf [license] master_uri = https://:8089 --> In all the above mentioned path I could not see the stanza called clustering, so kindly let me is this is the correct path to validate recoding the master node details. Thanks in Advance. ... View more

Thanks, Giuseppe, I had tried login into one of the search head --> settings --> Distributed Environment --> clustering. I could not see any configuration related to clustering, I could see Enable Clustering option. Correct if this is not the correct place to look into. Similarly under settings --> Distributed Environment --> Distributed search --> search peers --> I could see all the indexer are listed. Kindly let me is any other way I can find the master node details. ... View more

thanks Renjith, Let me try this and get back to you incase any issue occurs. ... View more

thanks Renjith. Let me try out the steps which you had described above, but before proceeding with this steps, need to know will there be any impact? when I am trying to remove/map the particular user to new role, as the user owns some of apps / Saved Scheduled search reports. ... View more

Ranjith, Yes we are using LDAP external authentication method for authentication in splunk. Yes Initially thought of creating a new role with all the capabilities of power user and excluding only the few permission as per the user requirement. But I am not sure on how to remove a role for a particular user in LDAP settings. I had followed below steps but correct me if this is not right way to do so? Splunk-->settings-->access control --> authentication method --> Configure Splunk to use LDAP and map groups --> action --> manage groups --> Splunk_Power_User. Here I could see many other user id's mapped to this role and my need is to remove only a particular user id from this role not for other users. Kindly guide me on this. thanks in advance . ... View more

Thanks mcarney, in both the heavy forwarder server we have huge number of directories and files into it. In heavy forwarder server 1 I could see there are 4200 directories, similarly in heavy forwarder server 2 there are 8703 directories. I have used the below command to get this information from Linux machine. [root@splunk01 generic]#ls -ltr | wc -l 4200 [root@splunk02 generic]# ls -ltr | wc -l 8703 This is the inputs.conf files present in the both heavy forwarder to read the syslogs details from the sourcetype (syslogs). [monitor:///opt/syslogs/generic/.../*.log] sourcetype = syslog host_segment = 4 blacklist = xxxx*ltm* index=unix_hosts With the help of UNIX admin we have executed a cron job for every half an hour but still the file system size is keep on increasing drastically. Cron Job detail: 00,30 * * * * /bin/find /opt/syslogs/generic -mtime +1 -type f -delete > /dev/null 2>&1 Kindly let us know if we need to change any configuration in Splunk to fix this permanently. ... View more

thanks, we have build additional splunk heavy forwarder server in order to manage the load. Now entire syslog loaded is pointed towards the newly build heavy forwarder server, but still files size is keep on increasing in this folder /opt/syslogs/generic. Since under generic folder there are many subdirectories its difficult to execute the log rotate as it consumes more SWAP memory. Question 1) How to find the volume of the logs getting injected every hour from the source (SYSLOGS) in both the heavy forwarder? I tried to execute this query but I did not get the correct result index =* source ="syslogs" sourcetype ="/opt/syslogs/generic" | eval indextime=strftime(_indextime, "%Y-%m-%d %H:%M:%S" ) | eval length = len (_raw) /1024 | stats sum(length) count by source indextime index host Need to get the volume of log getting injected in both the servers (Heavy Forwarder 1 & Heavy Forwarder 2). If I could find the exact volume of load in both the server, I can suggest to keep a separate server for syslogs. ... View more

Thanks jcoates, I am able to get this information in my splunk server, but I am not sure what I need to check in this files. Since I am unable to do the Service-now setup configuration I could see all the files are with default setting in default folder. Do I need to do manual configuration by editing all above mentioned conf files from default setting and copy the file to local folder path? splunk/etc/apps/splunk_TA_snow /local/ *.conf . Meanwhile I have taken this log file from Splunkd_access.log & Service_now_setup.log file attached. From the log what I could understand there is a problem in user name/password. Kindly let me know if you can help me on this. Splunkd_access.log e=json HTTP/1.1" 200 1504 - - - 1ms 127.0.0.1 - - [12/Oct/2015:12:35:51.306 +0530] "GET /services/server/info?output_mod e=json HTTP/1.1" 200 1504 - - - 1ms 127.0.0.1 - - [12/Oct/2015:12:35:56.306 +0530] "GET /services/server/info?output_mod e=json HTTP/1.1" 200 1504 - - - 1ms 127.0.0.1 - splunk-system-user [12/Oct/2015:12:35:56.761 +0530] "GET /servicesNS/-/- /configs/conf-service_now/_reload HTTP/1.0" 200 1858 - - - 10ms 127.0.0.1 - splunk-system-user [12/Oct/2015:12:35:56.780 +0530] "GET /servicesNS/nob ody/Splunk_TA_snow/configs/conf-service_now HTTP/1.0" 200 9058 - - - 2ms 127.0.0.1 - splunk-system-user [12/Oct/2015:12:35:56.786 +0530] "GET /servicesNS/nob ody/Splunk_TA_snow/storage/passwords/https%5C%3A%252F%252Fprod1cafetest.service-now. com%3Adummy%3A HTTP/1.0" 200 4860 - - - 1ms 127.0.0.1 - splunk-system-user [12/Oct/2015:12:35:56.790 +0530] "GET /servicesNS/nob ody/Splunk_TA_snow/storage/passwords/http%5C%3A%252F%252F192.168.20.31%3Adummy%3A HT TP/1.0" 200 4724 - - - 1ms 127.0.0.1 - - [12/Oct/2015:12:36:01.306 +0530] "GET /services/server/info?output_mod e=json HTTP/1.1" 200 1504 - - - 1ms service_now_setup.log 2015-10-12 12:34:50,337 INFO pid=27273 tid=MainThread file=snow_setup.py:detect_release:227 | Detect ServiceNow release and verify credentials 2015-10-12 12:34:50,343 INFO pid=27273 tid=MainThread file=snow_setup.py:_decrypt_username_password:217 | decrypting 2015-10-12 12:34:50,348 INFO pid=27273 tid=MainThread file=snow_setup.py:_decrypt_username_password:217 | decrypting 2015-10-12 12:35:06,312 ERROR pid=27273 tid=MainThread file=snow_setup.py:_detect_release:251 | Failed to validate ServiceNow account or detect release info. Please verify credentials, urls for ServiceNow and proxy, and try again. Reason=Traceback (most recent call last): File "/opt/splunk/etc/apps/Splunk_TA_snow/bin/snow_setup.py", line 243, in _detect_release conf_mgr, config, self.appName, False) File "/opt/splunk/etc/apps/Splunk_TA_snow/bin/snow_config.py", line 126, in fix_snow_release fixed_release = SnowConfig.get_snow_release(defaults) File "/opt/splunk/etc/apps/Splunk_TA_snow/bin/snow_config.py", line 188, in get_snow_release SnowConfig.verify_user_pass(defaults) File "/opt/splunk/etc/apps/Splunk_TA_snow/bin/snow_config.py", line 178, in verify_user_pass raise Exception(msg) Exception: Failed to verify ServiceNow username and password, reason=Traceback (most recent call last): File "/opt/splunk/etc/apps/Splunk_TA_snow/bin/snow_config.py", line 173, in verify_user_pass resp, content = http.request(url) File "/opt/splunk/etc/apps/Splunk_TA_snow/bin/httplib2/init.py", line 1593, in request (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) File "/opt/splunk/etc/apps/Splunk_TA_snow/bin/httplib2/init.py", line 1335, in _request (response, content) = self._conn_request(conn, request_uri, method, body, headers) File "/opt/splunk/etc/apps/Splunk_TA_snow/bin/httplib2/init_.py", line 1263, in _conn_request raise ServerNotFoundError("Unable to find the server at %s" % conn.host) ... View more

Thanks jcoates, I am having admin privilege for both Splunk/Linux Machine. when I tried to find out the path mentioned in the above error "/servicesNS/nobody/Splunk_TA_snow/service_now_setup/snow_proxy/snow_proxy” I could not find them under Splunk folder. kindly let me where we can find this path and is this causing an issue. ... View more