Currently we have an issue in getting the data into the heavy forwarder. We could see that below stanza is configured in the heavy forwarders, When checked under the path as mentioned in the stanza, we could not see logs getting into the server from the source. Heavy forwarder stanza: [monitor:///opt/syslogs/symantec/SymantecServer/...] whitelist = \.log index = Symantec sourcetype = sep host_segment = 5 Indexer inputs.conf stanza: [udp://hostname.com:8501] connection_host = dns index = Symantec source = hostname.com:8501 sourcetype = sep Source where Splunk monitors the logs from the heavy forwarder. Currently there are no logs under this folder: source="/opt/syslogs/symantec/SymantecServer/hostname/hostname.log" Splunkd.log from the Universal Forwarder server version 6.2 06-22-2016 01:31:13.857 -0400 ERROR TcpOutputFd - Connection to host=x.x.x.x:9997 failed 06-22-2016 01:31:43.615 -0400 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997 Initially the logs were getting into this heavy forwarder server from the universal forwarder server, but somehow this got broken. Kindly guide us in fixing this issue. Thanks in advance ... View more

Currently _internal is enabled, but we wanted to disable this from Splunk Web? I tried to do so by getting into splunk -->settings -->Data -->Indexes --> _internal --> status --Disable. When I disabled it, it threw out the following error: Error occurred attempting to disable _internal: **In handler 'indexes': cannot disable idx=_internal, is internal. Kindly let us know how to disable this index from the search. thanks in Advance. ... View more

Currently in our environment, we have 5 indexer instances, four search heads, a scheduled search Job instance, 2 Heavy Forwarders, and a deployment manager/ License master running on the same instance? Kindly let me know in which instance/configuration file can I find out the cluster-master details. ... View more

Regarding default root certificates for release pre 6.2 version for Splunk enterprise , I have some questions which needs to be clarified before executing the renewcert.zip script provided by the Splunk. Currently we have the below Splunk Enterprise set up in our Environment. 1) Five Clustered Indexer Version Splunk 6.2.1 2) Four Search Head in which three are build with Version Splunk 6.0.3 and one with Version 6.2.1 3) Scheduled Search Job with version 6.0.3 4) Two Heavy Forwarder one with latest 6.4.1 and another with 6.2.1 5) we have 1131 Universal forwarder configured in our Environment. Questions are : 1) In which order we should run the script first, I mean should we execute the script firs in UF, HF,SH and then in Indexer? 2) How to validate whether the current Splunk Environment is using the default certificate or not and it going to expire by 21 July 2016? Note - Under the path /opt/splunk/etc/auth/ all the certificate details like ca.pem, ca.default.pem,cacert.pem and cacert.default.pem are encrypted , so unable to find the Date of Expire details. In this case how to validate the details 3) I had run the below SPL query to validate the UF is using the default certificate or not? But I am not sure what does it mean from the out put stating SSL is false. index=_internal source=metrics.log group=tcpin_connections | dedup hostname | table hostname sourceIp fwdType version destPort ssl Output Details : hostname source ip fwdtype version destport ssl xxxxxx xxxxxx uf 6.2.0 9997 FALSE xxxxxx xxxxxx uf 6.2.0 9997 FALSE xxxxxx xxxxxx uf 4.3 9997 FALSE 4) In order to test the script, I had followed the below steps in my test environment and it worked fine. Individual Splunk instance with version 6.2.1 trail OS Red hat Linux 6.5, VM machine Steps - 1) Checked the Splunk Environment is set or not by executing the echo $ SPLUNK_HOME, echo $LD_LIBRARY_PATH and echo $ OPENSSL_CONF and Found it was not set. 2) Setup a Splunk Environment variable by executing the setSplunkEnV and to do this, splunk user should have permission to execute the script in /opt/splunk/bin. Used chmod -R 777 /opt/splunk/ . After running the setSplunkEnv script, checked the Splunk Environment variable was set properly by executing the echo $SPLUNK_HOME = /opt/splunk 3) Checked for the write permission for the path /opt/splunk/auth directory and it should be set with write permission. 4) Before executing the script, taken the backup of /opt/splunk by executing the cp -rp /opt/splunk /etc /temp 5) Validate all the certificate details before executing the s-renewcerts.sh and found date of expiry as 21 July 2016. 6) After executing the script validate all the certificate details in the path /opt/splunk/etc/auth by executing the cat command and found the date of expiry as 22 July 2026. kindly let me know should I need to follow the same steps in the cluster environment also. Thanks in Advance ... View more

Currently I have request raised by a user who wants to remove his view access to the internal indexes in Splunk (anything that starts with _) for his account. He is currently assigned with three different roles and they are Basic_user, Power_user and Security_user. When I verified the permission assigned for all the above three roles, I could find for both "Basic_user/Security_user" are provided with view access only for the user related indexes (Organization related indexes created by user), but in case of Power_user, the user has a privilege to view all the default Splunk internal indexes apart from Organization related indexes. Below is a list of permissions that need to be removed from the Power_user role for a particular user. 1) _audit 2) _blocksignature 3) _internal 4) _thefishbucket 5) _introspection 6) main 7) All non-internal indexes. When tried to remove a role against his name, the role was displayed greyed out. Kindly guide me how to get this permission removed for this user. Thanks in Advance. ... View more

Currently I am facing a file system /opt issue in the splunk heavyforwarder server, this server is used to monitor and forward the syslogs information to the indexer clusters. Files size is keep on increasing in this folder /opt/syslogs/generic/... under generic folder there are many subdirectories and each subdirectoris contains some .logs. When validated the splunkd.log I could see the below Info 04-22-2016 12:18:07.201 -0400 INFO AggregatorMiningProcessor - Got done message for: source::/opt/syslogs/generic/xxxx/2838552.log|host::xxxx|syslog|9947901 04-22-2016 12:18:12.904 -0400 INFO AggregatorMiningProcessor - Setting up line merging apparatus for: source::/opt/syslogs/generic/xxxopt/sport.log|host::xxxopt|syslog|9946037 I have tried toexecute the log rotate but when execute the log rotate its consuming the Swap memory has many log rotate process are running and if I kill the process then space increases, as temprorary solution i am trying to add space to this opt file system. Since all the data are critical (network data) deleting the files will be create a probelm while auditing. My question what will be the permanent solution to fix this issue. 1) Do I need to change any configuration inside the /opt/splunk/etc/apps/local/input.conf 2) Can I move the files to some other location in the same server? ... View more