Hi All thanks for your inputs on this issue, I had tried to access the URL individually but it failed, I am able to access one of the Splunk search head URL instance which is showing up in the F5 balancer. https ://hostname.xxx.com:8443 --- splunk search head 1 (Status showing UP in F5) https://hostname.xxx.com:8443 --- splunk search head 2 (Status showing down in F5) I am getting page can not be displayed error How / where to find the web port /URL configuration details for accessing the search head. thanks in advance ... View more

thanks for your input. ... View more

Hi Krishna, we had this issue, because of the proxy setting which was blocking the port and I am not sure right now, as it was done long back in different project. ... View more

thanks for all the people who guide me on this issue, after changing the limits.conf file and restarted the service issue got fixed. path = /opt/splunk/etc/apps/yourapp/local/limits.conf stanza [inputproc] max_fd = 1000 ... View more

thanks lakshman, We have two heavy forwarder/syslog instances running in the same server. HF is used to forward the data (syslog) to the 5 individual indexer instances and we have an F5 load balancer that is placed before the two HF servers to route the traffic. Problem - Most of the time we get an alert from Unix team stating that the splunkd process is consuming more CPU/Swap memory. Sometimes the swap memory becomes almost zero and kills the splunkd process. In a month we are getting almost 20 alerts for this issue. Kindly let guide us in overcoming this problem. System details: Splunk version 6.2.1 OS - RedHat 6.6 Memory - 6GB CPU - 3 VMware current status of swap memory free -m total used free shared buffers cached Mem: 15947 15664 282 0 357 6361 -/+ buffers/cache: 8945 7001 Swap: 3323 304 3019 kindly let me know how to fix this issue. ... View more

thanks lakshman for getting into this issue. Currently I am facing an swap memory issue in the HF serves, can you guide me on this. ... View more

thanks wolverine for putting some effort in this issue, this is the limits.conf detail which is the in the HF server /opt/splunk/etc/apps/all_indexer_base/local/limits.conf [search] max_rawsize_perchunk = 314572800 /opt/splunk/etc/system/default/limits.conf [inputproc] Threshold size (in mb) to trigger fishbucket rolling to a new db file_tracking_db_threshold_mb = 500 Approximate ceiling on sourcetypes & fingerprints in learned app learned_sourcetypes_limit = 1000 Maximum size (in mb) of heap allowed to be created by Splunk modular input Mon itorNoHandle. monitornohandle_max_heap_mb = 0 do you want us to add this stanza ? and what value should be provided in the stanza. [inputproc] max_fd = thanks in advance. ... View more

thanks mmodestino. But I did not understand the below sentence from your comment. "If you go to settings > inputs, did you set your number high enough? I quickly ended up needing a cronjob to reap the old files I didn't want splunk to monitor. (one time ingestion)" R u asking about the limits.conf file details ? this is the cronjob we have placed in the server to delete the large files 00,30 * * * * /bin/find /opt/syslogs/generic -mtime +1 -type f -delete > /dev/null 2>&1 00,30 * * * * /bin/find /opt/syslogs/web_access -mtime +1 -type f -delete > /dev/null 2>&1 so kindly guide us to fix this issue. thanks in advance. ... View more

Hi mmodestino, thanks for putting your effort in this issue. Yes we are using Root user to run splunk. We have restarted only the splunk services not the server ? So should we need to restart the server in order to reflect the changes. As its in production environment, need to get some approval before executing init script or restarting the server. grep ulimit splunkd.log.* splunkd.log.1:08-12-2016 14:39:32.564 -0400 INFO ulimit - Limit: virtual address space size: unlimited splunkd.log.1:08-12-2016 14:39:32.564 -0400 INFO ulimit - Limit: data segment size: unlimited splunkd.log.1:08-12-2016 14:39:32.564 -0400 INFO ulimit - Limit: resident memory size: unlimited splunkd.log.1:08-12-2016 14:39:32.564 -0400 INFO ulimit - Limit: stack size: 10485760 bytes [hard maximum: unlimited] splunkd.log.1:08-12-2016 14:39:32.564 -0400 INFO ulimit - Limit: core file size: 0 bytes [hard maximum: unlimited] splunkd.log.1:08-12-2016 14:39:32.564 -0400 WARN ulimit - Core file generation disabled splunkd.log.1:08-12-2016 14:39:32.564 -0400 INFO ulimit - Limit: data file size: unlimited splunkd.log.1:08-12-2016 14:39:32.564 -0400 INFO ulimit - Limit: open files: 102400 files splunkd.log.1:08-12-2016 14:39:32.564 -0400 INFO ulimit - Limit: user processes: 18000 processes splunkd.log.1:08-12-2016 14:39:32.564 -0400 INFO ulimit - Limit: cpu time: unlimited splunkd.log.1:08-12-2016 14:39:32.564 -0400 INFO ulimit - Linux transparent hugetables support, enabled="never" defrag="never" splunkd.log.2:08-12-2016 08:44:52.564 -0400 INFO ulimit - Limit: virtual address space size: unlimited splunkd.log.2:08-12-2016 08:44:52.564 -0400 INFO ulimit - Limit: data segment size: unlimited splunkd.log.2:08-12-2016 08:44:52.564 -0400 INFO ulimit - Limit: resident memory size: unlimited splunkd.log.2:08-12-2016 08:44:52.564 -0400 INFO ulimit - Limit: stack size: 10485760 bytes [hard maximum: unlimited] splunkd.log.2:08-12-2016 08:44:52.564 -0400 INFO ulimit - Limit: core file size: 0 bytes [hard maximum: unlimited] splunkd.log.2:08-12-2016 08:44:52.564 -0400 WARN ulimit - Core file generation disabled splunkd.log.2:08-12-2016 08:44:52.564 -0400 INFO ulimit - Limit: data file size: unlimited splunkd.log.2:08-12-2016 08:44:52.564 -0400 INFO ulimit - Limit: open files: 102400 files splunkd.log.2:08-12-2016 08:44:52.564 -0400 INFO ulimit - Limit: user processes: 18000 processes splunkd.log.2:08-12-2016 08:44:52.564 -0400 INFO ulimit - Limit: cpu time: unlimited splunkd.log.2:08-12-2016 08:44:52.564 -0400 INFO ulimit - Linux transparent hugetables support, enabled="never" defrag="never" splunkd.log.3:08-11-2016 18:47:10.873 -0400 INFO ulimit - Limit: virtual address space size: unlimited splunkd.log.3:08-11-2016 18:47:10.873 -0400 INFO ulimit - Limit: data segment size: unlimited splunkd.log.3:08-11-2016 18:47:10.873 -0400 INFO ulimit - Limit: resident memory size: unlimited splunkd.log.3:08-11-2016 18:47:10.873 -0400 INFO ulimit - Limit: stack size: 10485760 bytes [hard maximum: unlimited] splunkd.log.3:08-11-2016 18:47:10.873 -0400 INFO ulimit - Limit: core file size: 0 bytes [hard maximum: unlimited] splunkd.log.3:08-11-2016 18:47:10.873 -0400 WARN ulimit - Core file generation disabled splunkd.log.3:08-11-2016 18:47:10.873 -0400 INFO ulimit - Limit: data file size: unlimited splunkd.log.3:08-11-2016 18:47:10.873 -0400 INFO ulimit - Limit: open files: 4096 files splunkd.log.3:08-11-2016 18:47:10.873 -0400 INFO ulimit - Limit: user processes: 63681 processes splunkd.log.3:08-11-2016 18:47:10.873 -0400 INFO ulimit - Limit: cpu time: unlimited splunkd.log.3:08-11-2016 18:47:10.873 -0400 INFO ulimit - Linux transparent hugetables support, enabled="always" defrag="always" splunkd.log.3:08-11-2016 23:22:48.038 -0400 INFO ulimit - Limit: virtual address space size: unlimited splunkd.log.3:08-11-2016 23:22:48.038 -0400 INFO ulimit - Limit: data segment size: unlimited splunkd.log.3:08-11-2016 23:22:48.038 -0400 INFO ulimit - Limit: resident memory size: unlimited splunkd.log.3:08-11-2016 23:22:48.038 -0400 INFO ulimit - Limit: stack size: 10485760 bytes [hard maximum: unlimited] splunkd.log.3:08-11-2016 23:22:48.038 -0400 INFO ulimit - Limit: core file size: 0 bytes [hard maximum: unlimited] splunkd.log.3:08-11-2016 23:22:48.038 -0400 WARN ulimit - Core file generation disabled splunkd.log.3:08-11-2016 23:22:48.038 -0400 INFO ulimit - Limit: data file size: unlimited splunkd.log.3:08-11-2016 23:22:48.038 -0400 INFO ulimit - Limit: open files: 4096 files splunkd.log.3:08-11-2016 23:22:48.038 -0400 INFO ulimit - Limit: user processes: 63681 processes splunkd.log.3:08-11-2016 23:22:48.038 -0400 INFO ulimit - Limit: cpu time: unlimited splunkd.log.3:08-11-2016 23:22:48.038 -0400 INFO ulimit - Linux transparent hugetables support, enabled="never" defrag="never" splunkd.log.4:08-11-2016 13:12:52.508 -0400 INFO ulimit - Limit: virtual address space size: unlimited splunkd.log.4:08-11-2016 13:12:52.508 -0400 INFO ulimit - Limit: data segment size: unlimited splunkd.log.4:08-11-2016 13:12:52.508 -0400 INFO ulimit - Limit: resident memory size: unlimited splunkd.log.4:08-11-2016 13:12:52.508 -0400 INFO ulimit - Limit: stack size: 10485760 bytes [hard maximum: unlimited] splunkd.log.4:08-11-2016 13:12:52.508 -0400 INFO ulimit - Limit: core file size: 0 bytes [hard maximum: unlimited] splunkd.log.4:08-11-2016 13:12:52.508 -0400 WARN ulimit - Core file generation disabled splunkd.log.4:08-11-2016 13:12:52.508 -0400 INFO ulimit - Limit: data file size: unlimited splunkd.log.4:08-11-2016 13:12:52.508 -0400 INFO ulimit - Limit: open files: 4096 files splunkd.log.4:08-11-2016 13:12:52.508 -0400 INFO ulimit - Limit: user processes: 63681 processes splunkd.log.4:08-11-2016 13:12:52.508 -0400 INFO ulimit - Limit: cpu time: unlimited splunkd.log.4:08-11-2016 13:12:52.508 -0400 INFO ulimit - Linux transparent hugetables support, enabled="always" defrag="always" splunkd.log.5:08-10-2016 23:36:48.825 -0400 INFO ulimit - Limit: virtual address space size: unlimited splunkd.log.5:08-10-2016 23:36:48.825 -0400 INFO ulimit - Limit: data segment size: unlimited splunkd.log.5:08-10-2016 23:36:48.825 -0400 INFO ulimit - Limit: resident memory size: unlimited splunkd.log.5:08-10-2016 23:36:48.825 -0400 INFO ulimit - Limit: stack size: 10485760 bytes [hard maximum: unlimited] splunkd.log.5:08-10-2016 23:36:48.825 -0400 INFO ulimit - Limit: core file size: 0 bytes [hard maximum: unlimited] splunkd.log.5:08-10-2016 23:36:48.825 -0400 WARN ulimit - Core file generation disabled splunkd.log.5:08-10-2016 23:36:48.825 -0400 INFO ulimit - Limit: data file size: unlimited splunkd.log.5:08-10-2016 23:36:48.825 -0400 INFO ulimit - Limit: open files: 4096 files splunkd.log.5:08-10-2016 23:36:48.825 -0400 INFO ulimit - Limit: user processes: 63681 processes splunkd.log.5:08-10-2016 23:36:48.825 -0400 INFO ulimit - Limit: cpu time: unlimited splunkd.log.5:08-10-2016 23:36:48.825 -0400 INFO ulimit - Linux transparent hugetables support, enabled="always" defrag="always" splunkd.log.5:08-11-2016 08:41:56.173 -0400 INFO ulimit - Limit: virtual address space size: unlimited splunkd.log.5:08-11-2016 08:41:56.173 -0400 INFO ulimit - Limit: data segment size: unlimited splunkd.log.5:08-11-2016 08:41:56.173 -0400 INFO ulimit - Limit: resident memory size: unlimited splunkd.log.5:08-11-2016 08:41:56.173 -0400 INFO ulimit - Limit: stack size: 10485760 bytes [hard maximum: unlimited] splunkd.log.5:08-11-2016 08:41:56.173 -0400 INFO ulimit - Limit: core file size: 0 bytes [hard maximum: unlimited] splunkd.log.5:08-11-2016 08:41:56.173 -0400 WARN ulimit - Core file generation disabled splunkd.log.5:08-11-2016 08:41:56.173 -0400 INFO ulimit - Limit: data file size: unlimited splunkd.log.5:08-11-2016 08:41:56.173 -0400 INFO ulimit - Limit: open files: 4096 files splunkd.log.5:08-11-2016 08:41:56.173 -0400 INFO ulimit - Limit: user processes: 63681 processes splunkd.log.5:08-11-2016 08:41:56.173 -0400 INFO ulimit - Limit: cpu time: unlimited splunkd.log.5:08-11-2016 08:41:56.174 -0400 INFO ulimit - Linux transparent hugetables support, enabled="always" defrag="always" kindly let me know is there a fix without restarting the server ? thanks in advance. ... View more

thanks mmodestino, for guiding us, below are the parameter value set. System details: Splunk version 6.4.0 (HF instance) OS - RedHat 6.6 Memory - 6GB CPU - 3 VMware free -m total used free shared buffers cached Mem: 15947 8958 6988 0 732 3124 -/+ buffers/cache: 5101 10846 Swap: 3323 52 3271 I have changes the limits.conf value under /etc/security/limits.conf and restart the splunk service but still the changes did reflect, should I restart the servers ? #* soft core 0 #* hard rss 10000 #@student hard nproc 20 #@faculty soft nproc 20 #@faculty hard nproc 50 #ftp hard nproc 0 #@student - maxlogins 4 root soft nofile 1024000 root hard nofile 1024000 root soft nproc 180000 root hard nproc 180000 ulimit -u 18000 ulimit -n 102400 ulimit -f unlimited I have also disabled THP. cat /sys/kernel/mm/redhat_transparent_hugepage/defrag always madvise [never] cat /sys/kernel/mm/redhat_transparent_hugepage/enabled always madvise [never] But still I could see the same INFO into the splunkd.log . Kindly guide me how to fix this issue. 08-13-2016 01:27:04.217 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-13-2016 01:27:04.354 -0400 INFO TcpOutputProc - Closing stream for idx=x.x.x.x:9997 08-13-2016 01:27:04.354 -0400 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997 08-13-2016 01:27:09.990 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-13-2016 01:27:13.983 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-13-2016 01:27:17.972 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-13-2016 01:27:21.381 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-13-2016 01:27:24.766 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-13-2016 01:27:28.250 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-13-2016 01:27:33.473 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-13-2016 01:27:34.355 -0400 INFO TcpOutputProc - Closing stream for idx=x.x.x.x:9997 08-13-2016 01:27:34.355 -0400 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997 08-13-2016 01:27:38.629 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-13-2016 01:27:42.389 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-13-2016 01:27:46.801 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-13-2016 01:27:51.482 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-13-2016 01:27:54.734 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-13-2016 01:27:58.412 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-13-2016 01:28:02.596 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-13-2016 01:28:04.353 -0400 INFO TcpOutputProc - Closing stream for idx=x.x.x.x:9997 08-13-2016 01:28:04.353 -0400 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997 08-13-2016 01:28:06.651 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-13-2016 01:28:10.857 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-13-2016 01:28:15.696 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-13-2016 01:28:19.354 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-13-2016 01:28:23.987 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-13-2016 01:28:28.085 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-13-2016 01:28:32.075 -0400 INFO TailReader - File descriptor cache is full (100), trimming... thanks in advance. ... View more

yes, it reads the syslog's data which is running in the same server where the HF instance is running. /opt/splunk/etc/apps/ADMIN-hvy-forwarders/default/inputs.conf [monitor:///opt/syslogs/web_access/.../Common/*.log] [monitor:///opt/syslogs/symantec/SymantecServer/...] [monitor:///opt/syslogs/symantec/*semp*/...] [monitor:///opt/syslogs/symantec/.../ID.log] [monitor:///opt/syslogs/proxy/...] sourcetype = bluecoat_syslog [monitor:///opt/syslogs/dns/.../*.log] sourcetype = syslog [monitor:///opt/syslogs/webops_security/.../*.log] sourcetype = syslog [monitor:///opt/syslogs/firewall/.../*.log] sourcetype = syslog [monitor:///opt/syslogs/esx/.../*.log] sourcetype = syslog [monitor:///opt/syslogs/generic/DCESX*/*.log] sourcetype = syslog [monitor:///opt/syslogs/generic/dcqip*/*.log] sourcetype = syslog [monitor:///opt/syslogs/generic/dcesx*/*.log] sourcetype = syslog [monitor:///opt/syslogs/generic/GTSPCFW*/*.log] sourcetype = syslog [monitor:///opt/syslogs/generic/.../*.log] sourcetype = syslog [monitor:///opt/syslogs/generic/idsmgt/*.log] [monitor:///opt/syslogs/yammer/Messages.csv] so what will be the solution to make it stable ? I mean to make the swap memory stable. we have asked the Unix team to increase the swap memory from 3 GB to 8 GB will this work ? we are getting almost 20GB of syslog data in each of the two heavy-forwarder for every 24 hours and we have set a log rotate/corn job to delete the files after hour-an hour to control the disk space. Kindly guide us what will be the permanent solution for this. ... View more

lycollicott, Once again I could see syslogs data are not being captured from the server 01. By executing the host=splunk01* sourcetype=splunkd index=_internal "syslog" with the time frame for last 60 min, I got no result found. I have noticed that swap memory are almost 100% utilized by splunkd process. Kindly help me on this to fix the issue. INFO from splunkd.log 08-11-2016 13:24:57.261 -0400 INFO TcpOutputProc - Closing stream for idx=x.x.x.x:9997 08-11-2016 13:24:57.261 -0400 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997 08-11-2016 13:25:00.434 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:05.790 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:10.407 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:15.343 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:20.427 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:24.591 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:27.262 -0400 INFO TcpOutputProc - Closing stream for idx=x.x.x.x:9997 08-11-2016 13:25:27.263 -0400 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997 08-11-2016 13:25:29.224 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:34.030 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:38.908 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:43.324 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:47.946 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:53.707 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:57.262 -0400 INFO TcpOutputProc - Closing stream for idx=x.x.x.x:9997 08-11-2016 13:25:57.262 -0400 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997 08-11-2016 13:25:58.585 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:26:03.290 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:26:08.387 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:26:13.522 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:26:17.714 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:26:22.073 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:26:26.530 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:26:31.304 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:26:36.384 -0400 INFO TailReader - File descriptor cache is full (100), trimming... thanks in advance . ... View more

lycollicott, Once again I could see syslogs data are not being captured from the server 01. By executing the host=splunk01* sourcetype=splunkd index=_internal "syslog" with the time frame for last 60 min, I got no result found. I have noticed that swap memory are almost 100% utilized by splunkd process. Kindly help me on this to fix the issue. INFO from splunkd.log 08-11-2016 13:24:57.261 -0400 INFO TcpOutputProc - Closing stream for idx=x.x.x.x:9997 08-11-2016 13:24:57.261 -0400 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997 08-11-2016 13:25:00.434 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:05.790 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:10.407 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:15.343 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:20.427 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:24.591 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:27.262 -0400 INFO TcpOutputProc - Closing stream for idx=x.x.x.x:9997 08-11-2016 13:25:27.263 -0400 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997 08-11-2016 13:25:29.224 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:34.030 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:38.908 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:43.324 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:47.946 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:53.707 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:25:57.262 -0400 INFO TcpOutputProc - Closing stream for idx=x.x.x.x:9997 08-11-2016 13:25:57.262 -0400 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997 08-11-2016 13:25:58.585 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:26:03.290 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:26:08.387 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:26:13.522 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:26:17.714 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:26:22.073 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:26:26.530 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:26:31.304 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 13:26:36.384 -0400 INFO TailReader - File descriptor cache is full (100), trimming... thanks in advance ... View more

thanks martin, actually we don't have a test machine in our environment, so I have planning to raise a ticket with splunk support team to seek their help on this. ... View more

thank lycollicott, the query worked and I got the required output. host=splunk0* sourcetype=splunkd index=_internal "syslog" | stats sum(kb) by series, host | rename sum(kb) as KB | table series, host, KB | sort host, series | appendpipe [stats sum(KB) as Total by host ] with Time frame set to last 24 hours Today again data was not getting captured in splunk01 server for more than 3 hours, when executed the below query, I got no data found with time frame set to last 15 min. host=splunk0* sourcetype=splunkd index=_internal "syslog when verified splunk status, syslog-ng services and port are fine but still there was not result, but I had noticed below info in splunkd.log and also swap memory was 0. 08-11-2016 07:06:58.118 -0400 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_x.x.x.x_8089_splunk01.xxxx.com_splunk01.xxx.com_7xxxx1-XXXXX-XXX-XXX-XXXX 08-11-2016 07:06:58.128 -0400 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_x.x.x.x_8089_splunk01.xxxx.com_splunk01.xxx.com_7xxxx1-XXXXX-XXX-XXX-XXXX 08-11-2016 07:06:58.156 -0400 INFO HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_x.x.x.x_8089_splunk01.xxxx.com_splunk01.xxx.com_7xxxx1-XXXXX-XXX-XXX-XXXX 08-11-2016 07:07:45.496 -0400 INFO TailReader - File descriptor cache is full (100), trimming... 08-11-2016 07:07:48.220 -0400 INFO TcpOutputProc - Closing stream for idx=X.X.X.X:9997 08-11-2016 07:07:48.220 -0400 INFO TcpOutputProc - Connected to idx=X.X.X.X:9997 08-11-2016 07:08:17.406 -0400 INFO TcpOutputProc - Closing stream for idx=X.X.X.X:9997 08-11-2016 07:08:17.406 -0400 INFO TcpOutputProc - Connected to idx=X.X.X.X:9997 08-11-2016 07:08:47.566 -0400 INFO TcpOutputProc - Closing stream for idx=X.X.X.X:9997 08-11-2016 07:08:47.566 -0400 INFO TcpOutputProc - Connected to idx=X.X.X.X:9997 08-11-2016 07:08:52.863 -0400 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-nessus/bin/nessus2splunk.py" usage: nessus2splunk.py [-h] [-s SRCDIR] [-t TGTDIR] 08-11-2016 07:08:52.863 -0400 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-nessus/bin/nessus2splunk.py" nessus2splunk.py: error: argument -s/--srcdir: Invalid path specified ($SPLUNK_HOME may not be set). 08-11-2016 07:09:17.565 -0400 INFO TcpOutputProc - Closing stream for idx=X.X.X.45:9997 08-11-2016 07:09:17.565 -0400 INFO TcpOutputProc - Connected to idx=X.X.X.X:9997 08-11-2016 07:09:47.859 -0400 INFO TcpOutputProc - Closing stream for idx=X.X.X.X:9997 08-11-2016 07:09:47.958 -0400 INFO TcpOutputProc - Connected to idx=X.X.X.X:9997 08-11-2016 07:10:18.029 -0400 INFO TcpOutputProc - Closing stream for idx=X.X.X.X:9997 08-11-2016 07:10:18.029 -0400 INFO TcpOutputProc - Connected to idx=X.X.X.X:9997 so restarted the splunk service and it start working normally and now i am able to see the result by executing the query. thanks in advance. ... View more

Ya, but i am yet to get the permission to delete data from the organization. where I can find the limits.conf file as there are 46 Indexes.conf file are configured in all the 5 individual servers. So do you want me to change in all the 5 server under the path /opt/splunk/etc/system/local/limits.conf thanks in advance. ... View more

thanks martin for helping me. But I am not sure how to change the configuration, as it was done by a splunk expert, who is currently not there in the organization. Is it possible to guide me on steps to change the configuration. ... View more

Martin, can you guide me whether I can still go head and delete the content that are older data present under /splogs/net_fw/colddb/ older data. ... View more

thank lycollicott. we are getting the amount of date injected in both the servers 01 and 02. I had included command addcoltotals labelfield=series to get the total amount of syslog data per day. host=splunk01* sourcetype=splunkd index=_internal "syslog" | stats sum(kb) by series | addcoltotals labelfield=series from server01 the total amount of syslog data is about 15895692.557388 KB and its about 15.8 GB for last 24 hours from server 02 the total amount of syslog data is about 18893695.258755 KB and its about 18 GB of data. It seems more traffic are routed to server 02 and it took some time to search this information not incase of HF splunk01. Hey I tired the below query but it was difficult to identify from which server the data is being injected, is it possible to highlight the both server 01 and 02 in separate filed or column. host=splunk0* sourcetype=splunkd index=_internal "syslog" | stats sum(kb) by series | addcoltotals labelfield=series thanks in advance. ... View more

thank lycollicott, we got the output in kb and I have addcoltotals to get the sum of all the syslogs data injected from the two servers. host=splunk01* sourcetype=splunkd index=_internal "syslog" | stats sum(kb) by series | addcoltotals labelfield=series with time frame for last 24 hours from server 01- total = 17805533.451029 KB of data which is approximately 17.8GB of data being injected. from server 02 - total = 17546902.822219 KB of data which is approximately 17.5GB of data being injected. I guess now the data are being properly distributed from F5 load balancer. Instead of running two separate is it possible to get both the result in a single query ? thanks in advance. ... View more