When sent data from my cloudbees syslog java client using tcp data input, only some data is making it to splunk. Exactly 206 records are making it to Splunk even though I am sending data in excess to 1000 records. Is there a limit that needs to be configured? please advise.
Tcp and UDP inputs have a receive buffer that doesn't flush to disk until they overflow or Splunk restarts.
Check out rcvbuffer (I think it's called) in inputs.conf
I have searched the inputs.conf documentation and couldn't find anything related to outrcvbuffer. I tried changing the queueSize to 50mb and restarted the splunk instance but even that didn't help. Could you please check to find the property?
Yeah ok it's queueSize but try something much smaller like 1KB
->added the following configuration in \system\local\inputs.conf
queueSize = 50MB
persistentQueueSize = 100MB
-> Ran the test and still the no.of events remains 206 (where as the events available to send via TCP are 6500).
*Note:- When i ran "splunk cmd btool --app=search inputs list", my output was
connectionhost = ip
connectionhost = dns
disabled = 0
sourcetype = syslog
Looks like my config changes have not been picked up even after the restart.
What could possibly be the issue here ?
Try this instead:
have you problems of disk performace?
One of my customer had this problem because used very slow disks so Indexers didn't index all data and someone of them were loosed.
You can verify this using a simple search
index=_internal source=*metrics.log sourcetype=splunkd group=queue host=”your_indexer” blocked | timechart count by name
You can verify disk performances using an external tool (like Bonnie++), Splunk requires at least 800 iops.
To avoid this problem I configured a persistent queue on my indexer:
in /opt/splunk/etc/apps/search/local/inputs.conf file, in udp and tcp rooms, I inserted
persistentQueueSize = 10MB parameter.
queueSize = 5KB
persistentQueueSize = 10KB
in inputs.conf at $SPLUNK_HOME/etc/apps/"your app name"/local location did the trick for me.
After adding the configuration, you can also check to see if they got picked up or not by typing "splunk cmd btool --app=search inputs list" command in command prompt at Splunk\bin
Btool shows the configuration that exists on disk, not what's loaded into memory.
Appreciate the credit for all the help...
It's like you came up with the answer all on your own...