TCP data input: Why is splunk receiving only some data? Is there a limit that needs to be configured?

[jimrantoday]

Hello team,

When sent data from my cloudbees syslog java client using tcp data input, only some data is making it to splunk. Exactly 206 records are making it to Splunk even though I am sending data in excess to 1000 records. Is there a limit that needs to be configured? please advise.

Thank you.

[jkat54]

@jimrantoday,

Btool shows the configuration that exists on disk, not what's loaded into memory.

Appreciate the credit for all the help...

It's like you came up with the answer all on your own...

[jimrantoday · Accepted solution]

Adding

[tcp://5540]
queueSize = 5KB
persistentQueueSize = 10KB

in inputs.conf at $SPLUNK_HOME/etc/apps/"your app name"/local location did the trick for me.

After adding the configuration, you can also check to see if they got picked up or not by typing "splunk cmd btool --app=search inputs list" command in command prompt at Splunk\bin

[gcusello]

Hi jimrantoday,
have you problems of disk performace?
One of my customer had this problem because used very slow disks so Indexers didn't index all data and someone of them were loosed.
You can verify this using a simple search

index=_internal source=*metrics.log sourcetype=splunkd group=queue host=”your_indexer” blocked | timechart count by name

You can verify disk performances using an external tool (like Bonnie++), Splunk requires at least 800 iops.

To avoid this problem I configured a persistent queue on my indexer:
in /opt/splunk/etc/apps/search/local/inputs.conf file, in udp and tcp rooms, I inserted persistentQueueSize = 10MB parameter.

Bye.
Giuseppe

[jkat54]

Tcp and UDP inputs have a receive buffer that doesn't flush to disk until they overflow or Splunk restarts.

Check out _rcv_buffer (I think it's called) in inputs.conf

[jkat54]

Try this instead:

[tcp://5540]
queueSize= 1KB

Or

[tcp:5540]
queueSize= 1KB

OR
[tcp://*:5540]
queueSize= 1KB

[jkat54]

Yeah ok it's queueSize but try something much smaller like 1KB

[jimrantoday]

1. Increased the limits to see if there is an increase in no.of events indexed in Splunk.

->added the following configuration in \system\local\inputs.conf
[tcp://5540]
queueSize = 50MB
persistentQueueSize = 100MB

->Restarted Splunk
-> Ran the test and still the no.of events remains 206 (where as the events available to send via TCP are 6500).

*Note:- When i ran "splunk cmd btool --app=search inputs list", my output was
[splunktcp://9997]
connection_host = ip
[tcp://5540]
connection_host = dns
disabled = 0
sourcetype = syslog

Looks like my config changes have not been picked up even after the restart.

1. Decreased the limits to 1KB and the no.of events remain the same (206).

What could possibly be the issue here ?

[jimrantoday]

I have searched the inputs.conf documentation and couldn't find anything related to out_rcv_buffer. I tried changing the queueSize to 50mb and restarted the splunk instance but even that didn't help. Could you please check to find the property?