topic TCP data feed issue in Getting Data In

TCP data feed issue

yg — Tue, 16 Oct 2012 18:31:32 GMT

1 out of 20 records is broken with TCP connection from geographically distant locations such as Japan. No problem when connecting from the US servers. From London: about 1 out of 40.

Does it have anything to do with buffering the complete record before writing it to the Splunk database when the packets of the TCP/IP connection are being resent?

Thank you.

Re: TCP data feed issue

tskinnerivsec — Tue, 16 Oct 2012 19:17:35 GMT

Are you sending the data using the splunk universal forwarder or are you sending data via syslog? Can you also specify what you mean with a record being broken?

Re: TCP data feed issue

yg — Tue, 16 Oct 2012 19:58:41 GMT

I am sending the data with

tail -F | nc -v

By record being broken I mean a partial record. As far as I see, they're chopped off from the beginning in various places. I don't see the missing parts of the records so I am not sure if they're lost or appear on some other pages.

Re: TCP data feed issue

tskinnerivsec — Tue, 16 Oct 2012 21:13:21 GMT

Do you have the option of deploying a Splunk Heavy Forwarder in your remote location to aggregate your remote inputs?

Re: TCP data feed issue

yg — Wed, 17 Oct 2012 16:20:44 GMT

I just installed Splunk Universal Forwarder to the remote machine and added forwarding to my Splunk primary server through the opened port for the TCP data feed with

splunk add forward-server :

How do I start forwarding the log info from the remote server? What is the syntax? For some reason I was unable to find.

I am running it on 64-bit Linux.

Thank you.

Re: TCP data feed issue

tskinnerivsec — Wed, 17 Oct 2012 16:50:17 GMT

you will need to configure an inputs.conf file on the forwarder to monitor the file location of your log and send it to your splunk server.

in your inputs.conf file on the universal forwarder you would have a stanza something like this:

[monitor:///var/log/logfilename]
sourcetype = logfile
disabled = 0

In this stanza, you basically want to specify the file location of the log file you are monitoring and give it a source type. You can name the sourcetype anything you want, just name it something that makes sense for your environment.

You can deploy this inputs.conf file in a couple of different ways. If you are manually configuring everything, you could locate this file in the /etc/system/local area under the universal forwarder file path. If you wan't more granular control you could deploy this configuration as its own app to the universal forwarder in which case it would live under the /etc/apps/app-name/ area under the universal forwarder file path. You can name the app anything you like, it is good to have a functional naming scheme so you know what your apps do just by looking at them. This gets into a whole other area of splunk configurations. A good guide to look through is the splunk "Getting Data in Correctly" guide.