topic Re: Fetch data from FTP Server in Reporting

Fetch data from FTP Server

som_80in — Thu, 01 Nov 2012 14:41:43 GMT

what is configuration to be done with TCP in splunk in add data in web or in inputs.conf file to fetch proxy logs logs which are collected in FTP server.
I cannot mount the disk nor i can install forwarder in the FTP server.I want to configure with TCP.

Please help

Re: Fetch data from FTP Server

dwaddle — Fri, 02 Nov 2012 01:56:01 GMT

You will need to write some form of program/script to pull the logs from the FTP server and store them where Splunk can read them. Splunk has no built-in way of connecting to a server, retrieving files via FTP, and indexing them.

If this is linux, you might possibly be able to use something like CurlFtpFS to make the remote FTP server mountable. However, it is highly unknown how well (or if at all) this will work with Splunk.

Re: Fetch data from FTP Server

som_80in — Fri, 02 Nov 2012 09:27:26 GMT

Thanks a lot dwaddle.Can you please let me know any sample scripts to run on.I will also try CurlFtpFS and get back to u.

Re: Fetch data from FTP Server

dwaddle — Sun, 04 Nov 2012 01:18:35 GMT

I have no such script, sorry ...

Re: Fetch data from FTP Server

kallu — Sun, 04 Nov 2012 17:53:52 GMT

Scripting ftp isn't that difficult. Here is some examples to get you started
http://www.stratigery.com/scripting.ftp.html

Re: Fetch data from FTP Server

nicholasgrabows — Thu, 27 Dec 2012 19:39:39 GMT

There is a new splunkbase app called "importutil". It lets you import csv files (or any input) from an http url via the splunk search command line. Also works for ftp. sftp is experimental.

http://splunk-base.splunk.com/apps/69078/importutil

Here is an ftp example. Pulling from the bureau of labor stats:

|importutil ftp ftp://ftp.bls.gov/pub/time.series/ce/ce.data.102.WeeklyEarningsHist
| multikv
| table series_id, year, period, value, footnote_codes

Here is an example that imports data from the federal reserve economic data website:

|importutil http http://research.stlouisfed.org/fred2/data/PAYEMS.csv
| multikv
| table DATE, VALUE