How can I pull logs from a shared hosting account and get it into Splunk to index?

[walderbachj1]

The hosting provider is Rackspace Cloud Sites. In the root of each site is a logs dir, ex. somesite.com/logs. There are two different logs I want to grab each day:

1) ourlog_posts_YESTERDAYSDATE.csv
2) access_log_YESTERDAYSDATE.zip

Note I am unable to install a Universal Forwarder as this is essentially a shared hosting site. So I want to grab each log, each morning, and get it into Splunk to index.

• I can set up cron jobs on the shared hosting site.
  • We have Linux and Windows servers which currently send logs to Splunk via forwarders, but I want these 2 logs to have a unique hostname/source so they don't get thrown in with any other servers logs. So any solution that pulls these logs via FTP to a location already being monitored, needs to have that addressed.

I'd think someone has come up against this problem before and I'd like to hear your solution.

[jterry · Accepted solution]

Assuming your cron scripts/actions can get the files off the hosting system & on to a system that splunk has access to, you could configure a local dir input on the splunk server (or forwarder, etc) & associate the desired metadata to any data that's indexed from that source. The cron actions could then just put the data there.

[walderbachj1]

That's my initial thought. But thought I'd solicit other ideas too.