topic How can I pull logs from a shared hosting account and get it into Splunk to index? in Getting Data In

How can I pull logs from a shared hosting account and get it into Splunk to index?

walderbachj1 — Tue, 29 Sep 2020 09:50:50 GMT

The hosting provider is Rackspace Cloud Sites. In the root of each site is a logs dir, ex. somesite.com/logs. There are two different logs I want to grab each day:

1) ourlog_posts_YESTERDAYSDATE.csv
2) access_log_YESTERDAYSDATE.zip

Note I am unable to install a Universal Forwarder as this is essentially a shared hosting site. So I want to grab each log, each morning, and get it into Splunk to index.

I can set up cron jobs on the shared hosting site.
- We have Linux and Windows servers which currently send logs to Splunk via forwarders, but I want these 2 logs to have a unique hostname/source so they don't get thrown in with any other servers logs. So any solution that pulls these logs via FTP to a location already being monitored, needs to have that addressed.

I'd think someone has come up against this problem before and I'd like to hear your solution.

Re: How can I pull logs from a shared hosting account and get it into Splunk to index?

jterry — Thu, 02 Jun 2016 17:24:56 GMT

Assuming your cron scripts/actions can get the files off the hosting system & on to a system that splunk has access to, you could configure a local dir input on the splunk server (or forwarder, etc) & associate the desired metadata to any data that's indexed from that source. The cron actions could then just put the data there.

Re: How can I pull logs from a shared hosting account and get it into Splunk to index?

walderbachj1 — Thu, 02 Jun 2016 17:46:24 GMT

That's my initial thought. But thought I'd solicit other ideas too.