topic Re: How to collect Apache logs in Splunk without a forwarder? in Getting Data In

How to collect Apache logs in Splunk without a forwarder?

sloshburch — Thu, 19 Nov 2015 20:35:25 GMT

Someone just asked me an interesting question that I don't have the answer to...but I bet this community does 😉

Has anyone ever created an interesting way of getting apache logs off a server without actually installing a forwarder?

If it requires installing or running any script, then we might as well use a forwarder. But you never know if someone out there found some native way to send this log data into splunk using some other means than the forwarder.

Thanks for any ideas!

Re: How to collect Apache logs in Splunk without a forwarder?

ltrand — Thu, 19 Nov 2015 20:39:29 GMT

You can syslog the data out
or
You can have some kind of scp/scripted pull action from some centralized entity
or
you can write them to a local SQL/DB instance and use dbconnect to pull it
or
you can install the forwarder
:)

Re: How to collect Apache logs in Splunk without a forwarder?

mreynov_splunk — Fri, 20 Nov 2015 00:19:14 GMT

or you can figure out how to use the fancy new HTTP Event Collector: http://blogs.splunk.com/2015/10/06/http-event-collector-your-direct-event-pipe-to-splunk-6-3/

Re: How to collect Apache logs in Splunk without a forwarder?

hylam — Fri, 20 Nov 2015 05:07:07 GMT

NFS (Network File System)

Re: How to collect Apache logs in Splunk without a forwarder?

woodcock — Fri, 20 Nov 2015 15:56:03 GMT

You can use Stream and snoop the wire:

https://www.splunk.com/content/dam/splunk2/pdfs/fact-sheets/splunk-for-stream.pdf

Re: How to collect Apache logs in Splunk without a forwarder?

dhuseau — Mon, 01 Feb 2016 01:45:28 GMT

I have this issue too, but I am working on a python script that will send events to Splunk using the HTTP Event Collector. It's not finished yet but, I should get an initial version done this week. When it's done you can call the script via a cron job and it will read the latest logs and send them over to Splunk.

I have the first few commits here: https://github.com/alecdhuse/VPS-Log-Watch

Re: How to collect Apache logs in Splunk without a forwarder?

sloshburch — Mon, 01 Feb 2016 14:04:09 GMT

I like that you're using the HTTP Event Collector but I've been burned before with custom scripts. If I was able to have anything run on the endpoint apache server, I would go with a forwarder (proven, small, "real-time"). Are you able to share the circumstances that motivated this effort?

Re: How to collect Apache logs in Splunk without a forwarder?

dhuseau — Mon, 01 Feb 2016 15:20:26 GMT

I have a website on a shared VPS server. I do not manage the server and I do not have rights to install any software including the Splunk forwarder. However, I am allowed to run scripts and schedule cron jobs.

This seemed like an acceptable way to fill this gap. I am open to other methods of moving the data though.

Re: How to collect Apache logs in Splunk without a forwarder?

sloshburch — Mon, 01 Feb 2016 23:05:34 GMT

Oh ok. I have a similar situation and so I used the tar (not rpm) version to place the splunk forwarder. Then I have a nightly cronjob that simply runs ./splunk start to makes sure it didn't get killed. So far so good and more stable than my poor programming.

I figured I'd share in case my similar scenario inspires something.

Re: How to collect Apache logs in Splunk without a forwarder?

dhuseau — Tue, 02 Feb 2016 14:00:48 GMT

Thanks, that is probably a better solution.

Re: How to collect Apache logs in Splunk without a forwarder?

VinayakJamadar — Sun, 31 Mar 2024 18:32:38 GMT

Can you please explain a little bit more about this approach?

Re: How to collect Apache logs in Splunk without a forwarder?

PickleRick — Sun, 31 Mar 2024 21:50:55 GMT

You're trying do dig out a thread from some 8 years ago. Most probably most of the participants are no longer actively following Answers.

Your best bet would be to create a new thread and desciribe your problem there (possibly providing a link to this one if your case is similar.