https://community.splunk.com/t5/Getting-Data-In/Scripting-to-pull-in-logs-from-a-URL/m-p/55160#M10740 <P>I am trying to create a script that will index retrieve Apache server logs but have been unable to figure out how to do it. I am not able to place a forwarder on the machine, but I do have HTTP access to the log directory. I have tried creating a script to pull down the log files using WGET (DOS, Windows), and I get it to pull down the files, but I can not figure out how to get Splunk to index them. The files are compressed, so I access_log_1.gz, access_log_2.gz, etc. I have places the script in the $Splunk_Home\bin\scripts dir, and it points to a .bat file in $Splunk_Home\bin. The only line of the bat file is: "wget -r -nv -nH -A "*.gz" <A href="http://10.10.10.10:8001/logs/" target="_blank">http://10.10.10.10:8001/logs/</A>"</P> <P>Can someone point me to documentation or examples that show how to do this? Do I have to create an App to do it? Or can I just use a script only?<BR /><BR /> Thank you for any insight you can provide. </P> Mon, 28 Sep 2020 09:47:21 GMT minton55 2020-09-28T09:47:21Z https://community.splunk.com/t5/Getting-Data-In/Scripting-to-pull-in-logs-from-a-URL/m-p/55160#M10740 <P>I am trying to create a script that will index retrieve Apache server logs but have been unable to figure out how to do it. I am not able to place a forwarder on the machine, but I do have HTTP access to the log directory. I have tried creating a script to pull down the log files using WGET (DOS, Windows), and I get it to pull down the files, but I can not figure out how to get Splunk to index them. The files are compressed, so I access_log_1.gz, access_log_2.gz, etc. I have places the script in the $Splunk_Home\bin\scripts dir, and it points to a .bat file in $Splunk_Home\bin. The only line of the bat file is: "wget -r -nv -nH -A "*.gz" <A href="http://10.10.10.10:8001/logs/" target="_blank">http://10.10.10.10:8001/logs/</A>"</P> <P>Can someone point me to documentation or examples that show how to do this? Do I have to create an App to do it? Or can I just use a script only?<BR /><BR /> Thank you for any insight you can provide. </P> Mon, 28 Sep 2020 09:47:21 GMT https://community.splunk.com/t5/Getting-Data-In/Scripting-to-pull-in-logs-from-a-URL/m-p/55160#M10740 minton55 2020-09-28T09:47:21Z https://community.splunk.com/t5/Getting-Data-In/Scripting-to-pull-in-logs-from-a-URL/m-p/55161#M10741 <P>Getting the logs to your Splunk server is only 1/2 of the battle. You have to set up a source to actually index the files. I'm making the assumption you are sticking these files somewhere else other then the scripts directory.</P> <P>You will want to check out <A href="http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitorfilesanddirectories">http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitorfilesanddirectories</A> for more information on setting up a monitor..</P> Tue, 09 Aug 2011 17:42:56 GMT https://community.splunk.com/t5/Getting-Data-In/Scripting-to-pull-in-logs-from-a-URL/m-p/55161#M10741 Brian_Osburn 2011-08-09T17:42:56Z https://community.splunk.com/t5/Getting-Data-In/Scripting-to-pull-in-logs-from-a-URL/m-p/55162#M10742 <P>This is what I was missing. I am not able to do this in one script. I have to create a script that will pull over the files, and then set up a monitor on the directory to pull in the logs as they are written there.</P> <P>Thank you.</P> Tue, 09 Aug 2011 20:17:45 GMT https://community.splunk.com/t5/Getting-Data-In/Scripting-to-pull-in-logs-from-a-URL/m-p/55162#M10742 minton55 2011-08-09T20:17:45Z https://community.splunk.com/t5/Getting-Data-In/Scripting-to-pull-in-logs-from-a-URL/m-p/55163#M10743 <P>We use rsync to copy the apache logs from our web servers to our splunk server...</P> <P>You can download a windows version of rsync from <A href="http://rsync.samba.org/download.html">http://rsync.samba.org/download.html</A></P> <P>FYI: cwRsync is a packaging of rsync for MS Windows</P> Wed, 10 Aug 2011 01:11:52 GMT https://community.splunk.com/t5/Getting-Data-In/Scripting-to-pull-in-logs-from-a-URL/m-p/55163#M10743 lukeh 2011-08-10T01:11:52Z