Hello. I have a HF and I want it to download a .csv file from another internal server. Right now, I can download it as the splunk user using
wget on CLI so I know connectivity and permissions are no issue. I looked at scripted inputs but I don't think that's the right way about it, as I can't get it to work. Not sure how to go about this?
I just want to download a csv file and then send it to my indexer tier.
/usr/bin/wget -O file.csv 'https://myserver.com/feeds/list?v=csv&f=indicator&tr=1' exit 0
[script://./bin/script.sh] index = main sourcetype = test interval = 600.0 disabled = 0
07-10-2019 00:08:18.082 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/my_app/bin/script.sh" file.csv: Permission denied
I checked and the app is owned by splunk:splunk. The script is 755. I ran the ad-hoc command below as the splunk user and it downloaded the file just fine
/opt/splunk/bin/splunk cmd ../etc/apps/my_app/bin/script.sh
I tried adding the input through the HF's gui (Settings > Data Inputs > Scripts > Add new) but my app and script are not showing up in the dropdown...
it's not the script that is the issue here, it's the output file
file.csv AND it's location the script is trying to create it. Set the output file to use a full path that you are sure the user
splunk can write into.
Hope this helps ...
If I just want splunk to read the stream and then send it to indexers rather than download a log file and then send it, how would I adjust the script?
Splunk can index whatever is returned to standard out as a scripted input.
wget -O - 'https://myserver.com/feeds/list?v=csv&f=indicator&tr=1' 2> /dev/null
or, if you want to redirect standard error output also:
wget -O - 'https://myserver.com/feeds/list?v=csv&f=indicator&tr=1' 2>&1
@rob_jordan I changed the script to just be the 1-liner that you put (as I didn't know how to send data to stdout). Now it shows this and no data in index=main
07-10-2019 02:07:56.729 +0000 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/my_app/bin/script.sh" 2019-07-10 02:07:56 (8.61 MB/s) - written to stdout 
-O file.csv possibly should have a specific path before the file name (where the file should be written) unless preceded with a
cd some dir command