topic Re: How to compress CSV file and email it on Unix platform? in Getting Data In

How to compress CSV file and email it on Unix platform?

amitkr0201 — Fri, 13 Jun 2014 09:24:45 GMT

Is there an app/script/something else available which compresses a csv (preferably output of outputcsv command ) and emails it? Bonus point if its able to split it into multiple zips if size exceeds threshold.

I am working on a set of data which, when extracted as CSV, sizes to approx 150 MB, which when compressed is >10 MB.

The email provider I use allows max 8 MB attachments, so I need compression as well as splitting into multiple zip.

EDIT: Platform = Unix

Re: How to compress CSV file and email it on Unix platform?

grijhwani — Fri, 13 Jun 2014 11:23:35 GMT

On what platform? You don't specify Windows/Unix.

Re: How to compress CSV file and email it on Unix platform?

linu1988 — Fri, 13 Jun 2014 12:35:30 GMT

You can use a script which will break the zip file into pieces. If it more than the specified limit, send it in another mail. Don't use the default splunk script to send mail.

_http://docs.splunk.com/Documentation/Splunk/6.1.1/alert/ConfiguringScriptedAlerts

Use the 8th argument to get the csv file then perform the zip and mailing.

Thanks,
L

Re: How to compress CSV file and email it on Unix platform?

grijhwani — Fri, 13 Jun 2014 13:40:12 GMT

Linux provides zip, gzip, bzip2 for compression, and split to (wait for it) split files into smaller files... Then there are a multiplicity of ways of despatching the results as e-mails.

Writing a shell script to compress the file, then chunk it into multiple fixed size attachments should be almost trivial. If you are concerned about portability, then the better alternative is to split the source file at a given threshold and using zip specifically (for Windows compatibility) on the chunks individually.

There you go. Off the top of my head, and not debugged or checked, and dependent on installing Mutt as a mail client (because of its superior command line attachment management).

#!/bin/bash
sMailTarget=you@your.domain
fSource=${SPLUNK_ARG_8}
[ -f ${fSource} ] || { echo "No file"; exit 1; }
# Comment the next line out of you want to send empty results
[ -s ${fSource} ] || { echo "Empty file"; exit 1; }
fPrefix="`echo ${fSource} | sed -e s'/.csv//'`-seg"

split -a 2 -l50000 ${fSource} ${fPrefix}
for ${fInter} in ${fPrefix}*; do
  mv ${fInter} ${fInter}.csv
  zip -9 ${fInter}.zip ${fInter}.csv
  mutt -s "${SPLUNK_ARG_4} result segment ${fInter}" -a ${fInter}.zip ${sMailTarget} << EoM
File attached
EoM
done

Re: How to compress CSV file and email it on Unix platform?

amitkr0201 — Fri, 13 Jun 2014 13:51:25 GMT

Sounds promising I'll try it out. Thanks