How to compress CSV file and email it on Unix platform?

[amitkr0201]

Is there an app/script/something else available which compresses a csv (preferably output of outputcsv command ) and emails it? Bonus point if its able to split it into multiple zips if size exceeds threshold.

I am working on a set of data which, when extracted as CSV, sizes to approx 150 MB, which when compressed is >10 MB.

The email provider I use allows max 8 MB attachments, so I need compression as well as splitting into multiple zip.

EDIT: Platform = Unix

[grijhwani]

Linux provides zip, gzip, bzip2 for compression, and split to (wait for it) split files into smaller files... Then there are a multiplicity of ways of despatching the results as e-mails.

Writing a shell script to compress the file, then chunk it into multiple fixed size attachments should be almost trivial. If you are concerned about portability, then the better alternative is to split the source file at a given threshold and using zip specifically (for Windows compatibility) on the chunks individually.

There you go. Off the top of my head, and not debugged or checked, and dependent on installing Mutt as a mail client (because of its superior command line attachment management).

#!/bin/bash
[email protected]
fSource=${SPLUNK_ARG_8}
[ -f ${fSource} ] || { echo "No file"; exit 1; }
# Comment the next line out of you want to send empty results
[ -s ${fSource} ] || { echo "Empty file"; exit 1; }
fPrefix="`echo ${fSource} | sed -e s'/.csv//'`-seg"

split -a 2 -l50000 ${fSource} ${fPrefix}
for ${fInter} in ${fPrefix}*; do
  mv ${fInter} ${fInter}.csv
  zip -9 ${fInter}.zip ${fInter}.csv
  mutt -s "${SPLUNK_ARG_4} result segment ${fInter}" -a ${fInter}.zip ${sMailTarget} << EoM
File attached
EoM
done

[linu1988 · Accepted solution]

You can use a script which will break the zip file into pieces. If it more than the specified limit, send it in another mail. Don't use the default splunk script to send mail.

_http://docs.splunk.com/Documentation/Splunk/6.1.1/alert/ConfiguringScriptedAlerts

Use the 8th argument to get the csv file then perform the zip and mailing.

Thanks,
L

[amitkr0201]

Sounds promising I'll try it out. Thanks

[grijhwani]

On what platform? You don't specify Windows/Unix.