Getting Data In

How to compress CSV file and email it on Unix platform?

amitkr0201
Explorer

Is there an app/script/something else available which compresses a csv (preferably output of outputcsv command ) and emails it? Bonus point if its able to split it into multiple zips if size exceeds threshold.

I am working on a set of data which, when extracted as CSV, sizes to approx 150 MB, which when compressed is >10 MB.

The email provider I use allows max 8 MB attachments, so I need compression as well as splitting into multiple zip.

EDIT: Platform = Unix

0 Karma
1 Solution

linu1988
Champion

You can use a script which will break the zip file into pieces. If it more than the specified limit, send it in another mail. Don't use the default splunk script to send mail.

_http://docs.splunk.com/Documentation/Splunk/6.1.1/alert/ConfiguringScriptedAlerts

Use the 8th argument to get the csv file then perform the zip and mailing.

Thanks,
L

View solution in original post

grijhwani
Motivator

Linux provides zip, gzip, bzip2 for compression, and split to (wait for it) split files into smaller files... Then there are a multiplicity of ways of despatching the results as e-mails.

Writing a shell script to compress the file, then chunk it into multiple fixed size attachments should be almost trivial. If you are concerned about portability, then the better alternative is to split the source file at a given threshold and using zip specifically (for Windows compatibility) on the chunks individually.

There you go. Off the top of my head, and not debugged or checked, and dependent on installing Mutt as a mail client (because of its superior command line attachment management).

#!/bin/bash
sMailTarget=you@your.domain
fSource=${SPLUNK_ARG_8}
[ -f ${fSource} ] || { echo "No file"; exit 1; }
# Comment the next line out of you want to send empty results
[ -s ${fSource} ] || { echo "Empty file"; exit 1; }
fPrefix="`echo ${fSource} | sed -e s'/.csv//'`-seg"

split -a 2 -l50000 ${fSource} ${fPrefix}
for ${fInter} in ${fPrefix}*; do
  mv ${fInter} ${fInter}.csv
  zip -9 ${fInter}.zip ${fInter}.csv
  mutt -s "${SPLUNK_ARG_4} result segment ${fInter}" -a ${fInter}.zip ${sMailTarget} << EoM
File attached
EoM
done
0 Karma

linu1988
Champion

You can use a script which will break the zip file into pieces. If it more than the specified limit, send it in another mail. Don't use the default splunk script to send mail.

_http://docs.splunk.com/Documentation/Splunk/6.1.1/alert/ConfiguringScriptedAlerts

Use the 8th argument to get the csv file then perform the zip and mailing.

Thanks,
L

amitkr0201
Explorer

Sounds promising I'll try it out. Thanks

0 Karma

grijhwani
Motivator

On what platform? You don't specify Windows/Unix.

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...