Getting Data In

How to compress CSV file and email it on Unix platform?

amitkr0201
Explorer

Is there an app/script/something else available which compresses a csv (preferably output of outputcsv command ) and emails it? Bonus point if its able to split it into multiple zips if size exceeds threshold.

I am working on a set of data which, when extracted as CSV, sizes to approx 150 MB, which when compressed is >10 MB.

The email provider I use allows max 8 MB attachments, so I need compression as well as splitting into multiple zip.

EDIT: Platform = Unix

0 Karma
1 Solution

linu1988
Champion

You can use a script which will break the zip file into pieces. If it more than the specified limit, send it in another mail. Don't use the default splunk script to send mail.

_http://docs.splunk.com/Documentation/Splunk/6.1.1/alert/ConfiguringScriptedAlerts

Use the 8th argument to get the csv file then perform the zip and mailing.

Thanks,
L

View solution in original post

grijhwani
Motivator

Linux provides zip, gzip, bzip2 for compression, and split to (wait for it) split files into smaller files... Then there are a multiplicity of ways of despatching the results as e-mails.

Writing a shell script to compress the file, then chunk it into multiple fixed size attachments should be almost trivial. If you are concerned about portability, then the better alternative is to split the source file at a given threshold and using zip specifically (for Windows compatibility) on the chunks individually.

There you go. Off the top of my head, and not debugged or checked, and dependent on installing Mutt as a mail client (because of its superior command line attachment management).

#!/bin/bash
sMailTarget=you@your.domain
fSource=${SPLUNK_ARG_8}
[ -f ${fSource} ] || { echo "No file"; exit 1; }
# Comment the next line out of you want to send empty results
[ -s ${fSource} ] || { echo "Empty file"; exit 1; }
fPrefix="`echo ${fSource} | sed -e s'/.csv//'`-seg"

split -a 2 -l50000 ${fSource} ${fPrefix}
for ${fInter} in ${fPrefix}*; do
  mv ${fInter} ${fInter}.csv
  zip -9 ${fInter}.zip ${fInter}.csv
  mutt -s "${SPLUNK_ARG_4} result segment ${fInter}" -a ${fInter}.zip ${sMailTarget} << EoM
File attached
EoM
done
0 Karma

linu1988
Champion

You can use a script which will break the zip file into pieces. If it more than the specified limit, send it in another mail. Don't use the default splunk script to send mail.

_http://docs.splunk.com/Documentation/Splunk/6.1.1/alert/ConfiguringScriptedAlerts

Use the 8th argument to get the csv file then perform the zip and mailing.

Thanks,
L

amitkr0201
Explorer

Sounds promising I'll try it out. Thanks

0 Karma

grijhwani
Motivator

On what platform? You don't specify Windows/Unix.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...