topic Re: unarchive_cmd doesn't work for CSV file in Getting Data In

unarchive_cmd doesn't work for CSV file

dmillis — Mon, 28 Sep 2020 18:24:38 GMT

In 6.2.0, I have written a pre-processor script for a particular CSV log format, to produce a useable timestamp. The script is designed to be used as a 'unarchive_cmd' script for the input. I.e., from the CLI, it functions like this:
cat | convert.pl

The problem: no matter how much I play with inputs.conf and props.conf, the unarchive_cmd is ignored. Here is inputs.conf:
[monitor:///var/cool_csv_logs/file.csv] disabled = 0 followTail = 0 host = some_host index = scratch sourcetype = cool_csv
and props.conf:
`[cool_csv]
SHOULD_LINEMERGE = false
pulldown_type = true
INDEXED_EXTRACTIONS = csv
KV_MODE = node
category = Structured
invalid_cause = archive

[source::/var/cool_csv_logs/file.csv]
unarchive_cmd = /opt/splunk/bin/convert.pl`

Why is the unarchive_cmd not working?

Re: unarchive_cmd doesn't work for CSV file

dmillis — Tue, 09 Dec 2014 00:03:30 GMT

Well, David, it turns out that you cannot use a file suffix of .csv (or .txt) with unarchive_cmd.

Try changing your source filename to file.log

(This works!)

Re: unarchive_cmd doesn't work for CSV file

dmillis — Tue, 09 Dec 2014 00:36:17 GMT

Here's a more readable version:

Here is inputs.conf:

[monitor:///var/cool_csv_logs/file.csv] 
disabled = 0 
followTail = 0 
host = some_host 
index = scratch 
sourcetype = cool_csv

and props.conf:

[cool_csv] 
SHOULD_LINEMERGE = false 
pulldown_type = true
INDEXED_EXTRACTIONS = csv
KV_MODE = node
category = Structured
invalid_cause = archive

[source::/var/cool_csv_logs/file.csv]
unarchive_cmd = /opt/splunk/bin/convert.pl

... should be changed to file.log