topic Re: CSV file ingestion not respecting column headings in Getting Data In

CSV file ingestion not respecting column headings

ericlarsen — Tue, 29 Sep 2020 13:57:18 GMT

I'm trying to monitor a CSV file (via a UF) with column headings included in the file. I want the column headings to be extracted at search time.

Sample file output:
"Name","DatabaseSize","UsedDatabaseSpace","AvailableNewMailboxSpace","NumMailboxes","TotalItemCount"
"SFG-DB01","306.9 GB (329,503,997,952 bytes)","257.1 GB (276,068,106,240 bytes)","49.77 GB (53,435,891,712 bytes)","223"
"SFG-DB02","350.4 GB (376,212,291,584 bytes)","300.7 GB (322,833,514,496 bytes)","49.71 GB (53,378,777,088 bytes)","362"
"SFG-DB03","308.6 GB (331,383,570,432 bytes)","236.1 GB (253,546,692,608 bytes)","72.49 GB (77,836,877,824 bytes)","151"

inputs.conf:
[monitor://E:\fileName*.csv]
index = test
sourcetype = mySourcetypeLog
ignoreOlderThan = 24h
crcSalt =

props.conf:
[mySourcetypeLog]
SHOULD_LINEMERGE = false
REPORT-getfields = mySourcetypeLog_fields

transforms.conf:
[mySourcetypeLog_fields]
DELIMS=","
FIELDS = "Name","DatabaseSize","UsedDatabaseSpace","AvailableNewMailboxSpace","NumMailboxes","TotalItemCount"

When I run a oneshot, the data is ingested correctly (one event per log record) but the extracted fields are not showing up.

Any help would be appreciated.
Thanks.

Re: CSV file ingestion not respecting column headings

adonio — Fri, 05 May 2017 01:18:42 GMT

why do you want the column heading extracted at search time?
any particular reason?
this doc: http://docs.splunk.com/Documentation/Splunk/6.5.3/Data/Extractfieldsfromfileswithstructureddata
explains in detail best practices indexing csv data with nice config samples and data samples to work with

Re: CSV file ingestion not respecting column headings

ericlarsen — Fri, 05 May 2017 11:11:20 GMT

I don't want to have the users to create extracted fields for every single field if the field names are already included in the csv file.

Re: CSV file ingestion not respecting column headings

adonio — Fri, 05 May 2017 12:32:48 GMT

when you will bring the data like mentioned in the docs, the users will not have to create fields at all.
pay attention that you have 6 fields in your example but values for only 5 of them.
in that case, per docs, splunk will not extract the field with no values.
also, some values are strings like: "DatabaseSize" "308.6 GB (331,383,570,432 bytes)" you will probably would want to extract numeric field based on these values, for example:
field name: DatabaseSizeGB value 308.6 there are multiple ways to do it.
submitting a full answer with screenshot here

Re: CSV file ingestion not respecting column headings

adonio — Fri, 05 May 2017 12:37:38 GMT

Will recommend follow docs on csv index here:
http://docs.splunk.com/Documentation/Splunk/6.5.3/Data/Extractfieldsfromfileswithstructureddata
inputs.conf: (like you already have)

[monitor://E:\fileName*.csv]
index = test
sourcetype = mySourcetypeLog
ignoreOlderThan = 24h
crcSalt =

props.conf (on indexer/s)

[mySourcetypeLog]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
CHARSET=AUTO
INDEXED_EXTRACTIONS=csv
KV_MODE=none
category=Structured
description=Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled=false
pulldown_type=true

screenshots:

you can see on the left hand side of the first screenshot the props.conf
on the second screenshot you can see all the fields extracted nicely from header
hope it helps

Re: CSV file ingestion not respecting column headings

ericlarsen — Fri, 05 May 2017 12:50:10 GMT

Ignore the sample file. It's just for illustrative purposes.

I was able to get it work by setting the sourcetype = csv in inputs.conf.

Re: CSV file ingestion not respecting column headings

adonio — Fri, 05 May 2017 12:59:42 GMT

great,
please mark question as answered and up vote any comments answers that you think helped with resolution
have a great weekend