topic Re: Field name changed while indexing csv file in splunk in Knowledge Management

Why did the field name change while indexing csv file in splunk?

sivaranjiniG — Fri, 22 Sep 2023 17:52:36 GMT

I am trying to index a csv file by uploading it through splunk web.... while setting up sourcetype i could see all my field names become changed

For example:

field.project become field_project

Can somebody help me in this?

Re: Field name changed while indexing csv file in splunk

damien_chillet — Tue, 12 Dec 2017 13:01:32 GMT

By default, Splunk replaces non alphanumeric characters in field names with underscore:

From http://docs.splunk.com/Documentation/Splunk/7.0.0/Knowledge/Configureadvancedextractionswithfieldtransforms

 CLEAN_KEYS

Controls whether the system strips leading underscores and 0-9 characters from the field names it extracts. Key cleaning is the practice of replacing any non-alphanumeric characters in field names with underscores, as well as the removal of leading underscores and 0-9 characters from field names.

You could try set it to false for your sourcetype transforms.

Re: Field name changed while indexing csv file in splunk

sivaranjiniG — Tue, 12 Dec 2017 14:20:53 GMT

Sorry i m totally new to this....
Can you help me in creating this transforms.conf file. not sure what should be the stanza name

Re: Field name changed while indexing csv file in splunk

damien_chillet — Tue, 29 Sep 2020 17:16:26 GMT

Here is the test csv I am using:

field.name,field.value
field1,splunk
field2,answers

If you ingest the file using the sourcetype detected by Splunk (that will be csv here), it will use indexed extractions and replace "field.attr" with "field_attr". It is not possible to prevent Splunk from doing that at indexing time (as far as i know).

So you need to turn index extractions off. I used the following props.conf / transforms.conf:

props.conf

[test_abc]
SHOULD_LINEMERGE = false
disabled = false
INDEXED_EXTRACTIONS=
REPORT-no_clean_keys = test_abc_transforms
TRANSFORMS-drop_header = drop_header

transforms.conf

[test_abc_transforms]
CLEAN_KEYS = false
DELIMS=,
FIELDS=field.name,field.value

[drop_header]
REGEX = field\.name,field\.value
DEST_KEY = queue
FORMAT = nullQueue

test_abc_transforms extracts fields from your csv and tell Splunk not to clean the keys (replace "." with "_" in this case).

drop_header transform is to tell Splunk not to index the header of your CSV file.

Restart Splunk then ingest your file using Splunk CLI:

$SPLUNK_HOME/bin/splunk add oneshot path/to/your/csv/abc.csv -sourcetype test_abc -index main

It worked for me, let me know if you need further help.
Also i welcome anyone reading this solution to suggest any ameliorations, there may be better ways to achieve that 🙂

Re: Field name changed while indexing csv file in splunk

sivaranjiniG — Mon, 18 Dec 2017 11:47:57 GMT

Thanks So much its working 🙂

Re: Field name changed while indexing csv file in splunk

robertlynch2020 — Fri, 22 Sep 2023 11:34:53 GMT

If you have some fields with and without the ., below is an example of how to get that to work.

However it only works going into an event index, it does not seem to work going into metrices.

[test_abc_transforms] CLEAN_KEYS = false DELIMS=, FIELDS=degraded.threshold,down.threshold [drop_header] REGEX = metric_timestamp,metric_name,_value,degraded\.threshold,down\.threshold DEST_KEY = queue FORMAT = nullQueue

metric_timestamp,metric_name,_value,degraded.threshold,down.threshold 1695201472,mx.process.cpu.utilization,1.373348018,30,300 1695201472,mx.process.cpu.utilization,1.373348018,30,300 1695201472,mx.process.cpu.utilization,1.373348018,30,300 1695201472,mx.process.cpu.utilization,1.373348018,30,300 1695201472,mx.process.cpu.utilization,1.373348018,30,300 1695201472,mx.process.cpu.utilization,1.373348018,30,300 1695201472,mx.process.cpu.utilization,1.373348018,30,300 1695201472,mx.process.cpu.utilization,1.373348018,30,300 1695201472,mx.process.cpu.utilization,1.373348018,30,300 1695201472,mx.process.cpu.utilization,1.373348018,30,300