Knowledge Management

Why did the field name change while indexing csv file in splunk?

sivaranjiniG
Path Finder

I am trying to index a csv file by uploading it through splunk web.... while setting up sourcetype i could see all my field names become changed

For example:

field.project become field_project

Can somebody help me in this?

Labels (2)
0 Karma
1 Solution

damien_chillet
Builder

By default, Splunk replaces non alphanumeric characters in field names with underscore:

From http://docs.splunk.com/Documentation/Splunk/7.0.0/Knowledge/Configureadvancedextractionswithfieldtra...

 CLEAN_KEYS

Controls whether the system strips leading underscores and 0-9 characters from the field names it extracts. Key cleaning is the practice of replacing any non-alphanumeric characters in field names with underscores, as well as the removal of leading underscores and 0-9 characters from field names.

You could try set it to false for your sourcetype transforms.

View solution in original post

damien_chillet
Builder

By default, Splunk replaces non alphanumeric characters in field names with underscore:

From http://docs.splunk.com/Documentation/Splunk/7.0.0/Knowledge/Configureadvancedextractionswithfieldtra...

 CLEAN_KEYS

Controls whether the system strips leading underscores and 0-9 characters from the field names it extracts. Key cleaning is the practice of replacing any non-alphanumeric characters in field names with underscores, as well as the removal of leading underscores and 0-9 characters from field names.

You could try set it to false for your sourcetype transforms.

sivaranjiniG
Path Finder

Sorry i m totally new to this....
Can you help me in creating this transforms.conf file. not sure what should be the stanza name

0 Karma

damien_chillet
Builder

Here is the test csv I am using:

field.name,field.value
field1,splunk
field2,answers

If you ingest the file using the sourcetype detected by Splunk (that will be csv here), it will use indexed extractions and replace "field.attr" with "field_attr". It is not possible to prevent Splunk from doing that at indexing time (as far as i know).

So you need to turn index extractions off. I used the following props.conf / transforms.conf:

props.conf

[test_abc]
SHOULD_LINEMERGE = false
disabled = false
INDEXED_EXTRACTIONS=
REPORT-no_clean_keys = test_abc_transforms
TRANSFORMS-drop_header = drop_header

transforms.conf

[test_abc_transforms]
CLEAN_KEYS = false
DELIMS=,
FIELDS=field.name,field.value

[drop_header]
REGEX = field\.name,field\.value
DEST_KEY = queue
FORMAT = nullQueue

test_abc_transforms extracts fields from your csv and tell Splunk not to clean the keys (replace "." with "_" in this case).

drop_header transform is to tell Splunk not to index the header of your CSV file.

Restart Splunk then ingest your file using Splunk CLI:

$SPLUNK_HOME/bin/splunk add oneshot path/to/your/csv/abc.csv -sourcetype test_abc -index main

It worked for me, let me know if you need further help.
Also i welcome anyone reading this solution to suggest any ameliorations, there may be better ways to achieve that 🙂

sivaranjiniG
Path Finder

Thanks So much its working 🙂

robertlynch2020
Motivator

Hi

If you have some fields with and without the ., below is an example of how to get that to work.

However it only works going into an event index, it does not seem to work going into metrices.

[test_abc_transforms]
CLEAN_KEYS = false
DELIMS=,
FIELDS=degraded.threshold,down.threshold

[drop_header]
REGEX = metric_timestamp,metric_name,_value,degraded\.threshold,down\.threshold
DEST_KEY = queue
FORMAT = nullQueue

 

 

metric_timestamp,metric_name,_value,degraded.threshold,down.threshold
1695201472,mx.process.cpu.utilization,1.373348018,30,300
1695201472,mx.process.cpu.utilization,1.373348018,30,300
1695201472,mx.process.cpu.utilization,1.373348018,30,300
1695201472,mx.process.cpu.utilization,1.373348018,30,300
1695201472,mx.process.cpu.utilization,1.373348018,30,300
1695201472,mx.process.cpu.utilization,1.373348018,30,300
1695201472,mx.process.cpu.utilization,1.373348018,30,300
1695201472,mx.process.cpu.utilization,1.373348018,30,300
1695201472,mx.process.cpu.utilization,1.373348018,30,300
1695201472,mx.process.cpu.utilization,1.373348018,30,300

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...