Knowledge Management

Why did the field name change while indexing csv file in splunk?

sivaranjiniG
Path Finder

I am trying to index a csv file by uploading it through splunk web.... while setting up sourcetype i could see all my field names become changed

For example:

field.project become field_project

Can somebody help me in this?

Labels (2)
0 Karma
1 Solution

damien_chillet
Builder

By default, Splunk replaces non alphanumeric characters in field names with underscore:

From http://docs.splunk.com/Documentation/Splunk/7.0.0/Knowledge/Configureadvancedextractionswithfieldtra...

 CLEAN_KEYS

Controls whether the system strips leading underscores and 0-9 characters from the field names it extracts. Key cleaning is the practice of replacing any non-alphanumeric characters in field names with underscores, as well as the removal of leading underscores and 0-9 characters from field names.

You could try set it to false for your sourcetype transforms.

View solution in original post

damien_chillet
Builder

By default, Splunk replaces non alphanumeric characters in field names with underscore:

From http://docs.splunk.com/Documentation/Splunk/7.0.0/Knowledge/Configureadvancedextractionswithfieldtra...

 CLEAN_KEYS

Controls whether the system strips leading underscores and 0-9 characters from the field names it extracts. Key cleaning is the practice of replacing any non-alphanumeric characters in field names with underscores, as well as the removal of leading underscores and 0-9 characters from field names.

You could try set it to false for your sourcetype transforms.

sivaranjiniG
Path Finder

Sorry i m totally new to this....
Can you help me in creating this transforms.conf file. not sure what should be the stanza name

0 Karma

damien_chillet
Builder

Here is the test csv I am using:

field.name,field.value
field1,splunk
field2,answers

If you ingest the file using the sourcetype detected by Splunk (that will be csv here), it will use indexed extractions and replace "field.attr" with "field_attr". It is not possible to prevent Splunk from doing that at indexing time (as far as i know).

So you need to turn index extractions off. I used the following props.conf / transforms.conf:

props.conf

[test_abc]
SHOULD_LINEMERGE = false
disabled = false
INDEXED_EXTRACTIONS=
REPORT-no_clean_keys = test_abc_transforms
TRANSFORMS-drop_header = drop_header

transforms.conf

[test_abc_transforms]
CLEAN_KEYS = false
DELIMS=,
FIELDS=field.name,field.value

[drop_header]
REGEX = field\.name,field\.value
DEST_KEY = queue
FORMAT = nullQueue

test_abc_transforms extracts fields from your csv and tell Splunk not to clean the keys (replace "." with "_" in this case).

drop_header transform is to tell Splunk not to index the header of your CSV file.

Restart Splunk then ingest your file using Splunk CLI:

$SPLUNK_HOME/bin/splunk add oneshot path/to/your/csv/abc.csv -sourcetype test_abc -index main

It worked for me, let me know if you need further help.
Also i welcome anyone reading this solution to suggest any ameliorations, there may be better ways to achieve that 🙂

sivaranjiniG
Path Finder

Thanks So much its working 🙂

robertlynch2020
Motivator

Hi

If you have some fields with and without the ., below is an example of how to get that to work.

However it only works going into an event index, it does not seem to work going into metrices.

[test_abc_transforms]
CLEAN_KEYS = false
DELIMS=,
FIELDS=degraded.threshold,down.threshold

[drop_header]
REGEX = metric_timestamp,metric_name,_value,degraded\.threshold,down\.threshold
DEST_KEY = queue
FORMAT = nullQueue

 

 

metric_timestamp,metric_name,_value,degraded.threshold,down.threshold
1695201472,mx.process.cpu.utilization,1.373348018,30,300
1695201472,mx.process.cpu.utilization,1.373348018,30,300
1695201472,mx.process.cpu.utilization,1.373348018,30,300
1695201472,mx.process.cpu.utilization,1.373348018,30,300
1695201472,mx.process.cpu.utilization,1.373348018,30,300
1695201472,mx.process.cpu.utilization,1.373348018,30,300
1695201472,mx.process.cpu.utilization,1.373348018,30,300
1695201472,mx.process.cpu.utilization,1.373348018,30,300
1695201472,mx.process.cpu.utilization,1.373348018,30,300
1695201472,mx.process.cpu.utilization,1.373348018,30,300

 

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...