Getting Data In

Why is field ingestion not working when ingesting custom CSV data?

grantccarlson
New Member

Hello,

I am trying to ingest some custom CSV data that I have created. In some of the data it extracts the correct fields, but then also shows duplicate fields as "field 1", "field 2", etc.

Any ideas how to have the data ingest correctly? Attached is a screenshot.alt text

I have tried creating field aliases and rules when ingesting but I am not having any luck.

Thank you in advance for any help!

0 Karma

behudelson
Path Finder

It looks like you set up a field extraction without editing the field names and then saved it.

To remove them, you will need to remove the field extraction. Go to Settings->Fields->Field Extractions. Make sure you have the right app selected. Filter by the name of your sourcetype. If you don't know that, sort by owner and look for your username. Delete the entry that you did not mean to create.

If you see multiple entries and are unsure which one is yours go to Settings->Fields->Field Transoformations. Sort by owner and click into the ones owned by your username. Look for the one that has a field list similar to the one below:
alt text
Take note of the name on that screen at the upper left hand corner. Go back to Settings->Fields->Field Extractions. Find the extraction that has your name as owner with that transformation name in its name. Delete it.

niketn
Legend

@grantccarlson you would need to share your props.conf and transforms.conf applicable for this sourcetype. Would it be possible for you to do so?

Also if possible share sample data file with CSV Header row and at-least one data row. Kindly mock/anonymize any sensitive information before sharing data/code.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

grantccarlson
New Member

Hello, I'm not sure where to find the props.conf and transforms.conf files. I will attach a screenshot to show the data file.

0 Karma

grantccarlson
New Member

"Timestamp proto saddr daddr sport dport spkts dpkts sbytes dbytes stcpb State
1473729384 tcp 94.255.165.163 140.226.22.228 23237 23 1 0 64 0 2363627212 REQ
1473729384 tcp 36.69.63.66 164.47.8.18 43092 2323 1 0 64 0 2754545844 REQ
1473729384 tcp 41.174.140.215 161.98.74.245 48716 2323 1 0 64 0 2707573431 REQ

0 Karma

woodcock
Esteemed Legend

Your screenshot and your description do not match. There are no duplicated fields shown.

0 Karma

grantccarlson
New Member

"Field 6" is the same as "dport"
"field 4" is the same as "daddr"
"event_code" is the same as "saddr"

All of the above are the same field values with 2 different names...Sorry! Duplicate fields is probably not the best wording.

0 Karma

woodcock
Esteemed Legend

What is the first line of the csv file and how are you getting it in? Are you using INDEXED_EXTRACTIONS=csv?

0 Karma

grantccarlson
New Member

the first line of the csv file is the CIM field names that correspond to the data. I don't think that I am using indexed_extractions=csv.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

What are the props.conf and transforms.conf settings for the sourcetype?

---
If this reply helps you, Karma would be appreciated.
0 Karma

grantccarlson
New Member

I'm not sure. How do you check?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

You read the .conf files or use btool. Since you asked, however, I'll assume you did not change them.
How did you ingest the CSV file? Did you use the Add Data wizard?

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...