topic Re: Why is field ingestion not working when ingesting custom CSV data? in Getting Data In

Why is field ingestion not working when ingesting custom CSV data?

grantccarlson — Sat, 20 Apr 2019 18:13:54 GMT

Hello,

I am trying to ingest some custom CSV data that I have created. In some of the data it extracts the correct fields, but then also shows duplicate fields as "field 1", "field 2", etc.

Any ideas how to have the data ingest correctly? Attached is a screenshot.

I have tried creating field aliases and rules when ingesting but I am not having any luck.

Thank you in advance for any help!

Re: Why is field ingestion not working when ingesting custom CSV data?

richgalloway — Sun, 21 Apr 2019 03:10:44 GMT

What are the props.conf and transforms.conf settings for the sourcetype?

Re: Why is field ingestion not working when ingesting custom CSV data?

grantccarlson — Sun, 21 Apr 2019 03:32:45 GMT

I'm not sure. How do you check?

Re: Why is field ingestion not working when ingesting custom CSV data?

richgalloway — Sun, 21 Apr 2019 12:44:55 GMT

You read the .conf files or use btool. Since you asked, however, I'll assume you did not change them.
How did you ingest the CSV file? Did you use the Add Data wizard?

Re: Why is field ingestion not working when ingesting custom CSV data?

woodcock — Sun, 21 Apr 2019 15:03:48 GMT

Your screenshot and your description do not match. There are no duplicated fields shown.

Re: Why is field ingestion not working when ingesting custom CSV data?

grantccarlson — Sun, 21 Apr 2019 15:14:39 GMT

"Field 6" is the same as "dport"
"field 4" is the same as "daddr"
"event_code" is the same as "saddr"

All of the above are the same field values with 2 different names...Sorry! Duplicate fields is probably not the best wording.

Re: Why is field ingestion not working when ingesting custom CSV data?

woodcock — Sun, 21 Apr 2019 15:41:40 GMT

What is the first line of the csv file and how are you getting it in? Are you using INDEXED_EXTRACTIONS=csv?

Re: Why is field ingestion not working when ingesting custom CSV data?

grantccarlson — Sun, 21 Apr 2019 16:37:07 GMT

the first line of the csv file is the CIM field names that correspond to the data. I don't think that I am using indexed_extractions=csv.

Re: Why is field ingestion not working when ingesting custom CSV data?

niketn — Mon, 22 Apr 2019 03:38:40 GMT

@grantccarlson you would need to share your props.conf and transforms.conf applicable for this sourcetype. Would it be possible for you to do so?

Also if possible share sample data file with CSV Header row and at-least one data row. Kindly mock/anonymize any sensitive information before sharing data/code.

Re: Why is field ingestion not working when ingesting custom CSV data?

grantccarlson — Mon, 22 Apr 2019 14:55:46 GMT

Hello, I'm not sure where to find the props.conf and transforms.conf files. I will attach a screenshot to show the data file.

Re: Why is field ingestion not working when ingesting custom CSV data?

grantccarlson — Mon, 22 Apr 2019 14:57:32 GMT

"Timestamp proto saddr daddr sport dport spkts dpkts sbytes dbytes stcpb State
1473729384 tcp 94.255.165.163 140.226.22.228 23237 23 1 0 64 0 2363627212 REQ
1473729384 tcp 36.69.63.66 164.47.8.18 43092 2323 1 0 64 0 2754545844 REQ
1473729384 tcp 41.174.140.215 161.98.74.245 48716 2323 1 0 64 0 2707573431 REQ

Re: Why is field ingestion not working when ingesting custom CSV data?

behudelson — Mon, 22 Apr 2019 16:18:36 GMT

It looks like you set up a field extraction without editing the field names and then saved it.

To remove them, you will need to remove the field extraction. Go to Settings->Fields->Field Extractions. Make sure you have the right app selected. Filter by the name of your sourcetype. If you don't know that, sort by owner and look for your username. Delete the entry that you did not mean to create.

If you see multiple entries and are unsure which one is yours go to Settings->Fields->Field Transoformations. Sort by owner and click into the ones owned by your username. Look for the one that has a field list similar to the one below:

Take note of the name on that screen at the upper left hand corner. Go back to Settings->Fields->Field Extractions. Find the extraction that has your name as owner with that transformation name in its name. Delete it.