topic Splunk - CSV data indexing again and again + Splunk is indexing field names as new event in Getting Data In

Splunk - CSV data indexing again and again + Splunk is indexing field names as new event

shahbhavin19 — Thu, 06 Aug 2020 19:28:57 GMT

Hi Everyone,
Below is my CSV fields and some values and I am doing continuous monitoring CSV file:

TIMESTAMP, NAME, AGE, PHONE_NO, ZIP

07/08/2020 12:00:00 PM, ABC, 20, XYZ, 123

07/07/2020 12:00:00 PM, XYZ, 18, XYZ, 456

1. Splunk stores as 3 event, as Splunk is also considering field names as a event.. which I do not want to index fieldname as a event.
I have tried several Splunk Answers but no luck or might be I am doing in a wrong way.
Please suggest how to fix this.

TIMESTAMP, NAME, AGE, PHONE_NO, ZIP

07/08/2020 12:00:00 PM, ABC, 20, XYZ, 123

07/08/2020 12:00:00 PM, PQR, 19, XYZ, 456
I have changed in 2nd row for NAME & AGE and modified Time so that Splunk can pick that latest time and display latest data on dashboard..
So problem is everytime saving excel, Splunk indexing all the data inside the excel including field name..
I do not want to index field names as a event and Splunk index only data for new entries or for those entries which I have make the changes to avoid duplicate data indexing again and again.

It would be good if anyone can help me out to fix this issue. Thanks!

Re: Splunk - CSV data indexing again and again + Splunk is indexing field names as new event

thambisetty — Thu, 06 Aug 2020 19:37:30 GMT

First convert excel to csv.

set sourcetype to csv if you are using monitor stanza in inputs.conf

select sourcetype as csv if you are adding data from Splunk web.

Re: Splunk - CSV data indexing again and again + Splunk is indexing field names as new event

shahbhavin19 — Fri, 07 Aug 2020 09:24:05 GMT

@thambisettyI have already converted excel to CSV also I am using monitoring stanza in inputs.conf and set sourcetype to csv only.
So whenever I make changes to CSV, like adding new entry with new TIMESTAMP or modifiying existing entry with new TIMESTAMP and after saving Splunk indexing whole CSV data again and this causes multiple duplicate data issue + consuming more indexing space.

Is there any way to fix this? Thanks!

Re: Splunk - CSV data indexing again and again + Splunk is indexing field names as new event

thambisetty — Fri, 07 Aug 2020 09:29:28 GMT

INDEXED EXTRACTIONS

https://community.splunk.com/t5/Splunk-Search/How-to-skip-header-in-CSV-files-before-indexing/td-p/336207

Re: Splunk - CSV data indexing again and again + Splunk is indexing field names as new event

shahbhavin19 — Fri, 07 Aug 2020 09:32:24 GMT

@thambisetty Thanks, this fixes my 1st issue.. Is there any solution for 2nd issue?

Re: Splunk - CSV data indexing again and again + Splunk is indexing field names as new event

thambisetty — Fri, 07 Aug 2020 09:51:41 GMT

I have recently tried adding new events to text file while the file is being monitored. I found only new events being indexed.

If you modify existing records, I think the pointer which is used to keep track of till where file is read might be changing. This could be one of the reason.

Splunk doesn’t recommend monitor stanza if you are keep changing file content.

you can upload once.