CSV File with 'timestamp' field - Splunk adds 'none' value

[krishnakesiraju]

Hi,

I am trying to ingest a CSV file using a Python script (getting it from an S3 bucket) from HF. The CSV file has a field called 'timestamp' (without the quotes). This is the timestamp when the resource snapshot was taken. The value in this filed is most of the times unique - a timestamp of %Y-%m-%dT%H:%M:%S.%6N format. It does not have any other value.

When I ingest the file using the script or manually, I notice that Splunk is appending 'none' to the timestamp field. If I change the column header value to anything other than 'timestamp' (for ex., ts), there is no problem. Unfortunately, i do not have enough points to attach files. Below is the configuration I'm using, please let me know if I'm doing anything wrong.

Splunk Enterprise version # 7.2.0

props.conf

DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = CSV
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true

Why does Splunk duplicate the ingestion - once with actual timestamp value and once with timestamp as none.

I tried to change the column header (before index time) using transforms.conf but with no luck using the below config, is there something i'm missing?

props.conf

DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = csv
KV_MODE = none
TRANSFORMS-rename-field = extract_csv
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true

transforms.conf
[extract_csv]
DELIMS = ","
FIELDS="field1_timestamp","field2","field3","field4",....

Thanks

[krishnakesiraju · Accepted solution]

The column header was changed to another value, which 'solved' the problem. Surprised Splunk makes it so difficult for manipulating fields

[richgalloway]

@krishnakesiraju If your problem is resolved, please accept an answer to help future readers.

[rupesh26]

Hi,

You are using DATETIME_CONFIG = CURRENT, that's why timestamp value is none.
Try removing it and add TIMESTAMP_FIELDS=timestampin your props.conf, this will extract the _time value from your CSV

Reference:
https://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileswithstructureddata

[krishnakesiraju]

Hi Rupesh,
Thanks for your reply, but i want the 'timestamp' field value to be different to _time when the event was indexed. That was the reason I chose DATETIME_CONFIG = CURRENT setting. I want to be able to differentiate between these 2 values - as each of these holds a significance. Let me know if my understanding is incorrect.

Thanks,