topic Re: CSV File with 'timestamp' field - Splunk adds 'none' value in Getting Data In

CSV File with 'timestamp' field - Splunk adds 'none' value

krishnakesiraju — Wed, 30 Sep 2020 02:18:28 GMT

Hi,

I am trying to ingest a CSV file using a Python script (getting it from an S3 bucket) from HF. The CSV file has a field called 'timestamp' (without the quotes). This is the timestamp when the resource snapshot was taken. The value in this filed is most of the times unique - a timestamp of %Y-%m-%dT%H:%M:%S.%6N format. It does not have any other value.

When I ingest the file using the script or manually, I notice that Splunk is appending 'none' to the timestamp field. If I change the column header value to anything other than 'timestamp' (for ex., ts), there is no problem. Unfortunately, i do not have enough points to attach files. Below is the configuration I'm using, please let me know if I'm doing anything wrong.

Splunk Enterprise version # 7.2.0

props.conf

DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = CSV
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true

Why does Splunk duplicate the ingestion - once with actual timestamp value and once with timestamp as none.

I tried to change the column header (before index time) using transforms.conf but with no luck using the below config, is there something i'm missing?

props.conf

DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = csv
KV_MODE = none
TRANSFORMS-rename-field = extract_csv
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true

transforms.conf
[extract_csv]
DELIMS = ","
FIELDS="field1_timestamp","field2","field3","field4",....

Thanks

Re: CSV File with 'timestamp' field - Splunk adds 'none' value

rupesh26 — Fri, 20 Sep 2019 04:47:55 GMT

Hi,

You are using DATETIME_CONFIG = CURRENT, that's why timestamp value is none.
Try removing it and add TIMESTAMP_FIELDS=timestampin your props.conf, this will extract the _time value from your CSV

Reference:
https://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileswithstructureddata

Re: CSV File with 'timestamp' field - Splunk adds 'none' value

krishnakesiraju — Wed, 30 Sep 2020 02:18:37 GMT

Hi Rupesh,
Thanks for your reply, but i want the 'timestamp' field value to be different to _time when the event was indexed. That was the reason I chose DATETIME_CONFIG = CURRENT setting. I want to be able to differentiate between these 2 values - as each of these holds a significance. Let me know if my understanding is incorrect.

Thanks,

Re: CSV File with 'timestamp' field - Splunk adds 'none' value

krishnakesiraju — Wed, 25 Sep 2019 04:30:22 GMT

The column header was changed to another value, which 'solved' the problem. Surprised Splunk makes it so difficult for manipulating fields

Re: CSV File with 'timestamp' field - Splunk adds 'none' value

richgalloway — Wed, 25 Sep 2019 11:01:19 GMT

@krishnakesiraju If your problem is resolved, please accept an answer to help future readers.