Getting Data In

CSV File with 'timestamp' field - Splunk adds 'none' value

krishnakesiraju
Explorer

Hi,

I am trying to ingest a CSV file using a Python script (getting it from an S3 bucket) from HF. The CSV file has a field called 'timestamp' (without the quotes). This is the timestamp when the resource snapshot was taken. The value in this filed is most of the times unique - a timestamp of %Y-%m-%dT%H:%M:%S.%6N format. It does not have any other value.

When I ingest the file using the script or manually, I notice that Splunk is appending 'none' to the timestamp field. If I change the column header value to anything other than 'timestamp' (for ex., ts), there is no problem. Unfortunately, i do not have enough points to attach files. Below is the configuration I'm using, please let me know if I'm doing anything wrong.

Splunk Enterprise version # 7.2.0

props.conf

DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = CSV
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true

Why does Splunk duplicate the ingestion - once with actual timestamp value and once with timestamp as none.

I tried to change the column header (before index time) using transforms.conf but with no luck using the below config, is there something i'm missing?

props.conf

DATETIME_CONFIG = CURRENT
INDEXED_EXTRACTIONS = csv
KV_MODE = none
TRANSFORMS-rename-field = extract_csv
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
description = Comma-separated value format. Set header and other settings in "Delimited Settings"
disabled = false
pulldown_type = true

transforms.conf
[extract_csv]
DELIMS = ","
FIELDS="field1_timestamp","field2","field3","field4",....

Thanks

0 Karma
1 Solution

krishnakesiraju
Explorer

The column header was changed to another value, which 'solved' the problem. Surprised Splunk makes it so difficult for manipulating fields

View solution in original post

krishnakesiraju
Explorer

The column header was changed to another value, which 'solved' the problem. Surprised Splunk makes it so difficult for manipulating fields

richgalloway
SplunkTrust
SplunkTrust

@krishnakesiraju If your problem is resolved, please accept an answer to help future readers.

---
If this reply helps you, Karma would be appreciated.
0 Karma

rupesh26
Path Finder

Hi,

You are using DATETIME_CONFIG = CURRENT, that's why timestamp value is none.
Try removing it and add TIMESTAMP_FIELDS=timestampin your props.conf, this will extract the _time value from your CSV

Reference:
https://docs.splunk.com/Documentation/Splunk/latest/Data/Extractfieldsfromfileswithstructureddata

0 Karma

krishnakesiraju
Explorer

Hi Rupesh,
Thanks for your reply, but i want the 'timestamp' field value to be different to _time when the event was indexed. That was the reason I chose DATETIME_CONFIG = CURRENT setting. I want to be able to differentiate between these 2 values - as each of these holds a significance. Let me know if my understanding is incorrect.

Thanks,

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...