Getting Data In

How to index events (CSV file from universal forwarder) based on the time field?

pgadhari
Builder

I am trying to index a CSV file from UF, which contains some historical data. Below is the sample of the events. Somehow the events are not getting indexed based on the timestamp from the CSV file. Instead, they are getting indexed with current time (all the events having current timestamp) and not the timestamp from the time field in the CSV.

How can I fix this issue? I want to index the events based on the time field that goes to Jan to March 2020. Please help to resolve this? I have attached the screenshot also for your reference.

Time,Mbps_IN,Mbps_OUT
01/01/2020 0:00,17222030,874306
02/01/2020 0:00,19368200,1504505
03/01/2020 0:00,15194740,150084
04/01/2020 0:00,4768362,1790559
05/01/2020 0:00,57691290,6339732
06/01/2020 0:00,44419200,2114772
07/01/2020 0:00,16432560,1144577
08/01/2020 0:00,9053104,23321280
09/01/2020 0:00,16265580,12490060
10/01/2020 0:00,2274004,4886436
11/01/2020 0:00,28840920,1388473
12/01/2020 0:00,6569902,6743890
13/01/2020 0:00,9766315,31771390
14/01/2020 0:00,8418418,2619432
15/01/2020 0:00,8751632,4382776
16/01/2020 0:00,22305280,8519139
17/01/2020 0:00,2989921,157784
18/01/2020 0:00,5307088,225203
19/01/2020 0:00,21432030,22773270
20/01/2020 0:00,29338980,2971322
21/01/2020 0:00,9230931,2120051
22/01/2020 0:00,7299774,10691780
23/01/2020 0:00,50019440,6489089
24/01/2020 0:00,3431143,241807
25/01/2020 0:00,5989488,830827
26/01/2020 0:00,77886710,7772389
27/01/2020 0:00,6841259,23842100
28/01/2020 0:00,79912540,26599700
29/01/2020 0:00,50530910,5565867
30/01/2020 0:00,21047160,6741192
31/01/2020 0:00,10868270,784867
01/02/2020 0:00,7047898,1671952
02/02/2020 0:00,67265450,8155953
03/02/2020 0:00,36689240,5077973   

My inputs.conf:

[monitor:///home/splunk/bw_history/exec_hist.csv]
sourcetype = exec
index = testindex
crcSalt = <SOURCE>

My props.conf:

[exec]
DATETIME_CONFIG = NONE
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
disabled = false
pulldown_type = true
TIMESTAMP_FIELDS = Time

alt text

Labels (4)
0 Karma

pgadhari
Builder

Also I have tried these settings from heavy forwarder also, but same results. The events are taking current time as index time and not the Time field from CSV?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

It probably won't solve the problem, but there's an additional setting you should add to the props.conf stanza. At the very least, it will prevent Splunk from interpreting "03/02/2020" as 2 March 2020.

TIME_FORMAT = %d/%m/%Y %H:%M
---
If this reply helps you, Karma would be appreciated.
0 Karma

pgadhari
Builder

@richgalloway - tried putting that settings in props.conf, but still it is indexing as current time and not the Time field from CSV. Can you suggest any other alternative ?

[exec]
DATETIME_CONFIG = NONE
INDEXED_EXTRACTIONS = csv
KV_MODE = none
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
category = Structured
disabled = false
pulldown_type = true
TIMESTAMP_FIELDS = Time
TIME_FORMAT = %d/%m/%Y %H:%M
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Remove the DATETIME_CONFIG = NONE line.
If that doesn't help, try removing INDEXED_EXTRACTIONS on the indexer/HF and setting KV_MODE = json on the SH.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...