Getting Data In
×

Are you a member of the Splunk Community?

Sign in or Register with your Splunk account to get your questions answered, access valuable resources and connect with experts!
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Ask a Question

Ingested time for csv file on windows

hahhhaxin
Loves-to-Learn Lots
‎11-17-2024 06:12 PM

background -

the designed windows log flow is Splunk Agent of Universal forwarder -> Splunk Heavy Forwarder-> Splunk Indexer. the path are monitored with inputs.conf in Universal forwarder like this
[monitor://D:\test\*.csv]

disabled=0

index=asr_info

sourcetype=csv

source=asr:report

crcSalt=<SOURCE>

the example content for one of the csv file is like below - 
cn,comment_id,asr_number,created_by,created_date

zhy,15,2024-10-12-1,cc,2024-10-28 18:10

bj,10,2024-09-12-1,cc,2024-09-12 13:55

 

for the 2 indexed rows, the field extractions are good except _time.  for the first row, _time is 10/12/24 6:10:00.000 PM, for the second row, _time is 9/12/24 1:55:00.000 PM

Question -
How to make _time be the real ingested time instead of guessing from the row content? (tried with DATETIME_CONFIG = CURRENT in both HF and index in props like -
[source::asr:report]

DATATIME_CONFIG = CURRENT

but, it does not work
) 

Labels (4)
0 Karma
Reply

richgalloway
SplunkTrust
SplunkTrust
‎11-18-2024 06:30 AM

I'm assuming DATATYPE_CONFIG=CURRENT is a typo and the real setting is DATETIME_CONFIG=CURRENT.

Try changing the props stanza name to [csv] (the sourcetype).

FWIW, Splunk recommends not changing the source value in inputs.conf.

 

---
If this reply helps you, Karma would be appreciated.
0 Karma
Reply

hahhhaxin
Loves-to-Learn Lots
‎11-19-2024 01:40 AM

Yes it is typo.

We do not use sourcetype because various types of csv files are ingested, to keep this change only effectively in the current asset, we locate with sourcetype

0 Karma
Reply

gcusello
SplunkTrust
SplunkTrust
‎11-17-2024 11:19 PM

Hi @hahhhaxin ,

try to put the props.conf also on the UF.

Ciao.

Giuseppe

0 Karma
Reply

hahhhaxin
Loves-to-Learn Lots
‎11-18-2024 12:05 AM

@gcusello here is the btool output

PS C:\Program Files\SplunkUniversalForwarder> .\bin\splunk.exe btool props list  --debug | Select-String -Pattern "etc\\apps"

 

C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\props.conf [(::)?...]

C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\props.conf CHECK_FOR_HEADER = false

C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\props.conf priority = 10001

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    [20240821_131904]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    MAX_TIMESTAMP_LOOKAHEAD = 41

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    [WindowsUpdate]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    MAX_TIMESTAMP_LOOKAHEAD = 48

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    [WindowsUpdate-2]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    MAX_TIMESTAMP_LOOKAHEAD = 48

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    [WindowsUpdate-3]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    MAX_TIMESTAMP_LOOKAHEAD = 48

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    [WindowsUpdate-4]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    MAX_TIMESTAMP_LOOKAHEAD = 48

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    [WindowsUpdate-5]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    MAX_TIMESTAMP_LOOKAHEAD = 48

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    [WindowsUpdate-6]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    MAX_TIMESTAMP_LOOKAHEAD = 48

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    [WindowsUpdate-7]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    MAX_TIMESTAMP_LOOKAHEAD = 48

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    [WindowsUpdate-8]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    MAX_TIMESTAMP_LOOKAHEAD = 48

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    [WindowsUpdate-9]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    MAX_TIMESTAMP_LOOKAHEAD = 48

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    [first_install-too_small]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    PREFIX_SOURCETYPE = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    maxDist = 9999

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  [log2metrics_csv]

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  DATETIME_CONFIG =

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  INDEXED_EXTRACTIONS = csv

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  LINE_BREAKER = ([\r\n]+)

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  METRIC-SCHEMA-TRANSFORMS = metric-schema:log2metrics_default_csv

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  NO_BINARY_CHECK = true

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  category = Log to Metrics

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  description = Comma-separated value format. Log-to-metrics processing converts the numeric values in csv

events into metric data points.

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  pulldown_type = 1

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  [log2metrics_json]

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  DATETIME_CONFIG =

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  INDEXED_EXTRACTIONS = json

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  LINE_BREAKER = ([\r\n]+)

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  METRIC-SCHEMA-TRANSFORMS = metric-schema:log2metrics_default_json

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  NO_BINARY_CHECK = true

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  category = Log to Metrics

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  description = JSON-formatted data. Log-to-metrics processing converts the numeric values in json keys into

metric data points.

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  pulldown_type = 1

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  [log2metrics_keyvalue]

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  DATETIME_CONFIG =

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  LINE_BREAKER = ([\r\n]+)

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  METRIC-SCHEMA-TRANSFORMS = metric-schema:log2metrics_default_keyvalue

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  NO_BINARY_CHECK = true

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  TRANSFORMS-EXTRACT = metrics_field_extraction

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  category = Log to Metrics

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  description = '<key>=<value>' formatted data. Log-to-metrics processing converts the keys with numeric values

into metric data points.

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  pulldown_type = 1

C:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\props.conf                   [scheduler]

C:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\props.conf                   EXTRACT-fields = (?i)^(?:[^ ]* ){2}(?:[+\-]\d+ )?(?P<log_level>[^ ]*)\s+(?P<component>[^ ]+) -

(?P<event_message>.+)

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  [source::...\\var\\log\\introspection\\disk_objects.log(.\d+)?]

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  TRANSFORMS-diskobjectsclone = introspection_disk_objects_log_clone

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  [source::...\\var\\log\\introspection\\resource_usage.log(.\d+)?]

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  TRANSFORMS-resourceusageclone = introspection_resource_usage_log_clone

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  [source::...\\var\\log\\splunk\\metrics.log(.\d+)?]

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  TRANSFORMS-metricslogclone = metrics_log_clone

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    [splunk-powershell.ps-2]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    MAX_TIMESTAMP_LOOKAHEAD = 49

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    [splunk-powershell.ps-too_small]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    PREFIX_SOURCETYPE = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf                    maxDist = 9999

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  [splunk_intro_disk_objects]

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  METRIC-SCHEMA-TRANSFORMS = metric-schema:introspection_disk_objects

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  TRANSFORMS-blah = metrics_index_redirect,introspection_disk_objects_metric_name

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  [splunk_intro_resource_usage]

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  METRIC-SCHEMA-TRANSFORMS = metric-schema:introspection_resource_usage

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  TRANSFORMS-bloo = metrics_index_redirect,introspection_resource_usage_metric_name

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  [splunk_metrics_log]

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  METRIC-SCHEMA-TRANSFORMS = metric-schema:metrics_dot_log

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf  TRANSFORMS-metricslog = metrics_index_redirect,metrics_field_extraction,metrics_log_metric_name

C:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\props.conf                   [splunk_web_service]

C:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\props.conf                   EXTRACT-useragent = userAgent=(?P<browser>[^ (]+)

C:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\props.conf                   [splunkd]

C:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\props.conf                   EXTRACT-fields = (?i)^(?:[^ ]* ){2}(?:[+\-]\d+ )?(?P<log_level>[^ ]*)\s+(?P<component>[^ ]+)

(?:\[(?P<thread_id>\d+)\s)?(?:(?P<thread_name>[^\]]+)\]\s)?- (?P<event_message>.+)

0 Karma
Reply

gcusello
SplunkTrust
SplunkTrust
‎11-18-2024 01:14 AM

Hi @hahhhaxin ,

it's really difficoult to red the output of the btool, have this configuration on the UF?

I don't see DATATIME_CONFIG = CURRENT in your output on the UF.

Ciao.

Giuseppe

0 Karma
Reply

hahhhaxin
Loves-to-Learn Lots
‎11-18-2024 03:07 AM

No, it is on the HF and indexer, UF here is only targeted for getting data in.

the configuration in HF&indexer is -

[source::asr:report]

DATATIME_CONFIG = CURRENT
0 Karma
Reply

gcusello
SplunkTrust
SplunkTrust
‎11-18-2024 05:59 AM

Hi @hahhhaxin ,

for csv files, put the props.conf and transforms.conf  also on UF.

Ciao.

Giuseppe

0 Karma
Reply

hahhhaxin
Loves-to-Learn Lots
‎11-19-2024 01:45 AM

Hi @gcusello 

except for default props and trans, we do not add any another stanza in local UF props and transforms 

0 Karma
Reply

gcusello
SplunkTrust
SplunkTrust
‎11-19-2024 01:56 AM

Hi @hahhhaxin ,

for csv files you have to add also to UFs.

Ciao.

Giuseppe

0 Karma
Reply
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Take the Survey

Can’t make it to .conf25? Join us online!

Watch Global Broadcast
Get Updates on the Splunk Community!

Community Content Calendar, September edition

Welcome to another insightful post from our Community Content Calendar! We're thrilled to continue bringing ...

Splunkbase Unveils New App Listing Management Public Preview

Splunkbase Unveils New App Listing Management Public PreviewWe're thrilled to announce the public preview of ...

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Are you leveraging automation to its fullest potential in your threat detection strategy?Our upcoming Security ...