topic Re: Ingested time for csv file on windows in Getting Data In

Ingested time for csv file on windows

hahhhaxin — Mon, 18 Nov 2024 02:12:57 GMT

background -

the designed windows log flow is Splunk Agent of Universal forwarder -> Splunk Heavy Forwarder-> Splunk Indexer. the path are monitored with inputs.conf in Universal forwarder like this
[monitor://D:\test\*.csv]

disabled=0

index=asr_info

sourcetype=csv

source=asr:report

crcSalt=<SOURCE>

the example content for one of the csv file is like below -
cn,comment_id,asr_number,created_by,created_date

zhy,15,2024-10-12-1,cc,2024-10-28 18:10

bj,10,2024-09-12-1,cc,2024-09-12 13:55

for the 2 indexed rows, the field extractions are good except _time. for the first row, _time is 10/12/24 6:10:00.000 PM, for the second row, _time is 9/12/24 1:55:00.000 PM

Question -
How to make _time be the real ingested time instead of guessing from the row content? (tried with DATETIME_CONFIG = CURRENT in both HF and index in props like -
[source::asr:report]

DATATIME_CONFIG = CURRENT

but, it does not work
)

Re: Ingested time for csv file on windows

gcusello — Mon, 18 Nov 2024 07:19:11 GMT

Hi @hahhhaxin ,

try to put the props.conf also on the UF.

Ciao.

Giuseppe

Re: Ingested time for csv file on windows

hahhhaxin — Mon, 18 Nov 2024 08:05:25 GMT

@gcusello here is the btool output

PS C:\Program Files\SplunkUniversalForwarder> .\bin\splunk.exe btool props list --debug | Select-String -Pattern "etc\\apps"

C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\props.conf [(::)?...]

C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\props.conf CHECK_FOR_HEADER = false

C:\Program Files\SplunkUniversalForwarder\etc\apps\SplunkUniversalForwarder\default\props.conf priority = 10001

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf [20240821_131904]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf MAX_TIMESTAMP_LOOKAHEAD = 41

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf [WindowsUpdate]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf MAX_TIMESTAMP_LOOKAHEAD = 48

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf [WindowsUpdate-2]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf MAX_TIMESTAMP_LOOKAHEAD = 48

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf [WindowsUpdate-3]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf MAX_TIMESTAMP_LOOKAHEAD = 48

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf [WindowsUpdate-4]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf MAX_TIMESTAMP_LOOKAHEAD = 48

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf [WindowsUpdate-5]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf MAX_TIMESTAMP_LOOKAHEAD = 48

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf [WindowsUpdate-6]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf MAX_TIMESTAMP_LOOKAHEAD = 48

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf [WindowsUpdate-7]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf MAX_TIMESTAMP_LOOKAHEAD = 48

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf [WindowsUpdate-8]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf MAX_TIMESTAMP_LOOKAHEAD = 48

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf [WindowsUpdate-9]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf MAX_TIMESTAMP_LOOKAHEAD = 48

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf [first_install-too_small]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf PREFIX_SOURCETYPE = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf maxDist = 9999

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf [log2metrics_csv]

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf DATETIME_CONFIG =

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf INDEXED_EXTRACTIONS = csv

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf LINE_BREAKER = ([\r\n]+)

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf METRIC-SCHEMA-TRANSFORMS = metric-schema:log2metrics_default_csv

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf NO_BINARY_CHECK = true

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf category = Log to Metrics

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf description = Comma-separated value format. Log-to-metrics processing converts the numeric values in csv

events into metric data points.

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf pulldown_type = 1

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf [log2metrics_json]

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf DATETIME_CONFIG =

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf INDEXED_EXTRACTIONS = json

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf LINE_BREAKER = ([\r\n]+)

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf METRIC-SCHEMA-TRANSFORMS = metric-schema:log2metrics_default_json

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf NO_BINARY_CHECK = true

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf category = Log to Metrics

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf description = JSON-formatted data. Log-to-metrics processing converts the numeric values in json keys into

metric data points.

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf pulldown_type = 1

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf [log2metrics_keyvalue]

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf DATETIME_CONFIG =

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf LINE_BREAKER = ([\r\n]+)

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf METRIC-SCHEMA-TRANSFORMS = metric-schema:log2metrics_default_keyvalue

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf NO_BINARY_CHECK = true

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf TRANSFORMS-EXTRACT = metrics_field_extraction

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf category = Log to Metrics

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf description = '<key>=<value>' formatted data. Log-to-metrics processing converts the keys with numeric values

into metric data points.

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf pulldown_type = 1

C:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\props.conf [scheduler]

C:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\props.conf EXTRACT-fields = (?i)^(?:[^ ]* ){2}(?:[+\-]\d+ )?(?P<log_level>[^ ]*)\s+(?P<component>[^ ]+) -

(?P<event_message>.+)

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf [source::...\\var\\log\\introspection\\disk_objects.log(.\d+)?]

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf TRANSFORMS-diskobjectsclone = introspection_disk_objects_log_clone

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf [source::...\\var\\log\\introspection\\resource_usage.log(.\d+)?]

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf TRANSFORMS-resourceusageclone = introspection_resource_usage_log_clone

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf [source::...\\var\\log\\splunk\\metrics.log(.\d+)?]

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf TRANSFORMS-metricslogclone = metrics_log_clone

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf [splunk-powershell.ps-2]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf MAX_TIMESTAMP_LOOKAHEAD = 49

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf [splunk-powershell.ps-too_small]

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf PREFIX_SOURCETYPE = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf SHOULD_LINEMERGE = False

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf is_valid = True

C:\Program Files\SplunkUniversalForwarder\etc\apps\learned\local\props.conf maxDist = 9999

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf [splunk_intro_disk_objects]

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf METRIC-SCHEMA-TRANSFORMS = metric-schema:introspection_disk_objects

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf TRANSFORMS-blah = metrics_index_redirect,introspection_disk_objects_metric_name

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf [splunk_intro_resource_usage]

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf METRIC-SCHEMA-TRANSFORMS = metric-schema:introspection_resource_usage

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf TRANSFORMS-bloo = metrics_index_redirect,introspection_resource_usage_metric_name

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf [splunk_metrics_log]

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf METRIC-SCHEMA-TRANSFORMS = metric-schema:metrics_dot_log

C:\Program Files\SplunkUniversalForwarder\etc\apps\splunk_internal_metrics\default\props.conf TRANSFORMS-metricslog = metrics_index_redirect,metrics_field_extraction,metrics_log_metric_name

C:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\props.conf [splunk_web_service]

C:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\props.conf EXTRACT-useragent = userAgent=(?P<browser>[^ (]+)

C:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\props.conf [splunkd]

C:\Program Files\SplunkUniversalForwarder\etc\apps\search\default\props.conf EXTRACT-fields = (?i)^(?:[^ ]* ){2}(?:[+\-]\d+ )?(?P<log_level>[^ ]*)\s+(?P<component>[^ ]+)

(?:\[(?P<thread_id>\d+)\s)?(?:(?P<thread_name>[^\]]+)\]\s)?- (?P<event_message>.+)

Re: Ingested time for csv file on windows

gcusello — Mon, 18 Nov 2024 09:14:35 GMT

Hi @hahhhaxin ,

it's really difficoult to red the output of the btool, have this configuration on the UF?

I don't see DATATIME_CONFIG = CURRENT in your output on the UF.

Ciao.

Giuseppe

Re: Ingested time for csv file on windows

hahhhaxin — Mon, 18 Nov 2024 15:24:32 GMT

No, it is on the HF and indexer, UF here is only targeted for getting data in.

the configuration in HF&indexer is -

[source::asr:report] DATATIME_CONFIG = CURRENT

Re: Ingested time for csv file on windows

gcusello — Mon, 18 Nov 2024 13:59:15 GMT

Hi @hahhhaxin ,

for csv files, put the props.conf and transforms.conf also on UF.

Ciao.

Giuseppe

Re: Ingested time for csv file on windows

richgalloway — Mon, 18 Nov 2024 14:30:54 GMT

I'm assuming DATATYPE_CONFIG=CURRENT is a typo and the real setting is DATETIME_CONFIG=CURRENT.

Try changing the props stanza name to [csv] (the sourcetype).

FWIW, Splunk recommends not changing the source value in inputs.conf.

Re: Ingested time for csv file on windows

hahhhaxin — Tue, 19 Nov 2024 09:40:31 GMT

Yes it is typo.

We do not use sourcetype because various types of csv files are ingested, to keep this change only effectively in the current asset, we locate with sourcetype

Re: Ingested time for csv file on windows

hahhhaxin — Tue, 19 Nov 2024 09:45:20 GMT

Hi @gcusello

except for default props and trans, we do not add any another stanza in local UF props and transforms

Re: Ingested time for csv file on windows

gcusello — Tue, 19 Nov 2024 09:56:02 GMT

Hi @hahhhaxin ,

for csv files you have to add also to UFs.

Ciao.

Giuseppe