Any idea why the timestamp of data that send via logstash, change when stored in Splunk index?

[indeed_2000]

Hi
timestamp of data that send via logstash change when store in splunk index. what is the reason?

index="influx2splunk" | spath input=_raw | table time _time @timestamp _raw

time                                          _time                               @timestamp
2023-06-15T06:06:55Z 2023-06-15 05:06:55 2023-06-15T01:36:55.000Z
2023-06-15T06:06:55Z                                               2023-06-15T01:36:55.000Z

here is the _raw data that get from logstash:

{"usage_irq":0,"usage_user":4.373757455295997,"results":{"statement_id":0},"@version":"1","@timestamp":"2023-06-15T01:36:55.000Z","usage_guest":0,"cpu":"cpu20","usage_iowait":0,"usage_softirq":0.39761431396001656,"http_poller_metadata":{"input":{"http_poller":{"response":{"status_code":200,"status_message":"OK","headers":{"date":"Sat, 17 Jun 2023 06:05:47 GMT","x-influxdb-build":"OSS","x-influxdb-version":"1.7.8","transfer-encoding":"chunked","x-request-id":"00a6ba2f-0cd5-11ee-981b-005056b7dda2","content-type":"application/json","request-id":"00a6ba2f-0cd5-11ee-981b-005056b7dda2"},"elapsed_time_ns":797045},"request":{"name":"cpu","original":{"url":"https://192.168.1.1:8086/query?pretty=true&db=mydb&q=myquery","headers":{"Authorization":"Token mytoken"},"method":"get"},"retry_count":0,"host":{"hostname":"srv"}}}}},"usage_idle":92.04771372774293,"usage_system":3.1809145128373424,"usage_steal":0,"time":"2023-06-15T06:06:55Z","name":"cpu","usage_nice":0,"usage_guest_nice":0}

logstash config:

filter {
split {
field => "results"
}

split {
field => "[results][series]"
}

split {
field => "[results][series][values]"
}

mutate {
rename => { "[results][series]" => "series" }
}

mutate {
rename => { "[series][name]" => "name" }
}

ruby {
code => 'series = event.get("series"); series["columns"].each_with_index {|val, index| event.set(val, event.get("[series][values][" + index.to_s() + "]"))}'
}

date {
match => ["time", "yyyy-MM-dd'T'HH:mm:ss:SSS'Z"]
target => "_time"

}

prune {
blacklist_names => [ "event", "host", "series" ]
}

}

Any idea?

Thanks

[PickleRick]

Timestamps don't "change" on their own. You can instruct splunk to adjust the _time field in some cases but you have to do so explicitly. Most probably your source is not properly onboarded and you don't have properly configured timestamp recognition so the parsing component assumes the timestamp of the indexing moment.

[indeed_2000]

@PickleRickis it possible to fix it in logstash without change config in splunk?

I mean set logstash in somehow to consider timestamp as field that use as "_time"

Any idea?

Thanks

[PickleRick]

I have no idea. We don't know what is your splunk config, how you're ingesting the logs (you only showed the filter part of the logstash config) and so we can't know whether making both components work with each other requires changes on one side or the other or both.

Also if you paste a piece of config or a raw event, use either the code block (the </> icon on top of the editor) or the preformatted paragraph style.

[indeed_2000]

@PickleRickone suspicious thing that happen here is  one field that called "time" and  "@timestamp" duplicate date but in raw log only one date exist!

when i run below SPL return two date!

index="influx2splunk" | spath input=_raw | table time _time @timestamp _raw

time                                           _time @                               @timestamp
2023-06-15T06:06:55Z    2023-06-15 05:06:55   2023-06-15T01:36:55.000Z
2023-06-15T06:06:55Z                                                   2023-06-15T01:36:55.000Z

on raw data the only one date store in these fields:

{"usage_irq":0,"usage_user":4.373757455295997,"results":{"statement_id":0},"@version":"1","@timestamp":"2023-06-15T01:36:55.000Z","usage_guest":0,"cpu":"cpu20","usage_iowait":0,"usage_softirq":0.39761431396001656,"http_poller_metadata":{"input":{"http_poller":{"response":{"status_code":200,"status_message":"OK","headers":{"date":"Sat, 17 Jun 2023 06:05:47 GMT","x-influxdb-build":"OSS","x-influxdb-version":"1.7.8","transfer-encoding":"chunked","x-request-id":"00a6ba2f-0cd5-11ee-981b-005056b7dda2","content-type":"application/json","request-id":"00a6ba2f-0cd5-11ee-981b-005056b7dda2"},"elapsed_time_ns":797045},"request":{"name":"cpu","original":{"url":"https://192.168.1.1:8086/query?pretty=true&db=mydb&q=myquery","headers":{"Authorization":"Token mytoken"},"method":"get"},"retry_count":0,"host":{"hostname":"srv"}}}}},"usage_idle":92.04771372774293,"usage_system":3.1809145128373424,"usage_steal":0,"time":"2023-06-15T06:06:55Z","name":"cpu","usage_nice":0,"usage_guest_nice":0}

Any idea?

Thanks

[isoutamo]

We have already discussed about this on https://community.splunk.com/t5/Splunk-Search/Is-it-possible-to-consider-quot-time-quot-as-quot-time.... If you have read and followed those instructions you should have correct _time field on your indexed data.

[richgalloway]

What is your question?

[indeed_2000]

@richgalloway timestamp of data that send via logstash change when store in splunk index. what is the reason?