Getting Data In

How can I index events with the real date while using the timestamp from the log?

ZimmermanC1
Explorer

I have a Splunk instance in a Development & Test lab that uses what we call "repeatable time" to test software updates against a known good checkpoint of data. During a test all of the server times are rolled back to March 16 2011. I have been struggling to figure out how to get Splunk to ignore the Date in the logged events and only care about the "time".

The logs are being monitored in a directory structure as follows /var/adm/splunk/2017_"today's_julian_date"/"server_name"/*.log

 Example Log: header,123,321,2011-03-16 17:35:36.035 +00:00,subject,.............

Effectively I am trying to get these events indexed with today's date but the Time from the log. so it would have:

  _time=2017-12-11 17:35:36.035

I have tried playing with the TIME_PREFIX & MAX_TIMESTAMP_LOOKAHEAD settings in the sourcetype but have not been successful.

 TIME_PREFIX = ^[^\s]+\s
 MAX_TIMESTAMP_LOOKAHEAD = 25

This is a critical issue for me to sort out so any help would be greatly appreciated.

0 Karma

ZimmermanC1
Explorer

Following some more research I was able to determine that the date portion of the timestamp was being generated based on the file name which was called ASCII.20110316....... So i created a script to rename each file to ACII.2017-12-13....... and it appears to now almost be working. The problem seems to be that Splunk is now timeshifting the logs one day into the future. I assume it has something to do with the timezone but I am not 100% sure.

Any other ideas would be greatly appreciated.

0 Karma

DalJeanis
Legend

You might want to check out the Splunk Event Gen app, which IIRC can take input data and modify the timestamp in the _raw data for you.

https://splunkbase.splunk.com/app/1924/

0 Karma
Get Updates on the Splunk Community!

Index This | What is broken 80% of the time by February?

December 2025 Edition   Hayyy Splunk Education Enthusiasts and the Eternally Curious!    We’re back with this ...

Unlock Faster Time-to-Value on Edge and Ingest Processor with New SPL2 Pipeline ...

Hello Splunk Community,   We're thrilled to share an exciting update that will help you manage your data more ...

Splunk MCP & Agentic AI: Machine Data Without Limits

Discover how the Splunk Model Context Protocol (MCP) Server can revolutionize the way your organization uses ...