Hi Oliver!
That timestamp is void of any date and only contains time, so while you still may have some issues to clean up in those logs (whats up with those chars proceeding the time??), I can provide some suggestions and good habits for data onboarding:
I would recommend using the add data wizard to help you experiment with new data sources and to help get the inputs and props configurations you need.
Here's how I did it:
I saved your sample event to a text file, then on my searchhead, I navigated to settings> add data.
I used the upload file option to get the data in, then I used the set sourcetype screen to get to know your data and how Splunk's settings interact with it.
Right away, you can see that Splunk's auto discovery of the timestamp is having issues with your event.
As you can see, linebreaking and timestamping are set to 'auto'.
Best practice is to explicitly define linebreaking and time stamp formatting so that Splunk doesn't have to guess....This will improve performance and accuracy.
So I started with changing the linebreaker to 'every line'
And I configured the timestamp particulars, assuming that the TZ was UTC (you want to verify on the machine logging), and providing Splunk with information about where to find the timestamp. PROTIP: Had to escape some of the characters in the jibberish chars in front of your timestamp becuase splunk is using regex to identify the timestamp prefix.
Then you can use the advanced tab to get familiar with the configurations to use in props.conf! The you can save the sourcetype or you can even copy to clipboard for easy manual creation of a props.conf file.
[ __auto__learned__ ]
SHOULD_LINEMERGE=false
NO_BINARY_CHECK=true
TIME_FORMAT=%H:%M:%S,%f
TIME_PREFIX=\^\[\[0m\^\[\[0m
TZ=UTC
MAX_TIMESTAMP_LOOKAHEAD=25
Because there is no date in the file, Splunk is simply using today's date.
This is a super helpful way to ensure you get some of the most important items right when ingesting your data, and gives you a workspace to experiment.
Now, as for what happened to your data, I would suggest searching alltime for the sourcetype you set, because the auto timestamp recognition placed the event in Jan 3rd, 2016 when I ran it through the add data wiz. Failing that, I would use:
splunker@n00b-splkufwd-01:/opt/splunkforwarder/bin$ ./splunk list inputstatus
or
https://<yourHost>:8089/services/admin/inputstatus/TailingProcessor:FileStatus
To see whats up with the file ingestion
You'll need to figure out whats up with the preamble to that time, or be ok with ingesting with the current date on these logs...I would follow up on these logs...perhaps it can be cleaned up??
... View more