Parsing Redis logs

[marksnelling]

Hi,
I'm having trouble getting my Redis logs parsed correctly by Splunk, it gets the timestamps messed up.
I have the following stanzas in my indexer props.conf

[source::.../redis/redis.log(.\d+(.gz)?)?]
sourcetype=redis

[redis]
NO_BINARY_CHECK=1
TIME_PREFIX=[\d+]\s+
TIME_FORMAT=%d %b %H:%M:%S

The log filenames are in the format


/var/log/redis/redis.log
/var/log/redis/redis.log.1
/var/log/redis/redis.log.2.gz
...

The problem seems to be that the indexer doesn't want to apply the correct sourcetype to the logs, instead it uses redis.log-too_small and redis-too_small.

[_d_ · Accepted solution]

I would try the following (not sure whether the editor here is omitting YOUR backslashes but please note the backslashes in TIME_PREFIX):

On the Universal Forwarder:

[source::.../redis/redis.log*]
sourcetype = redis

On the Indexer:

[redis]
NO_BINARY_CHECK = 1
TIME_PREFIX = \[\d+\]\s+
TIME_FORMAT = %d %b %H:%M:%S

OR

On the Indexer ONLY:

[source::.../redis/redis.log*]
sourcetype = redis
NO_BINARY_CHECK = 1
TIME_PREFIX = \[\d+\]\s+
TIME_FORMAT = %d %b %H:%M:%S

[marksnelling]

Thanks, the first option worked.
the editor was removing the back-slashes in my post.

[_d_]

Can you please post a few sample lines of your Redis log(s)?
Also, check the source stanza as it looks like you may need to escape the periods and the \d (digits). It is very likely that this is the problem - where logs are not being assigned sourcetype=redis and therefore the timestamps are not being extracted properly. Run a quick test with this and see if data comes in with correct timestamps:

[source::.../redis/redis.log*]

Hope this helps.

> please upvote and accept answer if you find it useful - thanks!

[marksnelling]

I've tried both escaping and raw periods in this stanza with no effect. I should also add that the Redis data is coming from a Universal Forwarder.

Here's a sample of the Redis logs...

[3223] 26 Feb 23:59:01 * Background append only file rewriting started by pid 19383
[19383] 26 Feb 23:59:01 * SYNC append only file rewrite performed
[3223] 26 Feb 23:59:01 * Background append only file rewriting terminated with success
rewritten.
[3223] 26 Feb 23:59:01 * The new append only file was selected for future appends.