topic Re: Parsing Redis logs in Splunk Search

Parsing Redis logs

marksnelling — Mon, 28 Sep 2020 11:26:49 GMT

Hi,
I'm having trouble getting my Redis logs parsed correctly by Splunk, it gets the timestamps messed up.
I have the following stanzas in my indexer props.conf

[source::.../redis/redis.log(.\d+(.gz)?)?] sourcetype=redis

[redis]
NO_BINARY_CHECK=1
TIME_PREFIX=[\d+]\s+
TIME_FORMAT=%d %b %H:%M:%S

The log filenames are in the format

/var/log/redis/redis.log /var/log/redis/redis.log.1 /var/log/redis/redis.log.2.gz ...

The problem seems to be that the indexer doesn't want to apply the correct sourcetype to the logs, instead it uses redis.log-too_small and redis-too_small.

Re: Parsing Redis logs

_d_ — Mon, 27 Feb 2012 14:25:05 GMT

Can you please post a few sample lines of your Redis log(s)?
Also, check the source stanza as it looks like you may need to escape the periods and the \d (digits). It is very likely that this is the problem - where logs are not being assigned sourcetype=redis and therefore the timestamps are not being extracted properly. Run a quick test with this and see if data comes in with correct timestamps:

[source::.../redis/redis.log*]

Hope this helps.

> please upvote and accept answer if you find it useful - thanks!

Re: Parsing Redis logs

marksnelling — Mon, 27 Feb 2012 14:33:13 GMT

I've tried both escaping and raw periods in this stanza with no effect. I should also add that the Redis data is coming from a Universal Forwarder.

Here's a sample of the Redis logs...

[3223] 26 Feb 23:59:01 * Background append only file rewriting started by pid 19383 [19383] 26 Feb 23:59:01 * SYNC append only file rewrite performed [3223] 26 Feb 23:59:01 * Background append only file rewriting terminated with success rewritten. [3223] 26 Feb 23:59:01 * The new append only file was selected for future appends.

Re: Parsing Redis logs

_d_ — Mon, 27 Feb 2012 14:50:19 GMT

I would try the following (not sure whether the editor here is omitting YOUR backslashes but please note the backslashes in TIME_PREFIX):

On the Universal Forwarder:

[source::.../redis/redis.log*] sourcetype = redis

On the Indexer:

[redis] NO_BINARY_CHECK = 1 TIME_PREFIX = \[\d+\]\s+ TIME_FORMAT = %d %b %H:%M:%S

On the Indexer ONLY:

[source::.../redis/redis.log*] sourcetype = redis NO_BINARY_CHECK = 1 TIME_PREFIX = \[\d+\]\s+ TIME_FORMAT = %d %b %H:%M:%S

Re: Parsing Redis logs

marksnelling — Tue, 28 Feb 2012 10:00:06 GMT

Thanks, the first option worked.
the editor was removing the back-slashes in my post.