Splunk Search

Parsing Redis logs

marksnelling
Communicator

Hi,
I'm having trouble getting my Redis logs parsed correctly by Splunk, it gets the timestamps messed up.
I have the following stanzas in my indexer props.conf

[source::.../redis/redis.log(.\d+(.gz)?)?]
sourcetype=redis

[redis]
NO_BINARY_CHECK=1
TIME_PREFIX=[\d+]\s+
TIME_FORMAT=%d %b %H:%M:%S

The log filenames are in the format


/var/log/redis/redis.log
/var/log/redis/redis.log.1
/var/log/redis/redis.log.2.gz
...


The problem seems to be that the indexer doesn't want to apply the correct sourcetype to the logs, instead it uses redis.log-too_small and redis-too_small.

0 Karma
1 Solution

_d_
Splunk Employee
Splunk Employee

I would try the following (not sure whether the editor here is omitting YOUR backslashes but please note the backslashes in TIME_PREFIX):

On the Universal Forwarder:

[source::.../redis/redis.log*]
sourcetype = redis

On the Indexer:

[redis]
NO_BINARY_CHECK = 1
TIME_PREFIX = \[\d+\]\s+
TIME_FORMAT = %d %b %H:%M:%S

OR

On the Indexer ONLY:

[source::.../redis/redis.log*]
sourcetype = redis
NO_BINARY_CHECK = 1
TIME_PREFIX = \[\d+\]\s+
TIME_FORMAT = %d %b %H:%M:%S

View solution in original post

_d_
Splunk Employee
Splunk Employee

I would try the following (not sure whether the editor here is omitting YOUR backslashes but please note the backslashes in TIME_PREFIX):

On the Universal Forwarder:

[source::.../redis/redis.log*]
sourcetype = redis

On the Indexer:

[redis]
NO_BINARY_CHECK = 1
TIME_PREFIX = \[\d+\]\s+
TIME_FORMAT = %d %b %H:%M:%S

OR

On the Indexer ONLY:

[source::.../redis/redis.log*]
sourcetype = redis
NO_BINARY_CHECK = 1
TIME_PREFIX = \[\d+\]\s+
TIME_FORMAT = %d %b %H:%M:%S

View solution in original post

marksnelling
Communicator

Thanks, the first option worked.
the editor was removing the back-slashes in my post.

0 Karma

_d_
Splunk Employee
Splunk Employee

Can you please post a few sample lines of your Redis log(s)?
Also, check the source stanza as it looks like you may need to escape the periods and the \d (digits). It is very likely that this is the problem - where logs are not being assigned sourcetype=redis and therefore the timestamps are not being extracted properly. Run a quick test with this and see if data comes in with correct timestamps:

[source::.../redis/redis.log*]

Hope this helps.

> please upvote and accept answer if you find it useful - thanks!

0 Karma

marksnelling
Communicator

I've tried both escaping and raw periods in this stanza with no effect. I should also add that the Redis data is coming from a Universal Forwarder.

Here's a sample of the Redis logs...

[3223] 26 Feb 23:59:01 * Background append only file rewriting started by pid 19383
[19383] 26 Feb 23:59:01 * SYNC append only file rewrite performed
[3223] 26 Feb 23:59:01 * Background append only file rewriting terminated with success
rewritten.
[3223] 26 Feb 23:59:01 * The new append only file was selected for future appends.

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.