SA-Eventgen: detect timestamp in replay- vs. sample-mode

[nekb1958]

when i take my eventgen conf in sample-mode the timestamp is replaced with the actual time in the defined format. when i take the same conf and only change mode to replay, error:

ValueError: Can't find a timestamp (using patterns '['\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}']') in this event: '2012-01-30 23:59:04 QQc-O3J7xve CPUSys 1.046711451335729

what did i miss????

thanks for any help (i´m feeling stupid)
norbert

CONFIG

[short.splunk]  
mode = sample

outputMode = file  
fileName = e://tmp/meas-all.log

token.0.token = \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}  
token.0.replacementType = timestamp  
token.0.replacement = %b %d %H:%M:%S

DATA (== short.splunk)

2012-01-30 23:59:04 QQc-O3J7xve CPUUser 2.3843930719158464

2012-01-30 23:59:04 QQc-O3J7xve CPUSys 1.046711451335729

2012-01-30 23:59:04 sIkqmRX7zai CPUUser 0.57221331953932253

2012-01-30 23:59:04 sIkqmRX7zai CPUSys 0.37301278622911521

2012-01-30 23:59:02 QQc-O3J7xve MemFree 4326506496

2012-01-30 23:59:02 sIkqmRX7zai MemFree 3537133568

2012-01-30 23:58:04 sSGBns8SkM7 CPUUser 0.74475287745429919

2012-01-30 23:58:04 sSGBns8SkM7 CPUSys 0.56507473569084943

2012-01-30 23:58:02 sSGBns8SkM7 MemFree 7351476224

[fortiwhall]

I was having the hardest time with this today - thank you for the tip, @nekb1958.

I had switched from "sample" to "replay" as well and kept getting the same error over and over. I didn't understand @nekb1958's answer at first, but now I see what I'm supposed to do. I'm documenting it further below.

In my example, the timestamp looked like this:

Oct 16 10:59:54

In SAMPLE mode, you would use the three lines as follows, where the token regex is completely separate from the replacement format.

token.0.token = \w{3} \d{2} \d{2}:\d{2}:\d{2}
token.2.replacementType = timestamp  
token.2.replacement = %d/%b/%Y:%H:%M:%S.%f  

Here, the token does not need to match the replacement - the replacement variable is the POSIX format of the timestamp you want EventGen to insert into the result, while the token is the string you want to match. In the above case, I changed the timestamp to a new format just as an example.

In REPLAY mode, however, it seems the token regex and the replacement need to match.

token.0.token = \w{3} \d{2} \d{2}:\d{2}:\d{2}  
token.0.replacementType = timestamp  
token.0.replacement = %b %d %H:%M:%S  

That way, it knows where to look for a timestamp (via token regex) and it knows the POSIX format of the timestamp (via replacement format).

[nekb1958]

feeling stupid?

reading helps 😞

token.0.replacement = %b %d %H:%M:%S

isn´t (like i thought) the format (strptime) of the replaced timestamp, NO it is the format of the "input". stupid? yes 🙂

bye norbert

[nekb1958]

my original timestamp is in epoch, taking token.0.replacement = %s detect the right value, but has a problem to print epoch in replacement.
2013-08-08 14:33:37,548 DEBUG Match '1325372400' Format '%s' result: '2012-01-01 00:00:00'
2013-08-08 14:33:37,548 DEBUG Match '1325372402' Format '%s' result: '2012-01-01 00:00:02'

...
File "C:\Program Files\Splunk\etc\apps\SA-EventGen\lib\eventgensamples.py", line 921, in _getReplacement
replacementTime = replacementTime.strftime(self.replacement)
ValueError: Invalid format string

ok, now i know what works.

norbert