All Apps and Add-ons

Eventgen: Is there a known bug with backfill and replaytimestamp?

japger_splunk
Splunk Employee
Splunk Employee

I am trying to model transactions such that they are replayed using similar timing between events that are contained in the sample data file. The pause between each "block" of transactions also needs to be spaced out which is being correctly handled (every 600 seconds) in the below eventgen.conf file when ignoring the backfill. The token substitutions are also working as desired.

PROBLEM: The problem shows up when I try to backfill these events over a previous window. It seems that the backfill process does not account for the 600 second pause and is defaulting back to a 1 second spacing between blocks of events. Is this a known bug? Any ideas for a workaround?

eventgen.conf:

[test2.raw]
index=demo
host=myHost
source=test2.raw
breaker = \r*\n\r*\n
mode = replay
sampletype=raw
timeMultiple = 600
backfill = -10m
backfillSearch = index=demo source=test2.raw

outputMode = splunkstream
token.0.token = \d{4}-\d{2}-\d{2}T\d{2}.\d{2}.\d{2}
token.0.replacementType = replaytimestamp
token.0.replacement = %Y-%m-%dT%H:%M:%S
token.1.token = @@src_ip@@
token.1.replacementType = random
token.1.replacement = ipv4

test2.raw:

{"timestamp":"2015-09-04T15:45:00.454143Z","src_ip":"@@src_ip@@","comment":"web click #1"}
{"timestamp":"2015-09-04T15:46:01.454143Z","src_ip":"@@src_ip@@","comment":"web click #2"}
{"timestamp":"2015-09-04T15:47:02.454143Z","src_ip":"@@src_ip@@","comment":"web click #3"}
{"timestamp":"2015-09-04T15:48:03.454143Z","src_ip":"@@src_ip@@","comment":"web click #4"}
0 Karma
1 Solution

japger_splunk
Splunk Employee
Splunk Employee

No bug. After increasing the size of the backfill window and performing a more thorough analysis, I determined that the backfill window is respecting the specific timemultiple=600. The small period of time between the end of the backfill window and the beginning of the ongoing eventgen just happened to be producing 2 "blocks" that were within 1 second of each other. The above eventgen will reliably produce both backfill and ongoining "blocks". I should have caught that the timing between the last event in the backfill and the first event in the ongoining eventgen would be handled differently than as specified in the eventgen.conf file.

I will leave this example up there as a reference for how to replay events that need to be modeled as transactions.

View solution in original post

0 Karma

csharp_splunk
Splunk Employee
Splunk Employee

You don't need replaytimestamp (that feature should probably be removed, but it's being used by one consumer right now). Replaytimestamp is when you want to take a group of transactions using bundlelines and then randomize the time between them. It's specifically targeted at use cases where you want to run the Splunk transaction command and have a varied duration.

This may be a bug. Can you try with the Dev branch from https://github.com/splunk/eventgen and see if you're seeing that there? This branch is about to become master after a few years of development, and I'd like to see if you can reproduce it there. I'm relatively certain it should work properly on that branch.

0 Karma

japger_splunk
Splunk Employee
Splunk Employee

No bug. After increasing the size of the backfill window and performing a more thorough analysis, I determined that the backfill window is respecting the specific timemultiple=600. The small period of time between the end of the backfill window and the beginning of the ongoing eventgen just happened to be producing 2 "blocks" that were within 1 second of each other. The above eventgen will reliably produce both backfill and ongoining "blocks". I should have caught that the timing between the last event in the backfill and the first event in the ongoining eventgen would be handled differently than as specified in the eventgen.conf file.

I will leave this example up there as a reference for how to replay events that need to be modeled as transactions.

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...