Eventgen: How do you specify the size of the output file and control the rotation?

[DavidGuarneri]

I tried fileSize=xxxx but it didn't work.

The file output seems to max at 10MB and then rotates out. My concern is that it happens so fast that it's rotating out before Splunk has the chance to read it, or that it overflows at a single burst. The output file is persisently at 0 KB, with the rotated files are 10,275KB, so it is hard to see what exactly is happening. I'd like to change it to at least 100MB.

Here's my apache config:

[job_output--apache_20m_data.csv]
mode = sample
outputMode=file
fileName=D:/eventgen_generated_files/apache/apache.log.txt
sampletype = csv
timeMultiple = 36
backfill = -20m
backfillSearch = index=apache sourcetype=splunkd
index=apache
# 10/07/2016 14:00:00
token.0.token = \d{1,2}/\d{2}/\d{4}\s\d{1,2}:\d{2}:\d{2}
token.0.replacementType = timestamp
token.0.replacement = %m/%d/%Y %H:%M:%S
[job_output--apache_20m_data.csv]

[DavidGuarneri · Accepted solution]

Came back to answer my own question. The root of my issues was that I was not including milliseconds in my logs. The reason I left them out at first was because I was struggling to find a regex that worked in Eventgen. I would test it in Splunk but, Eventgent would not take it. Eventually, I found and an extraction that worked for me:

# 10/07/2016 14:00:04.123456
token.0.token = \d{1,2}/\d{2}/\d{4}\s\d{1,2}:\d{2}:\d{2}.\d{6}
token.0.replacementType = timestamp
token.0.replacement = %m/%d/%Y %H:%M:%S.%f

After I did this, it worked much better, and I no longer had to output to file.

If you think about it, the time multiplier can't work very well if you have a large number of events in the same second.

Here is a tip: to have uniform time regexes in eventgen.conf, concatenate the datetime in a uniform format with the _raw field in the source data that you export. Eventgen will get the first date that it reads in the event line according to your regex. This eliminates a lot of the troubleshooting when trying to catch the date field, especially with logs that have multiple date time formats.