All Apps and Add-ons

Eventgen: How do you specify the size of the output file and control the rotation?

DavidGuarneri
Path Finder

I tried fileSize=xxxx but it didn't work.

The file output seems to max at 10MB and then rotates out. My concern is that it happens so fast that it's rotating out before Splunk has the chance to read it, or that it overflows at a single burst. The output file is persisently at 0 KB, with the rotated files are 10,275KB, so it is hard to see what exactly is happening. I'd like to change it to at least 100MB.

Here's my apache config:

[job_output--apache_20m_data.csv]
mode = sample
outputMode=file
fileName=D:/eventgen_generated_files/apache/apache.log.txt
sampletype = csv
timeMultiple = 36
backfill = -20m
backfillSearch = index=apache sourcetype=splunkd
index=apache
# 10/07/2016 14:00:00
token.0.token = \d{1,2}/\d{2}/\d{4}\s\d{1,2}:\d{2}:\d{2}
token.0.replacementType = timestamp
token.0.replacement = %m/%d/%Y %H:%M:%S
[job_output--apache_20m_data.csv]
Tags (1)
0 Karma
1 Solution

DavidGuarneri
Path Finder

Came back to answer my own question. The root of my issues was that I was not including milliseconds in my logs. The reason I left them out at first was because I was struggling to find a regex that worked in Eventgen. I would test it in Splunk but, Eventgent would not take it. Eventually, I found and an extraction that worked for me:

# 10/07/2016 14:00:04.123456
token.0.token = \d{1,2}/\d{2}/\d{4}\s\d{1,2}:\d{2}:\d{2}.\d{6}
token.0.replacementType = timestamp
token.0.replacement = %m/%d/%Y %H:%M:%S.%f

After I did this, it worked much better, and I no longer had to output to file.

If you think about it, the time multiplier can't work very well if you have a large number of events in the same second.

Here is a tip: to have uniform time regexes in eventgen.conf, concatenate the datetime in a uniform format with the _raw field in the source data that you export. Eventgen will get the first date that it reads in the event line according to your regex. This eliminates a lot of the troubleshooting when trying to catch the date field, especially with logs that have multiple date time formats.

View solution in original post

0 Karma

DavidGuarneri
Path Finder

Came back to answer my own question. The root of my issues was that I was not including milliseconds in my logs. The reason I left them out at first was because I was struggling to find a regex that worked in Eventgen. I would test it in Splunk but, Eventgent would not take it. Eventually, I found and an extraction that worked for me:

# 10/07/2016 14:00:04.123456
token.0.token = \d{1,2}/\d{2}/\d{4}\s\d{1,2}:\d{2}:\d{2}.\d{6}
token.0.replacementType = timestamp
token.0.replacement = %m/%d/%Y %H:%M:%S.%f

After I did this, it worked much better, and I no longer had to output to file.

If you think about it, the time multiplier can't work very well if you have a large number of events in the same second.

Here is a tip: to have uniform time regexes in eventgen.conf, concatenate the datetime in a uniform format with the _raw field in the source data that you export. Eventgen will get the first date that it reads in the event line according to your regex. This eliminates a lot of the troubleshooting when trying to catch the date field, especially with logs that have multiple date time formats.

0 Karma
Get Updates on the Splunk Community!

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...

Announcing General Availability of Splunk Incident Intelligence!

Digital transformation is real! Across industries, companies big and small are going through rapid digital ...