All Apps and Add-ons

Eventgen: How do you specify the size of the output file and control the rotation?

DavidGuarneri
Path Finder

I tried fileSize=xxxx but it didn't work.

The file output seems to max at 10MB and then rotates out. My concern is that it happens so fast that it's rotating out before Splunk has the chance to read it, or that it overflows at a single burst. The output file is persisently at 0 KB, with the rotated files are 10,275KB, so it is hard to see what exactly is happening. I'd like to change it to at least 100MB.

Here's my apache config:

[job_output--apache_20m_data.csv]
mode = sample
outputMode=file
fileName=D:/eventgen_generated_files/apache/apache.log.txt
sampletype = csv
timeMultiple = 36
backfill = -20m
backfillSearch = index=apache sourcetype=splunkd
index=apache
# 10/07/2016 14:00:00
token.0.token = \d{1,2}/\d{2}/\d{4}\s\d{1,2}:\d{2}:\d{2}
token.0.replacementType = timestamp
token.0.replacement = %m/%d/%Y %H:%M:%S
[job_output--apache_20m_data.csv]
Tags (1)
0 Karma
1 Solution

DavidGuarneri
Path Finder

Came back to answer my own question. The root of my issues was that I was not including milliseconds in my logs. The reason I left them out at first was because I was struggling to find a regex that worked in Eventgen. I would test it in Splunk but, Eventgent would not take it. Eventually, I found and an extraction that worked for me:

# 10/07/2016 14:00:04.123456
token.0.token = \d{1,2}/\d{2}/\d{4}\s\d{1,2}:\d{2}:\d{2}.\d{6}
token.0.replacementType = timestamp
token.0.replacement = %m/%d/%Y %H:%M:%S.%f

After I did this, it worked much better, and I no longer had to output to file.

If you think about it, the time multiplier can't work very well if you have a large number of events in the same second.

Here is a tip: to have uniform time regexes in eventgen.conf, concatenate the datetime in a uniform format with the _raw field in the source data that you export. Eventgen will get the first date that it reads in the event line according to your regex. This eliminates a lot of the troubleshooting when trying to catch the date field, especially with logs that have multiple date time formats.

View solution in original post

0 Karma

DavidGuarneri
Path Finder

Came back to answer my own question. The root of my issues was that I was not including milliseconds in my logs. The reason I left them out at first was because I was struggling to find a regex that worked in Eventgen. I would test it in Splunk but, Eventgent would not take it. Eventually, I found and an extraction that worked for me:

# 10/07/2016 14:00:04.123456
token.0.token = \d{1,2}/\d{2}/\d{4}\s\d{1,2}:\d{2}:\d{2}.\d{6}
token.0.replacementType = timestamp
token.0.replacement = %m/%d/%Y %H:%M:%S.%f

After I did this, it worked much better, and I no longer had to output to file.

If you think about it, the time multiplier can't work very well if you have a large number of events in the same second.

Here is a tip: to have uniform time regexes in eventgen.conf, concatenate the datetime in a uniform format with the _raw field in the source data that you export. Eventgen will get the first date that it reads in the event line according to your regex. This eliminates a lot of the troubleshooting when trying to catch the date field, especially with logs that have multiple date time formats.

0 Karma
Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...