All Apps and Add-ons

Eventgen: How do you specify the size of the output file and control the rotation?

DavidGuarneri
Path Finder

I tried fileSize=xxxx but it didn't work.

The file output seems to max at 10MB and then rotates out. My concern is that it happens so fast that it's rotating out before Splunk has the chance to read it, or that it overflows at a single burst. The output file is persisently at 0 KB, with the rotated files are 10,275KB, so it is hard to see what exactly is happening. I'd like to change it to at least 100MB.

Here's my apache config:

[job_output--apache_20m_data.csv]
mode = sample
outputMode=file
fileName=D:/eventgen_generated_files/apache/apache.log.txt
sampletype = csv
timeMultiple = 36
backfill = -20m
backfillSearch = index=apache sourcetype=splunkd
index=apache
# 10/07/2016 14:00:00
token.0.token = \d{1,2}/\d{2}/\d{4}\s\d{1,2}:\d{2}:\d{2}
token.0.replacementType = timestamp
token.0.replacement = %m/%d/%Y %H:%M:%S
[job_output--apache_20m_data.csv]
Tags (1)
0 Karma
1 Solution

DavidGuarneri
Path Finder

Came back to answer my own question. The root of my issues was that I was not including milliseconds in my logs. The reason I left them out at first was because I was struggling to find a regex that worked in Eventgen. I would test it in Splunk but, Eventgent would not take it. Eventually, I found and an extraction that worked for me:

# 10/07/2016 14:00:04.123456
token.0.token = \d{1,2}/\d{2}/\d{4}\s\d{1,2}:\d{2}:\d{2}.\d{6}
token.0.replacementType = timestamp
token.0.replacement = %m/%d/%Y %H:%M:%S.%f

After I did this, it worked much better, and I no longer had to output to file.

If you think about it, the time multiplier can't work very well if you have a large number of events in the same second.

Here is a tip: to have uniform time regexes in eventgen.conf, concatenate the datetime in a uniform format with the _raw field in the source data that you export. Eventgen will get the first date that it reads in the event line according to your regex. This eliminates a lot of the troubleshooting when trying to catch the date field, especially with logs that have multiple date time formats.

View solution in original post

0 Karma

DavidGuarneri
Path Finder

Came back to answer my own question. The root of my issues was that I was not including milliseconds in my logs. The reason I left them out at first was because I was struggling to find a regex that worked in Eventgen. I would test it in Splunk but, Eventgent would not take it. Eventually, I found and an extraction that worked for me:

# 10/07/2016 14:00:04.123456
token.0.token = \d{1,2}/\d{2}/\d{4}\s\d{1,2}:\d{2}:\d{2}.\d{6}
token.0.replacementType = timestamp
token.0.replacement = %m/%d/%Y %H:%M:%S.%f

After I did this, it worked much better, and I no longer had to output to file.

If you think about it, the time multiplier can't work very well if you have a large number of events in the same second.

Here is a tip: to have uniform time regexes in eventgen.conf, concatenate the datetime in a uniform format with the _raw field in the source data that you export. Eventgen will get the first date that it reads in the event line according to your regex. This eliminates a lot of the troubleshooting when trying to catch the date field, especially with logs that have multiple date time formats.

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...