Deployment Architecture

Custom EventGen samples not being indexed

scc00
Contributor

So I created a simple custom app called eventgen_test, created the samples directory and added a sample.txt. Then created an eventgen.conf in /local directory. Restarted splunk and then...nothing. Ran a search within the _internal index to see what happened. Responses are as follows:

Time Event
6/15/18
12:54:14.418 PM
2018-06-15 16:54:14,418 WARNING module='Sample' sample='webproxy-data.txt': Can't find a timestamp (using patterns '['\d{2}/\d{2}/\d{4} \d{2}:\d{2}:\d{2}']') in this event: '06/15/2018 12:22:18,index=test,source=/var/log/proxy.log,sourcetype=stream:http,bytes=14518888,bytes_in=14516861,bytes_out=2027,src_ip=192.168.45.120,dest_ip=37.48.223.25,uri=http://download.acronis.com/liveupdate/{B129C72A-9734-4128-A342-D6DA3BAE805F"
'.
host = ip-172-31-10-157 source = /opt/splunk/var/log/splunk/eventgen.log sourcetype = eventgen
6/15/18
12:54:14.418 PM
2018-06-15 16:54:14,418 WARNING module='Sample' sample='webproxy-data.txt': Match found ('06/15/2018 12:22:18') but time parse failed. Timeformat '%m/%d/%Y %H:%M:%S,' Event '06/15/2018 12:22:18,index=test,source=/var/log/proxy.log,sourcetype=stream:http,bytes=14518888,bytes_in=14516861,bytes_out=2027,src_ip=192.168.45.120,dest_ip=37.48.223.25,uri=http://download.acronis.com/liveupdate/{B129C72A-9734-4128-A342-D6DA3BAE805F"

I have adjusted the timestamp regex to match but still no luck.

Additional the eventgen.log has a bunch of these errors: I have removed SA-Eventgen so I don't know why it's trying to use it still.

2018-06-15 19:56:56,799 ERROR module='Token' sample='sample.txt': File '/opt/splunk/etc/apps/SA-Eventgen/samples/random_domains.sample' does not exist

Here are my eventgen.conf and sample txt

eventgen.conf

[sample.txt]
mode = replay
timeMultiple = 2
backfill = -15m
backfillSearch = index=test source=sample.log

outputMode = splunkstream
splunkHost = localhost
splunkUser = admin
splunkPass = changeme

token.0.token = \d{2}/\d{2}/\d{4} \d{2}:\d{2}:\d{2}
token.0.replacementType = timestamp
token.0.replacement = %m/%d/%Y %H:%M:%S,

sample.txt
06/15/2018 11:22:18,index=test,source=/var/log/proxy.log,sourcetype=stream:http,,bytes=2309,bytes_in=1953,bytes_out=356,src_ip=192.168.250.100,dest_ip=37.48.223.25,uri=http://crl.microsoft.com/pki/crl/products/CodeSignPCA.crl"
06/15/2018 12:22:18,index=test,source=/var/log/proxy.log,sourcetype=stream:http,,bytes=4259,bytes_in=478,byes_out=480,src_ip=192.168.250.100,dest_ip=37.45.22.25,uri=http://crl.microsoft.com/pki/crl/products/CodeSignPCA.crl"
06/15/2018 10:22:18,index=test,source=/var/log/proxy.log,sourcetype=stream:http,,bytes=116456991,bytes_in=55007608,bytes_out=116453856,src_ip=37.48.223.25,dest_ip=37.48.223.25,uri=http://jgp.live.com/DownloadData.aspx"
06/15/2018 11:25:18,index=test,source=/var/log/proxy.log,sourcetype=stream:http,,bytes=33008711,bytes_in=236453856,bytes_out=33007608,src_ip=67.16.140.142,dest_ip=37.48.223.25,uri=http://jgp.live.com/DownloadData.aspx"
06/15/2018 11:26:18,index=test,source=/var/log/proxy.log,sourcetype=stream:http,,bytes=33008711,bytes_in=1103,bytes_out=33007608,src_ip=172.31.10.108,dest_ip=37.48.223.25,uri=http://ssw.live.com/UploadData.aspx"
06/15/2018 11:45:18,index=test,source=/var/log/proxy.log,sourcetype=stream:http,,bytes=33008711,bytes_in=1103,bytes_out=33007608,src_ip=192.168.25.21,dest_ip=37.48.223.25,uri=http://ssw.live.com/UploadData.aspx"
06/15/2018

How do i get this to work?

Thanks.

0 Karma
1 Solution

scc00
Contributor

The answer was to put the eventgen.conf into the default directory. Doesn't work in the local directory for some reason.

View solution in original post

0 Karma

scc00
Contributor

The answer was to put the eventgen.conf into the default directory. Doesn't work in the local directory for some reason.

0 Karma

guythomasdavis
Explorer

What I've found is if you are on Windows copy eventgen.conf to locals folder and run icacls against it to change permissions. icacls C:\program files\Splunk\etc\apps\your app here\local\eventgen.conf /grant SYSTEM:F.

Took me forever to figure this out. Also every time I edit the file I have to change perms again. Became such a pain that I wrote a c# service to watch for eventgen.conf changes and change perms automatically.

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...