Archive

Custom EventGen samples not being indexed

Communicator

So I created a simple custom app called eventgen_test, created the samples directory and added a sample.txt. Then created an eventgen.conf in /local directory. Restarted splunk and then...nothing. Ran a search within the _internal index to see what happened. Responses are as follows:

Time Event
6/15/18
12:54:14.418 PM
2018-06-15 16:54:14,418 WARNING module='Sample' sample='webproxy-data.txt': Can't find a timestamp (using patterns '['\d{2}/\d{2}/\d{4} \d{2}:\d{2}:\d{2}']') in this event: '06/15/2018 12:22:18,index=test,source=/var/log/proxy.log,sourcetype=stream:http,bytes=14518888,bytesin=14516861,bytesout=2027,srcip=192.168.45.120,destip=37.48.223.25,uri=http://download.acronis.com/liveupdate/{B129C72A-9734-4128-A342-D6DA3BAE805F"
'.
host = ip-172-31-10-157 source = /opt/splunk/var/log/splunk/eventgen.log sourcetype = eventgen
6/15/18
12:54:14.418 PM
2018-06-15 16:54:14,418 WARNING module='Sample' sample='webproxy-data.txt': Match found ('06/15/2018 12:22:18') but time parse failed. Timeformat '%m/%d/%Y %H:%M:%S,' Event '06/15/2018 12:22:18,index=test,source=/var/log/proxy.log,sourcetype=stream:http,bytes=14518888,bytesin=14516861,bytesout=2027,srcip=192.168.45.120,destip=37.48.223.25,uri=http://download.acronis.com/liveupdate/{B129C72A-9734-4128-A342-D6DA3BAE805F"

I have adjusted the timestamp regex to match but still no luck.

Additional the eventgen.log has a bunch of these errors: I have removed SA-Eventgen so I don't know why it's trying to use it still.

2018-06-15 19:56:56,799 ERROR module='Token' sample='sample.txt': File '/opt/splunk/etc/apps/SA-Eventgen/samples/random_domains.sample' does not exist

Here are my eventgen.conf and sample txt

eventgen.conf

[sample.txt]
mode = replay
timeMultiple = 2
backfill = -15m
backfillSearch = index=test source=sample.log

outputMode = splunkstream
splunkHost = localhost
splunkUser = admin
splunkPass = changeme

token.0.token = \d{2}/\d{2}/\d{4} \d{2}:\d{2}:\d{2}
token.0.replacementType = timestamp
token.0.replacement = %m/%d/%Y %H:%M:%S,

sample.txt
06/15/2018 11:22:18,index=test,source=/var/log/proxy.log,sourcetype=stream:http,,bytes=2309,bytesin=1953,bytesout=356,srcip=192.168.250.100,destip=37.48.223.25,uri=http://crl.microsoft.com/pki/crl/products/CodeSignPCA.crl"
06/15/2018 12:22:18,index=test,source=/var/log/proxy.log,sourcetype=stream:http,,bytes=4259,bytesin=478,byesout=480,srcip=192.168.250.100,destip=37.45.22.25,uri=http://crl.microsoft.com/pki/crl/products/CodeSignPCA.crl"
06/15/2018 10:22:18,index=test,source=/var/log/proxy.log,sourcetype=stream:http,,bytes=116456991,bytesin=55007608,bytesout=116453856,srcip=37.48.223.25,destip=37.48.223.25,uri=http://jgp.live.com/DownloadData.aspx"
06/15/2018 11:25:18,index=test,source=/var/log/proxy.log,sourcetype=stream:http,,bytes=33008711,bytesin=236453856,bytesout=33007608,srcip=67.16.140.142,destip=37.48.223.25,uri=http://jgp.live.com/DownloadData.aspx"
06/15/2018 11:26:18,index=test,source=/var/log/proxy.log,sourcetype=stream:http,,bytes=33008711,bytesin=1103,bytesout=33007608,srcip=172.31.10.108,destip=37.48.223.25,uri=http://ssw.live.com/UploadData.aspx"
06/15/2018 11:45:18,index=test,source=/var/log/proxy.log,sourcetype=stream:http,,bytes=33008711,bytesin=1103,bytesout=33007608,srcip=192.168.25.21,destip=37.48.223.25,uri=http://ssw.live.com/UploadData.aspx"
06/15/2018

How do i get this to work?

Thanks.

0 Karma
1 Solution

Communicator

The answer was to put the eventgen.conf into the default directory. Doesn't work in the local directory for some reason.

View solution in original post

0 Karma

Communicator

The answer was to put the eventgen.conf into the default directory. Doesn't work in the local directory for some reason.

View solution in original post

0 Karma

Explorer

What I've found is if you are on Windows copy eventgen.conf to locals folder and run icacls against it to change permissions. icacls C:\program files\Splunk\etc\apps\your app here\local\eventgen.conf /grant SYSTEM:F.

Took me forever to figure this out. Also every time I edit the file I have to change perms again. Became such a pain that I wrote a c# service to watch for eventgen.conf changes and change perms automatically.

0 Karma