Hi,
Installed SA_Eventgen and configured it for with two different samples(one is a CSV and another a txt file with raw data) but it is not generating any data. In App's UI under "Eventgen Logs" tab I can see that the eventgen process has begun for both the samples. Here are some screenshots and the eventgen.conf file.
Logs:
2018-08-20 16:18:00.000 Splunk _internal 2018-08-20 16:18:00 eventgen INFO MainProcess All timers started, joining queue until it's empty.
2018-08-20 16:18:00.000 Splunk _internal 2018-08-20 16:18:00 eventgen INFO MainProcess Start '1' generatorWorkers for sample 'Threats.sophos'
2018-08-20 16:18:00.000 Splunk _internal 2018-08-20 16:18:00 eventgen INFO MainProcess Creating timer object for sample 'Threats.sophos' in app 'Sample_Data'
2018-08-20 16:18:00.000 Splunk _internal 2018-08-20 16:18:00 eventgen INFO MainProcess Start '1' generatorWorkers for sample 'isilon_auth.csv'
2018-08-20 16:18:00.000 Splunk _internal 2018-08-20 16:18:00 eventgen INFO MainProcess Creating timer object for sample 'isilon_auth.csv' in app 'Sample_Data'
2018-08-20 16:18:00.000 Splunk _internal 2018-08-20 16:18:00 eventgen ERROR MainProcess No module named jinja2 Traceback (most recent call last): File "/opt/splunk/etc/apps/SA-Eventgen/lib/splunk_eventgen/eventgen_core.py", line 437, in _initializePlugins module = imp.load_module(base, mod_name, mod_path, mod_desc) File "/opt/splunk/etc/apps/SA-Eventgen/lib/splunk_eventgen/lib/plugins/generator/jinja.py", line 9, in <module> from jinja2 import nodes ImportError: No module named jinja2
2018-08-20 16:18:00.000 Splunk _internal 2018-08-20 16:18:00 eventgen WARNING MainProcess Could not load plugin: /opt/splunk/etc/apps/SA-Eventgen/lib/splunk_eventgen/lib/plugins/generator/jinja.py, skipping
2018-08-20 16:18:00.000 Splunk _internal 2018-08-20 16:18:00 eventgen INFO MainProcess Key 'splunkUser' in stanza 'Threats.sophos' may not be a valid setting
2018-08-20 16:18:00.000 Splunk _internal 2018-08-20 16:18:00 eventgen INFO MainProcess Key 'splunkPass' in stanza 'Threats.sophos' may not be a valid setting
2018-08-20 16:18:00.000 Splunk _internal 2018-08-20 16:18:00 eventgen INFO MainProcess Key 'splunkHost' in stanza 'Threats.sophos' may not be a valid setting
2018-08-20 16:18:00.000 Splunk _internal 2018-08-20 16:18:00 eventgen INFO MainProcess Key 'splunkUser' in stanza 'isilon_auth.csv' may not be a valid setting
2018-08-20 16:18:00.000 Splunk _internal 2018-08-20 16:18:00 eventgen INFO MainProcess Key 'splunkPass' in stanza 'isilon_auth.csv' may not be a valid setting
2018-08-20 16:18:00.000 Splunk _internal 2018-08-20 16:18:00 eventgen INFO MainProcess Key 'splunkHost' in stanza 'isilon_auth.csv' may not be a valid setting
2018-08-20 16:18:00.560 Splunk _internal 2018-08-20 16:18:00,560 INFO [Eventgen] Finished setup pools
2018-08-20 16:18:00.549 Splunk _internal 2018-08-20 16:18:00,549 INFO [Eventgen] Finished reload
2018-08-20 16:18:00.541 Splunk _internal 2018-08-20 16:18:00,541 INFO [Eventgen] Finished parse
2018-08-20 16:18:00.541 Splunk _internal 2018-08-20 16:18:00,541 INFO [Eventgen] Finished config parsing
2018-08-20 16:18:00.487 Splunk _internal 2018-08-20 16:18:00,487 INFO [Eventgen] Config made Splunk Embedded
2018-08-20 16:18:00.487 Splunk _internal 2018-08-20 16:18:00,487 INFO [Eventgen] Config object generated
2018-08-20 16:18:00.486 Splunk _internal 2018-08-20 16:18:00,486 INFO [Eventgen] Eventgen object generated
2018-08-20 16:18:00.478 Splunk _internal 2018-08-20 16:18:00,478 INFO [Eventgen] Prepared Config
2018-08-20 16:18:00.478 Splunk _internal 2018-08-20 16:18:00,478 INFO [Eventgen] Input Config is: {'configuration': "{u'modinput_eventgen://default': {'name': u'modinput_eventgen://default', u'host': u'Splunk', u'disabled': u'0', u'VERBOSE': u'0', u'index': u'default'}}", 'checkpoint_dir': '/opt/splunk/var/lib/splunk/modinputs/modinput_eventgen', 'session_key': 'wv2kjziDCSHghZyvYGnSF519l41gzBCmd_euQyENd1P3eVfgMcOM^Lz8SMrmD63iRq_mWKt8NAX430ARnDQgfQGxvBpzyDlAX3PG^7sXEz9BB_E8U6ppQQC', 'server_uri': 'https://127.0.0.1:8089', 'server_host': 'Splunk'}
2018-08-20 16:18:00.478 Splunk _internal 2018-08-20 16:18:00,478 INFO [Eventgen] Initialized streaming
2018-08-20 16:18:00.476 Splunk _internal 2018-08-20 16:18:00,476 DEBUG [Eventgen] Setting up SA-Eventgen Modular Input
2018-08-20 16:18:00.475 Splunk _internal 2018-08-20 16:18:00,475 DEBUG [Eventgen] Initialized ModularInput Logger
2018-08-20 16:18:00.000 Splunk _internal 2018-08-20 16:18:00 eventgen INFO MainProcess Retrieving eventgen configurations from /configs/eventgen
2018-08-20 16:18:00.000 Splunk _internal 2018-08-20 16:18:00 eventgen INFO MainProcess Logging Setup Complete.
Two samples.
/opt/splunk/etc/apps/Sample_Data/local/eventgen.conf
[isilon_auth.csv]
mode = replay
timeMultiple = 1
backfill = -15m
sampletype = csv
outputMode = splunkstream
index = main
sourcetype = isilon:data
source = syslog
host = localhost
splunkHost = localhost
splunkUser = admin
splunkPass = password
token.0.token = \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}
token.0.replacementType = replaytimestamp
token.0.replacement = %Y-%m-%d %H:%M:%S
[Threats.sophos]
mode = replay
timeMultiple = 1
backfill = -15m
sampletype = raw
outputMode = splunkstream
index = Sophos
sourcetype = sophos:threats
source = eventgen
host = localhost
splunkHost = localhost
splunkUser = admin
splunkPass = password
token.0.token = \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}
token.0.replacementType = replaytimestamp
token.0.replacement = %Y-%m-%d %H:%M:%S
App even populates the performance dashboard with one of the inputs but there is no actual data to search.
[UPDATE]
After changing the mode from "replay" to "sample", errors are not coming up anymore and we get the following being repeated after every 1 minute. Although still no data in Splunk.
2018-08-21 13:17:05 Splunk _internal 2018-08-21 13:17:05 eventgen INFO MainProcess Worker# 0: Put -1 events in queue for sample 'isilon_auth.csv' with et '2018-08-21 13:17:05.360795' and lt '2018-08-21 13:17:05.360850'
2018-08-21 13:17:05 Splunk _internal 2018-08-21 13:17:05 eventgen INFO MainProcess Start '1' generatorWorkers for sample 'isilon_auth.csv'
2018-08-21 13:17:05 Splunk _internal 2018-08-21 13:17:05 eventgen INFO MainProcess Worker# 0: Put -1 events in queue for sample 'Threats.sophos' with et '2018-08-21 13:17:05.310390' and lt '2018-08-21 13:17:05.310451'
2018-08-21 13:17:05 Splunk _internal 2018-08-21 13:17:05 eventgen INFO MainProcess Start '1' generatorWorkers for sample 'Threats.sophos'
2018-08-21 13:16:05 Splunk _internal 2018-08-21 13:16:05 eventgen INFO MainProcess Worker# 0: Put -1 events in queue for sample 'isilon_auth.csv' with et '2018-08-21 13:16:05.293507' and lt '2018-08-21 13:16:05.293573'
2018-08-21 13:16:05 Splunk _internal 2018-08-21 13:16:05 eventgen INFO MainProcess Start '1' generatorWorkers for sample 'isilon_auth.csv'
2018-08-21 13:16:05 Splunk _internal 2018-08-21 13:16:05 eventgen INFO MainProcess Worker# 0: Put -1 events in queue for sample 'Threats.sophos' with et '2018-08-21 13:16:05.243097' and lt '2018-08-21 13:16:05.243143'
2018-08-21 13:16:05 Splunk _internal 2018-08-21 13:16:05 eventgen INFO MainProcess Start '1' generatorWorkers for sample 'Threats.sophos'
Thanks,
~ Abhi
