All Apps and Add-ons
Highlighted

SA-Eventgen not generating any data.

Contributor

Hi,

Installed SA_Eventgen and configured it for with two different samples(one is a CSV and another a txt file with raw data) but it is not generating any data. In App's UI under "Eventgen Logs" tab I can see that the eventgen process has begun for both the samples. Here are some screenshots and the eventgen.conf file.

Logs:

2018-08-20 16:18:00.000     Splunk  _internal   2018-08-20 16:18:00 eventgen INFO MainProcess All timers started, joining queue until it's empty.
2018-08-20 16:18:00.000     Splunk  _internal   2018-08-20 16:18:00 eventgen INFO MainProcess Start '1' generatorWorkers for sample 'Threats.sophos'
2018-08-20 16:18:00.000     Splunk  _internal   2018-08-20 16:18:00 eventgen INFO MainProcess Creating timer object for sample 'Threats.sophos' in app 'Sample_Data'
2018-08-20 16:18:00.000     Splunk  _internal   2018-08-20 16:18:00 eventgen INFO MainProcess Start '1' generatorWorkers for sample 'isilon_auth.csv'
2018-08-20 16:18:00.000     Splunk  _internal   2018-08-20 16:18:00 eventgen INFO MainProcess Creating timer object for sample 'isilon_auth.csv' in app 'Sample_Data'
2018-08-20 16:18:00.000     Splunk  _internal   2018-08-20 16:18:00 eventgen ERROR MainProcess No module named jinja2 Traceback (most recent call last): File "/opt/splunk/etc/apps/SA-Eventgen/lib/splunk_eventgen/eventgen_core.py", line 437, in _initializePlugins module = imp.load_module(base, mod_name, mod_path, mod_desc) File "/opt/splunk/etc/apps/SA-Eventgen/lib/splunk_eventgen/lib/plugins/generator/jinja.py", line 9, in <module> from jinja2 import nodes ImportError: No module named jinja2
2018-08-20 16:18:00.000     Splunk  _internal   2018-08-20 16:18:00 eventgen WARNING MainProcess Could not load plugin: /opt/splunk/etc/apps/SA-Eventgen/lib/splunk_eventgen/lib/plugins/generator/jinja.py, skipping
2018-08-20 16:18:00.000     Splunk  _internal   2018-08-20 16:18:00 eventgen INFO MainProcess Key 'splunkUser' in stanza 'Threats.sophos' may not be a valid setting
2018-08-20 16:18:00.000     Splunk  _internal   2018-08-20 16:18:00 eventgen INFO MainProcess Key 'splunkPass' in stanza 'Threats.sophos' may not be a valid setting
2018-08-20 16:18:00.000     Splunk  _internal   2018-08-20 16:18:00 eventgen INFO MainProcess Key 'splunkHost' in stanza 'Threats.sophos' may not be a valid setting
2018-08-20 16:18:00.000     Splunk  _internal   2018-08-20 16:18:00 eventgen INFO MainProcess Key 'splunkUser' in stanza 'isilon_auth.csv' may not be a valid setting
2018-08-20 16:18:00.000     Splunk  _internal   2018-08-20 16:18:00 eventgen INFO MainProcess Key 'splunkPass' in stanza 'isilon_auth.csv' may not be a valid setting
2018-08-20 16:18:00.000     Splunk  _internal   2018-08-20 16:18:00 eventgen INFO MainProcess Key 'splunkHost' in stanza 'isilon_auth.csv' may not be a valid setting
2018-08-20 16:18:00.560     Splunk  _internal   2018-08-20 16:18:00,560 INFO [Eventgen] Finished setup pools
2018-08-20 16:18:00.549     Splunk  _internal   2018-08-20 16:18:00,549 INFO [Eventgen] Finished reload
2018-08-20 16:18:00.541     Splunk  _internal   2018-08-20 16:18:00,541 INFO [Eventgen] Finished parse
2018-08-20 16:18:00.541     Splunk  _internal   2018-08-20 16:18:00,541 INFO [Eventgen] Finished config parsing
2018-08-20 16:18:00.487     Splunk  _internal   2018-08-20 16:18:00,487 INFO [Eventgen] Config made Splunk Embedded
2018-08-20 16:18:00.487     Splunk  _internal   2018-08-20 16:18:00,487 INFO [Eventgen] Config object generated
2018-08-20 16:18:00.486     Splunk  _internal   2018-08-20 16:18:00,486 INFO [Eventgen] Eventgen object generated
2018-08-20 16:18:00.478     Splunk  _internal   2018-08-20 16:18:00,478 INFO [Eventgen] Prepared Config
2018-08-20 16:18:00.478     Splunk  _internal   2018-08-20 16:18:00,478 INFO [Eventgen] Input Config is: {'configuration': "{u'modinput_eventgen://default': {'name': u'modinput_eventgen://default', u'host': u'Splunk', u'disabled': u'0', u'VERBOSE': u'0', u'index': u'default'}}", 'checkpoint_dir': '/opt/splunk/var/lib/splunk/modinputs/modinput_eventgen', 'session_key': 'wv2kjziDCSHghZyvYGnSF519l41gzBCmd_euQyENd1P3eVfgMcOM^Lz8SMrmD63iRq_mWKt8NAX430ARnDQgfQGxvBpzyDlAX3PG^7sXEz9BB_E8U6ppQQC', 'server_uri': 'https://127.0.0.1:8089', 'server_host': 'Splunk'}
2018-08-20 16:18:00.478     Splunk  _internal   2018-08-20 16:18:00,478 INFO [Eventgen] Initialized streaming
2018-08-20 16:18:00.476     Splunk  _internal   2018-08-20 16:18:00,476 DEBUG [Eventgen] Setting up SA-Eventgen Modular Input
2018-08-20 16:18:00.475     Splunk  _internal   2018-08-20 16:18:00,475 DEBUG [Eventgen] Initialized ModularInput Logger
2018-08-20 16:18:00.000     Splunk  _internal   2018-08-20 16:18:00 eventgen INFO MainProcess Retrieving eventgen configurations from /configs/eventgen
2018-08-20 16:18:00.000     Splunk  _internal   2018-08-20 16:18:00 eventgen INFO MainProcess Logging Setup Complete. 

Two samples.
alt text

/opt/splunk/etc/apps/Sample_Data/local/eventgen.conf

[isilon_auth.csv]
mode = replay
timeMultiple = 1
backfill = -15m
sampletype = csv

outputMode = splunkstream
index = main
sourcetype = isilon:data
source = syslog
host = localhost
splunkHost = localhost
splunkUser = admin
splunkPass = password

token.0.token = \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}
token.0.replacementType = replaytimestamp
token.0.replacement = %Y-%m-%d %H:%M:%S


[Threats.sophos]
mode = replay
timeMultiple = 1
backfill = -15m
sampletype = raw

outputMode = splunkstream
index = Sophos
sourcetype = sophos:threats
source = eventgen
host = localhost
splunkHost = localhost
splunkUser = admin
splunkPass = password

token.0.token = \d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}
token.0.replacementType = replaytimestamp
token.0.replacement = %Y-%m-%d %H:%M:%S

App even populates the performance dashboard with one of the inputs but there is no actual data to search.

alt text

[UPDATE]

After changing the mode from "replay" to "sample", errors are not coming up anymore and we get the following being repeated after every 1 minute. Although still no data in Splunk.

2018-08-21 13:17:05     Splunk  _internal   2018-08-21 13:17:05 eventgen INFO MainProcess Worker# 0: Put -1 events in queue for sample 'isilon_auth.csv' with et '2018-08-21 13:17:05.360795' and lt '2018-08-21 13:17:05.360850'
2018-08-21 13:17:05     Splunk  _internal   2018-08-21 13:17:05 eventgen INFO MainProcess Start '1' generatorWorkers for sample 'isilon_auth.csv'
2018-08-21 13:17:05     Splunk  _internal   2018-08-21 13:17:05 eventgen INFO MainProcess Worker# 0: Put -1 events in queue for sample 'Threats.sophos' with et '2018-08-21 13:17:05.310390' and lt '2018-08-21 13:17:05.310451'
2018-08-21 13:17:05     Splunk  _internal   2018-08-21 13:17:05 eventgen INFO MainProcess Start '1' generatorWorkers for sample 'Threats.sophos'
2018-08-21 13:16:05     Splunk  _internal   2018-08-21 13:16:05 eventgen INFO MainProcess Worker# 0: Put -1 events in queue for sample 'isilon_auth.csv' with et '2018-08-21 13:16:05.293507' and lt '2018-08-21 13:16:05.293573'
2018-08-21 13:16:05     Splunk  _internal   2018-08-21 13:16:05 eventgen INFO MainProcess Start '1' generatorWorkers for sample 'isilon_auth.csv'
2018-08-21 13:16:05     Splunk  _internal   2018-08-21 13:16:05 eventgen INFO MainProcess Worker# 0: Put -1 events in queue for sample 'Threats.sophos' with et '2018-08-21 13:16:05.243097' and lt '2018-08-21 13:16:05.243143'
2018-08-21 13:16:05     Splunk  _internal   2018-08-21 13:16:05 eventgen INFO MainProcess Start '1' generatorWorkers for sample 'Threats.sophos' 

Thanks,

~ Abhi

0 Karma
Highlighted

Re: SA-Eventgen not generating any data.

Contributor

Update:

Roughly 2 minutes after getting the above logs, following errors have popped up.

2018-08-20 16:20:41.000     Splunk  _internal   2018-08-20 16:20:41 eventgen ERROR MainProcess string indices must be integers, not str Traceback (most recent call last): File "/opt/splunk/etc/apps/SA-Eventgen/lib/splunk_eventgen/eventgen_core.py", line 350, in _worker_do_work item.run() File "/opt/splunk/etc/apps/SA-Eventgen/lib/splunk_eventgen/lib/generatorplugin.py", line 144, in run self.gen(count=self.count, earliest=self.start_time, latest=self.end_time, samplename=self._sample.name) File "/opt/splunk/etc/apps/SA-Eventgen/lib/splunk_eventgen/lib/plugins/generator/replay.py", line 93, in gen current_event_timestamp = self._sample.getTSFromEvent(line[self._sample.timeField]) TypeError: string indices must be integers, not str

2018-08-20 16:20:41.000     Splunk  _internal   2018-08-20 16:20:41 eventgen ERROR MainProcess string indices must be integers, not str Traceback (most recent call last): File "/opt/splunk/etc/apps/SA-Eventgen/lib/splunk_eventgen/eventgen_core.py", line 350, in _worker_do_work item.run() File "/opt/splunk/etc/apps/SA-Eventgen/lib/splunk_eventgen/lib/generatorplugin.py", line 144, in run self.gen(count=self.count, earliest=self.start_time, latest=self.end_time, samplename=self._sample.name) File "/opt/splunk/etc/apps/SA-Eventgen/lib/splunk_eventgen/lib/plugins/generator/replay.py", line 93, in gen current_event_timestamp = self._sample.getTSFromEvent(line[self._sample.timeField]) TypeError: string indices must be integers, not str 
0 Karma
Highlighted

Re: SA-Eventgen not generating any data.

SplunkTrust
SplunkTrust

@abhijittikekar,

Can you please share Splunk Version & SA EventGen Version??

0 Karma
Highlighted

Re: SA-Eventgen not generating any data.

Contributor

Sure.

Splunk Version: 7.1.2

SA-EventGen: 6.2.1

Thanks,

~ Abhi

0 Karma
Highlighted

Re: SA-Eventgen not generating any data.

Engager

Has the issue been resolved now?

0 Karma
Highlighted

Re: SA-Eventgen not generating any data.

New Member

I am also facing the same issue. Please help.

0 Karma
Highlighted

Re: SA-Eventgen not generating any data.

New Member

Splunk Version: 7.2.0
SA-EventGen: 6.2.1

0 Karma
Highlighted

Re: SA-Eventgen not generating any data.

New Member

I am also facing the same issue. please help.

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.