All Apps and Add-ons

Eventgen installation in a clustered environment

joelfourhman
New Member

Hey Splunkers

I'm trying to install and configure Eventgen in a distributed and clustered environment.

So far I have:
- installed the SA-Eventgen on the CM
- pushed this app (via master-apps) to all indexers
- added environment variables for the python scripts to reference.

export SPLUNK_HOME=/opt/splunk
export SPLUNK_DB=/opt/splunk/var/lib/splunk
export SPLUNK_ETC=/opt/splunk/etc/
export LD_LIBRARY_PATH=/opt/splunk/lib/

Now I am getting this message in the Splunk GUI, and I'm not sure why. I believe it has to do with /opt/bin/splunk/python struggling.

 Search peer Indexer1000 has the following message: Unable to initialize modular input "modinput_eventgen" defined inside the app "SA-Eventgen": Introspecting scheme=modinput_eventgen: script running failed (exited with code 1). 

Running /opt/splunk/etc/master-apps/SA-Eventgen/bin/modinput_eventgen.py manually gives me this stack trace.

[user@Indexer1000 bin]$ /opt/splunk/bin/python modinput_eventgen.py
Traceback (most recent call last):
  File "modinput_eventgen.py", line 13, in <module>
    from mod_input import ModularInput  # noqa isort:skip
  File "/opt/splunk/etc/apps/SA-Eventgen/lib/mod_input/__init__.py", line 30, in <module>
    if 'slave' in splunk.clilib.cli_common.getMergedConf('server').get('clustering', {}).get('mode', {}):
  File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli_common.py", line 267, in getMergedConf
    stdout = '%s' % getMergedConfRaw(confName) # how to make pylint believe it's a string
  File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli_common.py", line 320, in getMergedConfRaw
    return _get_conf_raw_internal(confName, ['btool', confName, 'list'])
  File "/opt/splunk/lib/python2.7/site-packages/splunk/clilib/cli_common.py", line 273, in _get_conf_raw_internal
    stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  File "/opt/splunk/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/opt/splunk/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

I'm out of ideas, does anyone have any insights on this?

It seems as if Eventgen really needs some massaging during configuration to get it to work in a distributed and clustered environment. Any and all tips are appreciated if you've done this.

Thanks!

0 Karma
1 Solution

adonio
SplunkTrust
SplunkTrust

i think its a terrible thing to do.
why would you want to generate events on all indexers?
why not generate events on a single splunk instance and send data to the indexers?

View solution in original post

0 Karma

adonio
SplunkTrust
SplunkTrust

i think its a terrible thing to do.
why would you want to generate events on all indexers?
why not generate events on a single splunk instance and send data to the indexers?

View solution in original post

0 Karma

joelfourhman
New Member

How would I go about doing that, if not on an indexer? I saw this:
https://answers.splunk.com/answers/489151/where-to-deploy-eventgen-in-a-distributed-deployme.html

So I put it on my indexers.

0 Karma

adonio
SplunkTrust
SplunkTrust

i think that answer is very wrong and working with moderators to fix that.
here is my recommendation:
install splunk on a small VM / separate machine
follow the eventgen instructions and install and verify data is there
create outputs.conf that sends the data to your indexers in the cluster
see the data in the cluster

0 Karma

joelfourhman
New Member

That makes sense, and we can definitely do that.
I'll do some research and see if that's the best path forward.

0 Karma

adonio
SplunkTrust
SplunkTrust

@joelfourhman, i added an elaborated answer on the link you provided and moderators will follow up and accept. thank you for pointing it out!
I am moving this thread from comment to an answer although it did not answered your question directly but pointed you to a better direction
if you feel the solution provided is helpful, kindly accept the answer so others will know it worked for you.

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!