All Apps and Add-ons

eventgen events stopped being indexed

splunk_zen
Builder

The event generation was flawlessly working for weeks but went fully quiet until a sole burst on yesterday at 4pm
The configuration file was not touched (generation frequency still the same) so what can cause the indexing to stop ?
Enabled the following but can only see the events going into the queue but nothing being indexed

debug = true
verbose = true

Taking a look into eventgen_main (there's nothing in eventgen_error )

2018-08-14 15:06:44 eventgen        INFO     MainProcess Start '2' generatorWorkers for sample 'test_sample.txt'
2018-08-14 15:06:44 eventgen        INFO     MainProcess Worker# 0: Put 50 events in queue for sample 'test_sample.txt' with et '2018-08-14 15:00:44.141400' and lt '2018-08-14 15:06:44.141451'
2018-08-14 15:06:44 eventgen        INFO     MainProcess Worker# 1: Put 50 events in queue for sample 'test_sample.txt' with et '2018-08-14 15:00:44.141400' and lt '2018-08-14 15:06:44.141451'
2018-08-14 15:06:44 eventgen        INFO     MainProcess Worker# 0: Put 50 events in queue for sample 'test_sample.txt' with et '2018-08-14 15:00:44.141400' and lt '2018-08-14 15:06:44.141451'
2018-08-14 15:06:44 eventgen        INFO     MainProcess Worker# 1: Put 50 events in queue for sample 'test_sample.txt' with et '2018-08-14 15:00:44.141400' and lt '2018-08-14 15:06:44.141451'
2018-08-14 15:12:44 eventgen        INFO     MainProcess Start '2' generatorWorkers for sample 'test_sample.txt'

Looking at splunkd,

08-14-2018 15:19:41.588 +0000 INFO  LicenseUsage - type=Usage s="/opt/splunk/var/log/splunk/test_service.log" st=test_service_log h="ip-172-31-36-143" o="" idx="default" i="EAE584D7-DBF7-4B6F-819B-36BAD9EEE258" pool="auto_generated_pool_enterprise" b=84 poolsz=53687091201
08-14-2018 15:19:41.588 +0000 INFO  LicenseUsage - type=Usage s="/opt/splunk/var/log/splunk/test_service.log" st=test_service_log h="ip-172-31-36-143" o="" idx="default" i="EAE584D7-DBF7-4B6F-819B-36BAD9EEE258" pool="auto_generated_pool_enterprise" b=84 poolsz=53687091201
08-14-2018 15:20:42.620 +0000 INFO  LicenseUsage - type=Usage s="/opt/splunk/var/log/splunk/test_service.log" st=test_service_log h="ip-172-31-36-143" o="" idx="default" i="EAE584D7-DBF7-4B6F-819B-36BAD9EEE258" pool="auto_generated_pool_enterprise" b=172203 poolsz=53687091201

Can you give me some pointers on what may be happening ?

kthammireddygar
Path Finder

if nothing changed on eventgen Config, it may be that there is no disk or disk full.

0 Karma

jkat54
SplunkTrust
SplunkTrust

search index=_internal log_level=warn* OR log_level=error see if that reveals more details.

If I had to guess, something could be sending everything to null queue.

Is it by chance "bursting" every time you restart splunk?

0 Karma
Get Updates on the Splunk Community!

New Year, New Changes for Splunk Certifications

As we embrace a new year, we’re making a small but important update to the Splunk Certification ...

Stay Connected: Your Guide to January Tech Talks, Office Hours, and Webinars!

What are Community Office Hours? Community Office Hours is an interactive 60-minute Zoom series where ...

[Puzzles] Solve, Learn, Repeat: Reprocessing XML into Fixed-Length Events

This challenge was first posted on Slack #puzzles channelFor a previous puzzle, I needed a set of fixed-length ...