Splunk Insights for Infrastructure - Data Logs

[ric878]

Hi, I recently installed Splunk Insights for Infrastructure on a virtual machine. After going through the setup process I proceeded to setup my first entity. I copied and pasted the generated script, modified it to ignore certificate errors and after a couple of minutes it was available in Splunk. Problem is that I'm only getting metrics data from collectd, but I do not see any log information.

I followed the troubleshooting information found here http://docs.splunk.com/Documentation/Infrastructure/1.0.1/Admin/Troubleshooting but was not able to resolve my issue. I can confirm that the Splunk forwarder is online, and that port 9997 is not blocked. On the SII machine, I see a repeated error in the splunk.log file:

06-11-2018 01:34:47.616 -0500 ERROR LMStack - Invalid License with infinite byte quota with non-infinite max_stack_quota byte, set the effective stack size to the min between them
06-11-2018 01:34:47.616 -0500 ERROR LMStack - Invalid License with infinite byte quota with non-infinite max_stack_quota byte, set the effective stack size to the min between them
06-11-2018 01:34:47.616 -0500 ERROR LMStack - Invalid License with infinite byte quota with non-infinite max_stack_quota byte, set the effective stack size to the min between them
06-11-2018 01:34:47.616 -0500 ERROR LMStack - Invalid License with infinite byte quota with non-infinite max_stack_quota byte, set the effective stack size to the min between them
06-11-2018 01:34:47.616 -0500 ERROR LMStack - Invalid License with infinite byte quota with non-infinite max_stack_quota byte, set the effective stack size to the min between them
06-11-2018 01:34:47.616 -0500 ERROR LMStack - Invalid License with infinite byte quota with non-infinite max_stack_quota byte, set the effective stack size to the min between them
06-11-2018 01:34:58.616 -0500 ERROR LMStack - Invalid License with infinite byte quota with non-infinite max_stack_quota byte, set the effective stack size to the min between them
06-11-2018 01:34:58.616 -0500 ERROR LMStack - Invalid License with infinite byte quota with non-infinite max_stack_quota byte, set the effective stack size to the min between them
06-11-2018 01:34:58.616 -0500 ERROR LMStack - Invalid License with infinite byte quota with non-infinite max_stack_quota byte, set the effective stack size to the min between them
06-11-2018 01:34:58.616 -0500 ERROR LMStack - Invalid License with infinite byte quota with non-infinite max_stack_quota byte, set the effective stack size to the min between them
06-11-2018 01:34:58.616 -0500 ERROR LMStack - Invalid License with infinite byte quota with non-infinite max_stack_quota byte, set the effective stack size to the min between them
06-11-2018 01:34:58.616 -0500 ERROR LMStack - Invalid License with infinite byte quota with non-infinite max_stack_quota byte, set the effective stack size to the min between them

I'm not sure if that is related at all. Any help would be greatly appreciated.

Thanks,
Ricardo

[degreeds]

Same problem - I can see logs in console but not in the interface
host count
srv-ad-1 55235
[root@srv-splunk bin]#,Same problem - I can see logs in console but not on the interface

host count
srv-ad-1 55235
[root@srv-splunk bin]#

[ric878]

Hi, does anyone have any suggestions?

[ntankersley_spl]

Was this installed on an instance with an existing splunk install?

Did you install any other licenses than the default license that came with the Splunk Insights for Infrastructure package?

[ric878]

Clean install on a new virtual machine. No other previous Splunk software. Should I just destroy the VM and try again?

[ntankersley_spl]

I don't think the errors for the license are associated. Let's check for log files on the instance. In the command line go to $SPLUNK_HOME/splunk/bin and run

./splunk search "index=main | stats count by host" and see what comes out. You should see the same host with a count of logs collected. If nothing returns then you aren't getting any data in and we'll have to try something different.

[ric878]

Okay, I ran the command and I do not see any hosts and log counts. It would seem that I am not getting any logs into the SII instance.

Also, just to be 100% sure, I deleted the VM and started over, re-added the machine I wanted to monitor and came up with the same results, I'm getting metrics but no logs. I also ran the command again and again did not see any hosts and log counts.

I've double check the firewall on the SII instance and confirmed that I have the following ports open:
8000
8088
8089
9997

Any other suggestions?
Thanks.

[ric878]

Hi,

After my attempted reinstall, I went back and ran the command again to double check and to my surprise, I see two hosts with index data. I still do not see logs in the interface but it would seem the SII instance is getting the log data. Maybe I tried to soon after adding a host. So, one step closer but still no log data visible in the web GUI.