Trouble configuring the forwarder when integrating SCOM and Splunk

[abdulhasnath]

Hi,

RE: https://splunkbase.splunk.com/app/2729/

We have SCOM installed on a server, and have Splunk Enterprise installed on another server. I am trying to get them both to intergrate.
I have followed the guide from the above link, but I cant seem to configure the forwarder. It says that Universal forwarder is not supported, but then how do I configure a heavy forwarder or a search head on the server that SCOM is installed on?

Please advise.

Many thanks

Abdul

[jplumsdaine22]

Have you followed the instructions here?
http://docs.splunk.com/Documentation/AddOns/released/MSSCOM/Install

The heavy forwarder should be configured to forward to ip:9997, where ip is the ip address of your indexer, NOT the ip address of the heavy forwarder. Receiving should NOT be enabled on the heavy forwarder, but must be enabled on the indexer

On the heavy forwarder, in the scom TA Inputs section on you will need to select "Enable" for each input you wish to collect after you have edited its configuration. (see here for details: http://docs.splunk.com/Documentation/AddOns/released/MSSCOM/Configureinputs)

When editing the configuration on the the scom input on the heavy forwarder, leave the cron as default - if you want to modify it bear in mind that this is quartz cron syntax, NOT regular cron. But the default should be fine. You will need to specify an index and also a start date to collect the data.

The index that you specify on the heavy forwarder must be configured on the indexer before you enable the inputs.

You should also install the TA on your search head prior to enabling the inputs.

[abdulhasnath]

Thanks for your reply.

I have installed the heavy forwarder (Splunk Enterprise) on the same machine we run SCOM operations console. (Server A)
I have then installed Splunk Enterprise on our receiving machine. (Server B)

On Server A, I have configured the forwarder to IP:9997 where IP is Server B's address.
On Server B, I have enabled receiving on port 9997.

On Server A, when I use the addon to search it cant find anything.

I have followed all the guides. Can you advise more on the search head as this is something I haven't done?

In addition, according to the logs. This is the error message I am getting:

017-10-05 04:46:01 -04:00 [ log_level=ERROR pid=17432 input=_Splunk_TA_microsoft_scom_internal_used_Events ] New SCOMManagementGroupConnection Fail: The request was aborted: Could not create SSL/TLS secure channel.
at newSCOMManagementGroupConnection, C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-scom\bin\scom_command_loader.ps1: line 737
at run, C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-scom\bin\scom_command_loader.ps1: line 562
at , C:\Program Files\Splunk\etc\apps\Splunk_TA_microsoft-scom\bin\scom_command_loader.ps1: line 813
at , : line 1
at , : line 46
at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.GetResponse(WebRequest request)
at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.ProcessRecord()
2017-10-05 04:46:01 -04:00 [ log_level=WARN pid=17432 input=_Splunk_TA_microsoft_scom_internal_used_Events ] End SCOM TA

[jplumsdaine22]

OK - first up you need to use Server B to search. Forwarding means that no events will be kept on the heavy forwarder (Server A)

That error message suggets that the console cannot connect to SCOM. Have you tried using the scom console on server A to connect to SCOM ? ie outside of splunk

[abdulhasnath]

Thanks for your reply. SCOM operations console connects to our management server and pulls through alerts which we can see in Operations Console. That side of connectivity is all working. I am searching on Server B but cannot get any information, I am searching with both Search & Reporting and using the Data Summary. I have also tried installing the Splunk add-on on Server B and try to search with that. In addition, when I go to Add Data>Forward, there is no forwarders configured.

Is there a way I can check to see if the data is actually being forwarded? Other than enabling the reciving port on Server B and installing the addon, is there anything else that needs to be done here?

We just want to get a understanding of how the process works and what our data looks like in Splunk, but it seems to be proving a bit difficult.

Many thanks for your help

[jplumsdaine22]

This is a Splunk built app so you should be able to raise a support request. I recommend doing so if you are still having trouble. That said I'll do my best to help:

On server B you should be able to search for index=_internal host=ServerA. That will confim that forwarding between A and B (lets call these by their proper names, the Heavy Forwarder(HF) and Indexer)

If you see no results then you have probably not configured forwarding correctly. Make sure that the ip:port of the Indexer is configured on the HF . Make sure that the Indexer is listening on the same port that you specified on the HF

The data flow will be as follows. The Splunk instance on the HF will execute a series of powershell scripts (you can configure which ones in the scom TA) on a schedule. It will parse the data and forward it to the Indexer where it will be available to search (assuming you don't have another search head)

So if you see no results then the most likely issues will be:

HFs connection to scom (check the server configuration in the TA. Also check for any connection errors in the logs)
Heavy Forwarders connection to Indexer
Misconfigured indexes on the Indexer

Like I said I definitely recommend reaching out to support - that's what you're paying for!

[abdulhasnath]

I appreciate all your help jplumsdaine22.

After a long google search, I added:
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
to the scom_command_loader.ps1 file and the SSL issue went away.

I can now see data coming into my Splunk Enterprise, but this seems like information about alerts. I'm was expecting to be able to see information relating to the status of devices that are managed in SCOM .. is this something I would have to create a dashboard or visual display of the data we want to view?

[kmorris_splunk]

A heavy forwarder is the same install as Splunk Enterprise. In fact most components of a distributed Splunk architecture are. That components role is determined by the configuration. Here are the docs on deploying a heavy forwarder.

http://docs.splunk.com/Documentation/SplunkCloud/6.6.1/Forwarding/Deployaheavyforwarder

[abdulhasnath]

Thanks for your answers. So I have installed Splunk Enterprise on our server where Operations Console is installed. I have followed the guide for heavy forwarder but I am unable to get any data coming in. That said, how does SCOM actually send the data in? I haven't configured anything in SCOM to do this?

[jplumsdaine22]

It runs a series of custom powershell scripts (have a dig around in the bin folder of the splunk scom app if you want to see what it does)

Have you enabled the inputs through the scom TA on the heavy forwarder?

[jplumsdaine22]

Just to clarify you need to install the scom add on for splunk on the heavy forwarder

[abdulhasnath]

This is where I get confused, as I have installed the SCOM-ADD-ON on Splunk Enterprise (as Splunk Enterprise is the same install as heavy forwarder - see above). Therefore I am not sure what you mean when you say install the addon on the heavy forwarder. I have enabled the inputs within the addon as well as configuring the server address.

Within the Forwarding and Receiving section:
Configured the receiving port to be 9997
Configured forwarding to be the IP:9997 where IP is the machine address

From all of this, I dont see how SCOM sends info in also how does port 9997 relate.

Thanks for your help so far, really appreciate it!

[maciep]

you need a heavy forwarder because the add-on does more than a uf can do. But it doesn't need to be installed on a SCOM server. I believe you just need the SCOM console installed on the heavy forwarder, so the add-on can access the powershell cmdlets.

[abdulhasnath]

How would I do that? Is there any instructions I can follow? This is our first integration with Splunk.
Would this mean that Splunk Enterprise would have to be on the same server that we run SCOM on?

[jplumsdaine22]

https://technet.microsoft.com/en-us/library/hh298607(v=sc.12).aspx

[abdulhasnath]

Thanks for your reply, I have this already installed on a server. My question was does Splunk Enterprise have to be on the same server as where SCOM is installed? I understand that the Heavy forwarder is a full Splunk install. I am trying to avoid installing Splunk on the same machine as SCOM due to limited resources on that server to run both applications.

Therefore, is it possible to have:

(Server A) - where SCOM is installed

talk to

(Server B) - where Splunk Enterprise is installed

and if so how would I do that, as the heavy forwarder suggests both applications have to be on the same machine?

Many thanks

[maciep]

On Server B, you need to install the SCOM console. Just the console. Not a management server, just the console. The same thing you would install on your desktop so you could manage your SCOM env.

I believe the link that jplumsdaine22 provided shows you how to do that.

Does that make sense?