How to disable the event coming from the source="rest://Solarwinds Nodes" ?

[Hemnaath]

Hi All, We have a request from a user to disable the events that are coming from the source="rest://Solarwinds Nodes". These events are extremely large and consume unnecessary disk space (every 5 minutes) and licensing. it appears to be a REST call originating on host1.

{[-]

{"results":[{"solarwinds_node_id":2,"polling_engine_id":12,"polling_engine":"VMTP01","solarwinds_prefix":"N:","src_ip":"10.X.X.X","host":"","percent_memory_used":66,"cpu_load":12,"up_since":"2016-03-02T16:52:00","host_tier":null},{"solarwinds_node_id":3,"p.....

]
}
Show as raw text

We are getting the events when we execute the below query in search head.
host=host1* source = rest://Solarwinds Nodes sourcetype = rest:solarwinds:nodes

Question :

1) How/where to disable this events temporally from the host machine, as I am not sure how to figure out from which source this events are being captured. that is from which inputs.conf file this source/sourcetype are configured? As I had searched the host1 machine for any configuration related to solar winds.

Kindly advise me how to identify from which inputs.conf file these source/sourcetype are being configured.

thanks in advance

[Hemnaath]

Hi Sierrax, I have found from how/where this events are getting into Splunk, I mean the exact inputs.conf location in the specific host by executing the splunk btool command and on finding the inputs.conf file, I added the disabled = I stanza and restarted the splunk services and this fixed my issue.

Solution :

1) executed ./splunk cmd btool inputs list rest -- > Path /opt/splunk/bin
2) On finding the location of the inputs.conf file added the disabled =1 stanza
path :/opt/splunk/etc/apps/search/local
[rest://Solarwinds Nodes]
auth_type = none
disable = 1
endpoint = https://ws.xxxx.com/sw/getnodes
http_method = GET
index = main
index_error_response_codes = 0
polling_interval = 300
response_type = json
sourcetype = rest:solarwinds:nodes
streaming_request = 0

3) Restarted the splunk service by executing ./splunk restart --> /opt/splunk/bin

thanks for sharing your trouble shooting skills.

[SierraX]

Hi,

a very interesting case... not sure I found the right solution:

I found in the props.conf a part called KEYS: there is a key queue with the possible Values nullQueue or indexQueue

I'm not sure, but I think it's :

[rest:solarwinds:nodes]

DEST_KEY = nullQueue

In a props.conf to solve this.

Kind Regards
SierraX

[Hemnaath]

thanks Sierrax, but not sure where I can find this configuration ? I mean whether we need to check the host machine where logs are getting generated ? Because when I run the below query we are getting events as show below

       host=host1* source = rest://Solarwinds Nodes sourcetype = rest:solarwinds:nodes  

Output
{[-]
results:[[+]
]
}
Show as raw text
And the moment when we hit the Shows as raw text it displays list of server details

"results":[{"solarwinds_node_id":2,"polling_engine_id":14,"polling_engine":"Test01","solarwinds_prefix":"N:","src_ip":"10.X.X.X","host":"","percent_memory_used":62,"cpu_load":11,"up_since":"2016-09-29T19:28:00","host_tier":null},{"solarwinds_node_id":3,"polling_engine_id":13,"polling_engine":"Test07","solarwinds_prefix":"N:","src_ip":"10..X.X.X","host":"","percent_memory_used":58,"cpu_load":11,"up_since":"2016-09-29T18:53:00","host_tier":null},{"solarwinds_node_id":4,"polling_engine_id":2,"polling_engine":"Test01","solarwinds_prefix":"N:","src_ip":"10.X.X.X","host":"","percent_memory_used":16,"cpu_load":6,"up_since":"2016-11-06T19:51:00","host_tier":null},

When checked with users they confirmed that the list of server are having Solar wind application running into it.

Question :

1) From the list of host machine Test01, under opt/SplunkUniversalforwarder/etc/apps/TA-Solarwinds/local/inputs.conf file are already disabled and also the above events are getting into default index main. And there is no Props or transform .conf file present under this folder.

Inputs.conf details
[WinEventLog://Microsoft-IIS-Configuration/Operational]
disabled = 1
ignoreOlderThan = 4d
index = win_srv

2) Whereas in host=host1 server (splunk--> license/deployment instances are running). We could see apps called TA-Solarwinds --> default --> under this folder there are props,transform,eventype,workflow_action.conf and app.conf files.

But not sure from how/where this events are getting into Splunk. so kindly guide me where exactly I need to trouble shoot this issue to disable the events getting into splunk.

thanks in advance.

[Hemnaath]

Hi Sierrax, thanks for your effort on this, I got doubt whether I need to add this below stanza into the props.conf, along with the stanza that are already present in the props.conf as mentioned above comments.

[rest:solarwinds:nodes]
DEST_KEY = nullQueue

thanks in advance.

[SierraX]

Ok again...

I'm not sure this is correct ... and I haven't the time to test it the next couple of days for free.

To test:
Build up a Test-Splunk Server in a Virtual Machine with a http-event collector and another Linux VM as sender. Send a few times events with this sourcetype from Linux VM to http-event collector to see it's running in... when this is working correct... write the 2-liner in a props.conf (e.g. in $SPLUNKHOME/search/local/props.conf) ... restart the server and look there are any error messages which are not there before...
Send a few times events with this sourcetype again, and see the source type is still running in or not.
When not check also the licensing, that the events are also not counted on your license.

Time to check this way... I think 1 or 2 hours.

[Hemnaath]

Hi Sierrax, Could please guide on this issue, as these events are taking unwanted space and huge licenses.
thanks in advance.

[SierraX]

At the moment I have not the time to build a similar setup to develop and test for Karma.

Possible immediate solutions for you:
- Learn how Splunk works and how to build an app.
- Ask Splunk or Solarwind for support.
- Hire a Splunk Consultant

[Hemnaath]

Hi All, Can any one guide me on the above problem as it consuming more licenses.

thanks in advance.