Splunk Add-on for Technology Inventory: I think this stanza is pulling unrelated data.

[lycollicott]

I added this stanza to inputs.conf per https://splunkbase.splunk.com/app/3032/#/details, but every minute it is indexing a lot of Symantec related events which have no IP information in them.

[WinRegMon://IPAddresses]
index = windows
hive = HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Services\\Tcpip\d*\\Parameters\\Interfaces\\?.*
baseline = true
baseline_interval = 86400
proc = .*
type =

Here's what ends up in the index:

[sloshburch]

Yea...getting the Windows IPs is a hard one. We may switch that to using a scripted powershell approach or something, but for the interim, we're checking the Registry for those details and because of the variability of the registry, we have to capture all interfaces that match.

That said, the stanza itself shouldn't be running every minute. You might want to use btool to make sure there's no other config causing there to be such a fast interval. The baseline_interval is one day (in seconds). And the empty type should exclude activity registry from getting caught.

What do you get when your run:
"C:\Program Files\SplunkUniversalForwarder\bin\splunk.exe" btool inputs list WinRegMon --debug
Replace C:\Program Files\SplunkUniversalForwarder\ with wherever Splunk is installed on the host.

I have a feeling there's other stanzas that might be capturing that registry config. You can also validate by disabling the new stanza and see what is still collected.

What is the value you get in the Network Interfaces panel for IPs? Would you show that screen shot?

[lycollicott]

I had to read the documentation about baseline_interval in that stanza - OMG, can you imagine? - because I had never used it before and I mistakenly made some assumptions about how it worked. Once I did read and face-palmed, I just threw an actual interval in.

[sloshburch]

Phew! Glad it was an easy fix!

[lycollicott]

I was pulled into other things, so I'm a little late getting back to this, but it still is twitchy. If I disable the [WinRegMon://IPAddresses] stanza then nothing is collected for sourcetype WinRegistry, but when enabled it runs each minute.

PS C:\SplunkUniversalForwarder\bin> .\splunk btool inputs list WinRegMon  --debug
C:\SplunkUniversalForwarder\etc\system\default\inputs.conf                 [WinRegMon]
C:\SplunkUniversalForwarder\etc\system\default\inputs.conf                 baseline = 0
host = xxxx
index = default
C:\SplunkUniversalForwarder\etc\system\default\inputs.conf                 interval = 60
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   [WinRegMon://IPAddresses]
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   baseline = true
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   baseline_interval = 86400
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   hive = HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Services\\Tcpip\d*\\Parameters\\Interfaces\\?.*
host = xxxx
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   index = windows
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   interval = 86400
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   proc = .*
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   type =
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf [WinRegMon://default]
C:\SplunkUniversalForwarder\etc\system\default\inputs.conf                 baseline = 0
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf disabled = 1
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf hive = .*
host = xxxx
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf index = windows
C:\SplunkUniversalForwarder\etc\system\default\inputs.conf                 interval = 60
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf proc = .*
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf type = rename|set|delete|create
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf [WinRegMon://hkcu_run]
C:\SplunkUniversalForwarder\etc\system\default\inputs.conf                 baseline = 0
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf disabled = 1
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf hive = \\REGISTRY\\USER\\.*\\Software\\Microsoft\\Windows\\CurrentVersion\\Run\\.*
host = xxxx
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf index = windows
C:\SplunkUniversalForwarder\etc\system\default\inputs.conf                 interval = 60
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf proc = .*
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf type = set|create|delete|rename
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf [WinRegMon://hklm_run]
C:\SplunkUniversalForwarder\etc\system\default\inputs.conf                 baseline = 0
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf disabled = 1
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf hive = \\REGISTRY\\MACHINE\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Run\\.*
host = xxxx
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf index = windows
C:\SplunkUniversalForwarder\etc\system\default\inputs.conf                 interval = 60
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf proc = .*
C:\SplunkUniversalForwarder\etc\apps\Splunk_TA_windows\default\inputs.conf type = set|create|delete|rename
PS C:\SplunkUniversalForwarder\bin>

[sloshburch]

Start by removing/commenting out the interval = 86400 in the WinRegMon://IPAddresses stanza. I think you added that in through this discussion but we don't want it (notice it not listed on https://splunkbase.splunk.com/app/3032/#/details)

[lycollicott]

Really? It makes sense to me and it's valid syntax per the docs.

I want to give it the 24 hours to see what happens before I tweak it any more.

[sloshburch]

I'm not seeing it as valid syntax in the docs. Peek here and you'll see what I mean, there's no interval option in the WinRegMon section. I could be overlooking something so really please let me know. My thought was that the global interval option is not honored here.

More importantly, the baselin_interval is meant to take periodic snapshots so I'm not sure what the interval would do in this context - especially since there's nothing set for type.

[lycollicott]

etc\system\default\inputs.conf lists it within [WinRegMon].

[WinRegMon]
interval=60
baseline=0

Perhaps it doesn't matter within [WinRegMon:IPAddresses], because it doesn't work there.

Maybe Splunk_TA_windows\local\inputs.conf needs this, but I'm undecided :

[WinRegMon]
interval=86400
baseline=0

I think it might be better to nullqueue the Symantec items.

[sloshburch]

I hear ya. But just to sanity check and get a baseline, would you be cool with reversing the stanza to what is documented in the details tab? I totally agree that you can always do the nullqueue, but I'd love to uncover the root cause to make sure it's not indicative of a larger issue.

[lycollicott]

I already did that yesterday. The Windows team has a case open with Symantec to see if this activity is valid on their part.

What's that start up message? "See your world. Maybe wish you hadn't." 🙂

[sloshburch]

If it helps, I did a little research and according to the btool output you provided, I see that your system\default\inputs.conf has a WinRegMon stanza. That seems odd. I don't have any such stanza in my system\default\inputs.conf nor does any such stanza exist in the TA out of the box. Did you edit that file directly to add that in? (It could just be different renderings of btool over different splunk versions, but I figured I'd ask to be safe).

If you see the key path of the symantec items, it's clear that they are in fact in the correct path for us to capture as per our stanza (some software creates software-based network interfaces, which is fine). What doesn't make sense to me is why it's captured every minute. The only logical (or easily identifiable) reason for that would be an interval = 60 (since you said it happens every minute) or if the type attribute was capturing actions (not just baseline).

So let's start by pinpointing the odd WinRegMon stanza that starts your btool output to see if that addresses this.

[lycollicott]

We never added that stanza and I checked a recent 6.6.2 install and it was there as well.

I agree that is the probable cause of the interval, but I think this WinRegMon input has deeper issues. This search index=windows tinv_nic_ipv4=* over 7 days returns events for only 2 hosts out of 1000+ in that environment. If I run it for all time it returns events for over 1000.

[sloshburch]

Hang on. The search part makes sense. That's because the tinv_nic_ipv4 field is a calculated field looking for IPs while the registry stanza must collect all keys since the keyname for ipv4 ipv6 and other items are varying. In other words, the rarity of the tinv_nic_ipv4 in the data is by design. (Also, you have many other sourcetypes in your Windows index, I assume).

What is disturbing is the fact that the data was being collected every minute. The registry stuff should only be collected once a day with the described settings. That's what you'll want to focus on if that helps.
Would you elaborate on "We never added that stanza and I checked a recent 6.6.2 install and it was there as well." <- Are you saying you didn't add it in the default file but when you look at the conf file you see it in there?

[lycollicott]

Correct. We did not add that stanza.

[lycollicott]

I've been testing on one server.

I added the following stanza to c:\splunkuniversalforwarder\etc\apps\Splunk_TA_windows\local\inputs.conf:

[WinRegMon]
interval=86400

Despite what btool says below, it still ran every minute.

PS H:\> c:\splunkuniversalforwarder\bin\splunk btool inputs list WinRegMon --debug
c:\splunkuniversalforwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   [WinRegMon]
c:\splunkuniversalforwarder\etc\system\default\inputs.conf                                baseline = 0
c:\splunkuniversalforwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   interval = 86400
c:\splunkuniversalforwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   [WinRegMon://IPAddresses]
c:\splunkuniversalforwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   baseline = true
c:\splunkuniversalforwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   baseline_interval = 86400
c:\splunkuniversalforwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   hive = HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Services\\Tcpip\d*\\Parameters\\Interfaces\\?.*
c:\splunkuniversalforwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   index = windows
c:\splunkuniversalforwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   interval = 86400
c:\splunkuniversalforwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   proc = .*
c:\splunkuniversalforwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   type =

For comparison, here is a btool without that stanza.

PS H:\> c:\splunkuniversalforwarder\bin\splunk btool inputs list WinRegMon --debug
c:\splunkuniversalforwarder\etc\system\default\inputs.conf                               [WinRegMon]
c:\splunkuniversalforwarder\etc\system\default\inputs.conf                               baseline = 0
c:\splunkuniversalforwarder\etc\system\default\inputs.conf                                interval = 60
c:\splunkuniversalforwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   [WinRegMon://IPAddresses]
c:\splunkuniversalforwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   baseline = true
c:\splunkuniversalforwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   baseline_interval = 86400
c:\splunkuniversalforwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   hive = HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Services\\Tcpip\d*\\Parameters\\Interfaces\\?.*
c:\splunkuniversalforwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   index = windows
c:\splunkuniversalforwarder\etc\system\default\inputs.conf                                interval = 60
c:\splunkuniversalforwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   proc = .*
c:\splunkuniversalforwarder\etc\apps\Splunk_TA_windows\local\inputs.conf   type =

[sloshburch]

I was peeking at my data and I noticed that in my result set the only value I ever see for registry_type="baseline". In the screenshot you provided initially, we can see a value of CreateKey.
If you still see events for registry_type other than a baseline then there's certainly something still watching for all types.

I'm running 6.6.3 and when I btool I don't get any such [WinRegMon] stanza. I'm also not seeing any bugs within Splunk in regards to that feature.

Are these non baseline events also showing up for all Windows hosts (including the ones for 6.6.2) or do you see this behavior inconsistent? Also, do you still see events when you comment out or disable the WinRegMon://IPAddresses stanza?

Similarly, you might want to open a support ticket in regards to this feature of the forwarder. As you can infer, the Tech Add on itself is not doing anything other than using a feature of Splunk Forwarder. Support is better suited to conduct a webex and pinpoint the root cause live with you.

[lycollicott]

There have only been the following registry_type events on all hosts for that past 7 days:

1. CreateKey
2. DeleteKey
3. SetValue

There hasn't been a registry_type=baseline since July 30.

Disabling the stanza stops the nic collection from the registry everywhere.

baseline_interval is a minimum time between baselines, so this stanza seems to have created an initial baseline (despite and the only way to get the IP addresses is to search all the way back to that baseline.

I agree with you - this doesn't feel like the right way.

[sloshburch]

The fact that those three registry_types are showing up but not baseline is def odd. In my lab (and the desired behavior of this add on) is the opposite: you should see ONLY baseline and nothing else.

Would you open a case with support?

[sloshburch]

For what it's worth, I'm not really a huge fan of the registry approach to getting this data and very well may switch to writing a scripted input / powershell script to do it more effectively (and with less license usage).