I have ran across an issue that I've been banging my head against and it will not give.
I have a search that is trying to join to another search — easy enough. However, I seem to have some funky character at the end of some of my data that is messing up the join. For example, I have a search that produces this as the output:
name | value
vwilliams1 | 10000
I have a second search that LOOKS to be returning this output:
name | here
vwilliams1 | true
When I do a join on the name field it cannot join the two records. I have tried trim, rex sed commands, everything I can think of to get this to work. I have ran the len command and compared filed lengths, which show they are the same. I have even used substr(name,1,10) for both searches and that does not work. If I use substr(name,1,9) for both searches it will join, so it has to be something with that last character.
The only way I have been able to get the join to work is on the second search do this
|eval name= substr(name,1,9)
|strcat name "1" name
This obviously will not work for any other fields that don't have exactly 10 characters and end in a 1 so I cannot really use this solution. So I am back to trying to normalize the data in the second search to match the first search.
I have never came across a situation like this before and have used every trick I can think of to try and normalize this data. Are there any suggestions that people can give me on how to get this to work?
Thanks to all!
... View more
Thanks for the response adonio.
I think I have a much better understanding on all of this.
So about the system utilization. I finally got the answer from Splunk about this. They are very hesitant to say how enabling AD Monitoring will impact a DC, but the best answer I got is you can expect a 40-80Mb RSS on the memory side and 3-5% CPU utilization, depending on how busy your domain controller is.
Still have one question that seems inconsistent though. I have read that you need to run the Splunk Service with a Domain Admin account in order to capture all additions, changes, and deletions. If I just use the system account I would only get additions and change information. I have not consistently seen this though as an answer. Will the local SYSTEM account be adequate to run this or does the service need to run as a different user?
... View more
Hoping to get some clear guidance here.
What we are trying to do is to monitor Active Directory GPO changes and eventually Windows DNS. We currently have the universal forwarder installed on 80+ DCs collecting windows events and have the appropriate auditing enabled.
I have been reading a lot about Splunk App for Windows Infrastructure and AD Monitoring which seem to be what we need, but I am still unclear on several things. I see that Splunk App for Windows Infrastructure is supported in Splunk Cloud, but the app looks to be actually unavailable. Does that mean we just get the data and we have to make our own dashboards or do we need specific lookup tables or .conf files that have to be on the cloud search head to use this data?
That app requires several other apps to be installed on the forwarders. Splunk Add-on for Windows, Splunk Add-on for MS Active Directory, and we have future needs for Splunk Add-on for Windows DNS. Have no idea if this needs to be installed on every DC or just one. Also does anyone know the additional impact to the system if these apps are installed?
Then we have AD Monitoring. Not entirely sure if this is needed, or is this all I need. Also never have seen any directions on how to get this to work from just a forwarder, but in the description from the Splunk documents it should be possible.
Basically I am not sure just what I need installed where and how do I get this data in a usable format to Splunk Cloud.
Can anyone offer any helpful suggestions?
... View more
We have a setup where we have a syslog-ng server that forwards all events using a UF to a HF and then to the cloud. The issue we are having is that the host information is getting replaced with that of the UF name not the actual host that sent the syslog.
I don't have anything in the outputs.conf or inputs.conf on the UF setting the host. If I send directly to Splunk Cloud it will keep the correct host name. It is only when I send to the HF will this name get stripped and the host gets changed to the syslog server's name.
I have tried a regex to dynamically assign the host name in the inputs.conf by way of a regex based on the file path name on the UF, but cannot get it to work. An example of the file path is /var/log/splunk/network/hostname_log. I need just the hostname to be come the host.
My thought is that there must be a setting somewhere either on the UF or the HF that is doing this. Any ideas or is there another way of doing the.
... View more
We have recently moved from having an internal hosted Splunk setup to Splunk Cloud. Before the move to the Cloud all of our logs sent to syslog-ng kept the correct host information. Now though we have lost this and only see the syslog server as the host. All of the syslog-ng settings are the same, but the change is that we have added a Heavy Forwarder to act as a central spot to send all logs through.
We really would like to keep the original host information if we could. Is this possible with this setup and if so, what config needs to change on the Heavy Forwarder?
... View more
We have been having an issue with the Cisco IOS Add-on installed on a search head returning logs from a specific router. When we do any search that returns results from this one router, we get the error: "Streamed search execute failed because: vector::_M_range_check" from all indexers.
The search is simple: sourcetype="cisco:ios" over some time period that contains data from the device.
The problem though does not seem to be the indexers. Searches work with no errors once again if we disable the Cisco Networks Add-on for Splunk Enterprise on the search head. The documentation from the Cisco Networks App for Splunk Enterprise says it needs this add-on installed on all indexers and search heads. We have done that.
The router in question is a Cisco WS-C4500X-32.
Here is an example of the raw syslog data from this router that is causing issues:
2016 Jun 1 11:25:13 -04:00 192.168.64.1 Jun 1 07:25:09.822 PST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 10.123.81.10 (Vlan412) is up: new adjacency
2016 Jun 1 11:25:13 -04:00 192.168.64.1 Jun 1 07:25:09.948 PST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 10.123.81.6 (Vlan411) is up: new adjacency
2016 Jun 1 11:25:14 -04:00 192.168.64.1 Jun 1 07:25:11.245 PST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 10.123.97.17 (Port-channel82) is up: new adjacency
2016 Jun 1 11:25:14 -04:00 192.168.64.1 Jun 1 07:25:11.330 PST: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 10.123.97.13 (Port-channel81) is up: new adjacency
Any thoughts or ideas about this one or directions to help troubleshoot this?
... View more