Can I send windows security logs using UF over HTTP to Logstash ?

[vikas_gopal]

Hello Experts , 

I am trying to send windows security logs to logstash(http) receiver . Below is what I have based on my understanding from below splunk document 

https://docs.splunk.com/Documentation/Forwarder/latest/Forwarder/Configureforwardingwithoutputs.conf....

On UF I have

inputs.conf

[WinEventLog://Security]
disabled = 0

outputs.conf

[httpout]
httpEventCollectorToken = <token>
uri = http://127.0.0.1:8002
compressed = false
sendCookedData = false
compression = none

my logstash.conf ( I want to write the data into a file)

input {
http {
port => 8002
codec => plain
}
}

output {
file {
path => "C:\logstash_output\uf_debug_raw.txt"

}
}

The file is being created but it holds encoded data like encrypted data , symbols . Can someone suggest if this is even possible 

data in the file 
{"url":{"domain":"127.0.0.1","port":8002,"path":"/services/collector/s2s"},"@version":"1","event":{"original":"�x��V�n\u001CE\u0010�`@���@\u001C�����%

[isoutamo]

Are you sure that your character sets are correctly defined? Based on your example it seems that you have at least UTF escaped characters and probably real UTF or some other in your file?

[vikas_gopal]

Thank you for your response , I have tried below but with that also same problem . 

codec => plain { charset => "UTF-8" }

codec => plain { charset => "UTF-16LE" }

[richgalloway]

When a UF sends data via HTTP it uses the Splunk-to-Splunk protocol, which logstash doesn't support.

[vikas_gopal]

That is what I wanted to confirm 🙂 . Do you have any suggestion what could be the other way to send logs using UF to logstash , I have tested TCP which is working but somehow it is sending splunk UF  internal logs too to logstash which I need to filter later at logstash level 

[richgalloway]

Whether via HTTP or TCP, the UF only sends data using the Splunk-to-Splunk protocol so cannot send successfully to Logstash.  I suggest using a Logstash agent, instead.

The sending of UF internal logs is a setting in an inputs.conf file.  Turning that off will not solve the above problem, however.

[vikas_gopal]

exactly , stopping internal logs at UF level does not work however at logstash level it worked . but yeah via HEC it is not possible it seems so far . Still waiting for others to respond may be we crack something amazing here collectively 🙂 . Thank you for response though 

[PickleRick]

You can make events generated by local inputs be sent to just one output group. But that will not be pretty.

You need to set _TCP_ROUTING key for each input stanza that you want to selectively manage. That means adding this to every single Splunk's own input. I'd just create a separate app and create inputs.conf in that app containing just this one setting per each input stanza.

EDIT: And one more thing - you cannot use both tcpout and httpout at the same time.

[PickleRick]

@richgalloway Shouldn't UF send raw data when sendCookedData=false on tcpout? Never tried it myself but the docs say so.

[vikas_gopal]

Well I was using this already as mentioned in my original post . 

[PickleRick]

You used httpout which doesn't use this option at all so I completely missed that.

[richgalloway]

Good find, @PickleRick !  The docs do imply one should set sendCookedData=false when sending to third-party systems.

@vikas_gopalPlease try that and report the results.

[PickleRick]

It is expected. By default Splunk sends all data to all output groups. You'd need to fiddle with event routing which can be tricky since UF normally doesn't do transforms.

[vikas_gopal]

so I tried this but end up with same problem 
UF--> HF(routing) --> LS( writing to a file) 

httpout is definitely not working/supported for logstash . 

[PickleRick]

It's not that httpout is not supported for logstash, it's that logstash cannot do s2s. 😉

Yes, it is confusing but despite sharing some of the low-level mechanics, s2s over http (which is httpout) has nothing to do with "normal HEC" .