exporting all fields to CEF using real time output app

[awurster]

hey guys -

i'm having trouble working with the real-time output app. i have specified a search, and it appears to be working / outputting data via CEF, however the field map i specified is being ignored. i am relying on this app based on reading i've done on integrating splunk with 3rd party SIEMs:

http://splunk-base.splunk.com/answers/13795/cef-output-to-arcsight-where-can-i-find-rtoutputpy
http://www.splunk.com/web_assets/pdfs/resources/Integrating_Splunk_with_Arcsight.pdf

i'm note sure its a syntax issue - as i'm not clear on any documentation at all for the app.

i am exporting proxy logs from a cisco WSA into CEF format. i don't see much need to rename the fields, mostly because i don't understand much about CEF or this app.

here is my resulting RT search:

[4bb1c423-0e96-4a27-9680-7cae8bceed2c]
description = export data to arcsight using TCP 514
disabled = 0
file_backups = 5
file_size = 5242880
label = SIEM output
mode = cef
search = index=qa_cisco_wsa sourcetype=wsa_accesslogs | eval cef_field_map="dvc_time:dvc_time,duration:duration,dvc_ip:dvc_ip,http_status:http_status,result:result,bytes_in:bytes_in,http_method:http_method,dest_url:dest_url,user_id:user_id,user_domain:user_domain,hierarchy_domain:hierarchy_domain,mime_type:mime_type,action:action,cause:cause,x_access_policy:x_access_policy,x_identity:x_identity,x_routing_policy:x_routing_policy,user_agent:user_agent"
splunk_port = 8089
syslog_facility = None
syslog_host = <foo>
syslog_port = 514
syslog_proto = tcp
target = syslog
splunk_host = localhost
syslog_level = 5


[mlulmer]

I found that that the application performs CEF field validation. Please have a look at the following path: /etc/apps/SplunkRealTimeOutput/bin/real_time_output/cef
The file ceftool.py has the list of available supported CEF fields. I modified the Python and recompiled. This worked for me.

I've submitted a request to the SplunkRealTimeOutput developer to add all CEF fields.

Mark

[btran]

is this crazy or what? where is the rtoutput.py file?

[araitz]

http://splunk-base.splunk.com/apps/48082/splunk-real-time-output

[araitz]

That is a bit bizarre 🙂

[awurster]

and of course here is also the "new" search i added with a different GUID label in my realtime conf:

[4bb1c423-0e96-4a27-9680-7cae8bceed2c]
description = export data to arcsight using TCP 514
disabled = 0
file_backups = 5
file_size = 5242880
label = arcsight output
mode = cef
search = index=qa_cisco_wsa sourcetype=wsa_accesslogs | eval cef_field_map="dvc_time:dvc_time,duration:duration,dvc_ip:dvc_ip
...


[awurster]

weird... after 2 days or so of adding / re-adding the same search query - it somehow started working! you can see a transition mid-stream:


<29> Jan 23 14:26:35 pxyau101mel0001.globaltest.anz.com CEF:0|Splunk|wsa_accesslogs|1.0|100000|generic event|5|rt=1358911573 dvc_time=1358911595.879 user_id=- user_agent="Mozilla/5.0 (Windows NT 5.1; rv:17.0) Gecko/20100101 Firefox/17.0" bytes_in=0 dvchost=test-cef hierarchy_domain=NONE/- dest_url=http://www.theage.com.au/?reload\=true http_method=GET result=TCP_DENIED http_status=407 duration=0 mime_type=- dvc_ip=10.220.114.143


[awurster]

i'm not sure how to debug this either besides using a packet capture.

when i first installed it, it was exporting like 2-3 fields by default. once i modified the search... now it only appears to export "dvchost" which actually maps to the "host" in splunk (our proxy).

here's what the stream looks like:
<29> Jan 23 12:55:25 test-cef CEF:0|Splunk|wsa_accesslogs|1.0|100000|generic event|5|rt=1358906102 dvchost=test-cef
<29> Jan 23 12:55:25 test-cef CEF:0|Splunk|wsa_accesslogs|1.0|100000|generic event|5|rt=1358906102 dvchost=test-cef