All Apps and Add-ons

exporting all fields to CEF using real time output app

awurster
Contributor

hey guys -

i'm having trouble working with the real-time output app. i have specified a search, and it appears to be working / outputting data via CEF, however the field map i specified is being ignored. i am relying on this app based on reading i've done on integrating splunk with 3rd party SIEMs:

http://splunk-base.splunk.com/answers/13795/cef-output-to-arcsight-where-can-i-find-rtoutputpy
http://www.splunk.com/web_assets/pdfs/resources/Integrating_Splunk_with_Arcsight.pdf

i'm note sure its a syntax issue - as i'm not clear on any documentation at all for the app.

i am exporting proxy logs from a cisco WSA into CEF format. i don't see much need to rename the fields, mostly because i don't understand much about CEF or this app.

here is my resulting RT search:

[4bb1c423-0e96-4a27-9680-7cae8bceed2c]
description = export data to arcsight using TCP 514
disabled = 0
file_backups = 5
file_size = 5242880
label = SIEM output
mode = cef
search = index=qa_cisco_wsa sourcetype=wsa_accesslogs | eval cef_field_map="dvc_time:dvc_time,duration:duration,dvc_ip:dvc_ip,http_status:http_status,result:result,bytes_in:bytes_in,http_method:http_method,dest_url:dest_url,user_id:user_id,user_domain:user_domain,hierarchy_domain:hierarchy_domain,mime_type:mime_type,action:action,cause:cause,x_access_policy:x_access_policy,x_identity:x_identity,x_routing_policy:x_routing_policy,user_agent:user_agent"
splunk_port = 8089
syslog_facility = None
syslog_host = <foo>
syslog_port = 514
syslog_proto = tcp
target = syslog
splunk_host = localhost
syslog_level = 5

0 Karma

mlulmer
Explorer

I found that that the application performs CEF field validation. Please have a look at the following path: /etc/apps/SplunkRealTimeOutput/bin/real_time_output/cef
The file ceftool.py has the list of available supported CEF fields. I modified the Python and recompiled. This worked for me.

I've submitted a request to the SplunkRealTimeOutput developer to add all CEF fields.

Mark

0 Karma

btran
Explorer

is this crazy or what? where is the rtoutput.py file?

0 Karma

araitz
Splunk Employee
Splunk Employee
0 Karma

araitz
Splunk Employee
Splunk Employee

That is a bit bizarre 🙂

0 Karma

awurster
Contributor

and of course here is also the "new" search i added with a different GUID label in my realtime conf:

[4bb1c423-0e96-4a27-9680-7cae8bceed2c]
description = export data to arcsight using TCP 514
disabled = 0
file_backups = 5
file_size = 5242880
label = arcsight output
mode = cef
search = index=qa_cisco_wsa sourcetype=wsa_accesslogs | eval cef_field_map="dvc_time:dvc_time,duration:duration,dvc_ip:dvc_ip
...

0 Karma

awurster
Contributor

weird... after 2 days or so of adding / re-adding the same search query - it somehow started working! you can see a transition mid-stream:


<29> Jan 23 14:26:35 pxyau101mel0001.globaltest.anz.com CEF:0|Splunk|wsa_accesslogs|1.0|100000|generic event|5|rt=1358911573 dvc_time=1358911595.879 user_id=- user_agent="Mozilla/5.0 (Windows NT 5.1; rv:17.0) Gecko/20100101 Firefox/17.0" bytes_in=0 dvchost=test-cef hierarchy_domain=NONE/- dest_url=http://www.theage.com.au/?reload\=true http_method=GET result=TCP_DENIED http_status=407 duration=0 mime_type=- dvc_ip=10.220.114.143

0 Karma

awurster
Contributor

i'm not sure how to debug this either besides using a packet capture.

when i first installed it, it was exporting like 2-3 fields by default. once i modified the search... now it only appears to export "dvchost" which actually maps to the "host" in splunk (our proxy).

here's what the stream looks like:
<29> Jan 23 12:55:25 test-cef CEF:0|Splunk|wsa_accesslogs|1.0|100000|generic event|5|rt=1358906102 dvchost=test-cef
<29> Jan 23 12:55:25 test-cef CEF:0|Splunk|wsa_accesslogs|1.0|100000|generic event|5|rt=1358906102 dvchost=test-cef

0 Karma
Get Updates on the Splunk Community!

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...