How do I extract fields from data returned from HUNK?

[jwalzerpitt]

So we’re about to ingest Windows Event Logs to be queries via Hunk, and before doing so I loaded the Splunk add on for Windows app. It has an associated props.conf file.

How do I associate, or point the Windows Event Logs to that props.conf file so the events are parsed according to the props.conf file?

Thx,
Jeff

[jwalzerpitt]

Props.conf file attached as props.txt

[elin]

Are all the events coming from your virtual index the same type of event? If the add-on for windows has a sourcetype for the domain controller events coming from your virtual index, you could try setting the sourcetype for those events via the HDFS explorer.

[jwalzerpitt]

Props.conf file is as follows (we are ingesting the Windows event logs in XML format):

Copyright (C) 2005-2011 Splunk Inc. All Rights Reserved.

DO NOT EDIT THIS FILE!

Please make all changes to files in $SPLUNK_HOME/etc/apps/Splunk_TA_windows/local.

To make changes, copy the section/stanza you want to change from $SPLUNK_HOME/etc/apps/Splunk_TA_windows/default

into ../local and edit there.

DHCP

[source::....DhcpSrvLog]
sourcetype = DhcpSrvLog

[source::...\(DhcpSrvLog-)...]
sourcetype = DhcpSrvLog

[DhcpSrvLog]
SHOULD_LINEMERGE = false
REPORT-0auto_kv_for_microsoft_dhcp = auto_kv_for_microsoft_dhcp
REPORT-dest_for_microsoft_dhcp = dest_nt_host_as_dest,dest_mac_as_dest,dest_ip_as_dest
LOOKUP-signature_for_microsoft_dhcp = msdhcp_signature_lookup msdhcp_id OUTPUTNEW signature
LOOKUP-vendor_info_for_microsoft_dhcp = windows_vendor_info_lookup sourcetype OUTPUT vendor,product

Monitorware Windows Event Log

Apply the following properties to MonitorWare single-line text files (.monitorware)

[source::....monitorware]
SHOULD_LINEMERGE = false
TRANSFORMS-force_sourcetype_for_monitorware_txt = force_sourcetype_for_monitorware
TRANSFORMS-force_host_for_monitorware_txt = force_host_for_monitorware
TRANSFORMS-force_source_for_monitorware_txt = force_source_for_monitorware

Apply the following properties to incoming syslog data (udp/514)

Uncomment and modify the stanza ([source::udp:514]) below based on incoming MonitorWare data

[source::udp:514]

SHOULD_LINEMERGE = false

TRANSFORMS-force_sourcetype_for_monitorware_syslog = force_sourcetype_for_monitorware

TRANSFORMS-force_host_for_monitorware_syslog = force_host_for_monitorware

TRANSFORMS-force_source_for_monitorware_syslog = force_source_for_monitorware

Apply the following properties to all MonitorWare events

[source::MonitorWare...]

Using REPORT-0 to force alphanumeric precedence

REPORT-0kv_for_tab_monitorware = raw_kv_for_tab_monitorware,Message_kv_for_tab_monitorware

Using REPORT-1 to force alphanumeric precedence

REPORT-1Failure_Reason_for_monitorware = Failure_Reason_for_monitorware
REPORT-1User_for_monitorware = User_for_monitorware

NTSyslog Windows Event Log

Currently we only support NTSyslog:Security

[source::....ntsyslog]
SHOULD_LINEMERGE = false
TRANSFORMS-force_sourcetype_for_ntsyslog_txt = force_sourcetype_for_ntsyslog_security
TRANSFORMS-force_host_for_ntsyslog_txt = force_host_for_ntsyslog
TRANSFORMS-force_source_for_ntsyslog_txt = force_source_for_ntsyslog_security

Apply the following properties to incoming syslog data (udp/514)

Uncomment and modify the stanza ([source::udp:514]) below based on incoming NTSyslog data

[source::udp:514]

SHOULD_LINEMERGE = false

TRANSFORMS-force_sourcetype_for_ntsyslog_syslog = force_sourcetype_for_ntsyslog_security

TRANSFORMS-force_host_for_ntsyslog_syslog = force_host_for_ntsyslog

TRANSFORMS-force_source_for_ntsyslog_syslog = force_source_for_ntsyslog_security

Apply the following properties to NTsyslog window security event logs

[source::NTSyslog:Security]

Using REPORT-<0-2> to force alphanumeric precedence

Support for both verisions ([] and <>) of NTSyslog

REPORT-0raw_kv_for_ntsyslog = raw_kv_for_ntsyslog_square, raw_kv_for_ntsyslog_angle
REPORT-1message_kv_for_ntsyslog = message_kv_for_message_for_ntsyslog

Commenting in order to disable by default. If NTSyslog is used this should be enabled

LOOKUP-2action_EventCode_for_ntsyslog = ntsyslog_mappings NTSyslogID OUTPUTNEW action,EventCode,EventCode as signature_id

Snare Windows Event Log

Apply the following properties to Snare single-line text files (.snare)

[source::....snare]
SHOULD_LINEMERGE = false
TRANSFORMS-force_sourcetype_for_snare_txt = force_sourcetype_for_snare
TRANSFORMS-force_host_for_snare_txt = force_host_for_snare
TRANSFORMS-force_source_for_snare_txt = force_source_for_snare

Apply the following properties to incoming syslog data (udp/514)

Uncomment and modify the stanza ([source::udp:514]) below based on incoming Snare data

[source::udp:514]

SHOULD_LINEMERGE=false

TRANSFORMS-force_sourcetype_for_snare_syslog = force_sourcetype_for_snare

TRANSFORMS-force_host_for_snare_syslog = force_host_for_snare

TRANSFORMS-force_source_for_snare_syslog = force_source_for_snare

Apply the following properties to all Snare events

[source::Snare...]

Using REPORT-0 to force alphanumeric precedence

Support for both tab and comma delimitted Snare

Uncomment/Comment below based on Snare log type

REPORT-0kv_for_tab_snare = raw_kv_for_tab_snare,Message_kv_for_tab_snare

REPORT-0kv_for_comma_snare = raw_kv_for_comma_snare,Message_kv_for_comma_snare

Splunk Windows Event Log

Apply the following properties to Splunk multi-line text files (.windows)

[source::....windows]
SHOULD_LINEMERGE = false
LINE_BREAKER = (\r\n)
TRANSFORMS-force_sourcetype_for_windows_txt = force_sourcetype_for_windows_txt,force_sourcetype_application_sophos_for_windows_txt,force_sourcetype_application_sav_for_windows_txt,force_sourcetype_application_trendmicro_for_windows_txt,force_sourcetype_system_ias_for_windows_txt
TRANSFORMS-force_host_for_windows_txt = force_host_for_windows_txt
TRANSFORMS-force_source_for_windows_txt = force_source_for_windows_txt

windows eventlog modular input sourceing

[source::WinEventLog://*]
TRANSFORMS-force_source_for_wineventlog_modular = force_source_for_wineventlog_modular,force_sourcetype_system_ias_for_wineventlog

windows system sub-sourcetyping

[source::WinEventLog:System]
TRANSFORMS-force_sourcetype_system_ias_for_wineventlog = force_sourcetype_system_ias_for_wineventlog

Apply the following properties to all WinEventLog events

In addition to WinEventLog properties located in $SPLUNK_HOME/etc/system/default/props.conf

[source::(WMI:WinEventLog|WinEventLog)...]

Override default REPORT-MESSAGE with REPORT-0MESSAGE to force alphanumeric precedence

REPORT-0MESSAGE = wel-message, wel-eq-kv, wel-col-kv
REPORT-MESSAGE =

Windows XML Event Log

[(?::){0}XmlWinEventLog:*]
KV_MODE = none
REPORT-0xml_block_extract = system_xml_block,eventdata_xml_block,userdata_xml_block,debugdata_xml_block,renderinginfo_xml_block
REPORT-0xml_kv_extract = system_props_xml_kv,system_props_xml_attributes,eventdata_xml_data,rendering_info_xml_data

privilege

REPORT-0privilege_for_windows_security_xml= PrivilegeList_as_vendor_privilege

Extractions to add fields used by generic security extraction

REPORT-RecordNumber_from_xml = EventRecordID_as_RecordNumber
REPORT-EventCode_from_xml = EventID_as_EventCode
REPORT-Source_Port_from_xml = IpPort_as_Source_Port
REPORT-Token_Elevation_Type_from_xml = TokenElevationType_as_Token_Elevation_Type
REPORT-Target_Server_Name_from_xml = TargetServerName_as_Target_Server_Name
REPORT-Logon_Type_from_xml = LogonType_as_Logon_Type
REPORT-Logon_ID_from_xml = SubjectLogonId_as_Logon_ID
REPORT-Caller_Domain_from_xml = SubjectDomainName_as_Caller_Domain
REPORT-Target_Domain_from_xml = TargetDomainName_as_Target_Domain
REPORT-Caller_User_Name_from_xml = SubjectUserName_as_Caller_User_Name
REPORT-Target_User_Name_from_xml = TargetUserName_as_Target_User_Name
REPORT-Source_Workstation_from_xml = Workstation_as_Source_Workstation,WorkstationName_as_Source_Workstation,IpPort_as_Source_Workstation

Extractions to add fields used by generic system extraction

REPORT-signature_message_from_xml = updatelist_from_user_data
REPORT-signature_from_xml = updatetitle_from_user_data

All Windows Event Log

Apply the following properties to all Windows events

[source::(MonitorWare|NTSyslog|Snare|WinEventLog|WMI:WinEventLog)...]
LOOKUP-CategoryString_for_windows = windows_signature_lookup signature_id OUTPUTNEW CategoryString,action,result
FIELDALIAS-dvc_for_windows = host as dvc_nt_host,host as dvc
FIELDALIAS-event_id_for_windows = RecordNumber as event_id
FIELDALIAS-severity_for_windows = Type as severity
FIELDALIAS-severity_id_for_windows = EventType as severity
FIELDALIAS-id_for_windows = RecordNumber as id
REPORT-file_path-file_name_for_windows = file_path-file_name_for_windows

Attempt to map EventCodes that have sub statii ( i.e. EventCode=4625 + SubStatus=0xC0000064 = "User name does not exist" )

LOOKUP-signature_for_windows = windows_signature_lookup2 signature_id,Sub_Status OUTPUTNEW signature,signature as name, signature as subject

Default lookup for EventCode->signature mapping ( i.e. EventCode=4625 + SubStaus=null() = "An account failed to log on" )

LOOKUP-signature_for_windows3 = windows_signature_lookup signature_id OUTPUTNEW signature,signature as name, signature as subject

Since FIELDALIAS is destructive we need to preserve signature_id for certain SourceName values

EVAL-signature_id = if(SourceName="Microsoft-Windows-WindowsUpdateClient",signature_id,EventCode)

FIELDALIAS-user_group_id_for_windows = Primary_Group_ID as user_group_id

[elin]

It looks like the Windows TA expects the data to be named in separate files based on the type of windows event logs that the file contains. Is your data on HDFS also separated in similar fashion? If so, you should be able use the stanzas for the type of events you have, and replace the source with what they look like for your files in HDFS.

[elin]

Also, for reference, here's a link to docs on the HDFS explorer: http://docs.splunk.com/Documentation/Hunk/latest/Hunk/ExploreandconfigureHadoopsourcefiles

[jwalzerpitt]

When walking through Explore Data, how do I select a Windows sourcetype?

Before the Windows event logs, I created a virtual index for our Cisco ASA logs. For the Cisco ASA logs, I installed the Splunk Add-on for Cisco ASA and then walked through the 'Explore Data' HDFS explorer and select Cisco ASA as a sourcetype and the ASA fields are being extracted perfectly.

The props.conf/transforms.conf is confusing to me and I have yet to find a solid explanation, or event better, solid example on how to associate pre-defined props/transforms config files with log sources (and perhaps I have have yet to comes across that explanation/example).

Thx

[elin]

From the HDFS explorer, you should be able to select the windows event log sourcetypes from the pulldown in the uncategorized section - try using the filter box to type in your sourcetype.

There's more documentation on how props and transforms work in here: http://docs.splunk.com/Documentation/Splunk/6.2.5/Knowledge/Createandmaintainsearch-timefieldextract...

But generally, props.conf contains stanzas where the stanza name can be a number of things, including source:: and . So if you want to associate a source with an existing sourcetype, you can add a stanza for your new source. ex.

[source::...mylogsalllooklikethis.log.*]
sourcetype = WinEventLog:Application

[jwalzerpitt]

I am pulling the same type of event from the virtual index, Security (however, there are different types of Security events), but I'm not familiar with setting the sourcetype for the events via HDFS explorer.

Thx

[elin]

HDFS explorer allows you to browse through your files on HDFS for a virtual index via the UI. You can then set a sourcetype based on the source. If your security events are intermingled within a file, this might not work for you.

Can you provide a sample of the props.conf/transforms.conf?

[suarezry]

Sorry, I'm not understanding the question. Are you asking how to feed windows event logs to hadoop using the splunk windows app?

[jwalzerpitt]

No (and I apologize for the lack of clarity) - I have a virtual index set up that points to my domain controller events logs stored in HDFS. When I run a query against them, the fields (user, EventID, etc.) are not being parsed correctly.

How do I associate the Windows event logs to props.conf file from the Splunk add-on for Windows app so that the fields are extracted according to the props/transform config files of the app?

Thx

[suarezry]

Please supply a sample props.conf entry from the windows app that you wish to migrate to hunk, to give us a better understanding...

[jwalzerpitt]

What I did was copy the props.conf file (per the instructions within the props.conf file) from the add-on app at

/opt/hunk/etc/apps/Splunk_TA_windows/default/props.conf

to

/opt/hunk/etc/apps/search/local/Splunk_TA_windows.conf

Here is the first few lines from the props.conf file:

Copyright (C) 2005-2011 Splunk Inc. All Rights Reserved.

DO NOT EDIT THIS FILE!

Please make all changes to files in $SPLUNK_HOME/etc/apps/Splunk_TA_windows/local.

To make changes, copy the section/stanza you want to change from $SPLUNK_HOME/etc/apps/Splunk_TA_windows/default

into ../local and edit there.

DHCP

[source::....DhcpSrvLog]
sourcetype = DhcpSrvLog

[source::...\(DhcpSrvLog-)...]
sourcetype = DhcpSrvLog

[DhcpSrvLog]
SHOULD_LINEMERGE = false
REPORT-0auto_kv_for_microsoft_dhcp = auto_kv_for_microsoft_dhcp
REPORT-dest_for_microsoft_dhcp = dest_nt_host_as_dest,dest_mac_as_dest,dest_ip_as_dest
LOOKUP-signature_for_microsoft_dhcp = msdhcp_signature_lookup msdhcp_id OUTPUTNEW signature
LOOKUP-vendor_info_for_microsoft_dhcp = windows_vendor_info_lookup sourcetype OUTPUT vendor,product

Monitorware Windows Event Log

Apply the following properties to MonitorWare single-line text files (.monitorware)

[source::....monitorware]
SHOULD_LINEMERGE = false
TRANSFORMS-force_sourcetype_for_monitorware_txt = force_sourcetype_for_monitorware
TRANSFORMS-force_host_for_monitorware_txt = force_host_for_monitorware
TRANSFORMS-force_source_for_monitorware_txt = force_source_for_monitorware

Apply the following properties to incoming syslog data (udp/514)

Uncomment and modify the stanza ([source::udp:514]) below based on incoming MonitorWare data

[source::udp:514]

SHOULD_LINEMERGE = false

TRANSFORMS-force_sourcetype_for_monitorware_syslog = force_sourcetype_for_monitorware

TRANSFORMS-force_host_for_monitorware_syslog = force_host_for_monitorware

TRANSFORMS-force_source_for_monitorware_syslog = force_source_for_monitorware

Apply the following properties to all MonitorWare events

[source::MonitorWare...]

Using REPORT-0 to force alphanumeric precedence

REPORT-0kv_for_tab_monitorware = raw_kv_for_tab_monitorware,Message_kv_for_tab_monitorware

Using REPORT-1 to force alphanumeric precedence

REPORT-1Failure_Reason_for_monitorware = Failure_Reason_for_monitorware
REPORT-1User_for_monitorware = User_for_monitorware

NTSyslog Windows Event Log

Currently we only support NTSyslog:Security

[source::....ntsyslog]
SHOULD_LINEMERGE = false
TRANSFORMS-force_sourcetype_for_ntsyslog_txt = force_sourcetype_for_ntsyslog_security
TRANSFORMS-force_host_for_ntsyslog_txt = force_host_for_ntsyslog
[root@cdhhnk-prod-01 walzer]# cat /opt/hunk/etc/apps/search/local/Splunk_TA_windows.conf | more

Copyright (C) 2005-2011 Splunk Inc. All Rights Reserved.

DO NOT EDIT THIS FILE!

Please make all changes to files in $SPLUNK_HOME/etc/apps/Splunk_TA_windows/local.

To make changes, copy the section/stanza you want to change from $SPLUNK_HOME/etc/apps/Splunk_TA_windows/default

into ../local and edit there.

DHCP

[source::....DhcpSrvLog]
sourcetype = DhcpSrvLog

[source::...\(DhcpSrvLog-)...]
sourcetype = DhcpSrvLog

[DhcpSrvLog]
SHOULD_LINEMERGE = false
REPORT-0auto_kv_for_microsoft_dhcp = auto_kv_for_microsoft_dhcp
REPORT-dest_for_microsoft_dhcp = dest_nt_host_as_dest,dest_mac_as_dest,dest_ip_as_dest
LOOKUP-signature_for_microsoft_dhcp = msdhcp_signature_lookup msdhcp_id OUTPUTNEW signature
LOOKUP-vendor_info_for_microsoft_dhcp = windows_vendor_info_lookup sourcetype OUTPUT vendor,product

Monitorware Windows Event Log

Apply the following properties to MonitorWare single-line text files (.monitorware)

[source::....monitorware]
SHOULD_LINEMERGE = false
TRANSFORMS-force_sourcetype_for_monitorware_txt = force_sourcetype_for_monitorware
TRANSFORMS-force_host_for_monitorware_txt = force_host_for_monitorware
TRANSFORMS-force_source_for_monitorware_txt = force_source_for_monitorware

Apply the following properties to incoming syslog data (udp/514)

Uncomment and modify the stanza ([source::udp:514]) below based on incoming MonitorWare data

[source::udp:514]

SHOULD_LINEMERGE = false

TRANSFORMS-force_sourcetype_for_monitorware_syslog = force_sourcetype_for_monitorware

TRANSFORMS-force_host_for_monitorware_syslog = force_host_for_monitorware

TRANSFORMS-force_source_for_monitorware_syslog = force_source_for_monitorware

Apply the following properties to all MonitorWare events

[source::MonitorWare...]

Using REPORT-0 to force alphanumeric precedence

REPORT-0kv_for_tab_monitorware = raw_kv_for_tab_monitorware,Message_kv_for_tab_monitorware

Using REPORT-1 to force alphanumeric precedence

REPORT-1Failure_Reason_for_monitorware = Failure_Reason_for_monitorware
REPORT-1User_for_monitorware = User_for_monitorware

NTSyslog Windows Event Log

Currently we only support NTSyslog:Security

[source::....ntsyslog]
SHOULD_LINEMERGE = false
TRANSFORMS-force_sourcetype_for_ntsyslog_txt = force_sourcetype_for_ntsyslog_security
TRANSFORMS-force_host_for_ntsyslog_txt = force_host_for_ntsyslog
[root@cdhhnk-prod-01 walzer]# clear
[root@cdhhnk-prod-01 walzer]# cat /opt/hunk/etc/apps/search/local/Splunk_TA_windows.conf | more

Copyright (C) 2005-2011 Splunk Inc. All Rights Reserved.

DO NOT EDIT THIS FILE!

Please make all changes to files in $SPLUNK_HOME/etc/apps/Splunk_TA_windows/local.

To make changes, copy the section/stanza you want to change from $SPLUNK_HOME/etc/apps/Splunk_TA_windows/default

into ../local and edit there.

DHCP

[source::....DhcpSrvLog]
sourcetype = DhcpSrvLog

[source::...\(DhcpSrvLog-)...]
sourcetype = DhcpSrvLog

[DhcpSrvLog]
SHOULD_LINEMERGE = false
REPORT-0auto_kv_for_microsoft_dhcp = auto_kv_for_microsoft_dhcp
REPORT-dest_for_microsoft_dhcp = dest_nt_host_as_dest,dest_mac_as_dest,dest_ip_as_dest
LOOKUP-signature_for_microsoft_dhcp = msdhcp_signature_lookup msdhcp_id OUTPUTNEW signature
LOOKUP-vendor_info_for_microsoft_dhcp = windows_vendor_info_lookup sourcetype OUTPUT vendor,product

Monitorware Windows Event Log

Apply the following properties to MonitorWare single-line text files (.monitorware)

[source::....monitorware]
SHOULD_LINEMERGE = false
TRANSFORMS-force_sourcetype_for_monitorware_txt = force_sourcetype_for_monitorware
TRANSFORMS-force_host_for_monitorware_txt = force_host_for_monitorware
TRANSFORMS-force_source_for_monitorware_txt = force_source_for_monitorware

Apply the following properties to incoming syslog data (udp/514)

Uncomment and modify the stanza ([source::udp:514]) below based on incoming MonitorWare data

[source::udp:514]

SHOULD_LINEMERGE = false

TRANSFORMS-force_sourcetype_for_monitorware_syslog = force_sourcetype_for_monitorware

TRANSFORMS-force_host_for_monitorware_syslog = force_host_for_monitorware

TRANSFORMS-force_source_for_monitorware_syslog = force_source_for_monitorware

Apply the following properties to all MonitorWare events

[source::MonitorWare...]

Using REPORT-0 to force alphanumeric precedence

REPORT-0kv_for_tab_monitorware = raw_kv_for_tab_monitorware,Message_kv_for_tab_monitorware

Using REPORT-1 to force alphanumeric precedence

REPORT-1Failure_Reason_for_monitorware = Failure_Reason_for_monitorware
REPORT-1User_for_monitorware = User_for_monitorware

NTSyslog Windows Event Log

Currently we only support NTSyslog:Security

[source::....ntsyslog]
SHOULD_LINEMERGE = false
TRANSFORMS-force_sourcetype_for_ntsyslog_txt = force_sourcetype_for_ntsyslog_security
TRANSFORMS-force_host_for_ntsyslog_txt = force_host_for_ntsyslog

[suarezry]

/opt/hunk/etc/apps/search/local/Splunk_TA_windows.conf

That won't work. filename needs to be /opt/hunk/etc/apps/search/local/props.conf

Also you can't assume simply dropping the windows app props.conf file into the hunk app will work, there's too many variables. Most likely, the format that the winevents are stored as in HDFS are different than if the winevents were stored in a splunk indexer. I suggest you pick one props.conf stanza you want to test and copy it over to hunk and see what the results are.