All Apps and Add-ons

Splunk Add-on for ISC BIND: How to index a single data source and apply multiple sourcetypes in Hunk?

jwalzerpitt
Motivator

We use BlueCat for DNS/DHCP and we are forwarding the DNS/DHCP logs via CEF format to HDFS. I am trying to reverse engineer the Splunk Add-on for ISC BIND for Hunk, specifically the assigning of multiple sourcetypes to one index (assign events to their proper sourcetype - isc:bind:query, isc:bind:lameserver, isc:bind:network, isc:bind:transfer).

How would I go about configuring the props.conf file to assigning multiple sourcetypes to my source of BlueCat?

Thx

0 Karma

kschon_splunk
Splunk Employee
Splunk Employee

You should be able to do this with regular expressions. This page should help:
http://docs.splunk.com/Documentation/Splunk/6.4.3/Data/Advancedsourcetypeoverrides

0 Karma

jwalzerpitt
Motivator

I have my regexes ready to go, but do I list each sourcetype as follows in props.conf as:

[source::/xyz/dnslogs/...]
sourcetype = isc:bind:query

[isc:bind:query]
REPORT-1_extract_field = isc_bind_query_extract_field_0
EVAL-message_type = "Query"
EVAL-query_type = "Query"
EVAL-vendor_product = "ISC:Bind"
LOOKUP-2_look_up_extract = isc_bind_severities_lookup vendor_severity OUTPUT severity
LOOKUP-3_look_up_extract = isc_bind_category_lookup sourcetype OUTPUT vendor_category

transforms.conf
[isc_bind_query_extract_field_0]
REGEX = \s+client\s+([\w-.:]{1,100})#(\d{1,5})(?:\s+([^)]+))?:(?:\s+view\s+[^:]+:)?\s+query:\s+(?([\w-.:]{1,100}))?\s+([^\s]+)\s+([^\s]+)\s++-\s+(([\w-.:]{1,100}))$
FORMAT = vendor_severity::$1 src::$2 src_port::$3 query::$4 record_class::$5 record_type::$6 flag::$7 dest::$8

and so on?

[isc:bind:lameserver]
REPORT-1_extract_field = isc_bind_lameserver_extract_field_0
EVAL-app = "ISC:Bind"
EVAL-type = "alert"
LOOKUP-2_look_up_extract = isc_bind_severities_lookup vendor_severity OUTPUT severity
LOOKUP-3_look_up_extract = isc_bind_category_lookup sourcetype OUTPUT vendor_category

[source::/xyz/dnslogs/...]
sourcetype = isc:bind:lameserver

transforms.conf
[isc_bind_lameserver_extract_field_0]
REGEX = (?:\s+lame-servers:)?(?:\s+([^:]+):)?\s+(error\s+(([^)]+))\s+resolving\s+'([\w-.:]{1,100})/([^/]+)/([^']+)':\s+([\w-.:]{1,100})#(\d{1,5}))$
FORMAT = vendor_severity::$1 body::$2 error_type::$3 query::$4 record_type::$5 record_class::$6 dest::$7 dest_port::$8

Thx

0 Karma

kschon_splunk
Splunk Employee
Splunk Employee

(Take the following with a big grain of salt, as I haven't had a chance to test it myself)
I think what you want to do is something like the following. Choose one sourcetype to be the default for the index, i.e.:

[source::/xyz/dnslogs/...]
sourcetype = isc:bind:lameserver

Then for every other sourcetype you wish to assign, do the following:

transforms.conf
[disambiguate_isc_bind_query]
REGEX = \s+client\s+([\w-.:]{1,100})#(\d{1,5})(?:\s+([^)]+))?:(?:\s+view\s+[^:]+:)?\s+query:\s+(?([\w-.:]{1,100}))?\s+([^\s]+)\s+([^\s]+)\s++-\s+(([\w-.:]{1,100}))$
FORMAT = sourcetype:isc:bind:query
DEST_KEY = MetaData:Sourcetype

props.conf
[source::/xyz/dnslogs/...]

TRANSFORMS-isc_bind_query_transform = disambiguate_isc_bind_query

After combining all stanzas with the same header, your stanza in props should looks like:
[source::/xyz/dnslogs/...]

TRANSFORMS-isc_bind_query_transform = disambiguate_isc_bind_query
TRANSFORMS-isc_bind_network = disambiguate_isc_bind_network
......
......

0 Karma

ddrillic
Ultra Champion

In Hunk one assigns the sourcetype by the association to the file location in the HDFS.

Hunk - assigning sourcetype

Gives the example of -

[source::/user/xyz/ciscologfiles/...]
sourcetype = cisco_syslog

[source::/user/xyz/iislogfiles/...]
sourcetype = iis

Do you need to associate multiple sourcetypes for one source?

0 Karma

jwalzerpitt
Motivator

Yes, I would like to associate multiple sourcetypes to one source.

Thx

0 Karma
Get Updates on the Splunk Community!

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...

Five Subtly Different Ways of Adding Manual Instrumentation in Java

You can find the code of this example on GitHub here. Please feel free to star the repository to keep in ...

New Splunk APM Enhancements Help Troubleshoot Your MySQL and NoSQL Databases Faster

Splunk Observability has two new enhancements to make it quicker and easier to troubleshoot slow or frequently ...