All Apps and Add-ons

How to modify format of MS DNS server debug log events?

tomasmoser
Contributor

Hello,

I would like to modify format of MS DNS debug logs in order to get rid of some unimportant strings within domain names. I was playing with SEDCMD stanza in props.conf but not with success.

Log format as extracted by Splunk add-on for Microsoft DNS:
1. 2. 2017 20:19:22 0D80 PACKET 0000002548D040A0 UDP Rcv 10.17.81.32 7be7 Q [0001 D NOERROR] A (5)h42-m(3)sec(3)lab(0)

The problem is with (5)h42-m(3)sec(3)lab(0)"

I need to get events to look as follows:

2017 20:19:22 0D80 PACKET 0000002548D040A0 UDP Rcv 10.17.81.32 7be7 Q [0001 D NOERROR] A h42-m.sec.lab

When I implemented this ...
SEDCMD-remove_parens_num = s/((\d))/./g
SEDCMD-remove_first_period = s/^(.)//g
SEDCMD-remove_last_period = s/(.)$//g

... I stopped seeing my DNS logs in GUI permanently after the restart of Splunk. I do not understand. Any idea?

Tomas

0 Karma
1 Solution

skoelpin
SplunkTrust
SplunkTrust

Try testing this at search time before modifying props.conf..

Try this

index=whatever | rex mode=sed s/(\(\d)\)/./g

View solution in original post

gdavid
Path Finder

each solution had its own trouble for me.

if you use SEDCMD-remove-count = s/((\d+))/./g
you turn a PTR record into this: (.).(.).(.).(.).(.)in-addr(.)arpa(.)

also if read it correct SEDCMD happens at index time so your logs are lost if there are errors here. to have this modify the entire string like this is less than ideal.

0 Karma

gdavid
Path Finder

each solution had its own trouble for me.

if you use SEDCMD-remove-count = s/((\d+))/./g
you turn a PTR record into this: (.).(.).(.).(.).(.)in-addr(.)arpa(.)

also if read it correct SEDCMD happens at index time so your logs are lost if there are errors here. to have this modify the entire string like this is less than ideal.

0 Karma

gdavid
Path Finder

thanks for the fix here. this should be added standard to the app. i don't know why its logged this way and not sure why splunk app wouldn't normalize the data field.

0 Karma

gdavid
Path Finder

there are a lot of issues using the SEDCMD command to try to fix this. it applies to the whole string messing up other data in the DNS log.

i found issues with all the solutions suggested here, messing up PTR records or leaving a trailing . or leading (##) in the logs.

still searching for the fix here.

0 Karma

bengoerz
Explorer

Did you ever find a fix you liked?

I'm curious what problems you saw in PTR records? To me, looks like they have the same weird parenthetical-count formatting problem as A records, so this fix would be common to both.

0 Karma

mikaelbje
Motivator

Instead of recreating this by yourself, I believe the following add-on already does what you're trying to achieve: https://splunkbase.splunk.com/app/3377/

It's even CIM compliant, meaning the fields are normalized.

0 Karma

tomasmoser
Contributor

Well, it did help but I am not really happy. I ran into three problems:

  1. Why "SEDCMD-remove1 = s/((\d))/./g" and not "SEDCMD-remove1 = s/((\d))/./g"? I am not getting logic. Seems it works the same.

  2. Once I modified props.conf with SEDCMD above, all of a sudden I am not extracting any other fields during search time (as defined in default/props.conf) - ALL OTHER FIELDS VANISHED. I am getting just host, source, sourcetype.

  3. adding "SEDCMD-remove-head-dot = s/\s(.)//g" into props.conf does not do anything (was working with rex in search bar)

  4. adding "SEDCMD-remove-tail-dot = s/(.)$//g" into props.conf does not do anything (was warking with rex in search bar)

I do not understand Splunk's logic. Simply not.

0 Karma

tomasmoser
Contributor

I can confirm now that skoelpin's solution works!

My props.conf

[MSAD:NT6:DNS]

Replace (3)www(6)google(3)com with www.google.com etc.

SEDCMD-remove-count = s/((\d+))/./g
SEDCMD-remove-head-dot = s/\s(.)//g
SEDCMD-remove-tail-dot = s/(.)$//g

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Try testing this at search time before modifying props.conf..

Try this

index=whatever | rex mode=sed s/(\(\d)\)/./g

View solution in original post

ssubhani
Explorer

I saw this solution today after almost 4 years and it works for me too . However I do get a trailing dot OR a number before each .

Original

7/26/2021 4:14:02 PM 1498 PACKET 0000020027AD4070 UDP Rcv 10.38.2.63 e92f Q [0001 D NOERROR] A (5)ctldl(13)windowsupdate(3)com(0)

7/26/2021 4:14:02 PM 1498 PACKET 000002002D51DCC0 UDP Rcv 10.34.23.50 7494 Q [0001 D NOERROR] A (5)ctldl(13)windowsupdate(3)com(0)

7/26/2021 4:14:02 PM 1498 PACKET 0000020024F70D10 UDP Rcv 10.38.5.167 30e6 Q [0001 D NOERROR] A (4)pypi(3)org(0)

After applying Sedcmd ,notice the dot


7/26/2021 4:14:02 PM 1498 PACKET 0000020027AD4070 UDP Rcv 10.38.2.63 e92f Q [0001 D NOERROR] A .ctldl(13)windowsupdate.com.

7/26/2021 4:14:02 PM 1498 PACKET 000002002D51DCC0 UDP Rcv 10.34.23.50 7494 Q [0001 D NOERROR] A .ctldl(13)windowsupdate.com.

7/26/2021 4:14:02 PM 1498 PACKET 0000020024F70D10 UDP Rcv 10.38.5.167 30e6 Q [0001 D NOERROR] A .pypi.org.

 

Tags (1)
0 Karma

tomasmoser
Contributor

Yes, it does work correctly in this first stage.

  1. 2. 2017 23:07:08 0D7C PACKET 0000002549C0C0A0 UDP Snd 10.18.1.51 b1aa Q [0000 NOERROR] SRV ._ldap._tcp(23)Default-First-Site-Name._sites.dc._msdcs.develop3.develop2.develop.local.
0 Karma

skoelpin
SplunkTrust
SplunkTrust

So you verified its working correctly at search time.. Do you want me to give you the SEDCMD so you can add it to your props.conf for index time now?

0 Karma

tomasmoser
Contributor

Yes. Please.

0 Karma

skoelpin
SplunkTrust
SplunkTrust

Place this in your props.conf under $SPLUNK_HOME\etc\apps\#APP_NAME\local

[YourSourcetype]
SEDCMD-remove_parens = s/(\(\d)\)/./g

Don't forget to restart the Splunk service after making this change.

Lastly, if this works for you then please accept the answer

.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!