topic Re: How to modify format of MS DNS server debug log events? in All Apps and Add-ons

How to modify format of MS DNS server debug log events?

tomasmoser — Tue, 29 Sep 2020 12:53:23 GMT

Hello,

I would like to modify format of MS DNS debug logs in order to get rid of some unimportant strings within domain names. I was playing with SEDCMD stanza in props.conf but not with success.

Log format as extracted by Splunk add-on for Microsoft DNS:
1. 2. 2017 20:19:22 0D80 PACKET 0000002548D040A0 UDP Rcv 10.17.81.32 7be7 Q [0001 D NOERROR] A (5)h42-m(3)sec(3)lab(0)

The problem is with (5)h42-m(3)sec(3)lab(0)"

I need to get events to look as follows:

2017 20:19:22 0D80 PACKET 0000002548D040A0 UDP Rcv 10.17.81.32 7be7 Q [0001 D NOERROR] A h42-m.sec.lab

When I implemented this ...
SEDCMD-remove_parens_num = s/((\d))/./g
SEDCMD-remove_first_period = s/^(.)//g
SEDCMD-remove_last_period = s/(.)$//g

... I stopped seeing my DNS logs in GUI permanently after the restart of Splunk. I do not understand. Any idea?

Tomas

Re: How to modify format of MS DNS server debug log events?

skoelpin — Tue, 14 Feb 2017 21:51:06 GMT

Try testing this at search time before modifying props.conf..

Try this

index=whatever | rex mode=sed s/($\d)$/./g

Re: How to modify format of MS DNS server debug log events?

tomasmoser — Tue, 29 Sep 2020 12:51:39 GMT

Yes, it does work correctly in this first stage.

2. 2017 23:07:08 0D7C PACKET 0000002549C0C0A0 UDP Snd 10.18.1.51 b1aa Q [0000 NOERROR] SRV ._ldap._tcp(23)Default-First-Site-Name._sites.dc._msdcs.develop3.develop2.develop.local.

Re: How to modify format of MS DNS server debug log events?

skoelpin — Tue, 14 Feb 2017 22:34:03 GMT

So you verified its working correctly at search time.. Do you want me to give you the SEDCMD so you can add it to your props.conf for index time now?

Re: How to modify format of MS DNS server debug log events?

tomasmoser — Tue, 14 Feb 2017 22:36:29 GMT

Yes. Please.

Re: How to modify format of MS DNS server debug log events?

skoelpin — Tue, 14 Feb 2017 23:37:56 GMT

Place this in your props.conf under $SPLUNK_HOME\etc\apps\#APP_NAME\local

[YourSourcetype]
SEDCMD-remove_parens = s/(\(\d)\)/./g

Don't forget to restart the Splunk service after making this change.

Lastly, if this works for you then please accept the answer

Re: How to modify format of MS DNS server debug log events?

tomasmoser — Wed, 15 Feb 2017 09:19:36 GMT

Well, it did help but I am not really happy. I ran into three problems:

Why "SEDCMD-remove1 = s/((\d))/./g" and not "SEDCMD-remove1 = s/((\d))/./g"? I am not getting logic. Seems it works the same.
Once I modified props.conf with SEDCMD above, all of a sudden I am not extracting any other fields during search time (as defined in default/props.conf) - ALL OTHER FIELDS VANISHED. I am getting just host, source, sourcetype.
adding "SEDCMD-remove-head-dot = s/\s(.)//g" into props.conf does not do anything (was working with rex in search bar)
adding "SEDCMD-remove-tail-dot = s/(.)$//g" into props.conf does not do anything (was warking with rex in search bar)

I do not understand Splunk's logic. Simply not.

Re: How to modify format of MS DNS server debug log events?

mikaelbje — Wed, 15 Feb 2017 09:47:43 GMT

Instead of recreating this by yourself, I believe the following add-on already does what you're trying to achieve: https://splunkbase.splunk.com/app/3377/

It's even CIM compliant, meaning the fields are normalized.

Re: How to modify format of MS DNS server debug log events?

tomasmoser — Wed, 15 Feb 2017 11:18:21 GMT

I can confirm now that skoelpin's solution works!

My props.conf

[MSAD:NT6:DNS]

Replace (3)www(6)google(3)com with www.google.com etc.

SEDCMD-remove-count = s/((\d+))/./g
SEDCMD-remove-head-dot = s/\s(.)//g
SEDCMD-remove-tail-dot = s/(.)$//g

Re: How to modify format of MS DNS server debug log events?

gdavid — Sun, 18 Feb 2018 20:24:42 GMT

thanks for the fix here. this should be added standard to the app. i don't know why its logged this way and not sure why splunk app wouldn't normalize the data field.

Re: How to modify format of MS DNS server debug log events?

gdavid — Mon, 19 Feb 2018 02:36:03 GMT

there are a lot of issues using the SEDCMD command to try to fix this. it applies to the whole string messing up other data in the DNS log.

i found issues with all the solutions suggested here, messing up PTR records or leaving a trailing . or leading (##) in the logs.

still searching for the fix here.

Re: How to modify format of MS DNS server debug log events?

bengoerz — Wed, 07 Mar 2018 21:45:13 GMT

Did you ever find a fix you liked?

I'm curious what problems you saw in PTR records? To me, looks like they have the same weird parenthetical-count formatting problem as A records, so this fix would be common to both.

Re: How to modify format of MS DNS server debug log events?

gdavid — Thu, 08 Mar 2018 23:05:45 GMT

each solution had its own trouble for me.

if you use SEDCMD-remove-count = s/((\d+))/./g
you turn a PTR record into this: (.).(.).(.).(.).(.)in-addr(.)arpa(.)

also if read it correct SEDCMD happens at index time so your logs are lost if there are errors here. to have this modify the entire string like this is less than ideal.

Re: How to modify format of MS DNS server debug log events?

gdavid — Thu, 08 Mar 2018 23:05:55 GMT

each solution had its own trouble for me.

if you use SEDCMD-remove-count = s/((\d+))/./g
you turn a PTR record into this: (.).(.).(.).(.).(.)in-addr(.)arpa(.)

also if read it correct SEDCMD happens at index time so your logs are lost if there are errors here. to have this modify the entire string like this is less than ideal.

Re: How to modify format of MS DNS server debug log events?

ssubhani — Mon, 26 Jul 2021 13:18:39 GMT

I saw this solution today after almost 4 years and it works for me too . However I do get a trailing dot OR a number before each .

Original

7/26/2021 4:14:02 PM 1498 PACKET 0000020027AD4070 UDP Rcv 10.38.2.63 e92f Q [0001 D NOERROR] A (5)ctldl(13)windowsupdate(3)com(0)

7/26/2021 4:14:02 PM 1498 PACKET 000002002D51DCC0 UDP Rcv 10.34.23.50 7494 Q [0001 D NOERROR] A (5)ctldl(13)windowsupdate(3)com(0)

7/26/2021 4:14:02 PM 1498 PACKET 0000020024F70D10 UDP Rcv 10.38.5.167 30e6 Q [0001 D NOERROR] A (4)pypi(3)org(0)

After applying Sedcmd ,notice the dot

7/26/2021 4:14:02 PM 1498 PACKET 0000020027AD4070 UDP Rcv 10.38.2.63 e92f Q [0001 D NOERROR] A .ctldl(13)windowsupdate.com.

7/26/2021 4:14:02 PM 1498 PACKET 000002002D51DCC0 UDP Rcv 10.34.23.50 7494 Q [0001 D NOERROR] A .ctldl(13)windowsupdate.com.

7/26/2021 4:14:02 PM 1498 PACKET 0000020024F70D10 UDP Rcv 10.38.5.167 30e6 Q [0001 D NOERROR] A .pypi.org.