About vsilchev

vsilchev · ‎03-15-2017

Search index=_internal sourcetype="dbx*" error gives me events like this: 2017-03-15 13:58:05.813 +0400 3764@TestWin2012 [main] ERROR io.dropwizard.cli.ServerCommand - Unable to start server, shutting down java.lang.RuntimeException: java.net.BindException: Address already in use: bind at org.eclipse.jetty.setuid.SetUIDListener.lifeCycleStarting(SetUIDListener.java:213) ... So, it seems like DB Connect cannot start task server using port 9998 because this port number is already occupied. Changing port setting actually solves the problem. However, it is still unclear what could cause it. For me this problem appeared right after I upgraded DB Connect from 2.4 to 3.0.

vsilchev · ‎03-03-2017

As I understand, "RecordId" increases with every new record. If it is so, then you can use "Rising Column" input type. When creating/editing DB Input, on "Choose and Preview Table" step you should set Input Type to "Rising Column". Then you can choose "Automatic Mode" and just set the appropriate table or switch to editor and use the following SQL: SELECT * FROM "myDB"."dbo"."myDBAuditLog" WHERE RecordId > ? ORDER BY RecordId ASC Finally, don't forget to choose Rising Column ("RecordId" in your case). Then you can leave "Checkpoint Value" field empty if you want DBConnect to load all the records during the first request. Otherwise you can specify the Checkpoint Value to start with, so DBConnect will load only those records, which "RecordId" is greater then the Checkpoint Value. After that, every time DBConnect will use the Checkpoint Value for requesting only new records and update it using the greatest RecordId received from database.

vsilchev · ‎03-03-2017

In general, hostname-ip resolving with DHCP logs is like a "reference" example of Splunk time-based lookup. However... Assuming, that (for some reason) you cannot dump and store the data required for your lookup (e.g. DHCP leases), but you can somehow perform the lookup "at the moment" (for example, by running external command like "netsh"). In this case, I believe, you should create a scripted input that will collect/receive your events, perform the lookup (and probably additional event processing routines) and return extended event records to Splunk. Scripted inputs overview section of "Developing Views and Apps for Splunk Web" Manual could be a good starting point, I guess.

vsilchev · ‎01-31-2017

Well, after a bunch of experiments I came up with the following solution: If an event source sends events separated with null-character, then, most likely, Splunk (or something else) replaces null-characters with "\x00" string before applying line breaking & merging rules. In this case, one should add "\x00" pattern (attention to double backslash, because it is supposed to match "\x00" string, not the null-character) to LINE_BREAKER property in props.conf. For example: LINE_BREAKER = (\\x00)

vsilchev · ‎01-30-2017

First of all, thank you for your suggestions! Unfortunately, using "LINE_BREAKER = (?:\x00)" doesn't sovlse the problem, at least, in my case. I also thought about charset, but according to packet dump, it is definitely ASCII and Splunk recognizes it correctly, so it is not an issue, from my point of view. Taking into account, that "LINE_BREAKER = (\x00)" actually works, I suppose, that Splunk (or something else) replaces null-character with '\' 'x' '0' '0' char sequence before line breaking-merging happens.

vsilchev · ‎01-20-2017

My log source generates events ended with null-character ('\x00') and sends them to Splunk via TCP in chunks every 10 seconds. So, one TCP connection can contain several events, separated with null-character. However, my Splunk instance (on Microsoft Windows Server machine) doesn't recognize it as event separator and replaces it literally with "\x00" string. I've tried to use several regex patterns as line separator: LINE_BREAKER = ([\r\n\0]+) LINE_BREAKER = ([\r\n\00]+) LINE_BREAKER = ([\r\n\x00]+) LINE_BREAKER = ([\r\n]+)|(\0+) LINE_BREAKER = ([\r\n]+)|(\00+) LINE_BREAKER = ([\r\n]+)|(\x00+) ... but unfortunately they don't work for me. Currently I'm using '\x00' pattern to separate events, but it is definitely not a good solution. Any suggestions? UPD: Sample hex dump (from WireShark capture) of a tcp connection containing 4 events 0000 3c 31 34 3e 4a 61 6e 20 32 37 20 31 37 3a 35 39 <14>Jan 27 17:59 0010 3a 34 38 20 43 45 46 3a 30 7c 53 61 6d 70 6c 65 :48 CEF:0|Sample 0020 20 45 76 65 6e 74 7c 31 2e 30 7c 48 54 54 50 20 Event|1.0|HTTP 0030 54 72 61 6e 73 61 63 74 69 6f 6e 7c 31 20 72 65 Transaction|1 re 0040 71 4d 65 74 68 6f 64 3d 50 4f 53 54 20 75 73 65 qMethod=POST use 0050 72 41 67 65 6e 74 3d 4d 69 63 72 6f 73 6f 66 74 rAgent=Microsoft 0060 20 4f 66 66 69 63 65 2f 31 35 2e 30 20 28 57 69 Office/15.0 (Wi 0070 6e 64 6f 77 73 20 4e 54 20 36 2e 32 3b 20 4d 69 ndows NT 6.2; Mi 0080 63 72 6f 73 6f 66 74 20 4f 75 74 6c 6f 6f 6b 20 crosoft Outlook 0090 31 35 2e 30 2e 34 34 32 30 3b 20 50 72 6f 29 20 15.0.4420; Pro) 00a0 69 6e 3d 33 35 38 00 3c 31 34 3e 4a 61 6e 20 32 in=358.<14>Jan 2 00b0 37 20 31 37 3a 35 39 3a 34 38 20 43 45 46 3a 30 7 17:59:48 CEF:0 00c0 7c 53 61 6d 70 6c 65 20 45 76 65 6e 74 7c 31 2e |Sample Event|1. 00d0 30 7c 48 54 54 50 20 54 72 61 6e 73 61 63 74 69 0|HTTP Transacti 00e0 6f 6e 7c 31 20 72 65 71 4d 65 74 68 6f 64 3d 50 on|1 reqMethod=P 00f0 4f 53 54 20 75 73 65 72 41 67 65 6e 74 3d 4d 69 OST userAgent=Mi 0100 63 72 6f 73 6f 66 74 20 4f 66 66 69 63 65 2f 31 crosoft Office/1 0110 35 2e 30 20 28 57 69 6e 64 6f 77 73 20 4e 54 20 5.0 (Windows NT 0120 36 2e 32 3b 20 4d 69 63 72 6f 73 6f 66 74 20 4f 6.2; Microsoft O 0130 75 74 6c 6f 6f 6b 20 31 35 2e 30 2e 34 34 32 30 utlook 15.0.4420 0140 3b 20 50 72 6f 29 20 69 6e 3d 34 35 31 00 3c 31 ; Pro) in=451.<1 0150 34 3e 4a 61 6e 20 32 37 20 31 37 3a 35 39 3a 34 4>Jan 27 17:59:4 0160 38 20 43 45 46 3a 30 7c 53 61 6d 70 6c 65 20 45 8 CEF:0|Sample E 0170 76 65 6e 74 7c 31 2e 30 7c 48 54 54 50 20 54 72 vent|1.0|HTTP Tr 0180 61 6e 73 61 63 74 69 6f 6e 7c 31 20 72 65 71 4d ansaction|1 reqM 0190 65 74 68 6f 64 3d 50 4f 53 54 20 75 73 65 72 41 ethod=POST userA 01a0 67 65 6e 74 3d 4d 69 63 72 6f 73 6f 66 74 20 4f gent=Microsoft O 01b0 66 66 69 63 65 2f 31 35 2e 30 20 28 57 69 6e 64 ffice/15.0 (Wind 01c0 6f 77 73 20 4e 54 20 36 2e 32 3b 20 4d 69 63 72 ows NT 6.2; Micr 01d0 6f 73 6f 66 74 20 4f 75 74 6c 6f 6f 6b 20 31 35 osoft Outlook 15 01e0 2e 30 2e 34 34 32 30 3b 20 50 72 6f 29 20 69 6e .0.4420; Pro) in 01f0 3d 33 35 38 00 3c 31 34 3e 4a 61 6e 20 32 37 20 =358.<14>Jan 27 0200 31 37 3a 35 39 3a 34 38 20 43 45 46 3a 30 7c 53 17:59:48 CEF:0|S 0210 61 6d 70 6c 65 20 45 76 65 6e 74 7c 31 2e 30 7c ample Event|1.0| 0220 48 54 54 50 20 54 72 61 6e 73 61 63 74 69 6f 6e HTTP Transaction 0230 7c 31 20 72 65 71 4d 65 74 68 6f 64 3d 50 4f 53 |1 reqMethod=POS 0240 54 20 75 73 65 72 41 67 65 6e 74 3d 4d 69 63 72 T userAgent=Micr 0250 6f 73 6f 66 74 20 4f 66 66 69 63 65 2f 31 35 2e osoft Office/15. 0260 30 20 28 57 69 6e 64 6f 77 73 20 4e 54 20 36 2e 0 (Windows NT 6. 0270 32 3b 20 4d 69 63 72 6f 73 6f 66 74 20 4f 75 74 2; Microsoft Out 0280 6c 6f 6f 6b 20 31 35 2e 30 2e 34 34 32 30 3b 20 look 15.0.4420; 0290 50 72 6f 29 20 69 6e 3d 34 35 31 00 Pro) in=451. That's what I see in Splunk Web. <14>Jan 27 17:59:48 CEF:0|Sample Event|1.0|HTTP Transaction|1 reqMethod=POST userAgent=Microsoft Office/15.0 (Windows NT 6.2; Microsoft Outlook 15.0.4420; Pro) in=358\x00<14>Jan 27 17:59:48 CEF:0|Sample Event|1.0|HTTP Transaction|1 reqMethod=POST userAgent=Microsoft Office/15.0 (Windows NT 6.2; Microsoft Outlook 15.0.4420; Pro) in=451\x00<14>Jan 27 17:59:48 CEF:0|Sample Event|1.0|HTTP Transaction|1 reqMethod=POST userAgent=Microsoft Office/15.0 (Windows NT 6.2; Microsoft Outlook 15.0.4420; Pro) in=358\x00<14>Jan 27 17:59:48 CEF:0|Sample Event|1.0|HTTP Transaction|1 reqMethod=POST userAgent=Microsoft Office/15.0 (Windows NT 6.2; Microsoft Outlook 15.0.4420; Pro) in=451\x00

Posts	6
Solutions	2
Karma Given	6
Karma Received	8
Member Since	‎01-19-2017

Online Status	Offline
Date Last Visited	‎06-05-2020 02:04 AM

How to configure line breaking on TCP events separ...

Re: Splunk DB Connect: How to resolve "Can not com...

Re: Splunk DB Connect: How to configure the checkp...

Re: Is it possible to use a lookup at index time, ...

Re: How to configure line breaking on TCP events s...

Re: How to configure line breaking on TCP events s...

How to configure line breaking on TCP events separ...