topic Re: Using DELIMS with a string instead of single character in Getting Data In

Using DELIMS with a string instead of single character

restevan — Mon, 29 May 2017 18:37:36 GMT

Hi,

I'm forwarding from an proxy logs using NX-log and nxlog is using string "#011" to separate fields, like that:

May 29 20:28:55 PROXY - 10.147.23.21#011anonymous#011Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36#0112017-05-29#01118:28:44#011WBCNGTY02#011-#011login.splunk.com#01154.191.103.192#011443#0110#0117408#0112032#011SSL-tunnel#011-#011login.splunk.com:443#011-#011Inet#0110#011Navigate#011Req ID: 05543040; Compression: client=No, server=No, compress rate=0% decompress rate=0%#011Internal#011Internal#0110x8#011Allowed#011-#011-#011-#011-#011Allowed#011Malware Inspection Disabled for the Matching Policy Rule#011Unknown#011-#0110#011-#0110#011-#011-#011Feature disabled#011Web Proxy#011login.splunk.com#0119958#015

I've created in my transforms.conf the following:

  DELIMS = "\#"
    FIELDS = "c-ip","cs-username","c-agent","date","time","s-computername","cs-referred","r-host","r-ip","r-port","time-taken","sc-bytes","cs-bytes","cs-protocol","s-operation","cs-uri","cs-mime-type","s-object-source","sc-status","rule","FilterInfo","cs-network","sc-network","error-info","action","AuthenticationServer","NIS_scan_result","NIS_signature","ThreatName","MalwareInspectionAction","MalwareInspectionResult","UrlCategory","MalwareInspectionContentDeliveryMethod","MalwareInspectionDuration","MalwareInspectionThreatLevel","internal-service-info","NIS_application_protocol","NAT_address","UrlCategorizationReason","SessionType","UrlDestHost","s-port"

And it works but it's not ok because every field value has at its begining the string "011".

Is there a way to specify a field delimiter that is not a single character but a string?
I've tried the following

DELIMS = "\#011"
FIELDS = "c-ip","cs-username","c-agent","date","time","s-computername","cs-referred","r-host","r-ip","r-port","time-taken","sc-bytes","cs-bytes","cs-protocol","s-operation","cs-uri","cs-mime-type","s-object-source","sc-status","rule","FilterInfo","cs-network","sc-network","error-info","action","AuthenticationServer","NIS_scan_result","NIS_signature","ThreatName","MalwareInspectionAction","MalwareInspectionResult","UrlCategory","MalwareInspectionContentDeliveryMethod","MalwareInspectionDuration","MalwareInspectionThreatLevel","internal-service-info","NIS_application_protocol","NAT_address","UrlCategorizationReason","SessionType","UrlDestHost","s-port"

But that seems not to work.
May be there is another way of doing what I need.
Any help?

Thanks.

Re: Using DELIMS with a string instead of single character

jkat54 — Tue, 30 May 2017 02:16:05 GMT

I would change the #011 to pipes with sedcmd in props.

SEDCMD-fixdelims= s/#011/|/g

Then I would use pipes as the delims.

Or I would use REGEX and FORMAT in transforms.conf if you only want this to work at search time / not manipulate the data at all. However that will be a bit more challenging.

Re: Using DELIMS with a string instead of single character

jkat54 — Tue, 30 May 2017 02:22:19 GMT

You could do it like this but it's laborious:

 REGEX=(?<field1>.+)\#011(?<field2>.+)\#011....

Re: Using DELIMS with a string instead of single character

restevan — Tue, 30 May 2017 07:19:14 GMT

Hi @jkat54, that's exactly what I would like to avoid 🙂

Your suggestion worked fine! Just a little detail for the record and future references.
In prop.conf I had to put:

[my_new_sourcetype]
REPORT-TMG_over_nxlog=TMG_over_nxlog
SEDCMD-fixdelims= s/#011/|/g
SEDCMD-fixNXlogHeader= s/PROXY \-/PROXY |/g

The second SEDCMD is to reflect the header added by nxlog to each TMG log.
And in transform.conf that:

[TMG_over_nxlog]
DELIMS = "\|"
FIELDS = "nxlog-header","c-ip","cs-username","c-agent","date","time","s-computername","cs-referred","r-host","r-ip","r-port","time-taken","sc-bytes","cs-bytes","cs-protocol","s-operation","cs-uri","cs-mime-type","s-object-source","sc-status","rule","FilterInfo","cs-network","sc-network","error-info","action","AuthenticationServer","NIS_scan_result","NIS_signature","ThreatName","MalwareInspectionAction","MalwareInspectionResult","UrlCategory","MalwareInspectionContentDeliveryMethod","MalwareInspectionDuration","MalwareInspectionThreatLevel","internal-service-info","NIS_application_protocol","NAT_address","UrlCategorizationReason","SessionType","UrlDestHost","s-port"

And restart splunk.

Re: Using DELIMS with a string instead of single character

jkat54 — Tue, 30 May 2017 12:56:49 GMT

Sweet then! I love it when folks can take my suggestion and "run with it". If you feel I helped you solve the answer, then feel free to mark mine as the answer. Thanks!

Re: Using DELIMS with a string instead of single character

DalJeanis — Tue, 30 May 2017 13:33:32 GMT

Excellent!. @restevan - please mark answer as accepted.