topic Re: How to read logs in Tabbed text format in Getting Data In

How to read logs in Tabbed text format

eritzman — Tue, 21 May 2013 18:11:29 GMT

OK - I'm a NUB here and experimenting with SPLUNK.
I have some log files that are saved in a TAB/Columned format.
[example: Logs_2013-05-17.txt

RecID       TimeStamp               ClientIPAddress  ServerName                                         ServerIPAddress  ServerPort Method     URIStem                                            URIQuery                                                                                                                                                                                                                                                         Status UserName                                           URLRoot
26405679    2013-05-17 01:35:05.000 67.190.82.148    HOMESITE                                                192.168.100.109  443        GET        /sites/ter/_layouts/listfeed.aspx                  List=%7BD94F33B2%2DB09D%2D4EEA%2D8D14%2D0B8A1ACAC8B1%7D                                                                                                                                                                                                          401    NULL                                               home3.mysite.com
26536158    2013-05-17 00:00:41.000 174.255.208.31   HOME-WWW                                            192.168.100.89   80         GET        /styles/default.css                                NULL                                                                                                                                                                                                                                                             200    NULL                                               www.mysite.com

]

When I import it to Splunk I am not able to get it to recognize the fields and information correctly (EventBreaks)
How do I apply a "sourcetype" that will read these logs correctly?

Re: How to read logs in Tabbed text format

jkat54 — Tue, 21 May 2013 18:41:38 GMT

Splunk auto line breaks based on time stamps. You should first give the preview tool a try and see if you can figure it out there, but here's an example of a tab delimited input I have:

Props.conf on indexer:

 [SMTP]
 REPORT-smtp = getsmtpfields

Transforms.conf on indexer:

 [getsmtpfields]
 DELIMS = "\t"
 FIELDS = Type,Number,MSG_ID,Date,IP,MSG

Inputs.conf on universal forwarder:

 [monitor://c:\Program Files (x86)\hMailServer\Logs]
 disabled = 0
 host = hostname
 index = main
 sourcetype = SMTP
 crcSalt = <SOURCE>
 whitelist = \.log$

http://docs.splunk.com/Documentation/Splunk/5.0.2/admin/Transformsconf
http://docs.splunk.com/Documentation/Splunk/5.0.2/Admin/Propsconf

So yours might look like this:

Props.conf on indexer:

 [custominput]
 REPORT-custominput = custominputfields

Transforms.conf on indexer:

 [custominputfields]
 DELIMS = "\t"
 FIELDS = RecID,TimeStamp,ClientIPAddress,ServerName,ServerIPAddress,ServerPort,Method,URIStem,URIQuery,Status,UserName,URLRoot

Re: How to read logs in Tabbed text format

lguinn2 — Tue, 21 May 2013 18:46:45 GMT

The docs will help: Create source types

But here is one way to do what you want

inputs.conf

[monitor:///path/ Logs_*.txt]
sourcetype=mynewsourcetypename

props.conf

[mynewsourcetypename]
SHOULD_LINEMERGE=false
TRANSFORMS=extract_tabbed_fields

transforms.conf

[extract_tabbed_fields]
DELIMS="\t"
FIELDS=RecID,TimeStamp,ClientIPAddress,ServerName,ServerIPAddress,ServerPort Method,URIStem

HTH