topic Re: Hunk - assigning sourcetype in Getting Data In

Hunk - assigning sourcetype

jwalzerpitt — Wed, 26 Nov 2014 15:56:39 GMT

I create two virtual indexes within Hunk that reads from two separate HDFS directory. One is for Cisco ASA logs, and the other is for IIS logs. Each HDFS directory contains a bunch of *.log files. Clicking on 'search' for either index starts to index/read all of the log files, but the sourcetype is set wrong for both indexes.

How do I assign the correct sourcetype to each index?

Thx

Re: Hunk - assigning sourcetype

jwalzerpitt — Wed, 26 Nov 2014 16:37:55 GMT

After some additional review, for the IIS logs I see they're being tagged as a sourcetype of IIS, but they're not being parsed correctly. Any ideas on how to troubleshoot that issue?

The Cisco ASA logs aren't being identified as the correct sourcetype at all.

Re: Hunk - assigning sourcetype

rdagan_splunk — Wed, 26 Nov 2014 17:38:08 GMT

Try this option - in props.conf use the source:: and HDFS location :

Go to /hunk/etc/apps/search/local -> Create props.conf

[source::/user/xyz/ciscologfiles/...]
sourcetype = cisco_syslog

[source::/user/xyz/iislogfiles/...]
sourcetype = iis

Re: Hunk - assigning sourcetype

jwalzerpitt — Wed, 26 Nov 2014 18:36:55 GMT

Thx for the reply and info.

Created a props.conf file, which now reads:

[source::hdfs://hostname:8020/logs/firewall] sourcetype = cisco_syslog
[source::hdfs://hostname:8020/logs/web] sourcetype = iis

Restarted splunk and when I search on IIS or the ASA logs, they're still not parsing correctly

Re: Hunk - assigning sourcetype

Ledion_Bitincka — Wed, 26 Nov 2014 18:42:19 GMT

You need to remove the "hdfs://hostname:8020" part from the stanza, ie try replacing them with the following verbatim (no need to restart)

[source::/logs/firewall/...] 
sourcetype = cisco_syslog

[source::/logs/web/...] 
sourcetype = iis

Re: Hunk - assigning sourcetype

jwalzerpitt — Wed, 26 Nov 2014 18:53:22 GMT

Thx. Now reads:

[source::logs/firewall/...]
sourcetype = cisco_syslog

[source::logs/web/...]
sourcetype = iis

Still not parsing (not seeing source/dst IPs, and so on)

Re: Hunk - assigning sourcetype

Ledion_Bitincka — Wed, 26 Nov 2014 19:02:02 GMT

What are the events being sourcetyped as by Hunk?

Re: Hunk - assigning sourcetype

jwalzerpitt — Wed, 26 Nov 2014 19:09:42 GMT

The IIS logs are being tagged correctly (sourcetype = iis), where as the Cisco ASA logs have no sourcetype associated with them at all.

If I go the 'Explore Data' route and select sourcetype = iis the preview data screen shows the logs being parsed correctly, but once I save and then search the fields aren't parsed like they should be.

When I do 'Explore Data' re: Cisco, I set the sourcetype to cisco:asa, but preview data screen doesn't show the ASA logs being parsed as they should. I can switch between System Defaults, Syslog, and cisco:asa and the files never change how they're parsed.

Re: Hunk - assigning sourcetype

Ledion_Bitincka — Wed, 26 Nov 2014 19:51:01 GMT

Looking at your stanzas again you seem to be missing the leading / in both of them - can you please add that?

As for the iis logs being sourcetyped correctly but not being parsed properly, it seems like the root cause here is that iis log parsing uses index time rules which are not usable in Hunk - here's how the iis sourcetype is defined. Can you post the first few lines (anonimized) of the iis log files?

[iis]
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
INDEXED_EXTRACTIONS = w3c
detect_trailing_nulls = auto

Re: Hunk - assigning sourcetype

jwalzerpitt — Wed, 26 Nov 2014 20:06:51 GMT

Fixed props.conf to:

[source::/logs/firewall/...]
sourcetype = cisco_syslog

[source::/logs/web/...]
sourcetype = iis

[source::/logs/web/ex140401.log]
sourcetype = iis

Re: Hunk - assigning sourcetype

jwalzerpitt — Mon, 28 Sep 2020 18:16:00 GMT

anonimized iis log files:

2014-04-01 04:00:00 W3SVC1 x.x.x.x GET /server.txt - 443 - x.x.x.x - 200 0 0
2014-04-01 04:00:00 W3SVC1 x.x.x.x GET /server.txt - 80 - x.x.x.x - 200 0 0
2014-04-01 04:00:00 W3SVC1 x.x.x.x GET /server.txt - 80 - x.x.x.x - 200 0 0
2014-04-01 04:00:01 W3SVC1 x.x.x.x GET /dir_name/dir_name/dir_name/ImapRedirect.aspx - 443 - x.x.x.x Mozilla/5.0+(Linux;+U;+Android+4.0.4;+en-ca;+MB886+Build/7.7.1Q-115_MB886_BELL_FFW-11)+AppleWebKit/534.30+(KHTML,+like+Gecko)+Version/4.0+Mobile+Safari/534.30 200 0 0
2014-04-01 04:00:01 W3SVC1 x.x.x.x GET /favicon.ico - 443 - x.x.x.x Mozilla/5.0+(Linux;+U;+Android+4.0.4;+en-ca;+MB886+Build/7.7.1Q-115_MB886_BELL_FFW-11)+AppleWebKit/534.30+(KHTML,+like+Gecko)+Version/4.0+Mobile+Safari/534.30 404 0 2
2014-04-01 04:00:01 W3SVC1 x.x.x.x GET /server.txt - 443 - x.x.x.x - 200 0 0
2014-04-01 04:00:02 W3SVC1 x.x.x.x GET /dir_name/dir_name/dir_name/dir_name/Default.aspx - 443 - 96.235.28.93 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:28.0)+Gecko/20100101+Firefox/28.0 302 0 0
2014-04-01 04:00:04 W3SVC1 127.0.0.1 POST /dir_name/AuthProviderSoapBinding.asmx - 80 - 127.0.0.1 Plumtree+OpenHTTP+Library+(version+2.0) 200 0 0
2014-04-01 04:00:04 W3SVC1 x.x.x.x POST /portal/server.pt - 443 - 24.3.69.132 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:27.0)+Gecko/20100101+Firefox/27.0 302 0 0

Re: Hunk - assigning sourcetype

Ledion_Bitincka — Wed, 26 Nov 2014 20:29:49 GMT

Can you also please include the header line that contains the list of fields? Once you have the fields you should be able to configure parsing using props/transforms.conf to extract the fields

props.conf
[source::/logs/web/ex140401.log]
sourcetype = new-iis

[new-iis]
REPORT-manual-iis = manual-iis

transforms.conf
[manual-iis]
FIELDS = <comma delimited list of fields from file header>
DELIMS = " "

Re: Hunk - assigning sourcetype

jwalzerpitt — Wed, 26 Nov 2014 20:34:53 GMT

header line:

#Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query    s-port cs-username c-ip    cs(User-Agent) sc-status sc-substatus    sc-win32-status

Re: Hunk - assigning sourcetype

Ledion_Bitincka — Wed, 26 Nov 2014 20:43:21 GMT

Okay, so give this a try:

 props.conf
 [source::/logs/web/ex140401.log]
 sourcetype = new-iis

 [new-iis]
 REPORT-manual-iis = manual-iis

 transforms.conf
 [manual-iis]
 FIELDS = date, time, s-sitename, s-ip, cs-method, cs-uri-stem, cs-uri-query, s-port, cs-username, c-ip,  cs-user-agent, sc-status, sc-substatus,  sc-win32-status
 DELIMS = " " 
 # if the fields are tab delimited try the following
 #DELIMS = "\t"

Re: Hunk - assigning sourcetype

jwalzerpitt — Wed, 26 Nov 2014 20:55:48 GMT

props.conf now reads:

[source::/logs/firewall/...]
sourcetype = cisco_syslog

[source::/logs/web/...]
sourcetype = iis

[new-iis]
REPORT-manual-iis = manual-iis

[source::/logs/web/ex140401.log]
sourcetype = iis
(pleas note - this is added to the file when I do 'Explore Data')

Do I create the transforms.conf file in /hunk/etc/apps/search/local?

Thx for all of your help Ledion

Re: Hunk - assigning sourcetype

Ledion_Bitincka — Wed, 26 Nov 2014 20:57:33 GMT

Yes, both (props/transforms) go in $SPLUNK_HOME/etc/apps/search/local/

Re: Hunk - assigning sourcetype

jwalzerpitt — Wed, 26 Nov 2014 20:58:04 GMT

Thx - let me add and test again

Re: Hunk - assigning sourcetype

jwalzerpitt — Wed, 26 Nov 2014 21:08:56 GMT

props.conf:

[source::/logs/firewall/...]
sourcetype = cisco_syslog

[source::/logs/web/...]
sourcetype = iis

[new-iis]
REPORT-manual-iis = manual-iis

[source::/logs/web/ex140401.log]
sourcetype = iis

transforms.conf:

[manual-iis]
FIELDS = date, time, s-sitename, s-ip, cs-method, cs-uri-stem, cs-uri-query, s-port, cs-username, c-ip, cs-user-agent, sc-status, sc-substatus, sc-win32-status
DELIMS = "\t"

(I tried with DELIMS = " " as well)

Still not parsing. Anything to be gleaned from the search log?

Re: Hunk - assigning sourcetype

jwalzerpitt — Wed, 26 Nov 2014 21:10:45 GMT

And to be safe I restart Splunk after every change

Re: Hunk - assigning sourcetype

Ledion_Bitincka — Wed, 26 Nov 2014 21:38:17 GMT

Please setup the props.conf correctly as below, to wire up transforms correctly:

[source::/logs/web/ex140401.log]
sourcetype = manual-iis