Getting Data In

Hunk - assigning sourcetype

jwalzerpitt
Motivator

I create two virtual indexes within Hunk that reads from two separate HDFS directory. One is for Cisco ASA logs, and the other is for IIS logs. Each HDFS directory contains a bunch of *.log files. Clicking on 'search' for either index starts to index/read all of the log files, but the sourcetype is set wrong for both indexes.

How do I assign the correct sourcetype to each index?

Thx

Tags (3)
0 Karma

rdagan_splunk
Splunk Employee
Splunk Employee

Try this option - in props.conf use the source:: and HDFS location :

Go to /hunk/etc/apps/search/local -> Create props.conf

[source::/user/xyz/ciscologfiles/...]
sourcetype = cisco_syslog

[source::/user/xyz/iislogfiles/...]
sourcetype = iis

jwalzerpitt
Motivator

Thx for the reply and info.

Created a props.conf file, which now reads:

[source::hdfs://hostname:8020/logs/firewall] sourcetype = cisco_syslog
[source::hdfs://hostname:8020/logs/web] sourcetype = iis

Restarted splunk and when I search on IIS or the ASA logs, they're still not parsing correctly

0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

You need to remove the "hdfs://hostname:8020" part from the stanza, ie try replacing them with the following verbatim (no need to restart)

[source::/logs/firewall/...] 
sourcetype = cisco_syslog

[source::/logs/web/...] 
sourcetype = iis
0 Karma

jwalzerpitt
Motivator

Thx. Now reads:

[source::logs/firewall/...]
sourcetype = cisco_syslog

[source::logs/web/...]
sourcetype = iis

Still not parsing (not seeing source/dst IPs, and so on)

0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

What are the events being sourcetyped as by Hunk?

0 Karma

jwalzerpitt
Motivator

The IIS logs are being tagged correctly (sourcetype = iis), where as the Cisco ASA logs have no sourcetype associated with them at all.

If I go the 'Explore Data' route and select sourcetype = iis the preview data screen shows the logs being parsed correctly, but once I save and then search the fields aren't parsed like they should be.

When I do 'Explore Data' re: Cisco, I set the sourcetype to cisco:asa, but preview data screen doesn't show the ASA logs being parsed as they should. I can switch between System Defaults, Syslog, and cisco:asa and the files never change how they're parsed.

0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

Looking at your stanzas again you seem to be missing the leading / in both of them - can you please add that?

As for the iis logs being sourcetyped correctly but not being parsed properly, it seems like the root cause here is that iis log parsing uses index time rules which are not usable in Hunk - here's how the iis sourcetype is defined. Can you post the first few lines (anonimized) of the iis log files?

[iis]
pulldown_type = true
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = False
INDEXED_EXTRACTIONS = w3c
detect_trailing_nulls = auto
0 Karma

jwalzerpitt
Motivator

Fixed props.conf to:

[source::/logs/firewall/...]
sourcetype = cisco_syslog

[source::/logs/web/...]
sourcetype = iis

[source::/logs/web/ex140401.log]
sourcetype = iis

0 Karma

jwalzerpitt
Motivator

anonimized iis log files:

2014-04-01 04:00:00 W3SVC1 x.x.x.x GET /server.txt - 443 - x.x.x.x - 200 0 0
2014-04-01 04:00:00 W3SVC1 x.x.x.x GET /server.txt - 80 - x.x.x.x - 200 0 0
2014-04-01 04:00:00 W3SVC1 x.x.x.x GET /server.txt - 80 - x.x.x.x - 200 0 0
2014-04-01 04:00:01 W3SVC1 x.x.x.x GET /dir_name/dir_name/dir_name/ImapRedirect.aspx - 443 - x.x.x.x Mozilla/5.0+(Linux;+U;+Android+4.0.4;+en-ca;+MB886+Build/7.7.1Q-115_MB886_BELL_FFW-11)+AppleWebKit/534.30+(KHTML,+like+Gecko)+Version/4.0+Mobile+Safari/534.30 200 0 0
2014-04-01 04:00:01 W3SVC1 x.x.x.x GET /favicon.ico - 443 - x.x.x.x Mozilla/5.0+(Linux;+U;+Android+4.0.4;+en-ca;+MB886+Build/7.7.1Q-115_MB886_BELL_FFW-11)+AppleWebKit/534.30+(KHTML,+like+Gecko)+Version/4.0+Mobile+Safari/534.30 404 0 2
2014-04-01 04:00:01 W3SVC1 x.x.x.x GET /server.txt - 443 - x.x.x.x - 200 0 0
2014-04-01 04:00:02 W3SVC1 x.x.x.x GET /dir_name/dir_name/dir_name/dir_name/Default.aspx - 443 - 96.235.28.93 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:28.0)+Gecko/20100101+Firefox/28.0 302 0 0
2014-04-01 04:00:04 W3SVC1 127.0.0.1 POST /dir_name/AuthProviderSoapBinding.asmx - 80 - 127.0.0.1 Plumtree+OpenHTTP+Library+(version+2.0) 200 0 0
2014-04-01 04:00:04 W3SVC1 x.x.x.x POST /portal/server.pt - 443 - 24.3.69.132 Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:27.0)+Gecko/20100101+Firefox/27.0 302 0 0

0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

Can you also please include the header line that contains the list of fields? Once you have the fields you should be able to configure parsing using props/transforms.conf to extract the fields

props.conf
[source::/logs/web/ex140401.log]
sourcetype = new-iis

[new-iis]
REPORT-manual-iis = manual-iis

transforms.conf
[manual-iis]
FIELDS = <comma delimited list of fields from file header>
DELIMS = " " 
0 Karma

jwalzerpitt
Motivator

header line:

#Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query    s-port cs-username c-ip    cs(User-Agent) sc-status sc-substatus    sc-win32-status
0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

Okay, so give this a try:

 props.conf
 [source::/logs/web/ex140401.log]
 sourcetype = new-iis

 [new-iis]
 REPORT-manual-iis = manual-iis

 transforms.conf
 [manual-iis]
 FIELDS = date, time, s-sitename, s-ip, cs-method, cs-uri-stem, cs-uri-query, s-port, cs-username, c-ip,  cs-user-agent, sc-status, sc-substatus,  sc-win32-status
 DELIMS = " " 
 # if the fields are tab delimited try the following
 #DELIMS = "\t" 
0 Karma

jwalzerpitt
Motivator

props.conf now reads:

[source::/logs/firewall/...]
sourcetype = cisco_syslog

[source::/logs/web/...]
sourcetype = iis

[new-iis]
REPORT-manual-iis = manual-iis

[source::/logs/web/ex140401.log]
sourcetype = iis
(pleas note - this is added to the file when I do 'Explore Data')

Do I create the transforms.conf file in /hunk/etc/apps/search/local?

Thx for all of your help Ledion

0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

Yes, both (props/transforms) go in $SPLUNK_HOME/etc/apps/search/local/

0 Karma

jwalzerpitt
Motivator

And to be safe I restart Splunk after every change

0 Karma

jwalzerpitt
Motivator

props.conf:

[source::/logs/firewall/...]
sourcetype = cisco_syslog

[source::/logs/web/...]
sourcetype = iis

[new-iis]
REPORT-manual-iis = manual-iis

[source::/logs/web/ex140401.log]
sourcetype = iis

transforms.conf:

[manual-iis]
FIELDS = date, time, s-sitename, s-ip, cs-method, cs-uri-stem, cs-uri-query, s-port, cs-username, c-ip, cs-user-agent, sc-status, sc-substatus, sc-win32-status
DELIMS = "\t"

(I tried with DELIMS = " " as well)

Still not parsing. Anything to be gleaned from the search log?

0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

Please setup the props.conf correctly as below, to wire up transforms correctly:

[source::/logs/web/ex140401.log]
sourcetype = manual-iis
0 Karma

jwalzerpitt
Motivator

Ledion,

Hope you had a nice Thanksgiving holiday.

Set the props.conf as follows:

cat /opt/hunk/etc/apps/search/local/props.conf
[source::/logs/firewall/...]
sourcetype = cisco_syslog

[source::/logs/web/...]
sourcetype = iis

[new-iis]
REPORT-manual-iis = manual-iis

[source::/logs/web/ex140401.log]
sourcetype = manual-iis

(note - I see two props.conf files being modified. One is /opt/hunk/etc/apps/search/local/props.conf - and the other is /opt/hunk/etc/users/admin/search/local/props.conf)

Restarted Splunk and searched on web logs in Virtual Indexes and the events still aren't being parsed correctly.

Thx

0 Karma

Ledion_Bitincka
Splunk Employee
Splunk Employee

You have another mistake in your props.conf - sourcetype=new-iis (not sourcetype=manual-iis) - can you please just copy and paste the below and try running this search: index=vix source=/logs/web/ex140401.log? What is the sourcetype of the events that come back?

  /opt/hunk/etc/apps/search/local/props.conf
  [source::/logs/web/ex140401.log]
  sourcetype = new-iis
  priority = 10 

  [new-iis]
  REPORT-manual-iis = manual-iis

  /opt/hunk/etc/apps/search/local/transforms.conf
  [manual-iis]
  FIELDS = date, time, s-sitename, s-ip, cs-method, cs-uri-stem, cs-uri-query, s-port, cs-username, c-ip,  cs-user-agent, sc-status, sc-substatus,  sc-win32-status
  DELIMS = " " 
  # if the fields are tab delimited try the following
  #DELIMS = "\t" 
0 Karma

jwalzerpitt
Motivator

Ledion,

Was wondering if you might have time to take a look at a different issue re: key/value pairs?

I download the IAS app and got it to work, and it really saved a ton of time for parsing the Microsoft NPS Radius logs. However, I did notice there are some errors and/or missing key/value pairs. For example, in the transforms.conf file I see the following stanza:

[ias-attr-45]
SOURCE_KEY = ias_message
REGEX = ^(([^,]+),){6}(([^,]+),([^,]+),)*(45),([^,]+)
FORMAT = acct_authentic::$7

I found a parsing of the IAS/NPS logs and for the acct_authentic key/value, I see the following:

'enum' => { '0' => 'None', '1' => 'RADIUS', '2' => 'Local', '3' => 'Remote' },
'name' => 'Acct-Authentic',

How do I modify the transforms.conf stanza to include the values (0 = 'None', 1 = 'Radius', 2 = 'Local', 3 = 'Remote')?

Link is http://answers.splunk.com/answers/217719/field-extractions-for-microsoft-ias-how-to-modify.html

Thx

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.