change index for some events

[yudzhin]

Hello splunkers. I am new to splunk and have a question on how to change index for events that e.g. have status 404 on index time?  
props.conf 

[weblogs]
LINE_BREAKER = (&&&)
NO_BINARY_CHECK = true
REPORT-access = access-extractions
SHOULD_LINEMERGE = false
maxDist = 28
...
TRANSFORMS-change = notfound,changesourcetype

transforms.conf

[notfound]
REGEX = ".+?"\s(404)
DEST_KEY = MetaData:Index
FORMAT = index::notfoundindex

[changesourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = ^(.*)
FORMAT = sourcetype::access_combined

example of event:

141.146.8.66 - - [13/Jan/2016 21:03:09:200] "POST /category.screen?category_id=SURPRISE&JSESSIONID=SD1SL2FF5ADFF3 HTTP 1.1" 200 3496 "http://www.myflowershop.com/cart.do?action=view&itemId=EST-16&product_id=RP-SN-01" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.38 Safari/533.4" 294
 

Changing of sourcetype works fine, but index changing doesn't and I really do not know where the mistake is.

[yudzhin]

Problem solved just by re-typing transforms.conf stanzas, most probably some unexpected character was present.
thank's all for help 🙂

[richgalloway]

Why are you trying to put all 404s in a separate index? What problem do you expect that to solve?

[yudzhin]

@richgalloway I have to do some labs and there is a task to do so 🙂 

[gcusello · Accepted solution]

Hi @yudzhin ,

At first I don't understand why you want to put 404 events in a different index!

Remember that usually in Splunk are used different indexes basing on two parameters:

• different grants of access,
• different retention periods.

You can differentiate events using sourcetype or another field, don't think to Splunk as a DB table with fields.

Anyway, I see two things probably not correct:

• in props.conf, you're using two transformations in one command, I prefer to use two different transformations:

TRANSFORMS-notfound = notfound
TRANSFORMS-change = changesourcetype

• I cannot check the regex in notfound stanza, but I don't see, in your sample, anything that could be matched by that regex, could you share a sample with 404?

Anyway, if in your sample 200 is the posizion of 404, you could try this regex in notfound stanza

\"\s+404\s+

Ciao.

Giuseppe

[yudzhin]

Hello @gcusello  and thank's for response.

example of event with 404
69.80.0.18 - - [13/Jan/2016 21:03:08:169] "GET /product.screen?product_id=K9-CW-01&JSESSIONID=SD7SL2FF10ADFF4 HTTP 1.1" 404 2038 "http://www.myflowershop.com/cart.do?action=view&itemId=EST-1&product_id=K9-CW-01" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" 261

Unfortunatelly changing both regex and separate TRANSFORMS didn't help, after data is uploaded all events are present for webindex, not for notfoundindex.

BTW: I am doing some labs for SE2 certification and I have such task. I understand this has no sense, but I have to do  it 🙂

[gcusello]

Hi @yudzhin ,

the regex I hinted is correct so try it.

To debug your problem, try if each transformation runs by itself.

Anyway, thinks to your idea to have different indexes!

Ciao.

Giuseppe

[yudzhin]

@gcusello  TRANSFORMS-notfound = notfound is not working separately.