topic Re: change index for some events in Getting Data In

change index for some events

yudzhin — Tue, 07 Jul 2020 13:43:58 GMT

Hello splunkers. I am new to splunk and have a question on how to change index for events that e.g. have status 404 on index time?
props.conf

[weblogs]
LINE_BREAKER = (&&&)
NO_BINARY_CHECK = true
REPORT-access = access-extractions
SHOULD_LINEMERGE = false
maxDist = 28
...
TRANSFORMS-change = notfound,changesourcetype

transforms.conf

[notfound]
REGEX = ".+?"\s(404)
DEST_KEY = MetaData:Index
FORMAT = index::notfoundindex

[changesourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = ^(.*)
FORMAT = sourcetype::access_combined

example of event:

141.146.8.66 - - [13/Jan/2016 21:03:09:200] "POST /category.screen?category_id=SURPRISE&JSESSIONID=SD1SL2FF5ADFF3 HTTP 1.1" 200 3496 "http://www.myflowershop.com/cart.do?action=view&itemId=EST-16&product_id=RP-SN-01" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.38 Safari/533.4" 294

Changing of sourcetype works fine, but index changing doesn't and I really do not know where the mistake is.

Re: change index for some events

gcusello — Tue, 07 Jul 2020 14:08:37 GMT

Hi @yudzhin ,

At first I don't understand why you want to put 404 events in a different index!

Remember that usually in Splunk are used different indexes basing on two parameters:

different grants of access,
different retention periods.

You can differentiate events using sourcetype or another field, don't think to Splunk as a DB table with fields.

Anyway, I see two things probably not correct:

in props.conf, you're using two transformations in one command, I prefer to use two different transformations:

TRANSFORMS-notfound = notfound TRANSFORMS-change = changesourcetype

I cannot check the regex in notfound stanza, but I don't see, in your sample, anything that could be matched by that regex, could you share a sample with 404?

Anyway, if in your sample 200 is the posizion of 404, you could try this regex in notfound stanza

\"\s+404\s+

Ciao.

Giuseppe

Re: change index for some events

richgalloway — Tue, 07 Jul 2020 14:03:58 GMT

Why are you trying to put all 404s in a separate index? What problem do you expect that to solve?

Re: change index for some events

yudzhin — Tue, 07 Jul 2020 14:11:16 GMT

Hello @gcusello and thank's for response.

example of event with 404
69.80.0.18 - - [13/Jan/2016 21:03:08:169] "GET /product.screen?product_id=K9-CW-01&JSESSIONID=SD7SL2FF10ADFF4 HTTP 1.1" 404 2038 "http://www.myflowershop.com/cart.do?action=view&itemId=EST-1&product_id=K9-CW-01" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" 261

Unfortunatelly changing both regex and separate TRANSFORMS didn't help, after data is uploaded all events are present for webindex, not for notfoundindex.

BTW: I am doing some labs for SE2 certification and I have such task. I understand this has no sense, but I have to do it 🙂

Re: change index for some events

gcusello — Tue, 07 Jul 2020 14:10:58 GMT

Hi @yudzhin ,

the regex I hinted is correct so try it.

To debug your problem, try if each transformation runs by itself.

Anyway, thinks to your idea to have different indexes!

Ciao.

Giuseppe

Re: change index for some events

yudzhin — Tue, 07 Jul 2020 14:12:46 GMT

@richgalloway I have to do some labs and there is a task to do so 🙂

Re: change index for some events

yudzhin — Tue, 07 Jul 2020 14:17:46 GMT

@gcusello TRANSFORMS-notfound = notfound is not working separately.

Re: change index for some events

yudzhin — Tue, 07 Jul 2020 14:58:38 GMT

Problem solved just by re-typing transforms.conf stanzas, most probably some unexpected character was present.
thank's all for help 🙂