topic Re: How to classify into multiple indexes based on one sourcetype in Splunk Enterprise

How to classify into multiple indexes based on one sourcetype

silverKi — Tue, 30 Jul 2024 05:50:51 GMT

Currently, my sourcetype contains a mix of bank logs and card logs. I would like to categorize this into `index=bank` and `index=card` respectively.

Currently, the search is done with index=main, and all data is displayed. If index=bank, I want only bank-related logs to be output.

We set the forwarder as follows and created bank, card, and error indexes on the server that will receive the data.

This is the code I have written so far... I need help,,,,,

splunk@heavy-forwarder:/opt/splunk/etc/apps/search/local:> cat inputs.conf [monitor:///opt/splunk/var/log/splunk/test.log] disabled = false host = heavy-forwarder sourcetype = test crcSalt = <SOURCE>

splunk@heavy-forwarder:/opt/splunk/etc/system/local:> cat props.conf [test] TRANSFORM-routing=bankRouting,cardRouting,errorRouting

splunk@heavy-forwarder:/opt/splunk/etc/system/local:> cat transform.conf [bankRouting] REGEX=bank DEST_KEY =_INDEX FORMAT = bankGroup [cardRouting] REGEX=card DEST_KEY =_INDEX FORMAT = cardGroup [errorGroup] REGEX=error DEST_KEY =_INDEX FORMAT = errorGroup

splunk@heavy-forwarder:/opt/splunk/etc/system/local:> cat outputs.conf [tcpout:bankGroup] server = 192.168.111.153:9997 [tcpout:cardGroup] server = 192.168.111.151:9997 [tcpout:errorGroup] server = 192.168.111.152:9997

Re: How to classify into multiple indexes based on one sourcetype

silverKi — Tue, 30 Jul 2024 05:54:53 GMT

This is my test.log
[07-30-2024 02:19:22] +0900 INFO LMTracker [14307 MainThread] username=fIg-Jvkf, Visa, cardtype=credit, cardnumber=7085-5579-5664-8197, cvc=794, expireday=05/26, user-phone=852-9765-3539, comapny=IBK, com-tel=02-885-8485, address=7547 0c2F1YA76CHEkgw Street, city=Seoul, Country=Korea, status=500 Internal Server Error, Server error. Please try again later card.

Re: How to classify into multiple indexes based on one sourcetype

KendallW — Tue, 30 Jul 2024 06:05:39 GMT

Hi @silverKi

To classify logs into multiple indexes based on one sourcetype:
props.conf:

[test] TRANSFORMS-routing = bankRouting,cardRouting,errorRouting

Note:
-the plural form TRANSFORMS-routing instead of TRANSFORM-routing.

transforms.conf:

[bankRouting] REGEX = (?i)bank DEST_KEY = _MetaData:Index FORMAT = bank [cardRouting] REGEX = (?i)card DEST_KEY = _MetaData:Index FORMAT = card [errorRouting] REGEX = (?i)error DEST_KEY = _MetaData:Index FORMAT = error

Note:
-Use (?i) for case-insensitive matching
-Change DEST_KEY to _MetaData:Index
-FORMAT should be the exact index name

outputs.conf:

[tcpout] defaultGroup = defaultGroup [tcpout:defaultGroup] server = 192.168.111.153:9997 [tcpout-server://192.168.111.151:9997] index = card [tcpout-server://192.168.111.152:9997] index = error

Re: How to classify into multiple indexes based on one sourcetype

silverKi — Tue, 30 Jul 2024 06:32:00 GMT

@KendallW ,,, Thank for your tips,, But when I search index=card in search app, The result is nothing..

Re: How to classify into multiple indexes based on one sourcetype

KendallW — Tue, 06 Aug 2024 00:03:14 GMT

1. Does the "card" index exist?

2. Is any data at all being ingested to that index?

3. Are there any parsing or connectivity issues in the _internal index?

Re: How to classify into multiple indexes based on one sourcetype

isoutamo — Wed, 07 Aug 2024 14:26:36 GMT

Do you really have 3 different indexers which each contains own indexes like 1st card, 2nd bank and 3rd error indexes?
Or do you have one indexer (or cluster) which contains all those separate indexes?

Re: How to classify into multiple indexes based on one sourcetype

silverKi — Thu, 08 Aug 2024 01:43:24 GMT

have one forwarder and three indexer servers.
Each indexer server holds the indexes index-card, index=bank, index=error.

Re: How to classify into multiple indexes based on one sourcetype

PickleRick — Thu, 08 Aug 2024 05:58:14 GMT

OK. Apart from the fact that you're routing to servers (which - if these are clustered indexers should replicate the buckets), not redirecting to indexes (indexer is not the same as index), let me point out two things

1) You should not use the main index. It comes configured by default so that something is created in the environment but you should rather have properly configured indexes created according to your needs

2) Do you _need_ to split the data into indexes? (Two main reasons for splitting data into indexes are access rights and retention periods). That's not the same as using two different sourcetypes for two different kinds of data (which you should definitely do if the data formats do indeed differ).

Re: How to classify into multiple indexes based on one sourcetype

isoutamo — Thu, 08 Aug 2024 06:23:12 GMT

Then @KendallW ‘s answer should work with minor change on outputs.conf. You should just use default group and put all those indexers there and no index definitions into it.

Re: How to classify into multiple indexes based on one sourcetype

silverKi — Thu, 08 Aug 2024 07:58:27 GMT

1) I would like to say thank you for your advice, I think I overlooked the main index.

2) The main reason what I want to split from one source type say test, into index=bank, index=card, index=error is because I need to different access permissions.