How to classify into multiple indexes based on one sourcetype

[silverKi]

Currently, my sourcetype contains a mix of bank logs and card logs. I would like to categorize this into `index=bank` and `index=card` respectively.

Currently, the search is done with index=main, and all data is displayed. If index=bank, I want only bank-related logs to be output.

We set the forwarder as follows and created bank, card, and error indexes on the server that will receive the data.

This is the code I have written so far... I need help,,,,,

splunk@heavy-forwarder:/opt/splunk/etc/apps/search/local:> cat inputs.conf
[monitor:///opt/splunk/var/log/splunk/test.log]
disabled = false
host = heavy-forwarder
sourcetype = test
crcSalt = <SOURCE>

splunk@heavy-forwarder:/opt/splunk/etc/system/local:> cat props.conf
[test]
TRANSFORM-routing=bankRouting,cardRouting,errorRouting

splunk@heavy-forwarder:/opt/splunk/etc/system/local:> cat transform.conf
[bankRouting]
REGEX=bank
DEST_KEY =_INDEX
FORMAT = bankGroup

[cardRouting]
REGEX=card
DEST_KEY =_INDEX
FORMAT = cardGroup

[errorGroup]
REGEX=error
DEST_KEY =_INDEX
FORMAT = errorGroup

splunk@heavy-forwarder:/opt/splunk/etc/system/local:> cat outputs.conf
[tcpout:bankGroup]
server = 192.168.111.153:9997

[tcpout:cardGroup]
server = 192.168.111.151:9997

[tcpout:errorGroup]
server = 192.168.111.152:9997

[PickleRick]

OK. Apart from the fact that you're routing to servers (which - if these are clustered indexers should replicate the buckets), not redirecting to indexes (indexer is not the same as index), let me point out two things

1) You should not use the main index. It comes configured by default so that something is created in the environment but you should rather have properly configured indexes created according to your needs

2) Do you _need_ to split the data into indexes? (Two main reasons for splitting data into indexes are access rights and retention periods). That's not the same as using two different sourcetypes for two different kinds of data (which you should definitely do if the data formats do indeed differ).

[silverKi]

1) I would like to say thank you for your advice, I think I overlooked the main index. 

2) The main reason what I want to split from one source type say test, into index=bank, index=card, index=error is because I need to different access permissions.   

[KendallW]

Hi @silverKi 

To classify logs into multiple indexes based on one sourcetype:
props.conf:

[test]
TRANSFORMS-routing = bankRouting,cardRouting,errorRouting

Note:
-the plural form TRANSFORMS-routing instead of TRANSFORM-routing.

transforms.conf:

[bankRouting]
REGEX = (?i)bank
DEST_KEY = _MetaData:Index
FORMAT = bank

[cardRouting]
REGEX = (?i)card
DEST_KEY = _MetaData:Index
FORMAT = card

[errorRouting]
REGEX = (?i)error
DEST_KEY = _MetaData:Index
FORMAT = error

Note:
-Use (?i) for case-insensitive matching
-Change DEST_KEY to _MetaData:Index
-FORMAT should be the exact index name

outputs.conf:

[tcpout]
defaultGroup = defaultGroup

[tcpout:defaultGroup]
server = 192.168.111.153:9997

[tcpout-server://192.168.111.151:9997]
index = card

[tcpout-server://192.168.111.152:9997]
index = error

[silverKi]

@KendallW ,,, Thank for your tips,, But when I search index=card in search app, The result is nothing.. 

[isoutamo]

Do you really have 3 different indexers which each contains own indexes like 1st card, 2nd bank and 3rd error indexes?
Or do you have one indexer (or cluster) which contains all those separate indexes?

[silverKi]

have one forwarder and three indexer servers.
Each indexer server holds the indexes index-card,  index=bank, index=error.

[isoutamo]

Then @KendallW ‘s answer should work with minor change on outputs.conf. You should just use default group and put all those indexers there and no index definitions into it. 

[KendallW]

1. Does the "card" index exist?

2. Is any data at all being ingested to that index?

3. Are there any parsing or connectivity issues in the _internal index?

[silverKi]

This is my test.log 
[07-30-2024 02:19:22] +0900 INFO LMTracker [14307 MainThread] username=fIg-Jvkf, Visa, cardtype=credit, cardnumber=7085-5579-5664-8197, cvc=794, expireday=05/26, user-phone=852-9765-3539, comapny=IBK, com-tel=02-885-8485, address=7547 0c2F1YA76CHEkgw Street, city=Seoul, Country=Korea, status=500 Internal Server Error, Server error. Please try again later card.