Routing data to specific index based on REGEX on Heavy forwarder , regex expression is from JSON data.

[mahesh423]

Hi All,
Unable to route the json logs based on a a keyword (regex ) "MyService_DataApp" on the event to a particular index testlogs_idx .Could you please point anything wrong with the below and these configurations are on Heavy forwarder ,SH's and Indexers.

To test the routing I've created an index=thisshouldneverhappen and added under the inputs , and set up an alert, whenever an event hits that index to know something is broken , all the events still route to the index=thisshouldneverhappen .

Props
[json_srctype]
SHOULD_LINEMERGE=false
LINE_BREAKER=([\r\n]+){
NO_BINARY_CHECK=true
KV_MODE=json
MAX_TIMESTAMP_LOOKAHEAD=45
TIME_PREFIX=\W+\w{8}
TIME_FORMAT=%s%3N
TRUNCATE=50000
ANNOTATE_PUNCT=false
disabled=false
pulldown_type =true
TRANSFORMS-01_testlogs= a1-testlogs-Route
TRANSFORMS-02_testlogs =a2-testlogs-SourceType

Transforms
[a1-testlogs-Route]
DEST_KEY = _MetaData:Index
REGEX = MyService_DataApp
FORMAT = testlogs_idx

[a2-testlogs-SourceType]
DEST_KEY = MetaData:Sourcetype
REGEX = MyService_DataApp
FORMAT = sourcetype::testlogs_srctype

[mahesh423 · Accepted solution]

the routing to index and sourcetpe are working after making changes to the json logs where the keyword (regex ) "MyService_DataApp" is at the last of the event under the nested xml and I've moved the keyword (regex ) "MyService_DataApp" to the top under metadata with json key value pair

All the configs are correct .

[gcusello]

Hi mahesh423,
to debug your situation, try the following checks (probably someone of them you already used!):

• insert the above props.conf and transforms.conf on your Heavy Forwarders and Indexers and restart Splunk;
• check if the regex really flags the events to route;
• try to insert in props.conf also a stanza for the new sourcetype "testlogs_srctype".

and let me know the new situation.

Bye.
Giuseppe

[mahesh423]

Thanks @gcusello.I'm giving a few points which I tried . please review and advice.

Steps 1 completed - I've props and transforms on the Heavy forwarder and search heads and indexers and restarted .

2 - Steps completed .validated using https://regexr.com/

Step 3 -
try to insert in props.conf also a stanza for the new sourcetype "testlogs_srctype" - Not sure how can I add the new sourcetype as per the props -[json_srctype] , TRANSFORMS-02_testlogs =a2-testlogs-SourceType
attribute needs to create a new sourcetype , same logic works for with other data.

[a2-testlogs-SourceType]
DEST_KEY = MetaData:Sourcetype
REGEX = MyService_DataApp
FORMAT = sourcetype::testlogs_srctype

The above same logic is working with xml format data . the new data has
using the keyword "MyService_DataApp" within a json format xml nested object - embedded with in the as below

"LogMessage": {
"request": "MyService_DataApp"

transforms.conf , regex used as below
REGEX = MyService_DataApp

my regex pattern in the json log is at 6089 column and ran btool on the heavy forwarder for props and validated the DEPTH LIMIT which is default 1000 and what could be the reason or needs increase ( chances of bad performance) and MATCH_LIMIT is default which is 100000.