Splunk Search

Transformation to index events to different index not working

markconlin
Path Finder

Goal
I wish to place some events into a longer living index "staging-boeing-audit" for audit purposes.
All other events I wish to continue to be indexed as before.

What I have tried
I fabricated a simple example to prove this method will work.

  • I added a TRANSFORMS-"name" line to my props.conf for a test sourcetype
  • I added a transforms.conf to regex some events into a new index
  • I used oneshot to place a test file into Splunk

What happened
All data was placed into the original index "marktransform1"
No events were matched by the transform and thus the target index "staging-boeing-audit" is empty.

My props.conf

root@myhost:/opt/splunk/etc/system/local# cat props.conf
[mectest]
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = json
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
TIMESTAMP_FIELDS = info.created
TIME_FORMAT = %Y-%d-%m %H:%M:%S.%3Q
KV_MODE= none
AUTO_KV_JSON= true
category = Custom
description = added via ui
disabled = false
pulldown_type = 1
TRANSFORMS-routing = route_boeing

My transforms.conf

root@myhost:/opt/splunk/etc/system/local# cat tranforms.conf
[route_boeing]
REGEX = .
DEST_KEY = _MetaData:Index
FORMAT = staging-boeing-audit

My oneshot command

root@myhost:/opt/splunk/bin# ./splunk add oneshot ulfs.log -sourcetype mectest -index marktransform1

Sample from ulfs.log file

root@myhost:/opt/splunk/bin# tail -2 ulfs.log
{"context": {}, "info": {"name": "starwood.core", "msg": "Got rate from Starwood", "levelname": "INFO", "levelno": 20, "pathname": "/home/ubuntu/envs/airborne/src/tune/tune/utils.py", "filename": "utils.py", "module": "utils", "exc_info": null, "exc_text": null, "stack_info": null, "lineno": 45, "funcName": "debug", "created": "2018-01-10 17:52:12.253", "msecs": 252.66528129577637, "relativeCreated": 33089947.247982025, "thread": 140501375943792, "threadName": "DummyThread-686", "processName": "MainProcess", "process": 21272, "currency": "DKK", "event_type": "get_rate", "message_type": "starwood", "content_type": "profiling", "time": 0.0021333694458007812, "message": "Got rate from Starwood", "asctime": "2018-01-10 09:52:12,252", "loggername": "starwood.core"}}
{"context": {}, "info": {"name": "tune.memory", "msg": "GC status", "levelname": "DEBUG", "levelno": 10, "pathname": "/home/ubuntu/envs/airborne/src/tune/tune/memory.py", "filename": "memory.py", "module": "memory", "exc_info": null, "exc_text": null, "stack_info": null, "lineno": 49, "funcName": "gc_monitor", "created": "2018-01-10 17:52:33.135", "msecs": 135.5295181274414, "relativeCreated": 33110830.112218857, "thread": 140501389641576, "threadName": "DummyThread-1081", "processName": "MainProcess", "process": 22337, "current_collections_count": [86, 1, 30], "current_frames": 5, "enabled": true, "garbage_count": 0, "gc_stats": [{"collections": 26842, "collected": 5718217, "uncollectable": 0}, {"collections": 2440, "collected": 2079011, "uncollectable": 0}, {"collections": 101, "collected": 1168870, "uncollectable": 0}], "max_rss": 351032, "total_objects": 326350, "message": "GC status", "asctime": "2018-01-10 09:52:33,135", "loggername": "tune.memory"}}
1 Solution

gfuente
Motivator

Hello

There is a typo in the transforms file name:

 root@myhost:/opt/splunk/etc/system/local# cat tranforms.conf

It should be: transforms.conf

Regards

View solution in original post

gfuente
Motivator

Hello

There is a typo in the transforms file name:

 root@myhost:/opt/splunk/etc/system/local# cat tranforms.conf

It should be: transforms.conf

Regards

micahkemp
Champion

Good catch. I suggest converting this to an answer so that if it helps the poster find the root cause it can be accepted.

micahkemp
Champion

Did you restart splunk after making those changes?

0 Karma

markconlin
Path Finder

yes I did restart

0 Karma

markconlin
Path Finder

I have tried other regex as well like REGEX = .*

0 Karma

cpetterborg
SplunkTrust
SplunkTrust

Is there the possibility that you have another route_boeing in another transforms.conf?

0 Karma

markconlin
Path Finder

no, not possible

0 Karma

micahkemp
Champion

Paste the output of:

./bin/splunk btool transforms list route_boeing --debug

./bin/splunk btool props list mectest --debug
0 Karma

markconlin
Path Finder

I am testing this on my splunk indexer/search head not on an instance with a forwarder sending logs.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Did you place your props.conf and transforms.conf on the instance where data parsing happens? It'll either Indexer OR heavy forwarder (whichever comes first in data flow). Also, I believe REGEX = . you're using for testing (it'll route all events), when test is done, you'd put actual regex matching required events?

0 Karma

markconlin
Path Finder

I am aware my regex matches all events but I wanted to make sure the problem was not in the regex itself and .* and . were the two dumbest simplest ways to try it I could think of.

0 Karma

micahkemp
Champion

That's a good question. If this is a standalone instance, running oneshot on the same splunk instance that props/transforms are defined should be sufficient. But if this server is a universal forwarder props/transforms need to be on the indexer (or heavy forwarder) it is forwarding to.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...