Knowledge Management

index time fields extraction from source??

rajasekhar14
Path Finder

Hi All,

I'm trying to do index field extractions from source files, here is the my settings
file names are like:

/tmp/test-raj/abc/bcd.log
/tmp/test-raj/xyz/cccc.log

i want to extract the 3rd directory as a fields called raj
raj=abc
raj=xyz
+
+ etc

Transforms.conf
i placed transforms.conf file in UF and HF and indexer
[netscreen-error]
SOURCE_KEY = MetaData:Source

REGEX = \/tmp\/test-raj\/(?\w+[^\/]+)\/\S+ in source
FORMAT = raj::"$1"
WRITE_META = true

Props.conf
i placed props.conf file in UF and HF and indexer
[test-123]
TRANSFORMS-netscreen = netscreen-error

fileds.conf
[raj]
INDEXED = true

let me know your thoughts?

Tags (1)
0 Karma
1 Solution

koshyk
Super Champion

Your regex seems incorrect

Please try in transforms.conf

[netscreen-error]
SOURCE_KEY = MetaData:Source 
REGEX = \/tmp\/test-raj\/([^\/]+)\/.+
FORMAT = raj::$1
WRITE_META = true

Regex101 => https://regex101.com/r/OK0pNj/1

View solution in original post

DavidHourani
Super Champion

Hi @rajasekhar14,

In addition to fixing your regex and format as @koshyk mentioned it :

 REGEX = \/tmp\/test-raj\/([^\/]+)\/.+
 FORMAT = raj::$1

Make sure you add the fields.conf file to the indexer as even if your indexed field is written in the metadata the indexer will not use it unless defined in the fields.conf file.

Cheers,
David

0 Karma

rajasekhar14
Path Finder

@DavidHourani i placed the fileds.conf file and other files in the indexers only, but look likes it nor working. any thoughts??

0 Karma

DavidHourani
Super Champion

yeah it depends on where your data is coming from. If its going through a Heavy Forwarder then you need props.conf and transforms.conf on the HF and fields.conf on the indexer. If there is no HF then putting all on the indexer should do the trick.

0 Karma

rajasekhar14
Path Finder

Thanks @DavidHourani

0 Karma

DavidHourani
Super Champion

Most welcome @rajasekhar14 ! Please accept or upvote my answer and comments ❤️

0 Karma

koshyk
Super Champion

Your regex seems incorrect

Please try in transforms.conf

[netscreen-error]
SOURCE_KEY = MetaData:Source 
REGEX = \/tmp\/test-raj\/([^\/]+)\/.+
FORMAT = raj::$1
WRITE_META = true

Regex101 => https://regex101.com/r/OK0pNj/1

rajasekhar14
Path Finder

@koshyk thanks for the answer, i changed my Regex but its not working.
now all 3 files are in the only indexers
[splunk@**** local]$ cat transforms.conf
[netscreen-error]
SOURCE_KEY = MetaData:Source
REGEX = \/tmp\/test-raj\/([^\/]+)\/.+
FORMAT = raj::$1
WRITE_META = true

[splunk@**** local]$ cat props.conf
[test-123]
TRANSFORMS-netscreen = netscreen-error

[splunk@*** local]$ cat ../../spl_fields/local/fields.conf
[raj]
INDEXED = true

do i need to change these .conf to HF or UF??

0 Karma

koshyk
Super Champion

if you have HF, you need to send to HF & Indexers

Need to restart HF & indexers too if possible

0 Karma

rajasekhar14
Path Finder

Hi @koshyk ,

i have a small question on this, the above settings will use for source file name right? if i want to extract a index filed extraction in side from source file,?

i changed like this but its not working. can you please take a look.
props.conf
[ms:iis:auto]
TRANSFORMS-raj_namee = test-raj

Transforms.conf
[test-raj]
REGEX = ^(?:[^ \n]* ){2}([^ ]+)
FORMAT = appname::$1
WRITE_META = true

filed.conf
INDEXED=true

and the log format is

2019-07-17 18:21:33 xx-xx.xxx test 10.185.162.2 GET /monitor/monitor.html ----

and i'm using the above regex bold text and it need extract as a appname.

0 Karma
Get Updates on the Splunk Community!

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...