index time fields extraction from source??

[rajasekhar14]

Hi All,

I'm trying to do index field extractions from source files, here is the my settings
file names are like:

/tmp/test-raj/abc/bcd.log
/tmp/test-raj/xyz/cccc.log

i want to extract the 3rd directory as a fields called raj
raj=abc
raj=xyz
+
+ etc

Transforms.conf
i placed transforms.conf file in UF and HF and indexer
[netscreen-error]
SOURCE_KEY = MetaData:Source

REGEX = \/tmp\/test-raj\/(?\w+[^\/]+)\/\S+ in source
FORMAT = raj::"$1"
WRITE_META = true

Props.conf
i placed props.conf file in UF and HF and indexer
[test-123]
TRANSFORMS-netscreen = netscreen-error

fileds.conf
[raj]
INDEXED = true

let me know your thoughts?

[DavidHourani]

Hi @rajasekhar14,

In addition to fixing your regex and format as @koshyk mentioned it :

 REGEX = \/tmp\/test-raj\/([^\/]+)\/.+
 FORMAT = raj::$1

Make sure you add the fields.conf file to the indexer as even if your indexed field is written in the metadata the indexer will not use it unless defined in the fields.conf file.

Cheers,
David

[rajasekhar14]

@DavidHourani i placed the fileds.conf file and other files in the indexers only, but look likes it nor working. any thoughts??

[DavidHourani]

yeah it depends on where your data is coming from. If its going through a Heavy Forwarder then you need props.conf and transforms.conf on the HF and fields.conf on the indexer. If there is no HF then putting all on the indexer should do the trick.

[rajasekhar14]

Thanks @DavidHourani

[DavidHourani]

Most welcome @rajasekhar14 ! Please accept or upvote my answer and comments ❤️

[koshyk · Accepted solution]

Your regex seems incorrect

Please try in transforms.conf

[netscreen-error]
SOURCE_KEY = MetaData:Source 
REGEX = \/tmp\/test-raj\/([^\/]+)\/.+
FORMAT = raj::$1
WRITE_META = true

Regex101 => https://regex101.com/r/OK0pNj/1

[rajasekhar14]

@koshyk thanks for the answer, i changed my Regex but its not working.
now all 3 files are in the only indexers
[splunk@**** local]$ cat transforms.conf
[netscreen-error]
SOURCE_KEY = MetaData:Source
REGEX = \/tmp\/test-raj\/([^\/]+)\/.+
FORMAT = raj::$1
WRITE_META = true

[splunk@**** local]$ cat props.conf
[test-123]
TRANSFORMS-netscreen = netscreen-error

[splunk@*** local]$ cat ../../spl_fields/local/fields.conf
[raj]
INDEXED = true

do i need to change these .conf to HF or UF??

[koshyk]

if you have HF, you need to send to HF & Indexers

Need to restart HF & indexers too if possible

[rajasekhar14]

Hi @koshyk ,

i have a small question on this, the above settings will use for source file name right? if i want to extract a index filed extraction in side from source file,?

i changed like this but its not working. can you please take a look.
props.conf
[ms:iis:auto]
TRANSFORMS-raj_namee = test-raj

Transforms.conf
[test-raj]
REGEX = ^(?:[^ \n]* ){2}([^ ]+)
FORMAT = appname::$1
WRITE_META = true

filed.conf
INDEXED=true

and the log format is

2019-07-17 18:21:33 xx-xx.xxx test 10.185.162.2 GET /monitor/monitor.html ----

and i'm using the above regex bold text and it need extract as a appname.