topic Re: how to create fields from _raw field in Splunk Search

how to create fields from _raw field

jangid — Mon, 28 Sep 2020 12:01:03 GMT

How to create a field from _raw field?

my _raw field have some common pattern
e.g.

I0703 15:07:20.627351 3108 logger_c.cpp:42] PROCINFO.b:72 [mjangid] [User] [PROCINFO] PORT_NUMBER=6 NUM_PROGRAMS=1 START_TIME=1341322102 PID=4064 ACCOUNT=mjangid USER=mjangid TERMINAL_JBASE=ntcon TERMINAL_OS=ntcon DATABASE=default TTY=CONIN$ LANGUAGE=C LISTENING_TIME=1341324438 MEM_FREE=0 MEM_USED=8769536 THREAD_TYPE_INT=1 THREAD_TYPE_TXT=Normal LICENSE=5122|100000 STATS_OPEN=8 STATS_READ=55 STATS_WRITE=17 STATS_DELETE=1 STATS_CLEARFILE=0 STATS_PERFORM=3 STATS_INPUT=63 UNUSED_1="" OPEN_FILES_VIRTUAL=4 OPEN_FILES_REAL=4 USER_ROOT="" PROCESS_TXT="" STATS_READFRM=0 STATS_WRITEFRM=0 STATS_TOTALFRM=0 STATS_LOCKRETRY=0 PROGRAM=jsh LINE_NUMBER=103 SOURCE_NAME=CommandNext.b UNUSED_2="" UNUSED_3="" STATUS_TXT="Program at keyboard INPUT" STATUS_INT=14 CPU_USR=54.09 CPU_SYS=0.00 CPU_USR_CHILD=0.00 CPU_SYS_CHILD=0.00 USER_THREAD=""

from initial part of the log
I0703 15:07:20.627351 3108 logger_c.cpp:42] PROCINFO.b:72 [mjangid] [User] [PROCINFO]

Now I want to create some fields for whole app

processid = 3108
user_name = mjangid
user_name_2 = User
section = PROCINFO

any idea how to extract above field?

Thanks

Re: how to create fields from _raw field

Drainy — Mon, 28 Sep 2020 12:01:06 GMT

So the format of your data means that Splunk can automatically extract the data as they are key=value pairs. E.g. you already have user extracted as USER.

Instead, what you want to do is create field aliases for those fields so Splunk creates a different field name for those fields which you can match, since Splunk is already doing the extraction there is little point in defining your own new extractions.

Have a look here for how to setup aliasing.

http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Addaliasestofields

Quick regex that will pull the bits out of the first line;

\w+\s\d+:\d+:\d+.\d+\s+(?<processid>\d+)\s\w+.\w+:\d+\]\s\w+.\w:\d+\s\[(?<user_name>[\w]+)\]\s\[(?<user_name_2>\w+)\]\s\[(?<section>\w+)\]

Bear in mind that I've created that based just on that one example and really you should follow my advice above in this answer and in the comment below.

Finally, when using Splunk you don't want to extract values into field names like user_name or user_name_2. You may not hit problems now and it may do what you want, but you should really look at the Common Information Model and change your field names to match;
http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/UnderstandandusetheCommonInformationModel

Re: how to create fields from _raw field

jangid — Mon, 28 Sep 2020 12:01:11 GMT

Thanks for your reply.
I know the key/pair extraction and don't want to create alias.

I want to create a new field from my _raw field

e.g.

I0703 15:07:20.627351 3108 logger_c.cpp:42] PROCINFO.b:72 [mjangid] [User] [PROCINFO] ...

processid = 3108 --> this is my processID will change very frequently

user_name = mjangid --> system user
user_name_2 = User --> application user
section = PROCINFO --> my section

all above information available in log but want to extract in field

processid SysUser AppUser Section

3108 mjangid User-1 PROCINFO
910 mjangid User-1 PROCINFO

Re: how to create fields from _raw field

Drainy — Thu, 05 Jul 2012 08:06:49 GMT

If the detail is available within the log already and Splunk is pulling it at search time, why do you then want to write your own extractions to pull the same information again? Thats not really how Splunk works, or rather how you should use it. The search time extraction is highly optimized and the K/V pair extraction is really efficient, creating aliases for those to your expected fields would be far more efficient than running an extraction twice for the same data set. If you're sure you want to then have a look at my updated answer

Re: how to create fields from _raw field

Ayn — Thu, 05 Jul 2012 08:20:29 GMT

Why don't you want to create aliases? It's hands down the smoothest and easiest solution to what you're trying to achieve.

Re: how to create fields from _raw field

jangid — Thu, 05 Jul 2012 10:26:28 GMT

Thanks
your updated answer resolve my issue 🙂

Thank you
Manoj

Re: how to create fields from _raw field

jangid — Thu, 05 Jul 2012 16:21:31 GMT

How to I set this in props.conf so it'll create a field at index time?