topic Re: Create new field for incoming logs in Splunk Search

Create new field for incoming logs

swdowiarz — Wed, 27 Dec 2017 08:43:20 GMT

I have some issue with creating field for my logs. I have logs which contain number. I wan extract this number as session_id. First I've indexed data and then I've used "Extract new field", but new logs that are coming don't have this field ? Should I create this field before I extract this data ?

I would be grateful for some advice.

Re: Create new field for incoming logs

Kwip — Wed, 27 Dec 2017 08:48:12 GMT

Did your extraction matched all the session_id pattern you are looking for?

Could you post some sample logs to see the format of your session_ID?

Re: Create new field for incoming logs

swdowiarz — Wed, 27 Dec 2017 08:57:54 GMT

12/27/17
8:24:49.599 AM  
1514363089599   Groupon   I will do my best to help you, please tell me more about the issue you are having.   
host =instance-1 source =/var/www/html/backend/log_chat/conversation.log

12/27/17
8:24:49.599 AM  
1514363089599   user   i need help   
host =instance-1 source =/var/www/html/backend/log_chat/conversation.log

12/27/17
8:24:49.599 AM  
1514363089599   Groupon   Hello! Thank you for contacting Groupon   
host =instance-1 source =/var/www/html/backend/log_chat/conversation.log

These are examples of my logs and as you can see there is a number the beginning, so I want this number to be my session_id field.

Re: Create new field for incoming logs

kunalmao — Wed, 27 Dec 2017 08:58:52 GMT

when you use automatic field extractor there are multiple example of logs, try scrolling down to select the one which you need, else its always easier to use rex command. Please paste the sample log i can help you with the regex required for the rex command.

Re: Create new field for incoming logs

nikita_p — Wed, 27 Dec 2017 09:01:52 GMT

Hi @swdowiarz,
You can also create custom field in index time in transforms.conf.
You can check below splunk docs for this.
http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Data/Configureindex-timefieldextraction

Re: Create new field for incoming logs

swdowiarz — Wed, 27 Dec 2017 09:04:01 GMT

so I can use rex command in search instead of extracting new field ? That's good point !

Re: Create new field for incoming logs

swdowiarz — Wed, 27 Dec 2017 09:06:02 GMT

Hi @nikita_p

I was thinking about it, but I've heard that is not good idea, that's why I'm asking about it.

Re: Create new field for incoming logs

nickhills — Wed, 27 Dec 2017 09:09:45 GMT

I would think:
(?<session_id>\d+)\s+\w\w\w
would be a good place to start for your regex. Please give that a spin, and let me know how it goes.

Re: Create new field for incoming logs

gcusello — Wed, 27 Dec 2017 09:10:54 GMT

Hi swdowiarz,
at first verify you regex,

(?ms)\d+\/\d+\/\d+\s+\d+:\d+:\d+\.\d+\s\w+\s+(?<session_io>\d+)

you can test it at https://regex101.com/r/ouA97r/1

After you can think to extract you fields at index time (see http://docs.splunk.com/Documentation/Splunk/latest/Data/Configureindex-timefieldextraction )
Bye.
Giuseppe

Re: Create new field for incoming logs

swdowiarz — Wed, 27 Dec 2017 09:51:00 GMT

I did what I wanted by this: index="chat_bot" source="/var/www/html/backend/log_chat/conversation.log" | rex "^(?P<session_id>\d+)" | dedup session_id | timechart span=1d count

I've used rex command and now it is working as I wanted so there is no need to do extraction at index time.
Thank you all for involvement and help.

Re: Create new field for incoming logs

nikita_p — Wed, 27 Dec 2017 13:45:11 GMT

@swdowiarz, you can try it will help you.

Re: Create new field for incoming logs

micahkemp — Wed, 27 Dec 2017 15:54:02 GMT

Thanks for posting your answer, so others can see how you accomplished this. You should also accept the answer, so the question looks closed to others.