Splunk Search

Create new field for incoming logs

Path Finder

Hi

I have some issue with creating field for my logs. I have logs which contain number. I wan extract this number as session_id. First I've indexed data and then I've used "Extract new field", but new logs that are coming don't have this field ? Should I create this field before I extract this data ?

I would be grateful for some advice.

0 Karma
1 Solution

Path Finder

Hi

I did what I wanted by this: index="chat_bot" source="/var/www/html/backend/log_chat/conversation.log" | rex "^(?P<session_id>\d+)" | dedup session_id | timechart span=1d count

I've used rex command and now it is working as I wanted so there is no need to do extraction at index time.
Thank you all for involvement and help.

View solution in original post

0 Karma

Path Finder

Hi

I did what I wanted by this: index="chat_bot" source="/var/www/html/backend/log_chat/conversation.log" | rex "^(?P<session_id>\d+)" | dedup session_id | timechart span=1d count

I've used rex command and now it is working as I wanted so there is no need to do extraction at index time.
Thank you all for involvement and help.

View solution in original post

0 Karma

Champion

Thanks for posting your answer, so others can see how you accomplished this. You should also accept the answer, so the question looks closed to others.

0 Karma

Legend

Hi swdowiarz,
at first verify you regex,

(?ms)\d+\/\d+\/\d+\s+\d+:\d+:\d+\.\d+\s\w+\s+(?<session_io>\d+)

you can test it at https://regex101.com/r/ouA97r/1

After you can think to extract you fields at index time (see http://docs.splunk.com/Documentation/Splunk/latest/Data/Configureindex-timefieldextraction )
Bye.
Giuseppe

0 Karma

Contributor

Hi @swdowiarz,
You can also create custom field in index time in transforms.conf.
You can check below splunk docs for this.
http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Data/Configureindex-timefieldextraction

0 Karma

Path Finder

Hi @nikita_p

I was thinking about it, but I've heard that is not good idea, that's why I'm asking about it.

0 Karma

Contributor

@swdowiarz, you can try it will help you.

0 Karma

Contributor

Did your extraction matched all the session_id pattern you are looking for?

Could you post some sample logs to see the format of your session_ID?

0 Karma

Communicator

when you use automatic field extractor there are multiple example of logs, try scrolling down to select the one which you need, else its always easier to use rex command. Please paste the sample log i can help you with the regex required for the rex command.

0 Karma

Path Finder

so I can use rex command in search instead of extracting new field ? That's good point !

0 Karma

Path Finder
12/27/17
8:24:49.599 AM  
1514363089599   Groupon   I will do my best to help you, please tell me more about the issue you are having.   
host =instance-1 source =/var/www/html/backend/log_chat/conversation.log

12/27/17
8:24:49.599 AM  
1514363089599   user   i need help   
host =instance-1 source =/var/www/html/backend/log_chat/conversation.log

12/27/17
8:24:49.599 AM  
1514363089599   Groupon   Hello! Thank you for contacting Groupon   
host =instance-1 source =/var/www/html/backend/log_chat/conversation.log

These are examples of my logs and as you can see there is a number the beginning, so I want this number to be my session_id field.

0 Karma

Ultra Champion

I would think:
(?<session_id>\d+)\s+\w\w\w
would be a good place to start for your regex. Please give that a spin, and let me know how it goes.

0 Karma