Create new field for incoming logs

[swdowiarz]

Hi

I have some issue with creating field for my logs. I have logs which contain number. I wan extract this number as session_id. First I've indexed data and then I've used "Extract new field", but new logs that are coming don't have this field ? Should I create this field before I extract this data ?

I would be grateful for some advice.

[swdowiarz · Accepted solution]

Hi

I did what I wanted by this: index="chat_bot" source="/var/www/html/backend/log_chat/conversation.log" | rex "^(?P<session_id>\d+)" | dedup session_id | timechart span=1d count

I've used rex command and now it is working as I wanted so there is no need to do extraction at index time.
Thank you all for involvement and help.

[micahkemp]

Thanks for posting your answer, so others can see how you accomplished this. You should also accept the answer, so the question looks closed to others.

[gcusello]

Hi swdowiarz,
at first verify you regex,

(?ms)\d+\/\d+\/\d+\s+\d+:\d+:\d+\.\d+\s\w+\s+(?<session_io>\d+)

you can test it at https://regex101.com/r/ouA97r/1

After you can think to extract you fields at index time (see http://docs.splunk.com/Documentation/Splunk/latest/Data/Configureindex-timefieldextraction )
Bye.
Giuseppe

[nikita_p]

Hi @swdowiarz,
You can also create custom field in index time in transforms.conf.
You can check below splunk docs for this.
http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Data/Configureindex-timefieldextraction

[swdowiarz]

Hi @nikita_p

I was thinking about it, but I've heard that is not good idea, that's why I'm asking about it.

[nikita_p]

@swdowiarz, you can try it will help you.

[Kwip]

Did your extraction matched all the session_id pattern you are looking for?

Could you post some sample logs to see the format of your session_ID?

[kunalmao]

when you use automatic field extractor there are multiple example of logs, try scrolling down to select the one which you need, else its always easier to use rex command. Please paste the sample log i can help you with the regex required for the rex command.

[swdowiarz]

so I can use rex command in search instead of extracting new field ? That's good point !

[swdowiarz]

12/27/17
8:24:49.599 AM  
1514363089599   Groupon   I will do my best to help you, please tell me more about the issue you are having.   
host =instance-1 source =/var/www/html/backend/log_chat/conversation.log

12/27/17
8:24:49.599 AM  
1514363089599   user   i need help   
host =instance-1 source =/var/www/html/backend/log_chat/conversation.log

12/27/17
8:24:49.599 AM  
1514363089599   Groupon   Hello! Thank you for contacting Groupon   
host =instance-1 source =/var/www/html/backend/log_chat/conversation.log

These are examples of my logs and as you can see there is a number the beginning, so I want this number to be my session_id field.

[nickhills]

I would think:
(?<session_id>\d+)\s+\w\w\w
would be a good place to start for your regex. Please give that a spin, and let me know how it goes.