topic field value substitution props.conf in Getting Data In

field value substitution props.conf

MaryvonneMB — Mon, 13 Jun 2016 09:41:22 GMT

I would like to transform some date fields in my file when indexing:
basically my file is a csv one and one line event looks like this:

"","Value1","Value2","","","","20160326"...

I want to tranform my date like this

26/03/2016

I succeed with SEDCMD and this regex:

s/((.*?),){7}([0-9]{4})(0[1-9]|1[0-2])(0[1-9]|[1-2][0-9]|3[0-1]))/\1\5\/\4\/\3/

But, when I'm doing some search, field values in the window have the new date format, but have the old date format when I select field in the left column( with selected fields and interesting fields)

I guess I misunderstand something relating with index-time and search-time, like my modification is not persistent at search-time.
Anyone can help?

Re: field value substitution props.conf

craigv_splunk — Mon, 13 Jun 2016 15:27:23 GMT

After you added the SEDCMD line to props.conf, did you purge the the existing events from the index?

SEDCMD is an index time operations which means it would not be retroactive. Any events added to Splunk from before the configuration was added would contain the old date format which could be causing the left column to show the old format.

Re: field value substitution props.conf

woodcock — Tue, 14 Jun 2016 00:54:19 GMT

I suspect that you have 2 different solutions in place.

When you use SEDCMD, then it changes the raw data before it is indexed and the original text is NEVER THERE so it cannot show up the way that you are describing.

You might also have another solution, (a search-time one) in place that is probably working on pre-SEDCMD events.
This is the correct approach because events that were indexed before SEDCMD was in place will not be touched by the SEDCMD solution.

Re: field value substitution props.conf

MaryvonneMB — Tue, 14 Jun 2016 07:31:48 GMT

I think you're right because I tested SEDCMD several times and at first I didn't purge the index

Re: field value substitution props.conf

MaryvonneMB — Tue, 14 Jun 2016 07:42:27 GMT

Thanks for your answer. Reading Splunk doc a little more I find that it's better to make these kind of transformations at search-time instead of index-time because at index-time it could decrease the indexer performance. In fact we create a csv file using different csv sources with a SPL query. First we wanted to format date after this csv creation (when we'll index this csv)
I try and rex command works great with very little negative impact on execution time when I'm created my file (22sc without rex for 170,000 lines, 25sc with rex modifications)