Getting Data In

field value substitution props.conf

MaryvonneMB
Path Finder

I would like to transform some date fields in my file when indexing:
basically my file is a csv one and one line event looks like this:

"","Value1","Value2","","","","20160326"...

I want to tranform my date like this

26/03/2016

I succeed with SEDCMD and this regex:

s/((.*?),){7}([0-9]{4})(0[1-9]|1[0-2])(0[1-9]|[1-2][0-9]|3[0-1]))/\1\5\/\4\/\3/

But, when I'm doing some search, field values in the window have the new date format, but have the old date format when I select field in the left column( with selected fields and interesting fields)

I guess I misunderstand something relating with index-time and search-time, like my modification is not persistent at search-time.
Anyone can help?

0 Karma
1 Solution

woodcock
Esteemed Legend

I suspect that you have 2 different solutions in place.

When you use SEDCMD, then it changes the raw data before it is indexed and the original text is NEVER THERE so it cannot show up the way that you are describing.

You might also have another solution, (a search-time one) in place that is probably working on pre-SEDCMD events.
This is the correct approach because events that were indexed before SEDCMD was in place will not be touched by the SEDCMD solution.

View solution in original post

woodcock
Esteemed Legend

I suspect that you have 2 different solutions in place.

When you use SEDCMD, then it changes the raw data before it is indexed and the original text is NEVER THERE so it cannot show up the way that you are describing.

You might also have another solution, (a search-time one) in place that is probably working on pre-SEDCMD events.
This is the correct approach because events that were indexed before SEDCMD was in place will not be touched by the SEDCMD solution.

MaryvonneMB
Path Finder

Thanks for your answer. Reading Splunk doc a little more I find that it's better to make these kind of transformations at search-time instead of index-time because at index-time it could decrease the indexer performance. In fact we create a csv file using different csv sources with a SPL query. First we wanted to format date after this csv creation (when we'll index this csv)
I try and rex command works great with very little negative impact on execution time when I'm created my file (22sc without rex for 170,000 lines, 25sc with rex modifications)

0 Karma

craigv_splunk
Splunk Employee
Splunk Employee

After you added the SEDCMD line to props.conf, did you purge the the existing events from the index?

SEDCMD is an index time operations which means it would not be retroactive. Any events added to Splunk from before the configuration was added would contain the old date format which could be causing the left column to show the old format.

0 Karma

MaryvonneMB
Path Finder

I think you're right because I tested SEDCMD several times and at first I didn't purge the index

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...