Why SEDCMD configured in props.conf is working during Data Preview but not during SEARCH?

[DavidHourani]

Hello,

I have configured a SEDCMD in props.conf to remove a few unwanted lines of logs. During data preview, the SEDCMD seems to be working fine and the unwanted lines disappear, but once I accept and start searching data it's as if nothing has been modified.

Can anyone help please?

Thanks.

[MuS · Accepted solution]

Hi DavidHourani,

the SEDCMD is only applied at index time, not at search time. See the docs http://docs.splunk.com/Documentation/Splunk/6.2.1/admin/Propsconf

SEDCMD-<class> = <sed script>
* Only used at index time.

This means, if the props.conf is applied on a heavy forwarder or an indexer, all new events will be considered.
You could also use the route to nullQueue approach http://docs.splunk.com/Documentation/Splunk/6.2.1/Forwarding/Routeandfilterdatad#Filter_event_data_a...

Hope this helps ...

cheers, MuS

[DavidHourani]

Thank you for the information MuS.

The problem I was having occuring was when I am adding data into splunk via the "upload files from my computer" section.
So first I drag and drop the file in order to upload it. Then I define the SEDCMD and I can actually see the results filtered in the preview. Once I save and accept the defined sourcetype and then go on to search my data, I find that the SEDCMD that was shown in the preview didn't get applied to the new data that has just been added.

Splunk only indexes the data once the steps in the data input have been saved and finished right ? If that's the case, the data shown in the preview would still get the SEDCMD applied to them.

[MuS]

I assume, that the file will be indexed at the end of the upload/preview but i must admit I never used it before ... I'm doing it the old fashioned vi edit config files way.

[DavidHourani]

yeah your way works fine I think, I only get this problem when I input via drag and drop. I guess it might be a visual bug because eventually the logs got filtered as I wanted them to, but I had to delete the index content and re-input a few times before it worked.

Anyway thanks a lot for your help 🙂

[somesoni2]

Could you provide your SEDCMD command with some sample data?

[DavidHourani]

Yes of course.

Here is the SEDCMD:

SEDCMD-<class> = s/^(?!.*\d{4}-\d{2}-\d{2}.*\s).*//g

It's supposed to erase all the lines that don't start with a date e.g.: 2015-03-02

So data is quite random and some lines start with the date. I only want the lines that start with that to be taken into consideration.

Thanks!