topic Why SEDCMD configured in props.conf is working during Data Preview but not during SEARCH? in Getting Data In

Why SEDCMD configured in props.conf is working during Data Preview but not during SEARCH?

DavidHourani — Mon, 02 Feb 2015 17:21:56 GMT

Hello,

I have configured a SEDCMD in props.conf to remove a few unwanted lines of logs. During data preview, the SEDCMD seems to be working fine and the unwanted lines disappear, but once I accept and start searching data it's as if nothing has been modified.

Can anyone help please?

Thanks.

Re: Why SEDCMD configured in props.conf is working during Data Preview but not during SEARCH?

somesoni2 — Mon, 02 Feb 2015 18:16:09 GMT

Could you provide your SEDCMD command with some sample data?

Re: Why SEDCMD configured in props.conf is working during Data Preview but not during SEARCH?

DavidHourani — Tue, 03 Feb 2015 10:23:37 GMT

Yes of course.

Here is the SEDCMD:

SEDCMD-<class> = s/^(?!.*\d{4}-\d{2}-\d{2}.*\s).*//g

It's supposed to erase all the lines that don't start with a date e.g.: 2015-03-02

So data is quite random and some lines start with the date. I only want the lines that start with that to be taken into consideration.

Thanks!

Re: Why SEDCMD configured in props.conf is working during Data Preview but not during SEARCH?

MuS — Wed, 11 Feb 2015 08:42:08 GMT

Hi DavidHourani,

the SEDCMD is only applied at index time, not at search time. See the docs http://docs.splunk.com/Documentation/Splunk/6.2.1/admin/Propsconf

SEDCMD-<class> = <sed script>
* Only used at index time.

This means, if the props.conf is applied on a heavy forwarder or an indexer, all new events will be considered.
You could also use the route to nullQueue approach http://docs.splunk.com/Documentation/Splunk/6.2.1/Forwarding/Routeandfilterdatad#Filter_event_data_and_send_to_queues

Hope this helps ...

cheers, MuS

Re: Why SEDCMD configured in props.conf is working during Data Preview but not during SEARCH?

DavidHourani — Wed, 11 Feb 2015 10:45:57 GMT

Thank you for the information MuS.

The problem I was having occuring was when I am adding data into splunk via the "upload files from my computer" section.
So first I drag and drop the file in order to upload it. Then I define the SEDCMD and I can actually see the results filtered in the preview. Once I save and accept the defined sourcetype and then go on to search my data, I find that the SEDCMD that was shown in the preview didn't get applied to the new data that has just been added.

Splunk only indexes the data once the steps in the data input have been saved and finished right ? If that's the case, the data shown in the preview would still get the SEDCMD applied to them.

Re: Why SEDCMD configured in props.conf is working during Data Preview but not during SEARCH?

MuS — Wed, 11 Feb 2015 12:52:01 GMT

I assume, that the file will be indexed at the end of the upload/preview but i must admit I never used it before ... I'm doing it the old fashioned vi edit config files way.

Re: Why SEDCMD configured in props.conf is working during Data Preview but not during SEARCH?

DavidHourani — Wed, 11 Feb 2015 12:57:52 GMT

yeah your way works fine I think, I only get this problem when I input via drag and drop. I guess it might be a visual bug because eventually the logs got filtered as I wanted them to, but I had to delete the index content and re-input a few times before it worked.

Anyway thanks a lot for your help 🙂