Getting Data In

Why SEDCMD configured in props.conf is working during Data Preview but not during SEARCH?

DavidHourani
Super Champion

Hello,

I have configured a SEDCMD in props.conf to remove a few unwanted lines of logs. During data preview, the SEDCMD seems to be working fine and the unwanted lines disappear, but once I accept and start searching data it's as if nothing has been modified.

Can anyone help please?

Thanks.

Tags (2)
0 Karma
1 Solution

MuS
SplunkTrust
SplunkTrust

Hi DavidHourani,

the SEDCMD is only applied at index time, not at search time. See the docs http://docs.splunk.com/Documentation/Splunk/6.2.1/admin/Propsconf

SEDCMD-<class> = <sed script>
* Only used at index time.

This means, if the props.conf is applied on a heavy forwarder or an indexer, all new events will be considered.
You could also use the route to nullQueue approach http://docs.splunk.com/Documentation/Splunk/6.2.1/Forwarding/Routeandfilterdatad#Filter_event_data_a...

Hope this helps ...

cheers, MuS

View solution in original post

MuS
SplunkTrust
SplunkTrust

Hi DavidHourani,

the SEDCMD is only applied at index time, not at search time. See the docs http://docs.splunk.com/Documentation/Splunk/6.2.1/admin/Propsconf

SEDCMD-<class> = <sed script>
* Only used at index time.

This means, if the props.conf is applied on a heavy forwarder or an indexer, all new events will be considered.
You could also use the route to nullQueue approach http://docs.splunk.com/Documentation/Splunk/6.2.1/Forwarding/Routeandfilterdatad#Filter_event_data_a...

Hope this helps ...

cheers, MuS

DavidHourani
Super Champion

Thank you for the information MuS.

The problem I was having occuring was when I am adding data into splunk via the "upload files from my computer" section.
So first I drag and drop the file in order to upload it. Then I define the SEDCMD and I can actually see the results filtered in the preview. Once I save and accept the defined sourcetype and then go on to search my data, I find that the SEDCMD that was shown in the preview didn't get applied to the new data that has just been added.

Splunk only indexes the data once the steps in the data input have been saved and finished right ? If that's the case, the data shown in the preview would still get the SEDCMD applied to them.

0 Karma

MuS
SplunkTrust
SplunkTrust

I assume, that the file will be indexed at the end of the upload/preview but i must admit I never used it before ... I'm doing it the old fashioned vi edit config files way.

0 Karma

DavidHourani
Super Champion

yeah your way works fine I think, I only get this problem when I input via drag and drop. I guess it might be a visual bug because eventually the logs got filtered as I wanted them to, but I had to delete the index content and re-input a few times before it worked.

Anyway thanks a lot for your help 🙂

somesoni2
Revered Legend

Could you provide your SEDCMD command with some sample data?

0 Karma

DavidHourani
Super Champion

Yes of course.

Here is the SEDCMD:

SEDCMD-<class> = s/^(?!.*\d{4}-\d{2}-\d{2}.*\s).*//g

It's supposed to erase all the lines that don't start with a date e.g.: 2015-03-02

So data is quite random and some lines start with the date. I only want the lines that start with that to be taken into consideration.

Thanks!

0 Karma
Get Updates on the Splunk Community!

Security Highlights: September 2022 Newsletter

 September 2022 The Splunk App for Fraud Analytics (SFA) is now Splunk SupportedUse your existing Splunk ...

Platform Highlights | September 2022 Newsletter

 September 2022 What’s New in 9.0 and How to UpgradeGet a walk through of what is new Splunk Enterprise 9.0 ...

Observability Highlights | September 2022 Newsletter

 September 2022 Splunk Observability SuiteAccess to "Classic" SignalFx Interface Will be Removed on Sept 30, ...