Field removed from logs but... it's still showing?

[secneer]

Hello all,

I have created and applied the configuration in props.conf file:

SEDCMD-XXXXX = s/XXXXXX//g

The field I wanted deleted is deleted from the logs... or it appears that way. 

Looking at the raw logs, the field/values are not there, but once I expand it (with the tab in the upper-left corner of the log entry), the field is there...? As if that field/value pair was not deleted but hidden??

The field shows in the left-side column of "All Fields" too.

Hope someone can guide/explain it - I have not been able to find an answer (if it even has one)...

Thanks!

[secneer]

Thank you @richgalloway and @gcusello for the response... but those unfortunately weren't the answers I was looking for. 

Now I realise I may have not explained it the best I could; I apologise for that.

The field that has been SEDCMD appears as an available field even if I search for data that does not have it in the logs. 

Say, it's been easily over 10 hours since the restart. Searching, right now, for the data of the last 15 minutes still shows that field, showing that it's in 100% of the logs of that search.

That's what I don't understand/know how to fix. 

Thanks!

[isoutamo]

Hi

can you show your props.conf for that part? 

Where you have defined that extractions for field which still have that data? Is there possibility that you have 1st defined additional field and after that you apply SEDCMD to raw data? That is common mistake on search time (of course you couldn't use SEDCMD on search time) that you forgot to mask/change both raw data and extracted field.

r. Ismo

[gcusello]

Hi @secneer,

to better escribe your question, could you share some screenshot?

then, where do you located the props.conf containing SEDCMD?

have you intermediate Heavy Forwarders between the Universal Forwarder and the Indexers?

Ciao.

Giuseppe

[gcusello]

Hi @secneer ,

as @richgalloway said, the SEDCMD command on props.conf works at index time on the new arriving data.

this means that you are masking your data from the moment in which you restarted Splunk after the SEDCMD insertion.

The masking will work on the new data, not on the old ones.

The old data (already indexed) cannot be modified until their deletion when the bucket will exceed the retention time.

Ciao.

Giuseppe

[richgalloway]

The SEDCMD operation is performed at index time so changes will not be applied to existing data.  IOW, you will continue to see the field in events indexed before SEDCMD was changed.