Knowledge Management

Field removed from logs but... it's still showing?

secneer
Loves-to-Learn

 

Hello all,

I have created and applied the configuration in props.conf file:

SEDCMD-XXXXX = s/XXXXXX//g

The field I wanted deleted is deleted from the logs... or it appears that way. 

Looking at the raw logs, the field/values are not there, but once I expand it (with the tab in the upper-left corner of the log entry), the field is there...? As if that field/value pair was not deleted but hidden??

The field shows in the left-side column of "All Fields" too.

Hope someone can guide/explain it - I have not been able to find an answer (if it even has one)...

Thanks!

Labels (1)
0 Karma

secneer
Loves-to-Learn

Thank you @richgalloway and @gcusello for the response... but those unfortunately weren't the answers I was looking for. 

Now I realise I may have not explained it the best I could; I apologise for that.

The field that has been SEDCMD appears as an available field even if I search for data that does not have it in the logs. 

Say, it's been easily over 10 hours since the restart. Searching, right now, for the data of the last 15 minutes still shows that field, showing that it's in 100% of the logs of that search.

That's what I don't understand/know how to fix. 

Thanks!

 

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

can you show your props.conf for that part? 

Where you have defined that extractions for field which still have that data? Is there possibility that you have 1st defined additional field and after that you apply SEDCMD to raw data? That is common mistake on search time (of course you couldn't use SEDCMD on search time) that you forgot to mask/change both raw data and extracted field.

r. Ismo

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @secneer,

to better escribe your question, could you share some screenshot?

then, where do you located the props.conf containing SEDCMD?

have you intermediate Heavy Forwarders between the Universal Forwarder and the Indexers?

Ciao.

Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @secneer ,

as @richgalloway said, the SEDCMD command on props.conf works at index time on the new arriving data.

this means that you are masking your data from the moment in which you restarted Splunk after the SEDCMD insertion.

The masking will work on the new data, not on the old ones.

The old data (already indexed) cannot be modified until their deletion when the bucket will exceed the retention time.

Ciao.

Giuseppe

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The SEDCMD operation is performed at index time so changes will not be applied to existing data.  IOW, you will continue to see the field in events indexed before SEDCMD was changed.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...