Delete/Ignore specified value -updated-

[Toups]

I have searched the documentation and have not yet found how to omit or delete specific fields from an input.

The input in question is a ASCII string and we would like to remove the "irellivant" data prior to indexing and segmentation for a couple reasons. As always, thank you all in advance for your assistance.

1. Few feilds indexing = optimized indexing and searching
2. Minimize the volume of data (where practical) to maximize the storage capabilities.

Below is a sample log entry actual telephone numbers are masked here for secuirty. An example of the information we would like to omit would be (\x00\x00\x00) which is a null value indicating the end of the previous line.

"\x00\x00\x00160900429  C9E820     28155512127135552121                        20080310  #850 1      010"

[ftk]

I've updated my answer based on your sample log

[ftk]

You can do this at index time using SEDCMD and a regular expression designed to capture the text you want to remove.

Now you didn't post any sample data, but let's consider this simple scenario.

Sample Data:

10.0.0.2 WEBHOST1 Error:0xf0388d Level:Severe User:foo (c)1999 Prince productions ltd. All right reserved. The format of this log message is proprietary haha!
10.0.0.2 WEBHOST1 Error:0xf9990c Level:Info User:foo (c)1999 Prince productions ltd. All right reserved. The format of this log message is proprietary haha!

Now here the superfluous string is obvious, it's the silly copyright message. So in our props.conf we would use SEDCMD to strip the copyright notice for our monitor. In this example I assume that the sourcetype of these events is "silly_source".

Props.conf:

[silly_source]
SEDCMD-stripcopyright = s/\(c\)1999 Prince productions ltd. All right reserved. The format of this log message is proprietary haha!//g

Of course you can use any kind of regular expression in the sedcmd, so you could also use something like s/\(c\).*!$//g

I recommend that you read through the doc page on SEDCMD before you get started on this, it will help you make sense out of this.

[EDIT] RE: New sample data you can remove \x00\x00\x00 as with the following replacement regex:

s/(\\x00){3}//g

[Toups]

I have updated the original post with a sample of the source log information.