Getting Data In

file monitor input only ingest last 200 bytes of the file

season88481
Contributor

Hi guys,

I need to on-board some log files, but only the last 200 bytes of each log files contain useful information.

Is there any settings from props.conf that will only ingest the last XX characters of each log file?

Each file only contains a single line of log.

Or is there any other workaround could do the trick?

Many thanks.
S

0 Karma
1 Solution

PavelP
Motivator

Hello @season88481 ,

there are several possible options, for example SEDCMD or LINE_BREAKER to name a few. For example if your log file looks like this:

STANDARD BLA 12 XXXXXXX Message: connection from A to B

you can use this line in props.conf, anything inside of the first capture group will be removed:

LINE_BREAKER = ([\r\n]+.* Message: )connection

To see what can be done you have to post a couple of examples.

View solution in original post

0 Karma

PavelP
Motivator

Hello @season88481 ,

there are several possible options, for example SEDCMD or LINE_BREAKER to name a few. For example if your log file looks like this:

STANDARD BLA 12 XXXXXXX Message: connection from A to B

you can use this line in props.conf, anything inside of the first capture group will be removed:

LINE_BREAKER = ([\r\n]+.* Message: )connection

To see what can be done you have to post a couple of examples.

0 Karma

season88481
Contributor

Thanks Pavel,

Yes, i found the same. There is no such setting in props.conf that allows us to only ingest the first/last xx bytes of the log.

I end up using SHOULD_LINEMERGE=false and LINE_BREAKER in props.conf, and filter out the unwanted part by using props.conf and transforms.conf

0 Karma

to4kawa
Ultra Champion
PREAMBLE_REGEX = <regex>
* A regular expression that lets Splunk software ignore "preamble lines",
  or lines that occur before lines that represent structured data.
* When set, Splunk software ignores these preamble lines,
  based on the pattern you specify.
* Default: not set
0 Karma
Get Updates on the Splunk Community!

Observe and Secure All Apps with Splunk

  Join Us for Our Next Tech Talk: Observe and Secure All Apps with SplunkAs organizations continue to innovate ...

Splunk Decoded: Business Transactions vs Business IQ

It’s the morning of Black Friday, and your e-commerce site is handling 10x normal traffic. Orders are flowing, ...

Fastest way to demo Observability

I’ve been having a lot of fun learning about Kubernetes and Observability. I set myself an interesting ...