How would you tackle Apache Logging Cleanup Help?

[daniel333]

Morning,

So we have about 100 application stacks. Many of them are fronted by various versions of Apache(httpd). Until now we have not really cared that field extractions had not been working. But it's coming up more and more.

The issue is we have more than 20 varying configs for the logs. Each server has a different apache logging config. Basically just 8+ years of good intention.

Now I can battle politics and go in and get all these stacks to align to one standard, maybe even force them to key value pairs while i am in there. But that could take weeks.

Alternatively I could go in write a separate inputs.conf with a different source-type for each standard I find.

Or is there another option I am not seeing?

[maciep]

I'd say the political route should probably be your long-term goal.

Are the different formats unambiguous? If so, you could create multiple extracts for each of the known formats under one sourcetype, numbering them to control the order they're applied.

[my_access_logs]
EXTRACT-1-access_fields = <format 1 regex pattern>
EXTRACT-2-access_fields = <format 2 regex pattern>
...
EXTRACT-20-access_fields = <format 20 regex pattern>

But if the formats are ambiguous, where one format would incorrectly match the regex of another (eg, size and bytes in swapped places), then maybe creating multiple sourcetypes is your only real option right now.