Deployment Architecture

HALP! My *nix deployment server is propagating corrupted configuration files to my Windows clients!

hexx
Splunk Employee
Splunk Employee

I've noticed that the configuration files propagated by my Linux deployment server to my Windows deployment clients seem to be corrupted. When I open them with Notepad, everything is mashed up on a single line!

Example :

On my Linux deployment, the props.conf file I destined for my Windows forwarders looks like this :

[source::...\\MMEngine*.log]
sourcetype = mmengine_log
LINE_BREAKER =([\r|\n])\d{4}\s\d{2}:\d{2}:\d{2}\.\d{3}
SHOULD_LINEMERGE= false
DATETIME_CONFIG = \etc\system\local\datetime.xml
MAX_TIMESTAMP_LOOKAHEAD = 18
TIME_PREFIX = \d{4}\s

But once the file is deployed to the Windows forwarder, it looks like this when I open it with notepad :

[source::...\\MMEngine*.log]sourcetype = mmengine_logLINE_BREAKER =([\r|\n])\d{4}\s\d{2}:\d{2}:\d{2}\.\d{3}SHOULD_LINEMERGE= falseDATETIME_CONFIG = \etc\system\local\datetime.xmlMAX_TIMESTAMP_LOOKAHEAD = 18TIME_PREFIX = \d{4}\s

How can this be prevented?

1 Solution

hexx
Splunk Employee
Splunk Employee

This is likely to occur if the text files you are propagating to your Windows clients have been created and edited on a Linux box, and the reason is that Windows uses it's own newline character.

This difference can be exposed with "cat -etv". This command will expose "$" as the newline character used on *nix systems :

# cat -etv props.conf.linux
[source::...\\MMEngine*.log]$
sourcetype = mmengine_log$
LINE_BREAKER =([\r|\n])\d{4}\s\d{2}:\d{2}:\d{2}\.\d{3}$
SHOULD_LINEMERGE= false$
DATETIME_CONFIG = \etc\system\local\datetime.xml$
MAX_TIMESTAMP_LOOKAHEAD = 18$
TIME_PREFIX = \d{4}\s$

...and will show "^M$" as the newline sequence used on Windows systems :

# cat -etv props.conf.windows
[source::...\\MMEngine*.log]^M$
sourcetype = mmengine_log^M$
LINE_BREAKER =([\r|\n])\d{4}\s\d{2}:\d{2}:\d{2}\.\d{3}^M$
SHOULD_LINEMERGE= false^M$
DATETIME_CONFIG = \etc\system\local\datetime.xml^M$
MAX_TIMESTAMP_LOOKAHEAD = 18^M$
TIME_PREFIX = \d{4}\s^M$

The fix is a simple one : on your Linux deployment server, use the unix2dos command (http://en.wikipedia.org/wiki/Unix2dos) to convert the newline sequences of the configuration files destined to your Windows clients. This will make them readable in Notepad once they've been deployed.

Do note that Splunk (as well as most text editors other than Notepad) is "*nix-newline-aware" and will still honor the configuration parameters in files that appear mashed up on one line when opened in Notepad.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

It may not be fun using Notepad, but it works fine in Splunk. You can still edit the files on Windows using Wordpad, which can handle the unix line ends just fine.

hexx
Splunk Employee
Splunk Employee

This is likely to occur if the text files you are propagating to your Windows clients have been created and edited on a Linux box, and the reason is that Windows uses it's own newline character.

This difference can be exposed with "cat -etv". This command will expose "$" as the newline character used on *nix systems :

# cat -etv props.conf.linux
[source::...\\MMEngine*.log]$
sourcetype = mmengine_log$
LINE_BREAKER =([\r|\n])\d{4}\s\d{2}:\d{2}:\d{2}\.\d{3}$
SHOULD_LINEMERGE= false$
DATETIME_CONFIG = \etc\system\local\datetime.xml$
MAX_TIMESTAMP_LOOKAHEAD = 18$
TIME_PREFIX = \d{4}\s$

...and will show "^M$" as the newline sequence used on Windows systems :

# cat -etv props.conf.windows
[source::...\\MMEngine*.log]^M$
sourcetype = mmengine_log^M$
LINE_BREAKER =([\r|\n])\d{4}\s\d{2}:\d{2}:\d{2}\.\d{3}^M$
SHOULD_LINEMERGE= false^M$
DATETIME_CONFIG = \etc\system\local\datetime.xml^M$
MAX_TIMESTAMP_LOOKAHEAD = 18^M$
TIME_PREFIX = \d{4}\s^M$

The fix is a simple one : on your Linux deployment server, use the unix2dos command (http://en.wikipedia.org/wiki/Unix2dos) to convert the newline sequences of the configuration files destined to your Windows clients. This will make them readable in Notepad once they've been deployed.

Do note that Splunk (as well as most text editors other than Notepad) is "*nix-newline-aware" and will still honor the configuration parameters in files that appear mashed up on one line when opened in Notepad.

piebob
Motivator
Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...