Can I remove extra newlines from my csv?

[kinkdotcom]

I have an alert set up that surfaces suspicious activity by ip addresses which triggers an extremely simple shell script that feeds ips to a separate webapp.

The _si field has a newline in it, which turns every other line in the script into garbage data. Is there any way to remove the newline or exclude some or all of the meta data from the csv output? I'd like to keep this as simple as possible; worst case scenario I can call another script, but I'd rather reduce the number of moving parts.

[Rob]

Hi Kinkdotcom,

There are a couple of different workarounds we can use that mostly deal with re-formatting the _si field.

For example you can include the following in the scheduled search to change _si="hostname\r\nindexname" to _si="hostname, indexname":

| rex field=_si "(?<siHostName>.*?)[\r\n](?<siIndexName>.*?)" | eval _si=if(siIndexName!='', siHostName.", ".siIndexName, siHostName)

Unfortunately, there is not really a good way to do this using props and transforms as we can not concatenate fields at that point. However, if you wanted to you could run the regex extracts with a props.conf line in the appropriate stanza and only use one eval in your search to provide the concatenation. This would look sort of like this;

#in props.conf
[<source|sourcetype|host>]
EXTRACT-myNew_si = (?<siHostName>.*?)[\r\n](?<siIndexName>.*?) in _si

then your search just needs to include the following eval;

| eval _si=if(siIndexName!='', siHostName.", ".siIndexName, siHostName)

This will help to shorten your search string a bit while maintaining the same fundamental extractions.

Finally, the reason for using the eval if() is to make sure not to add the comma separation when we have a blank index name value as this gets saved back to the _si field which is contained within $SPLUNK_ARG_8.