Hello,
I hope you are well. I’m having some issues with log parsing on the rsyslog side.
Let me explain: I’m working with three machines:
- Linux sources (where I store the files to be read)
- SC4S
- Splunk machine
- TA-postfix
SC4S :
I'm working with this configuration file, which allows me to define the index and the source type
- logos-postfix-parser.conf : /opt/sc4s/local/config/app_parsers/syslog
# the block parser is where the "parsing" of the event happens and enrichment of meta data
# sample: <111> Mar 24 10:45:00 osnixexample: this is a test
block parser logos-postfix-rie-parser() {
channel {
rewrite {
r_set_splunk_dest_default(
index("idx_messagerie_rie")
sourcetype("st_postfix")
vendor("postfix")
product("logos")
);
};
};
};
block parser logos-postfix-interne-parser() {
channel {
rewrite {
r_set_splunk_dest_default(
index("idx_messagerie_relaisinterne")
sourcetype("st_postfix")
vendor("postfix")
product("logos")
);
};
};
};
block parser logos-postfix-externe-parser() {
channel {
rewrite {
r_set_splunk_dest_default(
index("idx_messagerie_relaisexterne") # Index cible
sourcetype("st_postfix") # Sourcetype personnalisé
vendor("postfix")
product("logos")
);
};
};
};
block parser logos-postfix-externe-cle-parser() {
channel {
rewrite {
r_set_splunk_dest_default(
index("idx_messagerie_relaisexterne_cle") # Index cible
sourcetype("st_postfix") # Sourcetype personnalisé
vendor("postfix")
product("logos")
);
};
};
};
application logos-postfix-rie[sc4s-syslog] {
filter {
#program('f_rie' type(string) flags(prefix));
program('RIE' type(string) flags(prefix));
};
parser {
logos-postfix-rie-parser();
};
};
application logos-postfix-interne[sc4s-syslog] {
filter {
program('INTERNE' type(string) flags(prefix));
#program('f_interne' type(string) flags(prefix));
};
parser {
logos-postfix-interne-parser();
};
};
application logos-postfix-externe[sc4s-syslog] {
filter {
program('EXTERNE' type(string) flags(prefix)); # <-- Filtre sur le tag
#program('f_externe' type(string) flags(prefix));
};
parser {
logos-postfix-externe-parser(); # <-- Utilise le parser dédié
};
};
application logos-postfix-externe-cle[sc4s-syslog] {
filter {
program('CLE' type(string) flags(prefix)); # <-- Filtre sur le tag
#program('f_externe' type(string) flags(prefix));
};
parser {
logos-postfix-externe-cle-parser(); # <-- Utilise le parser dédié
};
};
LINUX sources :
- rsyslog.conf : /etc/rsyslog.conf
I managed to retrieve my data in Splunk, except that the tag appears at the start of the log like this; I found a way to remove it via the props.conf file on my TA:
# (EXTERNE, INTERNE, RIE, CLE)SEDCMD-remove_tag = s/^(EXTERNE|INTERNE|RIE|CLE)\s+//
So, great, the tag has been removed, but the parsing isn’t working for the timestamp.
The _time isn’t being parsed correctly; no matter how I modify props.conf, it doesn’t work. In terms of priority, _time is parsed before SDCMD.
So I tried treating the tag as if it were still there in order to parse the _time.
# Note: Extending Splunk default settings (See $SPLUNK_HOME/etc/system/default/props.conf
pulldown_type = 0
# Ignore le tag (EXTERNE/INTERNE/RIE/CLE) et pointe sur le timestamp
TIME_PREFIX = ^(?:EXTERNE|INTERNE|RIE|CLE)\s+
# Format du timestamp : "Apr 28 10:45:00"
TIME_FORMAT = %b %d %H:%M:%S
# Augmente la fenêtre de recherche pour inclure le timestamp + nom d'hôte
MAX_TIMESTAMP_LOOKAHEAD = 25
# Désactive la fusion de lignes
SHOULD_LINEMERGE = false
# Extract the subject if enabled in Postfix's configuration
REPORT-subject = postfix_subject
# Extract to/from/message-id/helo (without the '<>'s)
REPORT-angle_brackets = postfix_angle_brackets
I get the feeling the problem lies with sc4s, and that I might need to add something to rsyslog.conf, but I’m not sure how to go about it.
Can you help me?
Hello @livehybrid , I tried what you propose but unfortunately, it doesn't work :
That's what I did, on my .conf files :
props.conf :
# KSCONF-NO-SORT
[st_postfix]
# Note: Extending Splunk default settings (See $SPLUNK_HOME/etc/system/default/props.conf
pulldown_type = 0
# Ignore le tag (EXTERNE/INTERNE/RIE/CLE) et pointe sur le timestamp
TIME_PREFIX = ^(?:EXTERNE|INTERNE|RIE|CLE)\s+
# Format du timestamp : "Apr 28 10:45:00"
TIME_FORMAT = %b %d %H:%M:%S
# Augmente la fenêtre de recherche pour inclure le timestamp + nom d'hôte
MAX_TIMESTAMP_LOOKAHEAD = 25
# Désactive la fusion de lignes
SHOULD_LINEMERGE = false
# Extract the subject if enabled in Postfix's configuration
REPORT-subject = postfix_subject
# Extract to/from/message-id/helo (without the '<>'s)
REPORT-angle_brackets = postfix_angle_brackets
EXTRACT-queue_id = postfix/[\w/]+\[\d+\]:\s+(?<queue_id>[A-Fa-f0-9]{6,20}):
EXTRACT-bounce = postfix/bounce\[\d+\]: [a-fA-F0-9]{6,20}: (?<bounce_reason>[^:]+): (?<bounce_queue_id>[a-fA-F0-9]{6,20})$
EXTRACT-status_reject = postfix/smtpd\[\d+\]: (?:NOQUEUE|[A-Fa-f0-9]{6,20}): (?<status>reject):
EXTRACT-reason = status=[^\s]+\s+\((?<reason>.*)\)$
EXTRACT-reject_reason = : (?<reject_reason>[^;:]+);
EXTRACT-dest = relay=(?<dest_host>[^\[ ,]+)\[(?<dest_ip>[^: \]]+)\](?::(?<dest_port>\d+))?
EXTRACT-remote_queue = queued as (?<xref>[A-Fa-f0-9]+) in reason
EXTRACT-status_code = status=\w+ \((?:host \S+ said:\s*)?(?<status_code_short>\d+)
EXTRACT-src-connect = (?:dis)?connect(?:ion after (?:HELO|CONNECT))? from (?:(?<src_host>[^\[]+)\[(?<src_ip>[\d.]+)|(?<src>.*))
# Extration of the different delays (cf. http://logreporters.sourceforge.net/faq.html#percentiles)
EXTRACT-delays = ^(?<time_before_queue>[^/]+)/(?<time_in_queue>[^/]+)/(?<time_connecting>[^/]+)/(?<time_transmitting>[^$]+)$ in delays
# Rename fields for CIM compliance with the Email data model
FIELDALIAS-status_code = dsn as status_code
#FIELDALIAS-status_code = status_code_short as status_code
FIELDALIAS-protocol = proto as protocol
FIELDALIAS-filter_action = reject_reason as filter_action
FIELDALIAS-internal_message_id = queue_id AS internal_message_id
FIELDALIAS-process_id = pid AS process_id
FIELDALIAS-src_user = from as src_user
FIELDALIAS-recipient = to as recipient
FIELDALIAS-orig_recipient = orig_to as orig_recipient
FIELDALIAS-recipient_count = nrcpt as recipient_count
# Don't extract 'src_host' if "unknown" (typical with reverse DNS disabled)
EVAL-src=coalesce(src, nullif(src_host, "unknown"), src_ip)
# FIELDALIAS-src=src_host AS src, src_ip AS src
EVAL-dest = coalesce(dest, nullif(dest_host, "unknown"), dest_ip)
LOOKUP-consts = postfix_consts sourcetype OUTPUT protocol, vendor, product
LOOKUP-actions = postfix_actions status OUTPUT action
# Email CIM requires the delay field to be in milliseconds
EVAL-delay = delay*1000
# Suppression des tags injectés par rsyslog (EXTERNE, INTERNE, RIE, CLE)
SEDCMD-remove_tag = s/^(EXTERNE|INTERNE|RIE|CLE)\s+//
#Extraction du _time
TRANSFORMS-getCorrectSC4STime = getCorrectSC4STime
transforms.conf :
# KSCONF-NO-SORT
[postfix_angle_brackets]
# Strip out the '<' or '>' from the value of the postfix log messages.
# Examples:
# to=<jdoe@aol.com>
# from=<bob@example.com>
# message-id=<20360611180017.4944318FE39@webapp.example.com>
# helo=<localhost.localdomain>
REGEX = [ ](to|from|message-id|helo)=<([^<> ]+)>
FORMAT = $1::$2
[postfix_subject]
# Logging the subject header requires changes to postfix config (disabled by default)
# main.cf: header_checks = regexp:/etc/postfix/header_checks
# header_checks: /^subject:/ WARN
# Example event:
# Nov 4 10:57:01 localhost postfix/cleanup[22492]: 2290326720: warning: header subject: test email from localhost[127.0.0.1]; from= to= proto=SMTP helo=
REGEX = header [Ss]ubject: (?<subject>.+?) from [^;]+;
# Lookups
[postfix_consts]
# Constant fields applied uniformly to ALL events
# Note: More efficient than using an 'EVAL-*' for these fields.
filename = postfix_consts.csv
[postfix_actions]
# Convert Postfix's 'status' messages into CIM 'actions' (as best as possible)
filename = postfix_actions.csv
#Extraction du _time
[getCorrectSC4STime]
INGEST_EVAL = _time=strptime(_raw,"%B %d %H:%M:%S")
