Please pardon my ignorance here as i am new to Splunk. I am using Splunk 7.1 on a Windows server and forwarding syslog messages from Linux server (j01ftc) , now as you can see below example
<0>Oct 18 14:49:51 j01ftc Oct 19 02:02:23 akl-ftc-sbc3b.vfnz-ipsn akl-ftc-sbc3b sipd05[4f] ERROR could not identify psipcontact
There are 2 issues. One is that the wrong time stamp is in beginning of the message and 2nd server name added , i want to exclude both of these from my logs in Splunk or at least fix the time in the event.
would you be able to point me to ,how to remove "Oct 18 14:49:51 j01ftc " from my log messages
That is the syslog header added by syslog server for every event.
If you want to extract the second timestamp (Oct 19 02:02:23), configure the props.conf as below:
[sourcetype] TIME_PREFIX = j01ftc\s TIME_FORMAT = %b %d %H:%M:%S MAX_TIMESTAMP_LOOKAHEAD = 15
If you want to completely remove the syslog header from events then, configure the props.conf as below:
[sourcetype] SEDCMD-dropSyslogHeader = s/(^[\w\s\:]+j01ftc\s)//g TIME_PREFIX = ^ TIME_FORMAT = %b %d %H:%M:%S MAX_TIMESTAMP_LOOKAHEAD = 15
I would suggest using 1st props.conf as it is much simpler. HTH!
under D:\Splunk\etc\apps\search\local props.conf
NOBINARYCHECK = true
TZ = Antarctica/SouthPole
category = Custom
pulldowntype = 1
disabled = false
SHOULDLINEMERGE = false
SEDCMD-dropSyslogHeader = s/(^[\w\s:]+j01ftc\s)//g
TIMEPREFIX = ^
MAXTIMESTAMPLOOKAHEAD = 15
but still no luck 😞
<0>Oct 18 23:38:53 j01ftc Oct 19 12:38:46 akl-ftc-sbc3b.ipsn akl-ftc-sbc3b sipd01[4b] ERROR could not identify psipcontact
Few questions please,
Can you tell me if
<0> is part of the event or is it something that got added while posting the question?
Can you tell more about your architecture?
UF (on syslog server) --> Indexer then, these configs should be on indexer.
HF (on syslog server) --> Indexer then, these configs should be on HF.
Make sure that these configuration are present on parsing layer (HF and Indexer).
Finally, did you try the 1st props.conf. It will fix the timestamp but won't discard syslog header.
Also, as @FrankVl suggested, you should have a look at syslog configuration to troubleshoot further.
sudosplunk's answer is the solution that worked for me using your example string:
(Oct 18 23:38:53 j01ftc Oct 19 12:38:46 akl-ftc-sbc3b.ipsn akl-ftc-sbc3b sipd01[4b] ERROR could not identify psipcontact)
TIMEPREFIX = j01ftc\s
TIMEFORMAT = %b %d %H:%M:%S
MAXTIMESTAMPLOOKAHEAD = 15
I would very much suggest fixing this on syslog side, instead of pulling all kinds of tricks on Splunk side to fix problems introduced by the syslog setup.
What does your syslog setup look like? Because the duplicate headers seem to indicate there is some intermediate syslog server involved or so? And also: how exactly are you feeding this into Splunk?
The feed is like
device-->(tcp514)syslog server -->TCP9998 splunk.
i highly doubt the duplicate time stamp is added by Syslog server , i believe its added on splunk server
Unless you specifically configured Splunk to add that header, I don't think so. Splunk only does that by default for UDP inputs and even then, it does not add that
<xx> part, only timestamp and sending host.
You can validate that by running a networkcapture (e.g. using tcpdump) on either the outgoing traffic of the syslog server, or the incoming traffic on the splunk server. If the header is there already, it isn't Splunk who is adding it.
Feel free to share your relevant Splunk configs (inputs, possible props/transforms) and syslog daemon configs here for further troubleshooting.