topic Re: double host name and timestamps in Getting Data In

double host name and timestamps

bbear — Thu, 15 Jul 2010 01:19:20 GMT

Hi All,

I have been trying to get Splunk to strip off the timestamp and host of forwarded events but do not understand how to do this.

I have syslog-ng and Splunk on the same server. Syslog-ng is forwarding the events to udp port 1514 for Splunk to read and index. This all works fine. But the data Splunk receives has the timestamp and hostname of the syslog server in addition to the evendata host. I.E.

Jul 14 14:15:56 10.128.213.50 Jul 14 14:15:56 my-host-int02 snmpd[7777]: Received SNMP packet(s) from UDP: [10.128.30.20]:54900

Notice the additional timestamp above. I see in the docs and in the answers forum that this is possible to have Splunk strip this out before indexing, but I have not been able to get all the pieces worked out.

Does someone know the answer that has worked this out already? I would like Splunk to strip the event to look like:

Jul 14 14:15:56 my-host-int02 snmpd[7777]: Received SNMP packet(s) from UDP: [10.128.30.20]:54900

Re: double host name and timestamps

ziegfried — Thu, 15 Jul 2010 02:18:57 GMT

You could do this by modifying the raw event text with a SEDCMD your props.conf (eg. $SPLUNK_HOME/etc/system/local/props.conf)

[source::tcp:1514]
SEDCMD-remove-dbl-date=s/^[A-Z][a-z]+\s+\d+\s+\d+:\d+:\d+\s+([A-Z][a-z]+\s+\d+\s+\d+:\d+:\d+\s+)/\1/g

Re: double host name and timestamps

bbear — Thu, 15 Jul 2010 02:21:29 GMT

I guess I should just keep trying. I figured out how to get it to work.

in $SPLUNK_HOME/etc/system/local create props.conf

props.conf

[syslog] TRANSFORMS = syslog-header-stripper-ts-host REPORT-syslog = syslog-extractions

In $SPLUNK_HOME/etc/system/local create transforms.conf

transforms.conf

[syslog-header-stripper-ts-host] REGEX = ^[A-Z][a-z]+\s+\d+\s\d+:\d+:\d+\s[^\s]\s(.)$ FORMAT = $1 DEST_KEY = _raw

Restart Splunk and it works

Re: double host name and timestamps

gkanapathy — Thu, 15 Jul 2010 05:51:57 GMT

It would be best to avoid having the double added in the first place. In inputs.conf, under where you define the input port, add:

no_appending_timestamp = true

From inputs.conf.spec documentation file:

no_appending_timestamp = true
* If this attribute is set to true, then Splunk does NOT append a timestamp and host to received events.
* NOTE: Do NOT include this key if you want to append timestamp and host to received events.