Overwritten sourcetypes not searchable

[horsefez]

Hi fellow splunkers,

I ran into a problem regarding "Overwriting of an existing sourcetype via props and transforms".

Let me tell you more about my current scenario:
I have to connect some loadbalancers via syslog udp to a heavy forwarder.

I have the following stanza in inputs.conf

[udp://:1514]
connection_host=dns
index=loadbalancer
sourcetype=loadbalancer_syslog

So far the events get collected in the index=loadbalancer and sourcetype=loadbalancer_syslog.

I then noticed that the syslog-events contain a lot of messages from different system-deamons. So for me there is the task to split every format of every deamon into a different sourcetype.

So I did the following to the props.conf on the heavy forwarder:

[loadbalancer_syslog]
TRUNCATE = 10000
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE = (\w{3}\s+?\d{1,2}\s+?\d{2}\:+?\d{2}\:+?\d{2}\s+?\w+\.\w+\.\w+\.\w+\.\w+)
TRANSFORMS-changeSourcetype1 = logger-set-sourcetype, httpd-set-sourcetype, sshd-set-sourcetype, crond-set-sourcetype, tmm-set-sourcetype, mcpd-set-sourcetype, syslog-ng-set-sourcetype


[loadbalancer_logger]
[loadbalancer_httpd]
[loadbalancer_sshd]
[loadbalancer_crond]
[loadbalancer_tmm]
[loadbalancer_mcpd]
[loadbalancer_syslog-ng]

Then I did the corresponding stanz in the transforms.conf on the HF:

[logger-set-sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = (\w{3}\s+?\d{1,2}\s+?\d{2}\:+?\d{2}\:+?\d{2}\s+?\w+\.\w+\.\w+\.\w+\.\w+\s+?\w{3}\s+?\d{1,2}\s+?\d{2}\:+?\d{2}\:+?\d{2}\s+?\w+\s+?\w+\s+?logger)
FORMAT = loadbalancer_logger

[httpd-set-sourcetype]
DEST_KEY = MetaData:Sourcetype
REGEX = (\w{3}\s+?\d{1,2}\s+?\d{2}\:+?\d{2}\:+?\d{2}\s+?\w+\.\w+\.\w+\.\w+\.\w+\s+?\w{3}\s+?\d{1,2}\s+?\d{2}\:+?\d{2}\:+?\d{2}\s+?\w+\s+?\w+\s+?httpd)
FORMAT = loadbalancer_httpd

...[shortened]

---

When I search the data based on the index there are distinct values for the sourcetypes in the sourcetype-field, but I can't search for the sourcetype-values directly. It returns "0" events.

I can find data by searching via the original "loadbalancer_syslog" sourcetype, but not with the newly created ones. (very strange)

Anyone had this problem before?
Help is much apprechiated!

Regards,
pyro_wood

[skalliger · Accepted solution]

Hi,
had a similar problem and the FORMAT = is kinda confusing in the documentation.

FORMAT = sourcetype:: ... 

should do the trick. Notice the double "::". The extracted values from REGEX should be inserted properly then.

Skalli

[horsefez]

Worked like a charm!
Thanks!