Our Splunk server receives data via syslog. As a result, I need to transform the syslog data using transforms.conf and props.conf (Details in the question "Why does Splunk not recognize standard fields in my Apache data forwarded by syslog?". My question, can I transform the data and still do some field extraction on that data? I would like to preserve the process field. However, the default transform simply strips out the data. It doesn't save any of the fields. So, given the following transformation in local/props.conf : [syslog] TRANSFORMS-strip-syslog-header = syslog-header-stripper-ts-host-proc And this default transform from default/transforms.conf : # This will strip out date stamp, host, process with pid and just get the # actual message [syslog-header-stripper-ts-host-proc] REGEX = ^[A-Z][a-z]+\s+\d+\s\d+:\d+:\d+\s.*?:\s(.*)$ FORMAT = $1 DEST_KEY = _raw Can I somehow preserve one of the fields and save it to the name of process ? I have had some luck with the following pattern, saved at https://www.regex101.com/r/iK8iX5/1 . However, I am uncertain how to use this in a Splunk Transform. ^(?<SyslogPri><\d+>)(?<SyslogDate>[A-Z][a-z]+\s+\d+\s\d+:\d+:\d+)\s(?<SyslogHost>.*)\s(?<process>.*):\s(?<SyslogMessage>.*)$ ... View more

I have over 100 Apache webservers which forward their logs to a syslog-ng server, which then forwards the data a TCP data input on Splunk, as well as forwarding the data to other non-Splunk log-analysis servers. In Splunk Search, the data looks like this: Dec 16 10:29:59 192.168.99.100 httpd[10583]: site1.example.org 10.4.5.6 - - [16/Dec/2014:10:29:59 -0800] "GET /rest/somepath/12345" HTTP/1.1" 200 105066 "-" "-" Dec 16 10:29:59 192.168.99.101 httpd[22404]: site2.example.org 4.4.12.15 - someuser [16/Dec/2014:10:29:59 -0800] "GET /wiki/javascript/foo.js" HTTP/1.1" 304 - "https://site2.example.org/wiki/somepage.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36" Dec 16 10:29:59 192.168.6.100 httpd[6380]: site3.example.org 172.16.43.41 - - [16/Dec/2014:10:29:59 -0800] "GET /project/projectA/somescript.cgi?username=spiderman" 200 9048 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" However, Splunk recognizes only a few default fields in this data. It recognizes the host , process , source , sourcetype , data_hour , etc. It does not recognize Apache-specific fields like clientip status , method , etc. which are mentioned in the Splunk tutorial. It doesn't even recognize string like 4.4.12.15 as an IP address. As a result, I need to create a whole bunch of custom field extractions in order to do many useful tasks in Splunk. Why does Splunk not recognize fields in my Apache data? How can I transform the data so that Splunk will recognize the data correctly? Second question: Would it help if I used a Splunk Forwarder on our syslog server instead of using TCP for data input? ... View more

I know that Splunk can show me results for the last 24 hours. I also know that Splunk can show me results in real time, with times like a "1 minute window". I would like to see Splunk search results from the 24 hours, and also update these results in real time. This way, I can log into Splunk, bring up my dashboard and get a sense of what happened in the last day and show me what is going on from here on out. ... View more