I'm trying to determine why multiline events are not working when syslog sends the data over to my splunk indexer. The servers are setup to use log4j and send data to both the console (nfs) and syslog. I took the event from the console/nfs log and put it into a text file on the same system and used netcat to send the event to the splunk indexer and it worked just fine. Is there any way to determine what the problem is when the syslog server is sending the event? It's basically just creating single events for each line. Here the code multiline event: 2013-08-08 16:20:28,845 [instance=testserver.i1] [tomcat-http--49] INFO org.apache.cxf.interceptor.LoggingOutInterceptor (AbstractLoggingInterceptor.java:149) - Outbound Message ID: 190 Response-Code: 200 Content-Type: application/xml Headers: {Date=[Thu, 08 Aug 2013 20:20:28 GMT]} Payload: One line of data Another line of data data data data data data ... View more

I'm working on trying to get our application server's log4j working so that events will show up in searches for multiline events as a single event. There is also a problem where there are "..." breaks where the multiple lines should just continue on. Here is an example output: 2013-07-25 14:31:49,813 [pool-4-thread-1] INFO net.app.security.Filter (269) - Exiting ClusteredLogoutFilter#sessionCreated - Timestamp : 1374777109813 ---------------------------- ID: 843352 Address: /path/to/application Http-Method: GET Content-Type: application/json Headers: {content-type=[application/json], connection=[Keep-Alive], host=[myhost.com], accept-language=[en-US,en;q=0.8], x-forwarded-for=[1.1.1.1, 2.2.2.2], optr_cxt=[01000f0001eb4515a7-bdfc-42a0-8b4a-498d89c... ...05a4008cc0a81-7ab4-4e73-81b6-0e7358e69e6eb2edbc79-1b21-4e3a-bcf1-0e1ecfa297031 ;usraddr=0.0.0.0;usrname=;], x-forwarded-host=[myserver, mytest], user-agent=[Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.72 Safari/537.36], x-dds-access-path=[wan, prv], cookie=[JSESSIONID=C4E29C6715F5DDD7475CD2554EC4050B.myserver.env; __utma=43946937.444067536.1371496959.1371496959.1371562971.2; __utmz=43946937.1371496959.1.1.utmcsr=(... ...direct)|utmccn=(direct)|utmcmd=(none); LKC=22; LKCD=ltr; MODULEBASE=viewport; PAGEBASE=viewport%2Flogin%2F; MODULENAME=component; STGT=TGT-64657-z0sbkEt6gIbNzSobYudbakapZxoer5To3wI0hRkIsvU6mYPJLb-cas; osv-user-name=somebody; osv-user-remember=true; osv-user-org-continent=NA; SSOcookie=6C3E0895C362D9D2F76272B5930EF414.service; AppName=myapp; AppEnv=P], x-ssl=[0, 0], referer=... ...[somehost?os-cid=40006], accept-encoding=[gzip,deflate,sdch], x-forwarded-server=[fwd.server, otherhost], Content-Type=[application/json], Accept=[application/json]} -------------------------------------- I assume that this is a problem within my log4j/syslog configuration on the sending server that is causing this? The other thing, which I'm not sure that it's an issue is, say I search on "1.1.1.1" , only the line that has this will show up in the search, not the full set of lines that I'm expecting. I would have click on this entry and then do a "Show Source" to see the full lines. Any ideas would be great. I have this in my config: props.conf [log4j] TRANSFORMS--strip-syslog = syslog-header-stripper-ts-host [log4j] BREAK_ONLY_BEFORE = BREAK_ONLY_BEFORE_DATE = true SHOULD_LINEMERGE = true TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N TIME_PREFIX = ^ MAX_TIMESTAMP_LOOKAHEAD = 25 Thanks! Jeff ... View more

I have a logging share right on the splunk server where a number of webservers write a few logs to. The structure more or less would look like: (There are a number of "web" servers such as web-02, -03 ..etc) /opt/var/log/httpd/web-01.domain.com/access.log /opt/var/log/httpd/web-01.domain.com/access.log.1.gz /opt/var/log/httpd/web-01.domain.com/access.log.2.gz /opt/var/log/httpd/web-01.domain.com/tmp.dmp (There are a number of "app" servers such as app-02 , -03, ..etc) /opt/var/log/httpd/app-01.domain.com/access.log /opt/var/log/httpd/app-01.domain.com/access.log.1.gz /opt/var/log/httpd/app-01.domain.com/access.log.2.gz /opt/var/log/httpd/app-01.domain.com/tmp.dmp Note: the logs in each directory that I want to index are access.log , error.log, rewrite.log ..etc . Basically anything that is a current *.log file. What I want to do is only index the "web" servers and only the uncompressed *.log files. It should ignore the app server directories. I setup an input with the following: Host - regex on path Hostname (regex): /opt/var/log/httpd/([^/]+)/ Advanced Options Whitelist: I've tried the following: (all variations of that without escaping /) \/web-\d+\.domain\.com\/\w+\.log$ web-*\/*.log$ web-\d+\.*\/\w+\.log$ web-*\.log$ When I save the input and after a minute or so, the "Number of files" on the Input summary page shows 180 when it should only be indexing 24 files. Is that number on the Input summary page accurate? Is that the actual number of files that are being indexed, even after whitelisting? If so, what am I missing here? ... View more