Hi to all that read this, Hoping one of you might be able to provide some assistance. We have an app that is producing logs using Extended Common web format. Right now the source type we are using is linked to the access-extractions transform, but is not giving all the required fields. I have tried a number of different approaches to get the required values using regex, but due to the nature of the logs, it feels like I might need a large number of regex entries to capture all variations. After figuring out that we were using the access-extractions transform, I though a better approach would be to edit this to suit - however I'm still pretty new to regex and not really sure what the regex in this transform is actually doing or how it works. A sample of the logs we are working with: 10.x.x.x www.blah.au - [20/Aug/2018:08:06:19 +1000] "GET /ebs/picmi/picmirepository.nsf/PICMI?OpenForm&t=PI&k=D&r=http%3A%2F%2Fwww.assediomoral.org%2Findex.php%2Fspip.php%3Farticle106 HTTP/1.1" 200 53245 "http://a.bla.es/?u=https%3A%2F%2Fwww.ebs.tga.gov.au%2Febs%2Fpicmi%2Fpicmirepository.nsf%2FPICMI%3FOpenForm%26t%3DPI%26k%3DD%26r%3Dhttp%253A%252F%252Fwww.assediomoral.org%252Findex.php%252Fspip.php%253Farticle106" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.189 Safari/537.36 Vivaldi/1.95.1077.60" 422 "" "d:/Lotus/Domino/data/ebs/picmi/picmirepository.nsf" 10.x.x.x www.blah.au "107831_67744" [20/Aug/2018:08:06:19 +1000] "GET /ebs/lm/lmdrafts.nsf/xAgentUpdateValidationMonitoring.xsp?documentId=7D35903C63DAEB54CA2582C000426C09&dojo.preventCache=1534716380650 HTTP/1.1" 200 78 "https://www.ebs.tga.gov.au/ebs/LM/LMDrafts.nsf/GenApp.xsp?documentId=7d35903c63daeb54ca2582c000426c09&action=editDocument" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1.2 Safari/605.1.15" 31 "_ga=GA1.3.644697231.1517015993; _gid=GA1.3.1541615874.1534641115; DomAuthSessId=A004127B4D088BDBD4B14B7E1BF0928B; WelcomeDialogLM=1; SessionID=9E1B7E03146C77042992C7B008ABB7DB303BC2AD" "d:/Lotus/Domino/data/ebs/lm/lmdrafts.nsf" 10.x.x.x www.blah.au - [20/Aug/2018:08:06:15 +1000] "GET /ebs/picmi/picmirepository.nsf/PICMI?OpenForm&t=PI&k=D&r=http%3A%2F%2Fwww2.ogs.state.ny.us%2Fhelp%2Furlstatusgo.html%3Furl%3Dhttp%253A%252F%252Fpedagogie.ac-toulouse.fr%252Feco-golfech%252Fspip.php%253Farticle129 HTTP/1.1" 200 53566 "https://www.apemsa.es/web/guest/analisis-de-agua/-/asset_publisher/7OQq/content/dureza?redirect=https%3A%2F%2Fwww.ebs.tga.gov.au%2Febs%2Fpicmi%2Fpicmirepository.nsf%2FPICMI%3FOpenForm%26t%3DPI%26k%3DD%26r%3Dhttp%253A%252F%252Fwww2.ogs.state.ny.us%252Fhelp%252Furlstatusgo.html%253Furl%253Dhttp%25253A%25252F%25252Fpedagogie.ac-toulouse.fr%25252Feco-golfech%25252Fspip.php%25253Farticle129" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.170 Safari/537.36,gzip(gfe)" 282 "" "d:/Lotus/Domino/data/ebs/picmi/picmirepository.nsf" 10.x.x.x www.blah.au - [20/Aug/2018:08:06:15 +1000] "GET /ebs/picmi/picmirepository.nsf/PICMI?OpenForm&t=PI&k=D&r=http%3A%2F%2Fwww2.ogs.state.ny.us%2Fhelp%2Furlstatusgo.html%3Furl%3Dhttp%253A%252F%252Fpedagogie.ac-toulouse.fr%252Feco-golfech%252Fspip.php%253Farticle129 HTTP/1.1" 200 53566 "https://www.apemsa.es/web/guest/analisis-de-agua/-/asset_publisher/7OQq/content/dureza?redirect=https%3A%2F%2Fwww.ebs.tga.gov.au%2Febs%2Fpicmi%2Fpicmirepository.nsf%2FPICMI%3FOpenForm%26t%3DPI%26k%3DD%26r%3Dhttp%253A%252F%252Fwww2.ogs.state.ny.us%252Fhelp%252Furlstatusgo.html%253Furl%253Dhttp%25253A%25252F%25252Fpedagogie.ac-toulouse.fr%25252Feco-golfech%25252Fspip.php%25253Farticle129" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.170 Safari/537.36,gzip(gfe)" 282 "" "d:/Lotus/Domino/data/ebs/picmi/picmirepository.nsf" The particular fields that we are after are the last 3 which represent the time to process, cookie header and translated URL. Regex from access-extractions: ^[[nspaces:clientip]]\s++[[nspaces:ident]]\s++[[nspaces:user]]\s++[[sbstring:req_time]]\s++[[access-request]]\s++[[nspaces:status]]\s++[[nspaces:bytes]](?:\s++"(?<referer>[[bc_domain:referer_]]?+[^"]*+)"(?:\s++[[qstring:useragent]](?:\s++[[qstring:cookie]])?+)?+)?[[all:other]] I'm assuming I need to update the last part of this "[[all:other]]" but have tried running this in GUI search box and in regex101, neither seem to be able to work with it so struggling to understand how to update correctly. ... View more

I'm trying to on-board a new application and having issues from the get go. Application is IBM IIB and outputs logs in to a number of sub directories beneath 1 parent logs directory. Logs are single events per log file and a mix of XML and json formats. I have started with the xml versions and one of the sub folders to confirm that my inputs were working correctly. There are 3 files in the directory and all 3 show as being read, but only 2 made it to the index and show in search. Trying to understand the cause of the 3rd not making it to the indexer. Inputs: [monitor://Q:\IIB_Log\CustomerDominoConsumer\ActivateDeactivateClient\mqsiarchive\*.log] disabled = false index = hprg_applog_nonprod sourcetype = hprg:iib:xml After setting this and seeing the issue - I ran “splunk list inputstatus” on the server where the files reside and I can see the below output regarding the files: Q:\IIB_Log\CustomerDominoConsumer\ActivateDeactivateClient\mqsiarchive\*.log type = directory Q:\IIB_Log\CustomerDominoConsumer\ActivateDeactivateClient\mqsiarchive\20180410_214256_061062_CustomerDominoConsumer_ActivateDeateClient_SOAPRequest.log file position = 313 file size = 313 parent = Q:\IIB_Log\CustomerDominoConsumer\ActivateDeactivateClient\mqsiarchive\*.log percent = 100.00 type = finished reading Q:\IIB_Log\CustomerDominoConsumer\ActivateDeactivateClient\mqsiarchive\20180410_233834_470779_CustomerDominoConsumer_ActivateDeateClient_SOAPRequest.log file position = 313 file size = 313 parent = Q:\IIB_Log\CustomerDominoConsumer\ActivateDeactivateClient\mqsiarchive\*.log percent = 100.00 type = finished reading Q:\IIB_Log\CustomerDominoConsumer\ActivateDeactivateClient\mqsiarchive\20180410_234012_541048_CustomerDominoConsumer_ActivateDeateClient_SOAPRequest.log file position = 313 file size = 313 parent = Q:\IIB_Log\CustomerDominoConsumer\ActivateDeactivateClient\mqsiarchive\*.log percent = 100.00 type = finished reading which to me suggests that Splunk has read and process all 3 files, yet when I run the search “index=_internal host=DWxxxxxS31 q:\” I get the below: 05/07/2018 15:38:09.884 07-05-2018 15:38:09.884 +1000 INFO Metrics - group=per_source_thruput, series="q:\iib_log\customerdominoconsumer\activatedeactivateclient\mqsiarchive\20180410_234012_541048_customerdominoconsumer_activatedeactivateclient_soaprequest.log", kbps=0.009649, eps=0.063136, kb=0.305664, ev=2, avg_age=3682772.500000, max_age=7365545 host = DWxxxxxS31 source = C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log sourcetype = splunkd 05/07/2018 15:38:09.884 07-05-2018 15:38:09.884 +1000 INFO Metrics - group=per_source_thruput, series="q:\iib_log\customerdominoconsumer\activatedeactivateclient\mqsiarchive\20180410_214256_061062_customerdominoconsumer_activatedeactivateclient_soaprequest.log", kbps=0.009649, eps=0.063136, kb=0.305664, ev=2, avg_age=3687826.500000, max_age=7375653 host = DWxxxxxS31 source = C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log sourcetype = splunkd 05/07/2018 15:37:39.025 07-05-2018 15:37:39.025 +1000 INFO TailingProcessor - Adding watch on path: Q:\IIB_Log\CustomerDominoConsumer\ActivateDeactivateClient\mqsiarchive. host = DWxxxxxS31 source = C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log sourcetype = splunkd 05/07/2018 15:37:39.025 07-05-2018 15:37:39.025 +1000 INFO TailingProcessor - Parsing configuration stanza: monitor://Q:\IIB_Log\CustomerDominoConsumer\ActivateDeactivateClient\mqsiarchive\*.log. host = DWxxxxxS31 source = C:\Program Files\SplunkUniversalForwarder\var\log\splunk\splunkd.log sourcetype = splunkd which to me suggests Splunk has only processed 2 of the 3 files, which also fits with the fact that when I search the sourcetype for the input, I’m only getting 2 results. Looking for any tips / suggestions that might help me trouble shoot this issue. ... View more

Splunk Version 6.6.2 I am getting lack of space errors due to poor set-up of our Splunk environment and am trying to resolve, but having issues. The error I'm currently receiving (there were others, but this seems to be the last one) is below: Search peer server3 has the following message: Disk Monitor: The index processor has paused data flow. Current free disk space on partition '/' has fallen to 4492MB, below the minimum of 5000MB. Data writes to index path '/opt/splunkhot/_audit/db'cannot safely proceed. Increase free disk space on partition '/' by removing or relocating data. Steps taken so far: I have followed the steps outlined here "https://docs.splunk.com/Documentation/Splunk/7.0.1/Indexer/Moveanindex" to move the indexes to a new location. I have also updated /opt/splunk/etc/splunk-launch.conf with the following: SPLUNK_DB=/var/lib/db/splunkhot Splunk has been restarted on indexes and cluster master yet I'm still seeing the above error after restarting. Not sure what else to check / update. Update: So.... I have determined the cause of my issue - but now I'm not sure of the best steps to resolve. Within master-apps, I have a indexes app which is defining a number of apps specifically - eg not using $SPLUNK_DB in the path. How does one update this without breaking index integrity? Under normal processes, one would shutdown the indexer, relocate the indexes and update the path, restart service and all good. But when I deploy the app to update - this will restart the services on the index servers automatically - not giving me a chance to copy the indexes. Can I copy prior to pushing the update? or is there a method of deploying where the services are not restarted automatically? ... View more