Hello! We recently migrated our Splunk instance (running v7.1.2) to a new server, also running 7.1.2. Prior to migration, on the original server, we rolled hot buckets to warm, then synced all buckets (warm and cold) to the colddb indexes of the target server. This worked fine, except we have an 8-hour gap of missing logs. This gap was at the time of our final data sync, not surprisingly. We did another diff and found some buckets that weren't copied over. No big deal, we copied them over... but we're still missing 8 hours of data. How do we go out getting that 8-hour gap filled in with its events? I'm not sure where else to check, but am wondering if we missed a step somewhere. We still have access to the buckets on the original server if we need to get to them. Also, we are (were) planning to upgrade the new Splunk server to the latest 7.2 release. Would it be a bad idea to perform this upgrade while troubleshooting missing events? Sounds like it might be. Does anyone have any suggestions? Thanks! ... View more

Hi. We are migrating our Splunk instance to a new server. We do not want it to re-index a directory that we have as a monitor. It was recommended that I copy over our fishbucket. I'm looking in /opt/splunk/var/lib/splunk and I see _thefishbucket.dat, but it's empty. How can I prevent a re-index if my fishbucket is empty? Thanks! ... View more

Hello. I am having trouble with a complicated query. Here's what I'm trying to do: We have events from IIS w3svc1 logs which include requestor IPs. For example: 2018-09-03 15:47:10 yy.yy.yy.yy GET //Catalog/ year=2018-2019 80 - xx.xx.xx.xx WordPress/4.9.6;+https://somewhere.com https://somewhere.com/Public/Catalog/?year=2018-2019 200 0 0 62 We also have standard WinEventLog:Application events that include the username that authenticated but does not include the requestor IP address. When the WinEventLog:Application event has "MESSAGE="Login success", I want to correlate that event with the w3svc1 access log event at the same exact time stamp. In other words, I want to determine which user successfully logged in at a specific time by referencing two separate logs. I can create the two separate queries, I'm unsure how to connect the two. With the help of DalJeanis below, I have the following so far: (index=wineventlog host=somehost* USER=johndoe MESSAGE="Login success") OR (index=ag host=somehost* user="-" NOT requestor_ip=1.2.3.4 NOT requestor_ip=5.6.7.8 NOT username=*bot*) | fields USER, requestor_ip | eval new_time=if(match(sourcetype,"wineventlog"), _time-10,_time) | sort 0 new_time | streamstats time_window=30s last(USER) as USER | where match(sourcetype,"w3svc1") I see where this is going... but I'm having trouble with streamstats complaining: Error in 'streamstats' command: time_window can only be used on input that is sorted in time order (both ascending and descending order are ok) I think the issue is that I'm having trouble with the match function in the eval. I'm not sure what was meant in the comment by "Type A record" and Type B record. My guess was that it's a way to distinguish between the two searches, so I used their sourcetypes for comparison. Maybe that's the wrong approach? To work around the error, I tried this: (index=wineventlog host=somehost* USER=johndoe MESSAGE="Login success") OR (index=ag host=somehost* user="-" NOT requestor_ip=1.2.3.4 NOT requestor_ip=5.6.7.8 NOT username=*bot*) | transaction requestor_ip maxspan=30s | fields USER, requestor_ip | eval new_time=if(match(sourcetype,"wineventlog"), _time-10,_time) | sort 0 new_time | streamstats last(USER) as USER | where match(sourcetype,"w3svc1") The problem is it's returning over 5000 results for the past 24 hours. It should only be returning maybe 2 or 3. It's as if the USER=johndoe constraint isn't working... Upon closer inspection, it looks like it's not even performing search A... Thanks! ... View more

I have a log file of properly formatted JSON events, but the event break is not working properly. Sometimes it separates the JSON into separate events, sometimes it does not. There doesn't seem to be any rhyme or reason to this. I tried the solution here: https://answers.splunk.com/answers/80741/event-break-json.html but it did not work. I am unable to restart Splunk at this time, however, but my understanding is that I shouldn't need to. (Please correct me if I'm wrong.) Here's my props.conf entry: [s-web] KV_MODE = json LINE_BREAKER = "(^){" NO_BINARY_CHECK = 1 TRUNCATE = 0 SHOULD_LINEMERGE = false Here's a sample event: {"pid":17156,"hostname":"sub.hostname.com","name":"s-undefined","level":30,"time":1515143225539,"remoteAddr":"::ffff:99.99.99.99","remoteAddrs":[],"method":"GET","url":"/","sessionId":"abcd2b32-00e8-4e0b-97f6-23abcdef3233e","v":1} Am I missing something here? Thank you in advance for your assistance! ... View more

Hi. I have an JSON event that has nested arrays of objects within it. In the Search app, it "prettifies" the top level of the JSON event, but does not provide us with a "pretty" form of nested arrays/objects. Unfortunately, the data is sensitive so I cannot provide a screenshot. But basically I'd like to be able to click the +/- sign and drill down into the nested JSON event. Instead, nested JSON is represented by a single string which is difficult to read. I can use spath to extract the nested JSON fields, that's works, but we'd like to be able to explore the JSON in a drill-down fashion within the Search app. I have verified that our JSON is valid. Is there something I need to do to get the GUI to understand nested JSON? Sorry if this question seems vague, just not sure how else to ask. Thanks! ... View more

Can Splunk be configured to allow for interpreting JSON objects with multiple-levels of depth? Here's an example: { level: warn message: {"invalidPublication":"Publication is valid for indexing at Elasticsearch and will be updated, but has warnings.","authors":[{"lastName":"foo","initials":"fb","firstName":"bar","authorResourceID":99999}],"title":"Some Title","warningReasons":["Invalid value for 'publicationDate' field [Sat Apr 01 2006 00:00:00 GMT-0500 (EST)], year not found in citation - dateComponents: [{\"year\":\"2008\",\"month\":\"6\",\"day\":\"2\"}].]"]} pid: 2888 sourceHostname: somehostname.somewhere.com timestamp: 2017-03-13 09:55:40 } In the above example, I would like the “messages” field to be interpreted by Splunk so that I can expand/collapse each section inside the message. Right now, it just displays nested JSON as a single string. Is this possible? Thanks! ... View more