Update on this with additional interesting tidbits... If I manually update [My Alert] in etc/users/robert/search/local/savedsearches.conf, set action.email.from = [email protected], then: Use Web UI to Clone [My Alert]: the cloned alert has action.email.from = [email protected] Use Web UI to edit [My Alert]: Splunk reverts [My Alert] to action.email.from = [email protected] We have many Splunk Reports that are schedule for nightly email delivery, and they ALL go out from [email protected] Minor correction to original post: Some Alert emails go out with no sender at all. The behavior is consistent per alert - that is the alerts that go out from [email protected] or [no sender] always do so. Some Alerts go to: [email protected] (as specified in etc/system/local/alert_actions.conf) [email protected] (not sure why) no sender ... View more

Thank you for your quick reply. In case I wasn't clear we have done Settings > Server settings > Email settings as described in http://docs.splunk.com/Documentation/Splunk/6.5.3/Alert/Emailnotification#Configure_email_notification_settings The values we set there are apparently saved by Splunk in the etc/system/local/alert_actions.conf which I listed above, most importantly the 'from = [email protected]' All our other alerts go out from '[email protected]' except this one which goes out from '[email protected]'. There must be something special about this alert but I cannot see any difference. It may be older than the other alerts, created at a time when we did have 'from = [email protected]' but I don't see how that would matter now... ... View more

Thank you for your reply! Turns out your second guess was correct: OUTPUT was the culprit. For some reason when used with savedsearch, the lookup must have an explicit OUTPUT foo AS something. In my case this macro works in normal search, but not from savedsearch: lookup my_test.csv key OUTPUT value This macro works in normal search, AND from savedsearch: lookup my_test.csv key OUTPUT value as value Here is an interesting additional wrinkle that may help someone... If the lookup macro has a trailing space ("lookup my_test.csv OUTPUT value AS value ") it will again work in normal search but again NOT from savedsearch. Wow. Perhaps @ssievert did not see this behavior because there are some Splunk bugs here that have been fixed since 6.2.1... Thanks again! ... View more

Lookup file was added directly to the filesystem (as with all our lookups), and looks like (from Settings/Lookups/Lookup table files page) in that case Splunk defaults to Sharing=Global. ... View more

The only interesting lines from search.log are: ... 02-06-2017 11:12:35.917 INFO LookupOperator - Assuming table name 'my_test.csv' is implicit filename. 02-06-2017 11:12:35.918 ERROR LookupOperator - Error in 'lookup' command: Could not find all of the specified destination fields in the lookup table. ... 02-06-2017 11:12:35.930 INFO ShutdownHandler - Shutdown complete in 281 microseconds 02-06-2017 11:12:35.931 ERROR dispatchRunner - RunDispatch::runDispatchThread threw error: Error in 'lookup' command: Could not find all of the specified destination fields in the lookup table. [End] ... View more

Thank you for quick reply. Sorry I should have included in the details my_macro is saved as Sharing=Global, with Permissions: Read access for Everyone. ... View more

Thanks for the bump. No we did not get a solution for this problem. Currently we just living with the double extraction 😞 ... View more

FWIW this solution also solved a similar problem we had with different symptom. After doing some (extensive) crash testing on one of our systems with a Splunk forwarder, the forwarder started leaving crash logs (e.g. [splunkforwarder]/var/logs/splunk/crash-2016-09-27-21:25:27.log) like this: ... Backtrace: [0x00007F1C68F88CC9] gsignal + 57 (/lib/x86_64-linux-gnu/libc.so.6) [0x00007F1C68F8C0D8] abort + 328 (/lib/x86_64-linux-gnu/libc.so.6) [0x00007F1C68F81B86] ? (/lib/x86_64-linux-gnu/libc.so.6) [0x00007F1C68F81C32] ? (/lib/x86_64-linux-gnu/libc.so.6) [0x0000000000E8440D] _ZN2bt5BTree4Node6readLEERK14FileDescriptor + 525 (splunkd) [0x0000000000E848B1] _ZN2bt5BTree4NodeC2ERK14FileDescriptorPKNS0_3KeyEj + 97 (splunkd) [0x0000000000E84EC4] _ZNK2bt5BTree4Node6acceptERK14FileDescriptorS4_RNS0_7VisitorEj + 148 (splunkd) [0x0000000000E82BC2] _ZNK2bt5BTree10checkValidERNS0_6RecordE + 130 (splunkd) [0x0000000000E8833B] _ZN2bt7BTreeCP10checkpointEv + 27 (splunkd) [0x0000000000E8A07A] _ZN2bt7BTreeCP9addUpdateERKNS_5BTree3KeyERKNS1_6RecordE + 458 (splunkd) [0x0000000000C3C4E6] _ZN11FileTracker9addUpdateERK5CRC_tRK10FishRecordb + 214 (splunkd) [0x000000000099A7D8] _ZN3WTF11resetOffsetEv + 280 (splunkd) [0x000000000099CF42] _ZN3WTF13loadFishStateEb + 1058 (splunkd) [0x000000000098895F] _ZN10TailReader8readFileER15WatchedTailFileP11TailWatcherP11BatchReader + 191 (splunkd) [0x0000000000988B7F] _ZN11TailWatcher8readFileER15WatchedTailFile + 127 (splunkd) [0x000000000098ADFC] _ZN11TailWatcher11fileChangedEP16WatchedFileStateRK7Timeval + 700 (splunkd) [0x0000000000EB8AA2] _ZN30FilesystemChangeInternalWorker15callFileChangedER7TimevalP16WatchedFileState + 114 (splunkd) [0x0000000000EBA430] _ZN30FilesystemChangeInternalWorker12when_expiredERy + 464 (splunkd) [0x0000000000F49B5D] _ZN11TimeoutHeap18runExpiredTimeoutsER7Timeval + 301 (splunkd) [0x0000000000EB3CB8] _ZN9EventLoop3runEv + 744 (splunkd) [0x000000000098916D] _ZN11TailWatcher3runEv + 141 (splunkd) [0x000000000098E91A] _ZN13TailingThread4mainEv + 154 (splunkd) [0x0000000000F4768E] _ZN6Thread8callMainEPv + 62 (splunkd) [0x00007F1C6931F182] ? (/lib/x86_64-linux-gnu/libpthread.so.0) [0x00007F1C6904C47D] clone + 109 (/lib/x86_64-linux-gnu/libc.so.6) ... The suggested btprobe cmd fixed this too. ... View more

This sounds reasonable, as we do currently have $ulimit -n 1024 eryder suggested in direct email to us to run: $ ./bin/splunk cmd btprobe -d [your path to...]/fishbucket/splunk_private_db -r This appears to have fixed the problem. Errors no longer seen after Splunk restart. Thanks! ... View more

Thanks I saw that issue too, but it wasn't clear to me what their solution was. Also we have different concerns, since this error is on one of our forwarders, not the indexer. We don't have backups of the forwarder filesystem. I was hoping since this a forwarder the solution might be easier. For example re-indexing some of the data from the forwarder would be acceptable... ... View more

Disappointed with the resolution as there is plenty of whitespace in our PDF for row labels. But at least I know to quit tinkering with this. Thanks. ... View more

Thank you. Since I am constrained to column-name alphabetical sort order inside the bar I ended up doing it like this (our search goes back up to 4 weeks which explains the magic 4 in eval title_sort below): ... earliest=@w-3d ... | eval week_start=relative_time(_time,"@w") | eval sort_start=relative_time(now(),"@w-3w") | eval title_sort=4-round((week_start-sort_start) / (60*60*24*7), 0) | eval week_label="(".title_sort.") ".strftime(week_start, "Week of %m-%d") ... Which gives column titles like this that sort chronologically (reverse alphabetically) inside the bars: (1) Week of 12-06 (2) Week of 11-29 ... The title_sort index numbers are just a bit more palatable that the numbers generated by 10000-year_number-week_number. ... View more

That's it, thank you. Too bad the Splunk Scheulde Dashboard PDF UI doesn't let me know my email will fail. Also curious why sending report email and sending dashboard PDF do not have the same capability rules. But working now so I am happy. ... View more

Thanks for responding. We are not using INDEXED_EXTRACTIONS as we have large json events with many fields and we don't want all the fields indexed. But the problem is similar to the duplicate fields folks see when using both KV_MODE=json and INDEXED_EXTRACTIONS=json... We do intentionally index the "job" field only. And this is the only field for which we see the duplicate fields, which makes sense since the "job" field is being extracted at index time and then again at search time (with KV_MODE=json). Of course we want KV_MODE=json for search time field extraction on the many other fields of the json event. ... View more

Thanks for responding. There is no duplication of keys. The output described above (double values for "job" field, but not "other_field") can be see with data like this: {"time":"2016-01-18T22:35:39.000Z","job":"job1","other_field":"other_value1"} {"time":"2016-01-19T22:35:39.000Z","job":"job2","other_field":"other_value2"} I think the problem is job field is extracted twice: once with our intentional index-time extraction (as shown in fields/transforms/props.conf), then again at search time with KV_MODE=json. The KV_MODE=json works great for all our other json fields, but is redundant for "job" field which has already been extracted. ... View more

Thank you, this is a great answer. Also explains the further mystery of why our severity=F query sometimes did work. For example, severity=F would find this event: F0106 09:02:03.592142 4628 other.cc:1042] command="f" I guess because the "f" is a segment so Splunk fetches (and presumably field-extracts) the event, then discovers hey there is a match on severity=F field. But this does seem to be a pretty hidden and serious limitation on Splunk search time field extractions: search time field extractions only work on event segments. I suppose our only option would be make severity an indexed field (index-time extraction). ... View more