I faced the exact same issue in one of the multi-site indexer clusters when I upgraded the indexer cluster from version 9.0.x to 9.0.5. After upgrading Splunk on the indexer, the virtual machine (VM) running the indexer unexpectedly went down. When I restarted the VM, I discovered that the Splunk service was already running, and the version displayed was the latest one. However, I failed to notice that it was experiencing problems connecting to the cluster manager. I completed the upgrade, but after a few days (around 15 days), the vulnerability management team requested another Splunk version upgrade. When I checked the Splunk version using the command, it displayed version 9.0.5. However, upon inspecting the $SPLUNK_HOME/etc/splunk.version file, I found that it still had the old version, indicating an unsuccessful upgrade. Realizing this, put the cluster master in maintenance mode, I stopped the Splunk service on the faulty indexer, cleared the standalone buckets using the commands mentioned below. Unfortunately, while restarting the Splunk service on the faulty indexer, the server went down again.   # finding standaralone buckets find $SPLUNK_DB/ -type d -name "db*" | grep -P "db_\d*_\d*_\d*$" #converting standardalone buckets to clustered buckets # 5A0E298B-0AFB-4d56-9dD0-A64dfdfd19DA8 is the GUID of cluster manager(master) find $SPLUNK_DB/ -type d -name "db_*" | grep -P "db_\d*_\d*_\d*$" |xargs -I {} mv {} {}_5A0E298B-0AFB-4d56-9dD0-A64dfdfd19DA8   I repeated this process two to three times, but it did not resolve the issue. Finally, I cleared the $SPLUNK_HOME/etc/instances.cfg file on the faulty indexer and restarted the service. This time, the indexer successfully joined the cluster. ... View more

Try this, ..yoursearch| rex field=main "(?s) (?<user>.*) your file is attached" | stats count by user sample search |stats count | eval main="hello Vasanth Kumar your file is attached,hello User One your file is attached,hello User two your file is attached,hello vasanth3 your file is attached" | eval main=split(main,",") | mvexpand main | rex field=main "(?s) (?<user>.*) your file is attached" | table main, user Hope this will help you.... Cheers! ... View more

Roboform still works. One (less than convenient) solution is, of course, to switch. ... View more

That seems a bit convoluted to me, is there a reason for not using key-value extraction capabilities in transforms.conf directly? props.conf [your_sourcetype] TRANSFORMS-keyvalue = PosTransactionProperties ... transforms.conf [PosTransactionProperties] REGEX = PosTransactionProperties\[\d+\]\.PosTransactionPropertyCode\[\d+\]=(?<_KEY_1>[^\n\r]+)[\n\r]+PosTransactionProperties\[\d+\]\.PosTransactionPropertyValue\[\d+\]=(?<_VAL_1>[^\n\r]+) This will extract both the field name and the field value using the magic _KEY_n and _VAL_n names. Nothing to do in the search itself or in calculated fields, the fields will just be there automatically. Edit: The reason for your approach not working out is that calculated fields ( EVAL-foo in props.conf) are applied after field extractions, so your transforms.conf rules don't see those fields. ... View more

Yes, it will replicate the data to 4th indexers whenever it joins the cluster. ... View more

This is very interesting, I will have to play with this a bit more, and the persistent queue comment was helpful Thankyou ... View more

In Splunk 6 you can have a SH search both clustered peers and one-off indexers. See here for more info http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/Configurethesearchhead#Search_across_both_clustered_and_non-clustered_search_peers ... View more

In this specific case, ing_rule would be 0 if there is an ing_clientip field - you can use that in an eval or calculated field. As a more general approach, you could extract different ing_user1, ing_user2, ... fields and assign ing_rule depending on which field is filled and after that merge them into one ing_user field. ... View more