quoted

quoted

On 5/5/2015 at 08:10 PM, in message [ref], Abhijit

Bhopatkar [off-list ref] wrote: 

On 05/05/15 3:14 pm, Abhijit Bhopatkar wrote: 

quoted

On 05/05/15 2:52 pm, Lidong Zhong wrote: 

quoted

quoted

quoted

quoted

On 5/1/2015 at 02:36 AM, in message [ref], Abhijit 

Bhopatkar [off-list ref] wrote: 

 
<snip> 
 

quoted

quoted

quoted

To illustrate the problem consider timeline for two senders and one 
receiver (we will ignore receive part for Sender2 node) 

Sender1              Sender2                         Receiver 
Get EX on TOKEN       Get EX on TOKEN 
<Granted>                    <Wait till granted> 

Get EX on MSG 
write LVB 
down MSG to CR 
Get EX of ACK 
<wait till granted> 
       BAST for ACK 
                                                              Get CR on MSG 
                      read LVB 
                      process 
                      release ACK 
AST for ACK 
down ACK to CR 
release MSG 
release TOKEN 
                     <granted> 
                     Get EX on MSG 

I am afraid this corner case could not be achieved ever. Sender2 will be  

blocked on getting 

quoted

quoted

EX lock on MSG resource until the receivers release the lock. The  

receivers' request on 

quoted

quoted

upconverting CR to EX on MSG should be put into the convert queue before  

Sender2's 

quoted

quoted

request being put into the wait queue, because sender2 has to wait until  

the EX on TOKEN 

quoted

quoted

is released. 

Yes my initial though of losing a message is not correct. The EX on message  

won't be granted 

quoted

immediately to Sender2 However there is still a deadlock. 

Perhaps i am missing something, but according to me nothing prevents  

Sender2 from acquiring 

quoted

EX on TOKEN _and_ MESSAGE __before__ up convert from reciever is queued.   

Consider adding 

quoted

unusual delay right after ACK is released on receiver. The Sender1 will  

immediately release 

quoted

MESSAGE and TOKEN. The receiver is still delayed for whatever reason.  

Sender2 gets TOKEN grant 

quoted

and immediately queues EX for MESSAGE (note this is before EX for MESSAGE  

is queued by receiver). 

quoted

Yes, there is a possibility leading to deadlock here.

quoted

DLM will (should?) return error for the up convert saying there is deadlock  

(-EDEADLK ??) 

quoted

 
On further investigation in dlm code. Since we do not set DLM_LKF_CONVDEADLK  
flag on our locks, 
in above deadlock case receiver's request to up convert will be simply  
canceled. And the code 
will proceed as expected since receiver still holds CR on MESSAGE. And then  
after the processing 
we will release the CR. 
 
So now my question is changed to; 
 
Why do we up convert the MESSAGE to EX in the first place? 
 
Was receiver EX on MESSAGE intended to serialize all receivers before taking  
CR on ACK? 
 

Yes, it is. Otherwise, each receiver may get duplicate messages when they try to
get CR on ACK while the sender doesn't downconvert EX on ACK in time.

What I can think of a way to fix the deadlock now is setting the DLM_LKF_NOQUEUE
flag when the sender tries to get EX on MESSAGE. It should keep trying until all the 
receivers release their locks on MESSAGE. Do you have any better idea without adding
more lock resources? Since we already have three for transmitting messages.

Regards,
Lidong

Since there is a possibility that we might lose out on this up convert in a  
race  condition, can 
we simply eliminate this up conversion? (since CR is preventing the next  
Sender from taking 
EX on MESSAGE anyway). 
 
Regards, 
Abhijit 
 
-- 
To unsubscribe from this list: send the line "unsubscribe linux-raid" in 
the body of a message to majordomo@vger.kernel.org 
More majordomo info at  http://vger.kernel.org/majordomo-info.html