[ENBD] Ammended setup: -> what would happen if...

Mon, 21 Apr 2003 11:15:59 +0000

This is a multi-part message in MIME format.
--------------090301040904040106050501
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit

Peter T. Breuer wrote:

>I thought I explained in my previous message. There is no "local to
>remote" mirroring in enbd, but 2.4.31 does have a raid1-like feature
>that amounts to clientside raid1 amongst enbd connections. That support
>has moved outside of the enbd driver and should now be done by
>ordinary raid1 over enbd, and indeed must be done by ordinary raid1
>if you want a local miror. If you use the fr1 driver (freshmeat)
>then disconnect and reconnect is automatic, and it will do an
>intelligent (minimal) resync too.
>
But how intelligent? (this was the original question)

The server doesn't necessarily care who the client is. So, the nbd 
server has a client disconnect. Then another client connects. The disk 
is changed. That client disconnects.

What if the first and second clients are not the same? Now, the disk has 
changed, and the first nbd client reconnects. It has a local (or remote) 
mirror that it's been using in the meantime, and it wants to resynch. 
Will the client:

a) resynch the changed blocks regardless of whether the server has 
changed or not

b) detect the changes between the server's state now versus when it 
disconnected, and perform a full resynch, using the part of the mirror 
that it's maintained connection to (local) as the master

c) detect the changes between the server's state now versus when it 
disconnected, and perform a full resynch, using the server that it just 
reconnected to as the master

d) give an error and ask for user intervention

I don't see any other real choices.

>>-> this is no longer an enbd question. Which should make answers more 
>>clear. I'd probably have to do a userspace synch: you're right, that I 
>>    
>>
>Userspace sync should not be necessary - that is what raid1 does.
>
But raid-1 isn't going to be adequate for doing the types of file synch 
that are necessary for this application, where there's multiple clients, 
etc. What I need is either a full-time file-system level mirror, or a 
caching filesystem. Or a userspace process that is run on reconnect to 
resynch the files and ask the user what do do about conflicts.

>>should be just looking at this as a hard disc. What I was most 
>>interested in is what behaviour the raid would exhibit... From this 
>>case, what I think is that the raid is going to take the most recently 
>>    
>>
>I'm not sure what you mean. A raid1 mirror of two components, in which
>one has failed, continues using the other component. When the failed
>component is replaced, then the replacement is synced in the background
>from the good component. In fr1 (as opposed to raid1) that resync is
>intelligent, only consisting of blocks that have been updated locally
>but not remotely. In fr1 the replacement is automatic too. In no
>scenario do you do any resync manually.
>
So, that would point to option b, or option a above.

>>synched superblock and give it precedence... but won't integrate the two 
>>except with a manually issued command. Which is the best case scenario 
>>for me -> I can't deal with a suddenly corrupted disc very well, but I 
>>can create a userspace tool to perform a resynch of the raid.
>>    
>>
>Raid is always resynced by the raid driver. That's much of the the point!
>
Yes, the driver does the resynch, but it waits for the superblock to be 
written onto the failed disk before it will use it, thus requiring 
manual intervention. You can't take out the failed disk, put the new one 
in, and have it go -> You have to tell the kernel to use it.

>>Which makes me think that I've got this setup totally wrong for the 
>>application -> I need to be doing the mirroring from the filesystem 
>>layer or on top of it, rather than at the bottom level. So, maybe using 
>>    
>>
>
>I'm not sure what you mean... you can either mirror at the filesystem
>level or at the block level. There are advantages and disadvantages to
>both. Omirr is the standard fs mirroring solution, I think.
>
I looked up this one, 'cause I hadn't heard of it before. It looks like 
it might have been the functionality that I'm looking for... however, it 
looks like it got dropped from the kernel (kernel traffic quote from 1999):

/According to Matthew Kirkwood, with omirr * "two filesystems could be 
kept in sync with one being the canonical version and the other 
(possibly remote, or slow) as a backup or similar." * He added, * "It 
went into the kernel at about 2.1.3x and out again at about 5x, where it 
became obvious that it was causing too many complications and was better 
left to userspace."*/

>>coda, which has some automatic caching features, or just a simpler rsych 
>>type setup... where the cache is also on an encrypted volume. And, I 
>>need to get some way of effectively failing the disks without crashing 
>>the client. No really easy solution comes to mind though.
>>    
>>
>
>I'm not sure what you mean .. if you are looking a failover, the distro
>comes with heartbeat scripts to do it (let me know of any problems :-).
>
There's 3 jobs that I need: failover, backup, and file-level 
synchronization. 2 out of 3 aint bad, but I'm holding out for a way to 
get the third.

Thanks for your help!

Cheers,
Liam

--------------090301040904040106050501
Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: 7bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1">
  <title></title>
</head>
<body>
Peter T. Breuer wrote:<br>
<blockquote type="cite"
 cite="mid200304201659.h3KGxVV13749@oboe.it.uc3m.es">
  <pre wrap="">I thought I explained in my previous message. There is no "local to
remote" mirroring in enbd, but 2.4.31 does have a raid1-like feature
that amounts to clientside raid1 amongst enbd connections. That support
has moved outside of the enbd driver and should now be done by
ordinary raid1 over enbd, and indeed must be done by ordinary raid1
if you want a local miror. If you use the fr1 driver (freshmeat)
then disconnect and reconnect is automatic, and it will do an
intelligent (minimal) resync too.</pre>
</blockquote>
But how intelligent? (this was the original question)<br>
<br>
The server doesn't necessarily care who the client is. So, the nbd
server has a client disconnect. Then another client connects. The disk
is changed. That client disconnects.<br>
<br>
What if the first and second clients are not the same? Now, the disk
has changed, and the first nbd client reconnects. It has a local (or
remote) mirror that it's been using in the meantime, and it wants to
resynch. Will the client:<br>
<br>
a) resynch the changed blocks regardless of whether the server has
changed or not<br>
<br>
b) detect the changes between the server's state now versus when it
disconnected, and perform a full resynch, using the part of the mirror
that it's maintained connection to (local) as the master<br>
<br>
c) detect the changes between the server's state now versus when it
disconnected, and perform a full resynch, using the server that it just
reconnected to as the master<br>
<br>
d) give an error and ask for user intervention<br>
<br>
I don't see any other real choices.<br>
<blockquote type="cite"
 cite="mid200304201659.h3KGxVV13749@oboe.it.uc3m.es">
  <blockquote type="cite">
    <pre wrap="">-&gt; this is no longer an enbd question. Which should make answers more 
clear. I'd probably have to do a userspace synch: you're right, that I 
    </pre>
  </blockquote>
  <pre wrap=""><!---->Userspace sync should not be necessary - that is what raid1 does.</pre>
</blockquote>
But raid-1 isn't going to be adequate for doing the types of file synch
that are necessary for this application, where there's multiple
clients, etc. What I need is either a full-time file-system level
mirror, or a caching filesystem. Or a userspace process that is run on
reconnect to resynch the files and ask the user what do do about
conflicts.<br>
<blockquote type="cite"
 cite="mid200304201659.h3KGxVV13749@oboe.it.uc3m.es">
  <blockquote type="cite">
    <pre wrap="">should be just looking at this as a hard disc. What I was most 
interested in is what behaviour the raid would exhibit... From this 
case, what I think is that the raid is going to take the most recently 
    </pre>
  </blockquote>
  <pre wrap=""><!---->I'm not sure what you mean. A raid1 mirror of two components, in which
one has failed, continues using the other component. When the failed
component is replaced, then the replacement is synced in the background
from the good component. In fr1 (as opposed to raid1) that resync is
intelligent, only consisting of blocks that have been updated locally
but not remotely. In fr1 the replacement is automatic too. In no
scenario do you do any resync manually.</pre>
</blockquote>
So, that would point to option b, or option a above.<br>
<blockquote type="cite"
 cite="mid200304201659.h3KGxVV13749@oboe.it.uc3m.es">
  <blockquote type="cite">
    <pre wrap="">synched superblock and give it precedence... but won't integrate the two 
except with a manually issued command. Which is the best case scenario 
for me -&gt; I can't deal with a suddenly corrupted disc very well, but I 
can create a userspace tool to perform a resynch of the raid.
    </pre>
  </blockquote>
  <pre wrap=""><!---->Raid is always resynced by the raid driver. That's much of the the point!</pre>
</blockquote>
Yes, the driver does the resynch, but it waits for the superblock to be
written onto the failed disk before it will use it, thus requiring
manual intervention. You can't take out the failed disk, put the new
one in, and have it go -&gt; You have to tell the kernel to use it.<br>
<blockquote type="cite"
 cite="mid200304201659.h3KGxVV13749@oboe.it.uc3m.es">
  <blockquote type="cite">
    <pre wrap="">Which makes me think that I've got this setup totally wrong for the 
application -&gt; I need to be doing the mirroring from the filesystem 
layer or on top of it, rather than at the bottom level. So, maybe using 
    </pre>
  </blockquote>
  <pre wrap=""><!---->
I'm not sure what you mean... you can either mirror at the filesystem
level or at the block level. There are advantages and disadvantages to
both. Omirr is the standard fs mirroring solution, I think.</pre>
</blockquote>
I looked up this one, 'cause I hadn't heard of it before. It looks like
it might have been the functionality that I'm looking for... however,
it looks like it got dropped from the kernel (kernel traffic quote from
1999):<br>
<br>
<i>According to Matthew Kirkwood, with omirr <b><font color="#880000">
"two filesystems could be kept in sync with one being the canonical
version and the other (possibly remote, or slow) as a backup or
similar." </font></b> He added, <b><font color="#880000"> "It went into
the kernel at about 2.1.3x and out again at about 5x, where it became
obvious that it was causing too many complications and was better left
to userspace."</font></b></i>
<blockquote type="cite"
 cite="mid200304201659.h3KGxVV13749@oboe.it.uc3m.es">
  <blockquote type="cite">
    <pre wrap="">coda, which has some automatic caching features, or just a simpler rsych 
type setup... where the cache is also on an encrypted volume. And, I 
need to get some way of effectively failing the disks without crashing 
the client. No really easy solution comes to mind though.
    </pre>
  </blockquote>
  <pre wrap=""><!---->
I'm not sure what you mean .. if you are looking a failover, the distro
comes with heartbeat scripts to do it (let me know of any problems :-).</pre>
</blockquote>
There's 3 jobs that I need: failover, backup, and file-level
synchronization. 2 out of 3 aint bad, but I'm holding out for a way to
get the third.<br>
<br>
Thanks for your help!<br>
<br>
Cheers,<br>
Liam<br>
</body>
</html>

--------------090301040904040106050501--