[ENBD] Re: enbd + x86_64

Mon Jan 22 11:25:30 MST 2007

Peter T. Breuer wrote:

>Don't bother testing with anything except 2.6.15 as we know it works
>with that (I am running 2.6.15.5 on a perfectly standard debian sarge
>amd64 installation).
>  
>
All right, I'm using "linux-source-2.6.15_2.6.15-4_all.deb" for that now.

Your 2.6.15.5 is a vanilla kernel, isn't it?

>>gcc version 3.3.5 (Debian 1:3.3.5-13), all kernel sources are Debian ones.
>>    
>>
>
>It's too late night for me to want to go and see what my debian says
>(I have to trudge into another room and boot a noisy machine ...  maybe
>I'll do it in a little while) and what the compiler says about itself
>there, but I rather imagine it will be a gcc 4.1, no?
>  
>
No, 3.3.5. Hey, we're dealing with Debian/Sarge not Ubuntu or the like ;-)

>>3rd try was enbd 2.4.33 with a configured+built 2.6.18, didn't compile 
>>    
>>
>
>Don't try different things. Just one. We know it works against 2.6.15
>because that's what I used.
>  
>
Okay, I just tried 2.6.18 because 2.6.15 is no longer available in 
"testing".

>>(they removed split-include from scripts/basic).
>>    
>>
>
>Possibly - I wouldn't know! 2.6.15.5 is the most recent kernel I have
>looked at. If you can tell me what is needed under that source tree
>I'll be glad to modify the kernel Makefile used.
>  
>
First I have to understand the whole story when it's working. Then I can 
try it on a more recent kernel.

>>4th try was enbd 2.4.33 with a configured+built 2.6.15, it compiled with 
>>lots of "warning: long long int format, s64 arg" (seems to affect only 
>>    
>>
>
>No ... you shouldn't get any of those. You saw that my compile evoked
>nothing of the sort.
>
Yes, but your listing started with making enbd_base.o. The major part of 
the warnings I spoke about appears in mmap_cache.c and netserver.c which 
are treated before the start of your listing.

>Are you maybe compiling with DEBUG set, or
>something? Are you sure you are pointing at the right source directory?
>  
>
No and yes, respectively.

>Just copy one of the compile lines I showed in my log, and check you get
>the same result.
>  
>
There are only two differences (apart from different source- and 
tmp-paths, of course):

i) The names of my object files all start with ".tmp_" (e.g. 
"linux-2.6.x/drivers/block/enbd/.tmp_enbd_base.o" instead of your 
"linux-2.6.x/drivers/block/enbd/enbd_base.o").

ii) My modpost runs with option "-m" since I chose module versioning 
support when building the kernel.

>>message output, though).
>>
>>Module insertion doesn't show anything unusual (I suppose, the version 
>>mismatch "Enhanced Network Block Device 2.4.30 $Date: 2002/09/17 
>>16:33:22 $" in the log doesn't mean anything.).
>>    
>>
>
>Possibly. I forget where it comes from.
>  
>
kernel/linux-2.6.x/include/linux/enbd.h, line 52. Looks like 
discontinued version control.

>>Finally, upon "make test", the client suffers a SIGSEGV:
>>
>>/root/enbd-2.4.33/nbd/enbd-maketest: line 348:  4208 Segmentation 
>>fault      nice -19 $ENBD_CLIENT $OPTCLI
>>    
>>
>
>That's not the informative part. You need to look at your syslogs
>(kernel log for the oops). 
>
>  
>
kern.log says (no oops):

...
Jan 22 17:07:21 localhost kernel: ENBD #5600[0]: enbd_init registered 
device at major 43
Jan 22 17:08:06 localhost enbd-server[4199]: server (-2) sighandler 
propagates signal 15
Jan 22 17:08:06 localhost enbd-server[4199]: server (-2) sighandler 
terminates safely on signal 15
Jan 22 17:08:11 localhost enbd-server[7560]: server: will open resources 
in mode linear
Jan 22 17:08:11 localhost enbd-server[7560]: looking for size of fd 4 
with seek SEEK_END...
Jan 22 17:08:11 localhost enbd-server[7560]: set size of fd 4 to 4194304
Jan 22 17:08:11 localhost enbd-server[7560]: looking for size of fd 5 
with seek SEEK_END...
Jan 22 17:08:11 localhost enbd-server[7560]: set size of fd 5 to 4194304
Jan 22 17:08:11 localhost enbd-server[7560]: size of exported 
file/device is 8388608B (8192 blocks)
Jan 22 17:08:11 localhost enbd-server[7561]: server (-2) set new signal 
handlers
Jan 22 17:08:16 localhost enbd-client[7570]: enbd-client: client 
channels is 4
Jan 22 17:08:16 localhost enbd-client[7570]: enbd-client: client says 
blksize is 1024
Jan 22 17:08:16 localhost enbd-client[7570]: enbd-client: client 
identifier is NBDabcdefNBD
Jan 22 17:08:16 localhost enbd-client[7570]: enbd-client: client 
negotiation timeout is 120
Jan 22 17:08:16 localhost enbd-client[7570]: enbd-client: client pulse 
intvl is 5
Jan 22 17:08:16 localhost kernel: enbd-client[7570]: segfault at 
0000000000000000 rip 00002aaaaacfc05c rsp 00007fffffc63ae0 error 4
Jan 22 17:08:26 localhost kernel: ENBD #2442[1]: do_enbd_request device 
not enabled.
Jan 22 17:08:26 localhost kernel: printk: 999 messages suppressed.
...

>>Which details would you need for further investigation?
>>    
>>
>
>The very first error message shown (there are no errors here).
>  
>
I hope, the above is more helpful.

>
>OK, I'll go boot the machine. How does one tell which debian one has?
>  
>
/proc/version shows the kernel and its KCC.

> % uname -a
> Linux xilofon.it.uc3m.es 2.6.15.5 #2 PREEMPT Tue Mar 14 19:31:17 CET 2006 x86_64 GNU/Linux
>
> % gcc -v
>  Reading specs from /usr/lib/gcc-lib/x86_64-linux/3.3.5/specs
>  Configured with: ../src/configure -v
>  --enable-languages=c,c++,java,f77,pascal,objc,ada,treelang
>  --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info
>  --with-gxx-include-dir=/usr/include/c++/3.3 --enable-shared
>  --enable-__cxa_atexit --with-system-zlib --enable-nls
>  --without-included-gettext --enable-clocale=gnu --enable-debug
>  --enable-java-gc=boehm --enable-java-awt=xlib --enable-objc-gc
>  --disable-multilib x86_64-linux
>  Thread model: posix
>  gcc version 3.3.5 (Debian 1:3.3.5-13)
>
>That looks exactly the same as yours.
>  
>
Definitely identical here, yes.

>In the enbd 2.4.33 source directory, a "make" starts off ..
>
>xilofon:/usr/oboe/ptb/lang/c/nbd/enbd-2.4.33% make
>mkdir -p /tmp
>cp nbd/configure /tmp/configure && chmod +x /tmp/configure
>make config
>make[1]: Entering directory `/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33'
>export
>CONFIG_SITE=/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/conf/config.Linux;
>\
>cd /tmp;  ./configure
>--srcdir=/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd  \
>                                  --with-kernel-srcdir=/usr; \
>loading site script
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/conf/config.Linux
>creating cache ./config.cache
>checking for gcc... gcc
>checking whether the C compiler (gcc -D_LARGEFILE64_SOURCE=1
>-D_LARGEFILE_SOURCE=1 -D_GNU_SOURCE=1 -D_XOPEN_SOURCE=1
>-D_FILE_OFFSET_BITS=64 -Wall -Winline -O2 -I/usr/include ) works... yes
>checking whether the C compiler (gcc -D_LARGEFILE64_SOURCE=1
>-D_LARGEFILE_SOURCE=1 -D_GNU_SOURCE=1 -D_XOPEN_SOURCE=1
>-D_FILE_OFFSET_BITS=64 -Wall -Winline -O2 -I/usr/include ) is a
>cross-compiler... no
>checking whether we are using GNU C... yes
>checking whether gcc accepts -g... yes
>checking whether warnings should be enabled... yes
>...
>  
>
Same here, nice.

>Then all the compiles of the daemons start with ...
>
>make[1]: Leaving directory `/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33'
>make -C /tmp VPATH=/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd \
>                            CFLAGS="-D_LARGEFILE64_SOURCE=1 -D_LARGEFILE_SOURCE=1 -D_GNU_SOURCE=1 -D_XOPEN_SOURCE=1 -D_FILE_OFFSET_BITS=64 -Wall -Winline -O2 \
>                                  -I/tmp \
>                                  -I/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/kernel/linux-2.6.x/include \
>                                  -I/usr/include \
>                                     \
>                                  -DDEBUG=0" \
>                            EXTRA_LIBS=" \
>                                         "
>make[1]: Entering directory `/var/tmp'
>gcc -DCONFDIR="\"/etc\"" -DPIDDIR="\"/var/run\"" -DSTATEDIR="\"/var/state/enbd\"" -D_LARGEFILE64_SOURCE=1 -D_LARGEFILE_SOURCE=1 -D_GNU_SOURCE=1 -D_XOPEN_SOURCE=1 -D_FILE_OFFSET_BITS=64 -Wall -Winline -O2  -I/tmp  -I/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/kernel/linux-2.6.x/include  -I/usr/include   -DDEBUG=0 -o enbd-server.o -c /var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/enbd-server.c
>...
>
>There are a few warnings in mmap_cache.c now I look ..
>
>gcc -DCONFDIR="\"/etc\"" -DPIDDIR="\"/var/run\"" -DSTATEDIR="\"/var/state/enbd\"" -D_LARGEFILE64_SOURCE=1 -D_LARGEFILE_SOURCE=1 -D_GNU_SOURCE=1 -D_XOPEN_SOURCE=1 -D_FILE_OFFSET_BITS=64 -Wall -Winline -O2  -I/tmp  -I/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/kernel/linux-2.6.x/include  -I/usr/include   -DDEBUG=0 -o mmap_cache.o -c /var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c: In function `free_mmap_cache_entry':
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c:51: warning: long long int format, s64 arg (arg 4)
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c:51: warning: long long int format, s64 arg (arg 5)
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c:51: warning: long long int format, s64 arg (arg 4)
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c:51: warning: long long int format, s64 arg (arg 5)
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c: In function `add_mmap_cache_entry':
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c:163: warning: long long int format, s64 arg (arg 4)
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c:163: warning: long long int format, s64 arg (arg 5)
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c:163: warning: long long int format, s64 arg (arg 4)
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c:163: warning: long long int format, s64 arg (arg 5)
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c:183: warning: long long int format, s64 arg (arg 3)
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c:183: warning: long long int format, s64 arg (arg 4)
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c:183: warning: long long int format, s64 arg (arg 5)
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c:183: warning: long long int format, s64 arg (arg 6)
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c:183: warning: long long int format, s64 arg (arg 3)
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c:183: warning: long long int format, s64 arg (arg 4)
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c:183: warning: long long int format, s64 arg (arg 5)
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c:183: warning: long long int format, s64 arg (arg 6)
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c:243: warning: long long int format, s64 arg (arg 4)
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c:243: warning: long long int format, s64 arg (arg 5)
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c:243: warning: long long int format, s64 arg (arg 4)
>/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/nbd/mmap_cache.c:243: warning: long long int format, s64 arg (arg 5)
>
>I'll fix those.
>  
>
Those I meant, cf. above.

>By the time we get to the kernel source, it is saying ...
>
>make[1]: Entering directory `/usr/local/src/linux-2.6.15.5'
>*** Warning: Overriding SUBDIRS on the command line can cause
>***          inconsistencies
>mkdir -p /tmp/linux-2.6.x/.tmp_versions
>make -f /usr/local/src/linux-2.6.15.5/scripts/Makefile.build obj=/tmp/linux-2.6.x/drivers/block/enbd
>  gcc -I/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/kernel/linux-2.6.x/include -Wp,-MD,/tmp/linux-2.6.x/drivers/block/enbd/.enbd_base.o.d -nostdinc -iwithprefix include -D__KERNEL__ -Iinclude  -D__KERNEL__ -Iinclude  -Wall -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common  -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks        -Wno-sign-compare -fno-asynchronous-unwind-tables   -O2 -fomit-frame-pointer    -DMODULE -I"/var/home/oboe/ptb/lang/c/nbd/enbd-2.4.33/kernel/linux-2.6.x/include" -DKBUILD_BASENAME=enbd_base -DKBUILD_MODNAME=enbd -c -o /tmp/linux-2.6.x/drivers/block/enbd/enbd_base.o /tmp/linux-2.6.x/drivers/block/enbd/enbd_base.c
>
>Check your paths against that.
>  
>

Apart from different install- and tmp-paths (and the prepended ".tmp_"), 
the same here:

make[1]: Entering directory `/usr/src/linux-source-2.6.15'
*** Warning: Overriding SUBDIRS on the command line can cause
***          inconsistencies
mkdir -p /tmp/enbd/linux-2.6.x/.tmp_versions
make -f /usr/src/linux-source-2.6.15/scripts/Makefile.build 
obj=/tmp/enbd/linux-2.6.x/drivers/block/enbd
  gcc -I/root/enbd-2.4.33/kernel/linux-2.6.x/include 
-Wp,-MD,/tmp/enbd/linux-2.6.x/drivers/block/enbd/.enbd_base.o.d 
-nostdinc -iwithprefix include -D__KERNEL__ -Iinclude  -D__KERNEL__ 
-Iinclude  -Wall -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing 
-fno-common  -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks   
-Wno-sign-compare -fno-asynchronous-unwind-tables   -O2 
-fomit-frame-pointer    -DMODULE 
-I"/root/enbd-2.4.33/kernel/linux-2.6.x/include" 
-DKBUILD_BASENAME=enbd_base -DKBUILD_MODNAME=enbd -c -o 
/tmp/enbd/linux-2.6.x/drivers/block/enbd/.tmp_enbd_base.o 
/tmp/enbd/linux-2.6.x/drivers/block/enbd/enbd_base.c

>Now let's have a look at mmap_cache.c ... line 163 is
>
>           MSG ("1 adding separated mmap cache entry %d: %Ldk-%Ldk\n",
>                 mmap_cache_entry_count,
>                 mce->from >> 10, (mce->from + mce->len) >> 10);
>            return 0;
>
>and if it really wants me to cast those to long long, I guess I will!
>
>           MSG ("1 adding separated mmap cache entry %d: %Ldk-%Ldk\n",
>                 mmap_cache_entry_count,
>                 (long long)mce->from >> 10,
>                 (long long)(mce->from + mce->len) >> 10);
>            return 0;
>
>Lines 183 and 243 are just the same sort of thing. Fix the same way.
>Oh, line 51 too. That's it. Now all smooth.
>  
>
I changed also %d to %zd in PERR in netserver.c:366 and netserver.c:372 
and in printf("BAD...) in enbd-test.c:1245. In x86_64 world, the sizeof 
operator's results are no longer of type 'int' but 'unsigned long' :-(

>So show me the first error, and the compile line that goes with it, and 
>maybe the line that it refers to in the code.
>  
>
Oh, that's seems to have been a misunderstanding! No compile time error 
(except with kernel 2.6.18), just warnings. Alas, enbd-client crashes 
nevertheless.

That "line 348" is the last line in nbd/enbd-maketest.

>(BTW I fly to the UK tomorrow for a week .. comms may be spotty).
>  
>
No hurry, I highly appreciate your support in any case.

Cheers
          Lothar