Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 330470

Summary: TCF file system does not list all entries within a directory
Product: [Tools] TCF Reporter: Uwe Stieber <uwe.st>
Component: CoreAssignee: Project Inbox <dsdp.tm.tcf-inbox>
Status: RESOLVED INVALID QA Contact: Martin Oberhuber <mober.at+eclipse>
Severity: major    
Priority: P3 CC: cdtdoug, eugene, felix.burton
Version: unspecified   
Target Milestone: ---   
Hardware: PC   
OS: Windows XP   
Whiteboard:

Description Uwe Stieber CLA 2010-11-17 11:24:08 EST 
Have the default TCF agent running on a openSUSE 11.3 box. If asking the TCF file system service to list the content of "/bin", it misses quite some parts of the directory content. For this example things like "/bin/bash" or "/bin/basename" are missing.

This is what it does return:

...
17:16:29.918: C:  145>FileSystem#opendir [/bin] 
17:16:29.949: R:  145> [null, FS7] 
17:16:29.949: C:  146>FileSystem#readdir [FS7] 
17:16:29.949: R:  146> [[{FileName=vim, Attrs={ATime=1289644819000, UID=0, MTime=1290010317000, Permissions=33261, GID=0, Size=0}}, {FileName=dmesg, Attrs={ATime=1278332054000, UID=0, MTime=1278332054000, Permissions=33261, GID=0, Size=5700}}, {FileName=fsync, Attrs={ATime=1280335567000, UID=0, MTime=1280335567000, Permissions=33261, GID=0, Size=5696}}, {FileName=ex, Attrs={ATime=1289644819000, UID=0, MTime=1290010317000, Permissions=33261, GID=0, Size=0}}, {FileName=getkeycodes, Attrs={ATime=1278330309000, UID=0, MTime=1278330309000, Permissions=33261, GID=0, Size=9816}}, {FileName=setmetamode, Attrs={ATime=1278330310000, UID=0, MTime=1278330310000, Permissions=33261, GID=0, Size=5748}}, {FileName=uname, Attrs={ATime=1289925531000, UID=0, MTime=1278331843000, Permissions=33261, GID=0, Size=22260}}, {FileName=pidof, Attrs={ATime=1289925529000, UID=0, MTime=1290010317000, Permissions=33261, GID=0, Size=0}}, {FileName=gunzip, Attrs={ATime=1287991004000, UID=0, MTime=1278009405000, Permissions=33261, GID=0, Size=2247}}, {FileName=testutf8, Attrs={ATime=1278330308000, UID=0, MTime=1278330308000, Permissions=33261, GID=0, Size=149}}, {FileName=ln, Attrs={ATime=1288809968000, UID=0, MTime=1278331840000, Permissions=33261, GID=0, Size=42820}}, {FileName=chvt, Attrs={ATime=1278330308000, UID=0, MTime=1278330308000, Permissions=33261, GID=0, Size=9808}}, {FileName=usleep, Attrs={ATime=1289925529000, UID=0, MTime=1280335567000, Permissions=33261, GID=0, Size=5700}}, {FileName=deallocvt, Attrs={ATime=1278330308000, UID=0, MTime=1278330308000, Permissions=33261, GID=0, Size=9812}}, {FileName=loadkeys, Attrs={ATime=1289925549000, UID=0, MTime=1278330309000, Permissions=33261, GID=0, Size=82632}}, {FileName=csh, Attrs={ATime=1287992248000, UID=0, MTime=1290010317000, Permissions=33261, GID=0, Size=0}}, {FileName=psfaddtable, Attrs={ATime=1278330309000, UID=0, MTime=1290010317000, Permissions=33261, GID=0, Size=0}}, {FileName=netstat, Attrs={ATime=1278329711000, UID=0, MTime=1278329711000, Permissions=33261, GID=0, Size=117160}}, {FileName=sync, Attrs={ATime=1289933003000, UID=0, MTime=1278331842000, Permissions=33261, GID=0, Size=22236}}, {FileName=egrep, Attrs={ATime=1289921955000, UID=0, MTime=1278328359000, Permissions=33261, GID=0, Size=84080}}, {FileName=vim-normal, Attrs={ATime=1289644819000, UID=0, MTime=1290010317000, Permissions=33261, GID=0, Size=0}}, {FileName=pkill, Attrs={ATime=1289925541000, UID=0, MTime=1278328323000, Permissions=33133, GID=0, Size=38736}}, {FileName=cpio, Attrs={ATime=1288437545000, UID=0, MTime=1282048208000, Permissions=33261, GID=0, Size=139120}}, {FileName=chmod, Attrs={ATime=1289925533000, UID=0, MTime=1278331839000, Permissions=33261, GID=0, Size=46908}}, {FileName=dd, Attrs={ATime=1289925532000, UID=0, MTime=1278331840000, Permissions=33261, GID=0, Size=51076}}, {FileName=resizecons, Attrs={ATime=1278330310000, UID=0, MTime=1278330310000, Permissions=33261, GID=0, Size=14040}}, {FileName=domainname, Attrs={ATime=1289925533000, UID=0, MTime=1290010317000, Permissions=33261, GID=0, Size=0}}, {FileName=rmdir, Attrs={ATime=1289932999000, UID=0, MTime=1278331841000, Permissions=33261, GID=0, Size=22264}}, {FileName=tar, Attrs={ATime=1288764911000, UID=0, MTime=1277998226000, Permissions=33261, GID=0, Size=367512}}, {FileName=date, Attrs={ATime=1290008706000, UID=0, MTime=1278331840000, Permissions=33261, GID=0, Size=55116}}, {FileName=grep, Attrs={ATime=1289925531000, UID=0, MTime=1278328359000, Permissions=33261, GID=0, Size=88208}}, {FileName=echo, Attrs={ATime=1289308409000, UID=0, MTime=1278331840000, Permissions=33261, GID=0, Size=22236}}, {FileName=kill, Attrs={ATime=1289933000000, UID=0, MTime=1278331840000, Permissions=33261, GID=0, Size=26808}}, {FileName=dbus-cleanup-sockets, Attrs={ATime=1278335123000, UID=0, MTime=1278335123000, Permissions=33261, GID=0, Size=9852}}, {FileName=cp, Attrs={ATime=1289925528000, UID=0, MTime=1278331840000, Permissions=33261, GID=0, Size=96324}}, {FileName=mv, Attrs={ATime=1289925532000, UID=0, MTime=1278331841000, Permissions=33261, GID=0, Size=88124}}, {FileName=more, Attrs={ATime=1285155129000, UID=0, MTime=1278332055000, Permissions=33261, GID=0, Size=34644}}, {FileName=ip, Attrs={ATime=1289925533000, UID=0, MTime=1290010317000, Permissions=33261, GID=0, Size=0}}, {FileName=eject, Attrs={ATime=1288810207000, UID=0, MTime=1278330339000, Permissions=35309, GID=17, Size=22640}}, {FileName=setfont, Attrs={ATime=1289925549000, UID=0, MTime=1278330310000, Permissions=33261, GID=0, Size=34660}}, {FileName=true, Attrs={ATime=1289318197000, UID=0, MTime=1278331843000, Permissions=33261, GID=0, Size=22228}}, {FileName=ksh, Attrs={ATime=1287992247000, UID=0, MTime=1290010317000, Permissions=33261, GID=0, Size=0}}, {FileName=pgrep, Attrs={ATime=1289925541000, UID=0, MTime=1278328323000, Permissions=33133, GID=0, Size=38736}}, {FileName=rm, Attrs={ATime=1289925531000, UID=0, MTime=1278331841000, Permissions=33261, GID=0, Size=51052}}, {FileName=dumpkeys, Attrs={ATime=1289925550000, UID=0, MTime=1278330308000, Permissions=33261, GID=0, Size=57848}}, {FileName=dbus-daemon, Attrs={ATime=1289925535000, UID=0, MTime=1278335123000, Permissions=33261, GID=0, Size=358416}}, {FileName=stty, Attrs={ATime=1289925528000, UID=0, MTime=1278331842000, Permissions=33261, GID=0, Size=46884}}, {FileName=awk, Attrs={ATime=1289925530000, UID=0, MTime=1290010317000, Permissions=33261, GID=0, Size=0}}, {FileName=psfgettable, Attrs={ATime=1278330309000, UID=0, MTime=1290010317000, Permissions=33261, GID=0, Size=0}}, {FileName=gzip, Attrs={ATime=1289925549000, UID=0, MTime=1278009405000, Permissions=33261, GID=0, Size=82172}}, {FileName=login, Attrs={ATime=1278331951000, UID=0, MTime=1278331951000, Permissions=33261, GID=0, Size=30700}}, {FileName=setserial, Attrs={ATime=1278328393000, UID=0, MTime=1278328393000, Permissions=33261, GID=0, Size=23208}}, {FileName=fgconsole, Attrs={ATime=1278330309000, UID=0, MTime=1278330309000, Permissions=33261, GID=0, Size=9808}}, {FileName=rpm, Attrs={ATime=1288432049000, UID=0, MTime=1277998692000, Permissions=33261, GID=0, Size=85576}}, {FileName=initviocons, Attrs={ATime=1278328731000, UID=0, MTime=1278328731000, Permissions=33261, GID=0, Size=19272}}, {FileName=ps, Attrs={ATime=1290008401000, UID=0, MTime=1278328323000, Permissions=33133, GID=0, Size=120976}}, {FileName=su, Attrs={ATime=1290008703000, UID=0, MTime=1278331842000, Permissions=35309, GID=0, Size=34756}}, {FileName=pwd, Attrs={ATime=1289308552000, UID=0, MTime=1278331841000, Permissions=33261, GID=0, Size=26396}}, {FileName=dbus-send, Attrs={ATime=1289925543000, UID=0, MTime=1278335123000, Permissions=33261, GID=0, Size=18152}}, {FileName=fuser, Attrs={ATime=1278328634000, UID=0, MTime=1278328634000, Permissions=33261, GID=0, Size=30800}}, {FileName=fillup, Attrs={ATime=1288437679000, UID=0, MTime=1278329380000, Permissions=33261, GID=0, Size=34472}}, {FileName=stat, Attrs={ATime=1289925528000, UID=0, MTime=1278331842000, Permissions=33261, GID=0, Size=42804}}, {FileName=lsmod, Attrs={ATime=1289925531000, UID=0, MTime=1277996404000, Permissions=33261, GID=0, Size=5700}}, {FileName=showkey, Attrs={ATime=1278330310000, UID=0, MTime=1278330310000, Permissions=33261, GID=0, Size=9828}}], null, false] 
17:16:29.965: C:  147>FileSystem#close [FS7] 
17:16:29.965: R:  147> [null] 
...

This is what it should list:

-rwxr-xr-x 1 root root   22K 2010-07-05 14:10 arch
lrwxrwxrwx 1 root root    21 2010-07-17 17:03 awk -> /etc/alternatives/awk
-rwxr-xr-x 1 root root   22K 2010-07-05 14:10 basename
-rwxr-xr-x 1 root root  590K 2010-07-05 13:47 bash
-rwxr-xr-x 1 root root   46K 2010-07-05 14:10 cat
-rwxr-xr-x 1 root root   50K 2010-07-05 14:10 chgrp
-rwxr-xr-x 1 root root   46K 2010-07-05 14:10 chmod
-rwxr-xr-x 1 root root   50K 2010-07-05 14:10 chown
-rwxr-xr-x 1 root root  9.6K 2010-07-05 13:45 chvt
-rwxr-xr-x 1 root root   95K 2010-07-05 14:10 cp
-rwxr-xr-x 1 root root  136K 2010-08-17 14:30 cpio
lrwxrwxrwx 1 root root     4 2010-07-17 17:05 csh -> tcsh
-rwxr-xr-x 1 root root   54K 2010-07-05 14:10 date
-rwxr-xr-x 1 root root  9.7K 2010-07-05 15:05 dbus-cleanup-sockets
-rwxr-xr-x 1 root root  351K 2010-07-05 15:05 dbus-daemon
-rwxr-xr-x 1 root root   14K 2010-07-05 15:05 dbus-monitor
-rwxr-xr-x 1 root root   18K 2010-07-05 15:05 dbus-send
-rwxr-xr-x 1 root root  9.6K 2010-07-05 15:05 dbus-uuidgen
-rwxr-xr-x 1 root root   50K 2010-07-05 14:10 dd
-rwxr-xr-x 1 root root  9.6K 2010-07-05 13:45 deallocvt
-rwxr-xr-x 1 root root   66K 2010-07-05 14:10 df
-rwxr-xr-x 1 root root  5.6K 2010-07-05 14:14 dmesg
lrwxrwxrwx 1 root root     8 2010-07-17 16:59 dnsdomainname -> hostname
lrwxrwxrwx 1 root root     8 2010-07-17 16:59 domainname -> hostname
-rwxr-xr-x 1 root root   57K 2010-07-05 13:45 dumpkeys
-rwxr-xr-x 1 root root   22K 2010-07-05 14:10 echo
-rwxr-xr-x 1 root root   47K 2010-07-05 13:08 ed
-rwxr-xr-x 1 root root   83K 2010-07-05 13:12 egrep
-rwsr-xr-x 1 root audio  23K 2010-07-05 13:45 eject
lrwxrwxrwx 1 root root     3 2010-07-17 17:01 ex -> vim
-rwxr-xr-x 1 root root   22K 2010-07-05 14:10 false
-rwxr-xr-x 1 root root  9.6K 2010-07-05 13:45 fgconsole
-rwxr-xr-x 1 root root   51K 2010-07-05 13:12 fgrep
-rwxr-xr-x 1 root root   34K 2010-07-05 13:29 fillup
-rwxr-xr-x 1 root root  163K 2010-07-05 13:13 find
-rwxr-xr-x 1 root root  5.6K 2010-07-28 18:46 fsync
-rwxr-xr-x 1 root root   31K 2010-07-05 13:17 fuser
-rwxr-xr-x 1 root root     0 2010-11-17 17:11 gawk
-rwxr-xr-x 1 root root  9.6K 2010-07-05 13:45 getkeycodes
-rwxr-xr-x 1 root root   87K 2010-07-05 13:12 grep
-rwxr-xr-x 1 root root  8.0K 2010-07-05 13:45 guess_encoding
-rwxr-xr-x 1 root root  2.2K 2010-07-01 20:36 gunzip
-rwxr-xr-x 1 root root   81K 2010-07-01 20:36 gzip
-rwxr-xr-x 1 root root     0 2010-11-17 17:11 hostname
-rwxr-xr-x 1 root root   19K 2010-07-05 13:18 initviocons
lrwxrwxrwx 1 root root     8 2010-07-17 17:01 ip -> /sbin/ip
-rwxr-xr-x 1 root root   339 2010-07-28 18:36 ipg
-rwxr-xr-x 1 root root  9.6K 2010-07-05 13:45 kbd_mode
-rwxr-xr-x 1 root root  9.7K 2010-07-05 13:45 kbdrate
-rwxr-xr-x 1 root root   22K 2010-07-01 16:58 keyctl
-rwxr-xr-x 1 root root   27K 2010-07-05 14:10 kill
lrwxrwxrwx 1 root root    21 2010-07-17 17:08 ksh -> /etc/alternatives/ksh
lrwxrwxrwx 1 root root    16 2010-07-17 17:08 ksh93 -> /lib/ast/bin/ksh
-rwxr-xr-x 1 root root   42K 2010-07-05 14:10 ln
-rwxr-xr-x 1 root root   81K 2010-07-05 13:45 loadkeys
-rwxr-xr-x 1 root root   22K 2010-07-05 13:45 loadunimap
-rwxr-xr-x 1 root root   10K 2010-07-05 14:14 logger
-rwxr-xr-x 1 root root   30K 2010-07-05 14:12 login
-rwxr-xr-x 1 root root  103K 2010-07-05 14:10 ls
-rwxr-xr-x 1 root root  5.6K 2010-07-01 17:00 lsmod
lrwxrwxrwx 1 root root    14 2010-07-17 17:11 mail -> /usr/bin/mailx
-rwxr-xr-x 1 root root   18K 2010-07-05 13:45 mapscrn
-rwxr-xr-x 1 root root   30K 2010-07-05 14:10 md5sum
-rwxr-xr-x 1 root root   30K 2010-07-05 14:10 mkdir
-rwxr-xr-x 1 root root   30K 2010-07-05 14:10 mknod
-rwxr-xr-x 1 root root   30K 2010-07-05 14:10 mktemp
-rwxr-xr-x 1 root root   34K 2010-07-05 14:14 more
-rwsr-xr-x 1 root root   91K 2010-07-05 14:14 mount
-rwxr-xr-x 1 root root  5.6K 2010-07-28 18:46 mountpoint
-rwxr-xr-x 1 root root   87K 2010-07-05 14:10 mv
-rwxr-xr-x 1 root root  115K 2010-07-05 13:35 netstat
-rwxr-xr-x 2 root root  9.7K 2010-07-05 13:47 nisdomainname
-rwxr-xr-x 1 root root   14K 2010-07-05 13:45 openvt
-r-xr-xr-x 2 root root   38K 2010-07-05 13:12 pgrep
lrwxrwxrwx 1 root root    14 2010-08-31 18:24 pidof -> /sbin/killall5
-rwsr-xr-x 1 root root   39K 2010-07-28 18:36 ping
-rwsr-xr-x 1 root root   35K 2010-07-28 18:36 ping6
-r-xr-xr-x 2 root root   38K 2010-07-05 13:12 pkill
-r-xr-xr-x 1 root root  119K 2010-07-05 13:12 ps
lrwxrwxrwx 1 root root     9 2010-07-17 17:05 psfaddtable -> psfxtable
lrwxrwxrwx 1 root root     9 2010-07-17 17:05 psfgettable -> psfxtable
lrwxrwxrwx 1 root root     9 2010-07-17 17:05 psfstriptable -> psfxtable
-rwxr-xr-x 1 root root     0 2010-11-17 17:11 psfxtable
-rwxr-xr-x 1 root root   26K 2010-07-05 14:10 pwd
-rwxr-xr-x 1 root root   34K 2010-07-05 14:10 readlink
-rwxr-xr-x 1 root root   14K 2010-07-05 13:45 resizecons
-rwxr-xr-x 1 root root   50K 2010-07-05 14:10 rm
-rwxr-xr-x 1 root root   22K 2010-07-05 14:10 rmdir
-rwxr-xr-x 1 root root   84K 2010-07-01 17:38 rpm
-rwxr-xr-x 1 root root  135K 2010-07-05 13:41 sash
-rwxr-xr-x 1 root root   55K 2010-07-05 13:14 sed
-rwxr-xr-x 1 root root   34K 2010-07-05 13:45 setfont
-rwxr-xr-x 1 root root  9.6K 2010-07-05 13:45 setkeycodes
-rwxr-xr-x 1 root root  9.6K 2010-07-05 13:45 setleds
-rwxr-xr-x 1 root root  5.7K 2010-07-05 13:45 setmetamode
-rwxr-xr-x 1 root root   23K 2010-07-05 13:13 setserial
lrwxrwxrwx 1 root root     4 2010-07-17 17:01 sh -> bash
-rwxr-xr-x 1 root root   14K 2010-07-05 13:45 showconsolefont
-rwxr-xr-x 1 root root  9.6K 2010-07-05 13:45 showkey
-rwxr-xr-x 1 root root   22K 2010-07-05 14:10 sleep
-rwxr-xr-x 1 root root   87K 2010-07-05 14:10 sort
-rwxr-xr-x 1 root root   42K 2010-07-05 14:10 stat
-rwxr-xr-x 1 root root   46K 2010-07-05 14:10 stty
-rwsr-xr-x 1 root root   34K 2010-07-05 14:10 su
-rwxr-xr-x 1 root root   22K 2010-07-05 14:10 sync
-rwxr-xr-x 1 root root  359K 2010-07-01 17:30 tar
-rwxr-xr-x 1 root root     0 2010-11-17 17:11 tcsh
-rwxr-xr-x 1 root root   149 2010-07-05 13:45 testutf8
-rwxr-xr-x 1 root root   46K 2010-07-05 14:10 touch
-rwxr-xr-x 1 root root   22K 2010-07-05 14:10 true
-rwsr-xr-x 1 root root   67K 2010-07-05 14:14 umount
-rwxr-xr-x 1 root root   22K 2010-07-05 14:10 uname
-rwxr-xr-x 1 root root  3.5K 2010-07-05 13:45 unicode_start
-rwxr-xr-x 1 root root   363 2010-07-05 13:45 unicode_stop
-rwxr-xr-x 1 root root  5.6K 2010-07-28 18:46 usleep
lrwxrwxrwx 1 root root     3 2010-07-17 17:01 vi -> vim
lrwxrwxrwx 1 root root    21 2010-07-17 17:01 vim -> /etc/alternatives/vim
-rwxr-xr-x 1 root root     0 2010-11-17 17:11 vim-normal
-rwxr-xr-x 2 root root  9.7K 2010-07-05 13:47 ypdomainname
-rwxr-xr-x 1 root root  1.9K 2010-07-01 20:36 zcat
-rwxr-xr-x 1 root root  619K 2010-07-05 21:31 zsh
Comment 1 Eugene Tarassov CLA 2010-11-17 14:32:21 EST 
The log shows that the directory handle was closed before reading full directory contents. Reading a long directory needs multiple FileSystem#readdir commands. Clients are supposed to exit the reading loop and close the handle when readdir returns eof=true.

Not a bug.
Comment 2 Martin Oberhuber CLA 2010-11-17 17:13:54 EST 
So that's multiple client/server roundtrips? Hm... I cannot think of many use cases where a client would want to read only part of a directory. It's confusing. What's the rationale in not sending the entire directory?
Comment 3 Eugene Tarassov CLA 2010-11-17 17:41:35 EST 
(In reply to comment #2)
> So that's multiple client/server roundtrips? Hm... I cannot think of many use
> cases where a client would want to read only part of a directory. It's
> confusing. What's the rationale in not sending the entire directory?

Rationale is ability to limit buffer size and fair sharing of bandwidth.

In any case, TCF File System protocol is derived from SSH File Transfer Protocol, which, in turn, is derived from POSIX API. To me, deviation from well established standards would be confusing, while following them makes perfect sense.
Comment 4 Felix Burton CLA 2010-11-17 17:52:06 EST 
(In reply to comment #3)
> (In reply to comment #2)
> > So that's multiple client/server roundtrips? Hm... I cannot think of many use
> > cases where a client would want to read only part of a directory. It's
> > confusing. What's the rationale in not sending the entire directory?
> Rationale is ability to limit buffer size and fair sharing of bandwidth.
> In any case, TCF File System protocol is derived from SSH File Transfer
> Protocol, which, in turn, is derived from POSIX API. To me, deviation from well
> established standards would be confusing, while following them makes perfect
> sense.

In addition to Eugene's comment, since TCF is an asynchronous protocol, a client that is concerned about performance could issue multiple readdir (and other data retireval) requests to increase flow of data, for example issue two readdir requests at the time and as soon as one is complete issue a next one.  For really high latency connections, the client could have more than two simultaneous requests if necessary.

This allows badwidth sharing, through-put and partial directory reads.
Comment 5 Martin Oberhuber CLA 2010-11-18 00:28:32 EST 
(In reply to comment #4)
> In addition to Eugene's comment, since TCF is an asynchronous protocol, a
> client that is concerned about performance could issue multiple readdir (and

Right... that exactly addresses my concern, which was about latency. It felt odd forcing the client to wait for the client/server roundtrip time multiple times.

I agree that following the known POSIX semantics makes sense. Uwe, do you think anything else should be done to avoid the small trap that you apparently ran into?
Comment 6 Uwe Stieber CLA 2010-11-18 01:59:33 EST 
>Uwe, do you think anything else should be done to avoid the small trap that you 
>apparently ran into?

No, I'm fine with the explanations given here. It's unexpected from a standard I/O API point of view, but I'm completely fine sending multiple requests until EOF. It will populate the UI in multiple steps than.

Thanks! :)
Comment 7 Doug Schaefer CLA 2011-05-17 10:48:16 EDT 
Moving bugs to new home for IP log.