Hi,
We are writing a stackable file system. While testing this on NFS we
came across a situation where sometimes we are getting ASCII NUL bytes
in the buffer read from the lower file system (NFS in this case). This
happens when NFS client 1 is writing to a file and NFS client 2 is
continuously reading from the same file. A simple case like the
following also shows the same problem
----------------8<------------------8<-----------------------------
while (1)
{
if (vfs_getattr(filp->f_vfsmnt, filp->f_dentry, &stat) != 0)
{
printk(KERN_INFO "Error in getting attribute\n");
break;
}
if (no_inc == 60)
{
printk(KERN_INFO "File not increased for 1 min. Quitting\n");
break;
}
inc = stat.size - old_size;
if (inc)
{
old_size = stat.size;
no_inc = 0;
count = vfs_read(filp, buf,
sizeof(buf), &filp->f_pos);
if (count < 0)
{
printk(KERN_INFO "Error in reading file\n");
goto out;
}
printk(KERN_INFO "File increase = [%llu], total read = [%llu]"
" file_len = [%llu] File size from inode
= [%llu]\n",
inc, count, stat.size,
i_size_read(filp->f_dentry->d_inode));
printk(KERN_INFO
"----------------------------------------------------\n");
debug_dump("Read buffer", buf, total_count);
}
else
{
no_inc++;
}
}
----------------8<------------------8<-----------------------------
We have also noticed that the expected increase (inc) and the size
returned in (vfs_read()) is different. A similar program in user space
also reproduces this issue. A google search for this issue tells me
that others have seen this issue as well.
http://stackoverflow.com/questions/6814404/java-inputstream-read-methods-returning-ascii-nul-characters-for-file-in-a-nfs
Is this a known and expected behaviour of NFS?
PS : This is a very old kernel (2.6.18.274.) We still have to support RHEL 5 :)
--
Regards,
Ranjan
Hi Ranjan...
On Tue, Aug 14, 2012 at 2:53 PM, Ranjan Sinha [off-list ref] wrote:
Hi,
We are writing a stackable file system. While testing this on NFS we
came across a situation where sometimes we are getting ASCII NUL bytes
in the buffer read from the lower file system (NFS in this case). This
happens when NFS client 1 is writing to a file and NFS client 2 is
continuously reading from the same file.
are you using synchronous or asynchronous mode in the NFS?
--
regards,
Mulyadi Santosa
Freelance Linux trainer and consultant
blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com
Hi Mulyadi,
On Tue, Aug 14, 2012 at 3:33 PM, Mulyadi Santosa
[off-list ref] wrote:
Hi Ranjan...
On Tue, Aug 14, 2012 at 2:53 PM, Ranjan Sinha [off-list ref] wrote:
quoted
Hi,
We are writing a stackable file system. While testing this on NFS we
came across a situation where sometimes we are getting ASCII NUL bytes
in the buffer read from the lower file system (NFS in this case). This
happens when NFS client 1 is writing to a file and NFS client 2 is
continuously reading from the same file.
are you using synchronous or asynchronous mode in the NFS?
For now, /etc/export file has the following setting
*(rw,sync,no_root_squash)
On client side we have not specified any options explicitly. This is
from /proc/mounts entry
rw,vers=3,rsize=32768,wsize=32768,hard,proto=tcp,timeo=600,retrans=2,sec=sys
--
Regards,
Ranjan
Hi Ranjan...
On Tue, Aug 14, 2012 at 5:10 PM, Ranjan Sinha [off-list ref] wrote:
For now, /etc/export file has the following setting
*(rw,sync,no_root_squash)
hm, AFAIK that means synchronous method is selected. So,
theoritically, if there is no further data, the other end of NFS
should just wait.
Are you using blocking or non blocking read, btw? Sorry, i am not
really that good reading VFS code...
On client side we have not specified any options explicitly. This is
from /proc/mounts entry
hm, not sure, maybe in your case, read and write buffer should be
reduced so any new data should be transmitted ASAP. I was inspired by
bufferbloat handling, but maybe I am wrong here somewhere....
--
regards,
Mulyadi Santosa
Freelance Linux trainer and consultant
blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com
From: Rajat Sharma <hidden> Date: 2012-08-14 10:49:04
Try mounting with noac nfs mount option to disable attribute caching.
ac / noac
"Selects whether the client may cache file attributes. If neither
option is specified (or if ac is specified), the client caches file
attributes."
-Rajat
On Tue, Aug 14, 2012 at 3:51 PM, Mulyadi Santosa
[off-list ref] wrote:
Hi Ranjan...
On Tue, Aug 14, 2012 at 5:10 PM, Ranjan Sinha [off-list ref] wrote:
quoted
For now, /etc/export file has the following setting
*(rw,sync,no_root_squash)
hm, AFAIK that means synchronous method is selected. So,
theoritically, if there is no further data, the other end of NFS
should just wait.
Are you using blocking or non blocking read, btw? Sorry, i am not
really that good reading VFS code...
quoted
On client side we have not specified any options explicitly. This is
from /proc/mounts entry
hm, not sure, maybe in your case, read and write buffer should be
reduced so any new data should be transmitted ASAP. I was inspired by
bufferbloat handling, but maybe I am wrong here somewhere....
--
regards,
Mulyadi Santosa
Freelance Linux trainer and consultant
blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com
_______________________________________________
Kernelnewbies mailing list
Kernelnewbies at kernelnewbies.org
http://lists.kernelnewbies.org/mailman/listinfo/kernelnewbies
Hi,
On Tue, Aug 14, 2012 at 4:19 PM, Rajat Sharma [off-list ref] wrote:
Try mounting with noac nfs mount option to disable attribute caching.
ac / noac
"Selects whether the client may cache file attributes. If neither
option is specified (or if ac is specified), the client caches file
attributes."
i don't think this is because of attribute caching. The size does change and
that is why we go to the read call (think of this is a simplified case of
tail -f). The only problem is that sometimes when we read we get ASCII NUL bytes
at the end. If we read the same block again, we get the correct data.
In addition, we cannot force specific mount options in actual deployment
scenarios.
<edit>
quoted
On Tue, Aug 14, 2012 at 5:10 PM, Ranjan Sinha [off-list ref] wrote:
quoted
For now, /etc/export file has the following setting
*(rw,sync,no_root_squash)
hm, AFAIK that means synchronous method is selected. So,
theoritically, if there is no further data, the other end of NFS
should just wait.
Are you using blocking or non blocking read, btw? Sorry, i am not
really that good reading VFS code...
This is a blocking read call. I think this is not because there is no data,
rather somehow the updated data is not present in the VM buffers but the
inode size has changed. As I just said, if we read the file again from the
exact same location, we get the actual contents. Though after going through the
code I don't understand how is this possible.
quoted
quoted
On client side we have not specified any options explicitly. This is
from /proc/mounts entry
hm, not sure, maybe in your case, read and write buffer should be
reduced so any new data should be transmitted ASAP. I was inspired by
bufferbloat handling, but maybe I am wrong here somewhere....
From: Rajat Sharma <hidden> Date: 2012-08-16 07:30:37
Correct me if I am reading something wrong, in your program listing,
while printing the buffer you are passing a total_count variable,
while vfs_read returned value is collected in count variable.
debug_dump("Read buffer", buf, total_count);
One suggestion, please fill up buf with some fixed known pattern
before vfs_read.
We have also noticed that the expected increase (inc) and the size
returned in (vfs_read()) is different.
There is nothing which is blocking updates to file size between
vfs_getattr() and vfs_read(), right? no locking?
-Rajat
On Thu, Aug 16, 2012 at 12:01 PM, Ranjan Sinha [off-list ref] wrote:
Hi,
On Tue, Aug 14, 2012 at 4:19 PM, Rajat Sharma [off-list ref] wrote:
quoted
Try mounting with noac nfs mount option to disable attribute caching.
ac / noac
"Selects whether the client may cache file attributes. If neither
option is specified (or if ac is specified), the client caches file
attributes."
i don't think this is because of attribute caching. The size does change and
that is why we go to the read call (think of this is a simplified case of
tail -f). The only problem is that sometimes when we read we get ASCII NUL bytes
at the end. If we read the same block again, we get the correct data.
In addition, we cannot force specific mount options in actual deployment
scenarios.
<edit>
quoted
quoted
On Tue, Aug 14, 2012 at 5:10 PM, Ranjan Sinha [off-list ref] wrote:
quoted
For now, /etc/export file has the following setting
*(rw,sync,no_root_squash)
hm, AFAIK that means synchronous method is selected. So,
theoritically, if there is no further data, the other end of NFS
should just wait.
Are you using blocking or non blocking read, btw? Sorry, i am not
really that good reading VFS code...
This is a blocking read call. I think this is not because there is no data,
rather somehow the updated data is not present in the VM buffers but the
inode size has changed. As I just said, if we read the file again from the
exact same location, we get the actual contents. Though after going through the
code I don't understand how is this possible.
quoted
quoted
quoted
On client side we have not specified any options explicitly. This is
from /proc/mounts entry
hm, not sure, maybe in your case, read and write buffer should be
reduced so any new data should be transmitted ASAP. I was inspired by
bufferbloat handling, but maybe I am wrong here somewhere....
On Thu, Aug 16, 2012 at 1:00 PM, Rajat Sharma [off-list ref] wrote:
Correct me if I am reading something wrong, in your program listing,
while printing the buffer you are passing a total_count variable,
while vfs_read returned value is collected in count variable.
debug_dump("Read buffer", buf, total_count);
My apologies. Please read that as count only. A typo in the listing.
One suggestion, please fill up buf with some fixed known pattern
before vfs_read.
I tried that as well. It still comes out as ASCII NUL.
quoted
We have also noticed that the expected increase (inc) and the size
returned in (vfs_read()) is different.
There is nothing which is blocking updates to file size between
vfs_getattr() and vfs_read(), right? no locking?
No locking. On second thoughts I think this is ok since more data could be
available between the calls to vfs_getattr and vfs_read as the other NFS client
is continuously writing to that file.
--
Ranjan
-Rajat
On Thu, Aug 16, 2012 at 12:01 PM, Ranjan Sinha [off-list ref] wrote:
quoted
Hi,
On Tue, Aug 14, 2012 at 4:19 PM, Rajat Sharma [off-list ref] wrote:
quoted
Try mounting with noac nfs mount option to disable attribute caching.
ac / noac
"Selects whether the client may cache file attributes. If neither
option is specified (or if ac is specified), the client caches file
attributes."
i don't think this is because of attribute caching. The size does change and
that is why we go to the read call (think of this is a simplified case of
tail -f). The only problem is that sometimes when we read we get ASCII NUL bytes
at the end. If we read the same block again, we get the correct data.
In addition, we cannot force specific mount options in actual deployment
scenarios.
<edit>
quoted
quoted
On Tue, Aug 14, 2012 at 5:10 PM, Ranjan Sinha [off-list ref] wrote:
quoted
For now, /etc/export file has the following setting
*(rw,sync,no_root_squash)
hm, AFAIK that means synchronous method is selected. So,
theoritically, if there is no further data, the other end of NFS
should just wait.
Are you using blocking or non blocking read, btw? Sorry, i am not
really that good reading VFS code...
This is a blocking read call. I think this is not because there is no data,
rather somehow the updated data is not present in the VM buffers but the
inode size has changed. As I just said, if we read the file again from the
exact same location, we get the actual contents. Though after going through the
code I don't understand how is this possible.
quoted
quoted
quoted
On client side we have not specified any options explicitly. This is
from /proc/mounts entry
hm, not sure, maybe in your case, read and write buffer should be
reduced so any new data should be transmitted ASAP. I was inspired by
bufferbloat handling, but maybe I am wrong here somewhere....
From: Rajat Sharma <hidden> Date: 2012-08-16 08:03:57
What is the pattern other NFS client is writing to the file? Can't it
be a legitimate NUL by any chance?
On Thu, Aug 16, 2012 at 1:22 PM, Ranjan Sinha [off-list ref] wrote:
On Thu, Aug 16, 2012 at 1:00 PM, Rajat Sharma [off-list ref] wrote:
quoted
Correct me if I am reading something wrong, in your program listing,
while printing the buffer you are passing a total_count variable,
while vfs_read returned value is collected in count variable.
debug_dump("Read buffer", buf, total_count);
My apologies. Please read that as count only. A typo in the listing.
quoted
One suggestion, please fill up buf with some fixed known pattern
before vfs_read.
I tried that as well. It still comes out as ASCII NUL.
quoted
quoted
We have also noticed that the expected increase (inc) and the size
returned in (vfs_read()) is different.
There is nothing which is blocking updates to file size between
vfs_getattr() and vfs_read(), right? no locking?
No locking. On second thoughts I think this is ok since more data could be
available between the calls to vfs_getattr and vfs_read as the other NFS client
is continuously writing to that file.
--
Ranjan
quoted
-Rajat
On Thu, Aug 16, 2012 at 12:01 PM, Ranjan Sinha [off-list ref] wrote:
quoted
Hi,
On Tue, Aug 14, 2012 at 4:19 PM, Rajat Sharma [off-list ref] wrote:
quoted
Try mounting with noac nfs mount option to disable attribute caching.
ac / noac
"Selects whether the client may cache file attributes. If neither
option is specified (or if ac is specified), the client caches file
attributes."
i don't think this is because of attribute caching. The size does change and
that is why we go to the read call (think of this is a simplified case of
tail -f). The only problem is that sometimes when we read we get ASCII NUL bytes
at the end. If we read the same block again, we get the correct data.
In addition, we cannot force specific mount options in actual deployment
scenarios.
<edit>
quoted
quoted
On Tue, Aug 14, 2012 at 5:10 PM, Ranjan Sinha [off-list ref] wrote:
quoted
For now, /etc/export file has the following setting
*(rw,sync,no_root_squash)
hm, AFAIK that means synchronous method is selected. So,
theoritically, if there is no further data, the other end of NFS
should just wait.
Are you using blocking or non blocking read, btw? Sorry, i am not
really that good reading VFS code...
This is a blocking read call. I think this is not because there is no data,
rather somehow the updated data is not present in the VM buffers but the
inode size has changed. As I just said, if we read the file again from the
exact same location, we get the actual contents. Though after going through the
code I don't understand how is this possible.
quoted
quoted
quoted
On client side we have not specified any options explicitly. This is
from /proc/mounts entry
hm, not sure, maybe in your case, read and write buffer should be
reduced so any new data should be transmitted ASAP. I was inspired by
bufferbloat handling, but maybe I am wrong here somewhere....
On Thu, Aug 16, 2012 at 1:33 PM, Rajat Sharma [off-list ref] wrote:
What is the pattern other NFS client is writing to the file? Can't it
be a legitimate NUL by any chance?
Redirected output of ping.
On Thu, Aug 16, 2012 at 1:22 PM, Ranjan Sinha [off-list ref] wrote:
quoted
On Thu, Aug 16, 2012 at 1:00 PM, Rajat Sharma [off-list ref] wrote:
quoted
Correct me if I am reading something wrong, in your program listing,
while printing the buffer you are passing a total_count variable,
while vfs_read returned value is collected in count variable.
debug_dump("Read buffer", buf, total_count);
My apologies. Please read that as count only. A typo in the listing.
quoted
One suggestion, please fill up buf with some fixed known pattern
before vfs_read.
I tried that as well. It still comes out as ASCII NUL.
quoted
quoted
We have also noticed that the expected increase (inc) and the size
returned in (vfs_read()) is different.
There is nothing which is blocking updates to file size between
vfs_getattr() and vfs_read(), right? no locking?
No locking. On second thoughts I think this is ok since more data could be
available between the calls to vfs_getattr and vfs_read as the other NFS client
is continuously writing to that file.
--
Ranjan
quoted
-Rajat
On Thu, Aug 16, 2012 at 12:01 PM, Ranjan Sinha [off-list ref] wrote:
quoted
Hi,
On Tue, Aug 14, 2012 at 4:19 PM, Rajat Sharma [off-list ref] wrote:
quoted
Try mounting with noac nfs mount option to disable attribute caching.
ac / noac
"Selects whether the client may cache file attributes. If neither
option is specified (or if ac is specified), the client caches file
attributes."
i don't think this is because of attribute caching. The size does change and
that is why we go to the read call (think of this is a simplified case of
tail -f). The only problem is that sometimes when we read we get ASCII NUL bytes
at the end. If we read the same block again, we get the correct data.
In addition, we cannot force specific mount options in actual deployment
scenarios.
<edit>
quoted
quoted
On Tue, Aug 14, 2012 at 5:10 PM, Ranjan Sinha [off-list ref] wrote:
quoted
For now, /etc/export file has the following setting
*(rw,sync,no_root_squash)
hm, AFAIK that means synchronous method is selected. So,
theoritically, if there is no further data, the other end of NFS
should just wait.
Are you using blocking or non blocking read, btw? Sorry, i am not
really that good reading VFS code...
This is a blocking read call. I think this is not because there is no data,
rather somehow the updated data is not present in the VM buffers but the
inode size has changed. As I just said, if we read the file again from the
exact same location, we get the actual contents. Though after going through the
code I don't understand how is this possible.
quoted
quoted
quoted
On client side we have not specified any options explicitly. This is
from /proc/mounts entry
hm, not sure, maybe in your case, read and write buffer should be
reduced so any new data should be transmitted ASAP. I was inspired by
bufferbloat handling, but maybe I am wrong here somewhere....
From: Rajat Sharma <hidden> Date: 2012-08-16 16:59:07
So is it truncating the file? i.e.
# ping > /nfs/somefile
On Thu, Aug 16, 2012 at 2:46 PM, Ranjan Sinha [off-list ref] wrote:
On Thu, Aug 16, 2012 at 1:33 PM, Rajat Sharma [off-list ref] wrote:
quoted
What is the pattern other NFS client is writing to the file? Can't it
be a legitimate NUL by any chance?
Redirected output of ping.
quoted
On Thu, Aug 16, 2012 at 1:22 PM, Ranjan Sinha [off-list ref] wrote:
quoted
On Thu, Aug 16, 2012 at 1:00 PM, Rajat Sharma [off-list ref] wrote:
quoted
Correct me if I am reading something wrong, in your program listing,
while printing the buffer you are passing a total_count variable,
while vfs_read returned value is collected in count variable.
debug_dump("Read buffer", buf, total_count);
My apologies. Please read that as count only. A typo in the listing.
quoted
One suggestion, please fill up buf with some fixed known pattern
before vfs_read.
I tried that as well. It still comes out as ASCII NUL.
quoted
quoted
We have also noticed that the expected increase (inc) and the size
returned in (vfs_read()) is different.
There is nothing which is blocking updates to file size between
vfs_getattr() and vfs_read(), right? no locking?
No locking. On second thoughts I think this is ok since more data could be
available between the calls to vfs_getattr and vfs_read as the other NFS client
is continuously writing to that file.
--
Ranjan
quoted
-Rajat
On Thu, Aug 16, 2012 at 12:01 PM, Ranjan Sinha [off-list ref] wrote:
quoted
Hi,
On Tue, Aug 14, 2012 at 4:19 PM, Rajat Sharma [off-list ref] wrote:
quoted
Try mounting with noac nfs mount option to disable attribute caching.
ac / noac
"Selects whether the client may cache file attributes. If neither
option is specified (or if ac is specified), the client caches file
attributes."
i don't think this is because of attribute caching. The size does change and
that is why we go to the read call (think of this is a simplified case of
tail -f). The only problem is that sometimes when we read we get ASCII NUL bytes
at the end. If we read the same block again, we get the correct data.
In addition, we cannot force specific mount options in actual deployment
scenarios.
<edit>
quoted
quoted
On Tue, Aug 14, 2012 at 5:10 PM, Ranjan Sinha [off-list ref] wrote:
quoted
For now, /etc/export file has the following setting
*(rw,sync,no_root_squash)
hm, AFAIK that means synchronous method is selected. So,
theoritically, if there is no further data, the other end of NFS
should just wait.
Are you using blocking or non blocking read, btw? Sorry, i am not
really that good reading VFS code...
This is a blocking read call. I think this is not because there is no data,
rather somehow the updated data is not present in the VM buffers but the
inode size has changed. As I just said, if we read the file again from the
exact same location, we get the actual contents. Though after going through the
code I don't understand how is this possible.
quoted
quoted
quoted
On client side we have not specified any options explicitly. This is
from /proc/mounts entry
hm, not sure, maybe in your case, read and write buffer should be
reduced so any new data should be transmitted ASAP. I was inspired by
bufferbloat handling, but maybe I am wrong here somewhere....
On Thu, Aug 16, 2012 at 10:29 PM, Rajat Sharma [off-list ref] wrote:
So is it truncating the file? i.e.
# ping > /nfs/somefile
No appending ping >>/nfs/somefile
On Thu, Aug 16, 2012 at 2:46 PM, Ranjan Sinha [off-list ref] wrote:
quoted
On Thu, Aug 16, 2012 at 1:33 PM, Rajat Sharma [off-list ref] wrote:
quoted
What is the pattern other NFS client is writing to the file? Can't it
be a legitimate NUL by any chance?
Redirected output of ping.
quoted
On Thu, Aug 16, 2012 at 1:22 PM, Ranjan Sinha [off-list ref] wrote:
quoted
On Thu, Aug 16, 2012 at 1:00 PM, Rajat Sharma [off-list ref] wrote:
quoted
Correct me if I am reading something wrong, in your program listing,
while printing the buffer you are passing a total_count variable,
while vfs_read returned value is collected in count variable.
debug_dump("Read buffer", buf, total_count);
My apologies. Please read that as count only. A typo in the listing.
quoted
One suggestion, please fill up buf with some fixed known pattern
before vfs_read.
I tried that as well. It still comes out as ASCII NUL.
quoted
quoted
We have also noticed that the expected increase (inc) and the size
returned in (vfs_read()) is different.
There is nothing which is blocking updates to file size between
vfs_getattr() and vfs_read(), right? no locking?
No locking. On second thoughts I think this is ok since more data could be
available between the calls to vfs_getattr and vfs_read as the other NFS client
is continuously writing to that file.
--
Ranjan
quoted
-Rajat
On Thu, Aug 16, 2012 at 12:01 PM, Ranjan Sinha [off-list ref] wrote:
quoted
Hi,
On Tue, Aug 14, 2012 at 4:19 PM, Rajat Sharma [off-list ref] wrote:
quoted
Try mounting with noac nfs mount option to disable attribute caching.
ac / noac
"Selects whether the client may cache file attributes. If neither
option is specified (or if ac is specified), the client caches file
attributes."
i don't think this is because of attribute caching. The size does change and
that is why we go to the read call (think of this is a simplified case of
tail -f). The only problem is that sometimes when we read we get ASCII NUL bytes
at the end. If we read the same block again, we get the correct data.
In addition, we cannot force specific mount options in actual deployment
scenarios.
<edit>
quoted
quoted
On Tue, Aug 14, 2012 at 5:10 PM, Ranjan Sinha [off-list ref] wrote:
quoted
For now, /etc/export file has the following setting
*(rw,sync,no_root_squash)
hm, AFAIK that means synchronous method is selected. So,
theoritically, if there is no further data, the other end of NFS
should just wait.
Are you using blocking or non blocking read, btw? Sorry, i am not
really that good reading VFS code...
This is a blocking read call. I think this is not because there is no data,
rather somehow the updated data is not present in the VM buffers but the
inode size has changed. As I just said, if we read the file again from the
exact same location, we get the actual contents. Though after going through the
code I don't understand how is this possible.
quoted
quoted
quoted
On client side we have not specified any options explicitly. This is
from /proc/mounts entry
hm, not sure, maybe in your case, read and write buffer should be
reduced so any new data should be transmitted ASAP. I was inspired by
bufferbloat handling, but maybe I am wrong here somewhere....