I'm trying to think of a good way to figure out where space is going on
somewhat large and busy filesystems. Often I will be notified by nagios that I
am running out of space somewhere, and it will even tell me what the trend in
usage is. So now I know that I lost a bunch of space recently.
There are several things I can do at that point:
- Check quota usage. If quotas are working correctly, this can sometimes
help when one user is obviously using a ton of space. More often than not,
there is nothing obvious, quotas aren't working or the space is used by system
accounts that provide no help in knowing where the space went.
- Use du. This is just plain painful and time consuming. Doing this usually
causes too much load.
Without knowing other ways to find information, I was thinking it would be
really nice to have a way to see which files or directories grew or shrank
since some point in the past. Would it be possible to get the kernel to tell
me when files were opened for writing and closed so I could record the size
difference? I know there are various file notify mechanisms, but from what I
can see, they don't handle millions of files very well.
Of course I'm open to other ideas, but this was my curiosity.
On Wed, Aug 10, 2011 at 07:06:29PM -0400, Chris wrote:
I'm trying to think of a good way to figure out where space is going on
somewhat large and busy filesystems. Often I will be notified by nagios that I
am running out of space somewhere, and it will even tell me what the trend in
usage is. So now I know that I lost a bunch of space recently.
There are several things I can do at that point:
- Check quota usage. If quotas are working correctly, this can sometimes
help when one user is obviously using a ton of space. More often than not,
there is nothing obvious, quotas aren't working or the space is used by system
accounts that provide no help in knowing where the space went.
- Use du. This is just plain painful and time consuming. Doing this usually
causes too much load.
Without knowing other ways to find information, I was thinking it would be
really nice to have a way to see which files or directories grew or shrank
since some point in the past. Would it be possible to get the kernel to tell
me when files were opened for writing and closed so I could record the size
difference? I know there are various file notify mechanisms, but from what I
can see, they don't handle millions of files very well.
Of course I'm open to other ideas, but this was my curiosity.
From: Vaughn Clinton <hidden> Date: 2011-08-11 08:21:50
I like "inotify". I've become somewhat familiar with the interface and I
believe this will allow you to do what you want.
There are plenty "C" examples out there.
Cheers,
-----Original Message-----
From: Adam Lee
Sent: Thursday, August 11, 2011 12:17 PM
To: kernelnewbies at kernelnewbies.org
Subject: Re: Tracing file size changes
On Wed, Aug 10, 2011 at 07:06:29PM -0400, Chris wrote:
I'm trying to think of a good way to figure out where space is going on
somewhat large and busy filesystems. Often I will be notified by nagios
that I
am running out of space somewhere, and it will even tell me what the trend
in
usage is. So now I know that I lost a bunch of space recently.
There are several things I can do at that point:
- Check quota usage. If quotas are working correctly, this can sometimes
help when one user is obviously using a ton of space. More often than
not,
there is nothing obvious, quotas aren't working or the space is used by
system
accounts that provide no help in knowing where the space went.
- Use du. This is just plain painful and time consuming. Doing this
usually
causes too much load.
Without knowing other ways to find information, I was thinking it would be
really nice to have a way to see which files or directories grew or shrank
since some point in the past. Would it be possible to get the kernel to
tell
me when files were opened for writing and closed so I could record the
size
difference? I know there are various file notify mechanisms, but from
what I
can see, they don't handle millions of files very well.
Of course I'm open to other ideas, but this was my curiosity.
On Thu, Aug 11, 2011 at 10:47:00AM +0800, Adam Lee wrote:
quoted
Without knowing other ways to find information, I was thinking it would be
really nice to have a way to see which files or directories grew or shrank
since some point in the past. Would it be possible to get the kernel to tell
me when files were opened for writing and closed so I could record the size
difference? >
It is pretty much what I want to do, the problem with it is that it doesn't
appear to be recursive. I'd like to just put a watch on root, and get events
for the entire filesystem, not just that directory. From playing with this for
a couple minutes, it looks like I'd have to walk the entire filesystem setting
watches on every directory. So while it is very cool for what it does, I don't
think it will help me out much.
Is doing something with ld_preload to get in the way of every open call the
only way to do this?
It is pretty much what I want to do, the problem with it is that it doesn't
appear to be recursive.
I looked again tonight, and it appears that newer kernels with fanotify are
able to do this. So the feature is available, just too new to be on my
servers. Oh, well :P
Thanks for the input. I will probably just end up doing what I can with
ld_preload.
Chris
It is pretty much what I want to do, the problem with it is that it doesn't
appear to be recursive.
I looked again tonight, and it appears that newer kernels with fanotify are
able to do this. So the feature is available, just too new to be on my
servers. Oh, well :P
Thanks for the input. I will probably just end up doing what I can with
ld_preload.
Chris
Hi Chris,
I'm not sure whether that would help or not, but you could also do it in
kernel space using LSM (if you don't already use another LSM module).
With the file_security hook, you can track any changes on files and
store information in the extended attributes of the filesystem for
example. This might be a bit more flexible than a system wide
ld_preload.
My two cents.
--
Christophe