Re: high throughput storage server?

On Sun, Feb 27, 2011 at 3:30 PM, Ed W [off-list ref] wrote:

Your application appears to be an implementation of a queue processing
system?  ie each machine: pulls a file down, processes it, gets the next
file, etc?

Sort of.  It's not so much "each machine" as it is "each job".  A
machine can have multiple jobs.

At this point I'm not exactly sure what the jobs' specifics are; that
is, not sure if a job reads a bunch of files at once, then processes;
or, reads one file, then processes (as you described).

Can you share some information on
- the size of files you pull down (I saw something in another post)

They vary; they can be anywhere from about 100 MB to a few TB.
Average is probably on the order of a few hundred MB.

- how long each machine takes to process each file

I'm not sure how long a job takes to process a file; I'm trying to get
these answers from the people who design and run the jobs.

- whether there is any dependency between the processing machines? eg can
each machine operate completely independently of the others and start it's
job when it wishes (or does it need to sync?)

I'm fairly sure the jobs are independent.

Given the tentative assumption that
- processing each file takes many multiples of the time needed to download
the file, and
- files are processed independently

It would appear that you can use a much lower powered system to basically
push jobs out to the processing machines in advance, this way your bandwidth
basically only needs to be:
   size_of_job * num_machines / time_to_process_jobs

So if the time to process jobs is significant then you have quite some time
to push out the next job to local storage ready?

Firstly is this architecture workable?  If so then you have some new
performance parameters to target for the storage architecture?

That might be workable, but it would require me (or someone) to
develop and deploy the job dispatching system.  Which is certainly
doable, but it might meet some "political" resistance.  My boss
basically said, "find a system to buy or spec out a system to build
that meets [the requirements I've mentioned in this and other
emails]."
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html