Re: high throughput storage server?

Joe Landman put forth on 2/28/2011 9:46 AM:

On 02/27/2011 04:30 PM, Ed W wrote:

[...]

quoted

It would appear that you can use a much lower powered system to
basically push jobs out to the processing machines in advance, this way
your bandwidth basically only needs to be:
size_of_job * num_machines / time_to_process_jobs

This would be good.  Matt's original argument suggested he needed this
as his sustained bandwidth given the way the analysis proceeded.

And Joe has provided a nice mathematical model for quantifying it.

If we assume that the processing time is T_p, and the communication time
is T_c, ignoring other factors, the total time for 1 job is T_j = T_p +
T_c.  If T_c << T_p, then you can effectively ignore bandwidth related
issues (and use a much smaller bandwidth system).  For T_c << T_p, lets
(for laughs) say T_c = 0.1 x T_p (e.g. communication time is 1/10th the
processing time).  Then even if you halved your bandwidth, and doubled
T_c, you are making only an about 10% increase in your total execution
time for a job.

With Nmachines each with Ncores, you have Nmachines x Ncores jobs going
on all at once. If T_c << T_p (as in the above example), then most of
the time, on average, the machines will not be communicating.  In fact,
if we do a very rough first pass approximation to an answer (there are
more accurate statistical models) for this, one would expect the network
to be used T_c/T_p fraction of the time by each process.  Then the total
consumption of data for a run (assuming all runs are *approximately* of
equal duration)

    D = B x T_c

D being the amount of data in MB or GB, and B being the bandwidth
expressed in MB/s or GB/s.  Your effective bandwidth per run, Beff will be

    D = Beff x T = Beff x (T_c + T_p)

For Nmachines x Ncores jobs, Dtotal is the total data transfered

    Dtotal    = Nmachines x Ncores * D = Nmachines x Ncores x Beff
          x (T_c + T_p)


You know Dtotal (aggregate data needed for run).  You know Nmachines and
Ncores.  You know T_c and T_p (approximately).  From this, solve for
Beff.  Thats what you have to sustain (approximately).

This assumes his application is threaded and scales linearly across
multiple cores.  If not, running Ncores processes on each node should
achieve a similar result to the threaded case, assuming the application
is written such that multiple process instances don't trip over each
other by say, all using the same scratch file path/name, etc, etc.

-- 
Stan