Computing Marginals Using MapReduce

29 Sep 2015  ·  Afrati Foto, Sharma Shantanu, Ullman Jeffrey D., Ullman Jonathan R. ·

We consider the problem of computing the data-cube marginals of a fixed order $k$ (i.e., all marginals that aggregate over $k$ dimensions), using a single round of MapReduce. The focus is on the relationship between the reducer size (number of inputs allowed at a single reducer) and the replication rate (number of reducers to which an input is sent)... We show that the replication rate is minimized when the reducers receive all the inputs necessary to compute one marginal of higher order. That observation lets us view the problem as one of covering sets of $k$ dimensions with sets of a larger size $m$, a problem that has been studied under the name "covering numbers." We offer a number of constructions that, for different values of $k$ and $m$ meet or come close to yielding the minimum possible replication rate for a given reducer size. read more

PDF Abstract
No code implementations yet. Submit your code now




  Add Datasets introduced or used in this paper