Integration of the HDRF algorithm within PowerGraph.

[fabiopetroni]

HDRF is a novel stream-based graph partitioning algorithm that provides important performance improvements with respect to existing solutions in partitioning quality.
In particular, HDRF provides the smallest average replication factor with close to optimal load balance. These two characteristics put together allow HDRF to significantly reduce the time needed to perform computation on graphs and makes it the best choice for partitioning graph data.

The HDRF algorithm is extensively described in the following publication:

F. Petroni, L. Querzoni, K. Daudjee, S. Kamali and G. Iacoboni:
"HDRF: Stream-Based Partitioning for Power-Law Graphs".
CIKM, 2015.
http://www.dis.uniroma1.it/~midlab/articoli/PQDKI15CIKM.pdf

[dbickson]

Hi Fabio,
The patch looks good, but I did not find the documentation string where you allow the new partitioning method in the engine. Can you please add it as well?

[fabiopetroni]

On 7/29/15 5:54 PM, Danny Bickson wrote:

Hi Fabio,
The patch looks good, but I did not find the documentation string where
you allow the new partitioning method in the engine. Can you please add
it as well?

—
Reply to this email directly or view it on GitHub
#177 (comment).

Hi Danny,

the string is in the src/graphlab/graph/distributed_graph.hpp file:

...} else if (method == "hdrf") {
if (rpc.procid() == 0) logstream(LOG_EMPH) << "Use hdrf oblivious ingress, usehash: " << usehash
<< ", userecent: " << userecent << std::endl;
ingress_ptr = new distributed_hdrf_ingress<VertexData, EdgeData>(rpc.dc(), *this, usehash, userecent);
} else...

It is possible to use HDRF as input partitioner by specifing the following option:

--graph_opts ingress=hdrf

Best,
Fabio

[dbickson]

To clarify, need to add additional documentation here:
https://github.com/dato-code/PowerGraph/blob/18c21033d77208cedac7661e4fcf35698abb4021/src/graphlab/options/graph_help.txt

[fabiopetroni]

On 7/29/15 7:25 PM, Danny Bickson wrote:

To clarify, need to add additional documentation here:
https://github.com/dato-code/PowerGraph/blob/18c21033d77208cedac7661e4fcf35698abb4021/src/graphlab/options/graph_help.txt

—
Reply to this email directly or view it on GitHub
#177 (comment).

Ok, thanks.
I created a new pull request with the updated documentation.
:)