Graph analytics and mining
Author
Prof. Willy Zwaenepoel
University of Sydney
This project extracted inferences from a network and explored graph analytics on different platforms – including combinations of in-core and out-of-core processing.
The original plan was to explore graph analytics on different platforms, with the intent to explore whether it was possible to build a single platform that would provide good performance on all these platforms, including all combinations of in-core and out-of-core processing and single machine and cluster platforms.
While good progress was made on this front with the 2017 Usenix ATC best paper award and more generally in a thesis in the framework of this project, it quickly became obvious that that goal was too ambitious to be accomplished in the time frame of the project. As a result, the original goals were modified in a number of ways. The original focus on graph analytics was widened to also include graph mining, and this is pursued in a 2021 Eurosys paper.
Most importantly, the storage system underlying the original out-of-core graph processing system proved to have much wider applicability beyond graph processing, and was successfully used as an underlying storage system for general big data processing in a 2018 Eurosys paper. A very surprising outcome of this line of work was that it can be used with great success for distributing database workloads, as demonstrated in a 2020 ASPLOS paper and in a second Ph.D. thesis in the framework of this project.
Finally, we pursued an independent line of inquiry into scheduling for modern multicore computers, which among other things led to a very well received 2018 Usenix ATC paper comparing the most prevalent schedulers used in industry.