Parallel BZIP2
September 11th, 2008 by TheBonsai
m00!
Due to a new toy we got at work, I was searching how to massively speed up BZIP2 processing on GNU/Linux systems. It sucks when you have 16 cores but only one process/thread.
I finally found PBZIP2, which basically is just a new controll wrapper around the underlying library (the code has around 60 Kilobytes – with comments and all). It builds in about the half of a second.
Just to imagine a bit:
Test file was a 5.6 Gigabyte file with random binary data (/dev/urandom). A normal BZIP2 took 30 Minutes to finish, a parallelized PBZIP2 made the same in 3 Minutes and 30 seconds.
In case you need to speedup BZIP2 processing on your SMP-servers, consider to use PBZIP2 – it rocks!
This entry was posted on Thursday, September 11th, 2008 at 11:45 pm and is filed under Hobby, Linux, Work, english. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.