TheBonsai's Blog

About the days and nights of TheBonsai

Parallel BZIP2

September 11th, 2008 by TheBonsai

m00!

Due to a new toy we got at work, I was searching how to massively speed up BZIP2 processing on GNU/Linux systems. It sucks when you have 16 cores but only one process/thread.

I finally found PBZIP2, which basically is just a new controll wrapper around the underlying library (the code has around 60 Kilobytes – with comments and all). It builds in about the half of a second.

Just to imagine a bit:

Test file was a 5.6 Gigabyte file with random binary data (/dev/urandom). A normal BZIP2 took 30 Minutes to finish, a parallelized PBZIP2 made the same in 3 Minutes and 30 seconds.

In case you need to speedup BZIP2 processing on your SMP-servers, consider to use PBZIP2 – it rocks!

This entry was posted on Thursday, September 11th, 2008 at 23:45 and is filed under english, Hobby, Linux, Work. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Leave a Reply