LinuxGem
此处为老的 LinuxGem,新版 LinuxGem 请移步 www.linuxgem.org

多核版 gzip 和 bzip2

galeki posted @ 2008年2月20日 07:06 in Bash 和它的朋友们 with tags pbzip2 mzip , 9065 阅读

修改自: http://www.linux.com/feature/126412

Linux 下用得最普遍的两种压缩文件格式就是 .gz 和 .bz2 了,分别由 gzip 和 bzip2 命令创建。

说到压缩文件,除了压缩率之外,压缩和解压的速度也很关键,在创建或解压比较大的压缩文件时,如果时间过久,也挺郁闷的……

虽然多核 CPU 在这两年已经大面积普及了,但是标准的 gzip 和 bzip2 程序在压缩和解压的时候,只能用到 CPU 的其中一个核心,如果全部核心同时开动,一定可以大大加速压缩和解压的过程,于是,gzip 和 bzip2 的多核版: mgzippbzip2 就横空出世了~

mgzip 可以在压缩的时候充分利用多核,pbzip2 在压缩和解压的时候都可以充分利用多核。用 mgzip 压缩的压缩包,也可以通过标准的 gunzip 解压。pbzip2 只有在解压 pbzip2 创建的压缩文件时,才能利用多核。

mgzip 和 pbzip2 的用法和原来基本相同,只有一点小的差别,比如 mgzip 没有 --recursive 参数。

当然,为了获得多核速度上的提升,压缩文件也要记录一些额外的信息,导致文件要比标准的 gzip 和 bzip2 压缩后的文件要大一些,mgzip 和 gzip 的压缩结果就差不少,相比之下,pbzip2 和 bzip2 之间的差异要小很多,完全可以忽略:

$ bunzip2 linux-2.6.23.tar.bz2
$ gzip -c  linux-2.6.23.tar > linux-2.6.23.tar.gzip
$ mgzip -c linux-2.6.23.tar > linux-2.6.23.tar.mgzip
$ ls -lh
-rw-r----- 1 ben ben 253M 2008-01-19 18:55 linux-2.6.23.tar
-rw-rw-r-- 1 ben ben  56M 2008-01-19 18:57 linux-2.6.23.tar.gzip
-rw-rw-r-- 1 ben ben  67M 2008-01-19 18:57 linux-2.6.23.tar.mgzip

$ gunzip -c linux-2.6.23.tar.mgzip > linux-2.6.23.tar.mgzip-gunzip
$ md5sum linux-2.6.23.tar.mgzip-gunzip linux-2.6.23.tar
853c87de6fe51e57a0b10eb4dbb12113  linux-2.6.23.tar.mgzip-gunzip
853c87de6fe51e57a0b10eb4dbb12113  linux-2.6.23.tar

$ bzip2  -c -k -9 linux-2.6.23.tar > linux-2.6.23.tar.bzip2
$ pbzip2 -c -k -9 linux-2.6.23.tar > linux-2.6.23.tar.pbzip2

$ ls -lh
-rw-r----- 1 ben ben 253M 2008-01-19 18:55 linux-2.6.23.tar
-rw-rw-r-- 1 ben ben  56M 2008-01-19 18:57 linux-2.6.23.tar.gzip
-rw-rw-r-- 1 ben ben  67M 2008-01-19 18:57 linux-2.6.23.tar.mgzip
-rw-rw-r-- 1 ben ben  44M 2008-01-19 19:03 linux-2.6.23.tar.bzip2
-rw-rw-r-- 1 ben ben  44M 2008-01-19 19:01 linux-2.6.23.tar.pbzip2

$ ls -l
-rw-r----- 1 ben ben 264704000 2008-01-19 18:55 linux-2.6.23.tar
-rw-rw-r-- 1 ben ben  45488158 2008-01-19 19:03 linux-2.6.23.tar.bzip2
-rw-rw-r-- 1 ben ben  57928789 2008-01-19 18:57 linux-2.6.23.tar.gzip
-rw-rw-r-- 1 ben ben  69968799 2008-01-19 18:57 linux-2.6.23.tar.mgzip
-rw-rw-r-- 1 ben ben  45695449 2008-01-19 19:01 linux-2.6.23.tar.pbzip2

不过,pbzip2 有个缺点,就是不能从标准输入来获取数据,所以要想作出一个 .tar.pbzip2 的文件,得先创建一个 .tar 文件才行,不过相信再今后的版本中,肯定能实现这个功能。

家里的电脑连超线程都不是,没法测试了,偷懒拿来现成的测试结果图( CPU 为 四核 Q6600 2.4G,测试文件为 linux-2.6.23 内核源代码包 ) :

可以看出,在高压缩比的情况下,多核版和单核版的压缩文件结果差异会变小,并且不管压缩比是大是小,多核版都能大幅度提高压缩的时间 ( 四核情况下,快了 3 倍 )~

同样,解压缩的时间也大幅缩短:

  • 无匹配
  • 无匹配
Jarod 说:
2008年2月20日 17:10 可以用tar调用它们才有用,从来不直接调这两个命令
galeki 说:
2008年2月20日 19:30 做个符号链接不就得了~
joecen 说:
2008年3月03日 19:24 7zip已经可以使用多线程进行压缩。并且压缩率比bzip2还高。
yu 说:
2008年3月04日 01:41 7zip就是普及率不是很高
ee 说:
2008年5月13日 05:22 不过我个人使用pbzip2压缩文件后,后缀都是.bz2
Imamia Quran Academy 说:
2023年7月16日 02:25

Hi everyone,  I really love your story. You have a good taste of design, i hope i can be like you… i am from Imamia Quran Academy.Imamia Online Quran Academy has encompassed this field of Online Shia Quran Memorization too. They understand the urge of every Muslim to become Hafiz-e-Quran.

Imamia Quran Academy 说:
2023年7月16日 02:28
Hi everyone,  I really love your story. You have a good taste of design, i hope i can be like you… i am from Imamia Quran Academy. Imamia Online Quran Academy has encompassed this field of <a href="https://imamiaquranacademy.com/courses/online-shia-quran-memorization/">Online Shia Quran Memorization</a> too. They understand the urge of every Muslim to become Hafiz-e-Quran<br />
Imamia Quran Academy 说:
2023年7月16日 02:29

Very nice article, I enjoyed reading your post, very nice share, I want to twit this to my followers. Thanks!. I am from Imamia Quran Academy.Imamia Quran Center believes in the importance of a good teacher and has eminent Online Shia Male Quran Teachers.

forex market 说:
2023年8月21日 00:56

what matters most is the good deeds that we do on our fellow men, it does not matter what religion you have as long as you do good stuffs- UFABETแทงบอลบนมือถือ

haider 说:
2023年8月21日 19:31

Great site. Lots of useful info here. I’m sending it to several friends ans also sharing in delicious. And obviously, thanks for your sweat! แทงใบเหลืองใบแดงUFABET

forex market 说:
2023年8月22日 00:40

Definitely believe that which you said. Your favorite justification appeared to be on the net the simplest thing to be aware of. I say to you, I certainly get annoyed while people consider worries that they plainly don’t know about. You managed to hit the nail upon the top and defined out the whole thing without having side-effects , people can take a signal. Will probably be back to get more. Thanks UFABETแทงบอลทายผลผู้ชนะ

forex market 说:
2023年8月23日 23:00

Very nice info and straight to the point. I dont know if this is really the best place to ask but do you folks have any thoughts on where to hire some professional writers? Thanks dallas landscaping 안전놀이터

haider 说:
2023年8月23日 23:28

Hello! Good stuff, please keep us posted when you post again something like that! เว็บพนันบอลออนไลน์อันดับ1

haider 说:
2023年8月24日 19:34

I am curious to find out what blog platform you have been utilizing? I’m having some minor security problems with my latest website and I would like to find something more risk-free. Do you have any solutions? เว็บบอลออนไลน์UFA

forex market 说:
2023年8月24日 19:35

This web-site is actually a walk-through like the data it suited you about it and didn’t know who to question. Glimpse here, and you’ll absolutely discover it. UFABET เว็บพนันแตกง่าย

haider 说:
2023年8月25日 14:17

you are actually a good webmaster. The web site loading pace is incredible. It kind of feels that you’re doing any unique trick. Moreover, The contents are masterwork. you’ve done a great task in this topic! สมัครแทงบอลฟรี

forex market 说:
2023年8月25日 15:28

After study a few of the blog posts on your own internet site now, i genuinely as if your strategy for blogging. I bookmarked it to my bookmark site list and will be checking back soon. Pls check out my site also and let me know what you think. เว็บไซต์แทงบอลอันดับ1

forex market 说:
2023年8月26日 13:55

I’m impressed, I must say. Really rarely can i encounter a weblog that’s both educative and entertaining, and let me tell you, you have hit the nail for the head. Your idea is outstanding; the problem is something which there are not enough persons are speaking intelligently about. My business is very happy i found this around my search for something concerning this. เว็บพนันออนไลน์UFA

haider 说:
2023年8月27日 18:55

I’d must seek advice from you here. Which isn’t some thing It’s my job to do! I spend time reading an article that will make people think. Also, thank you allowing me to comment! 메이저사이트

haider 说:
2023年8月30日 17:40

It’s difficult to get knowledgeable men and women within this topic, but the truth is be understood as what happens you are speaking about! Thanks 메이저사이트

haider 说:
2023年9月04日 14:24

I’d must seek advice from you here. Which isn’t some thing It’s my job to do! I spend time reading an article that will make people think. Also, thank you allowing me to comment! 안전놀이터

forex market 说:
2023年9月05日 14:17

Undeniably believe that which you stated. Your favorite reason appeared to be on the net the simplest thing to be aware of. I say to you, I certainly get annoyed while people consider worries that they plainly do not know about. You managed to hit the nail upon the top as well as defined out the whole thing without having side effect , people can take a signal. Will likely be back to get more. Thanks salingsilang

sdsds 说:
2023年9月08日 23:13

Good day. Very cool website!! Guy .. Beautiful .. Amazing .. I will bookmark your blog and take the feeds additionally…I’m glad to find numerous useful information right here within the article. Thanks for sharing. sunflower hawaiian shirt

haider 说:
2023年9月08日 23:41

Ha ha… I was just surfing around and took a look at these feedback. I can’t believe that there’s still this much interest. Thanks for posting about this. filmyzilla 2023

haider 说:
2023年9月14日 22:03

What i don’t understood is if truth be told how you’re no longer really a lot more neatly-liked than you might be now. You’re so intelligent. You understand thus considerably when it comes to this topic, made me in my view consider it from so many numerous angles. Its like women and men don’t seem to be fascinated unless it’s something to accomplish with Woman gaga! Your own stuffs great. Always take care of it up! สล็อต

haider 说:
2023年9月20日 14:35

Hello! I know this is kind of off topic but I was wondering if you knew where I could get a captcha plugin for my comment form? I’m using the same blog platform as yours and I’m having trouble finding one? Thanks a lot! funny crocs slippers

jmaes 说:
2023年9月24日 15:00

There are a couple of interesting points over time here but I don’t know if I see every one of them center to heart. There’s some validity but Let me take hold opinion until I take a look at it further. Excellent post , thanks and now we want much more! Added onto FeedBurner also shillong teer common number

jmaes 说:
2023年10月03日 14:13

Can I merely say exactly what a relief to seek out somebody who actually knows what theyre discussing on-line. You definitely realize how to bring a difficulty to light and earn it critical. More people have to read this and see why side with the story. I cant believe youre no more well-liked simply because you absolutely have the gift. Kalaastar song lyrics

james 说:
2023年10月06日 19:36

You could certainly see your skills within the work you write. The arena hopes for even more passionate writers like you who aren’t afraid to say how they believe. All the time go after your heart. crocs chicken

ds 说:
2023年10月12日 19:43

I’m some sort of intended for the vast majority of content pieces, When i certainly savored, We would definitely desire far more facts about that, provided that it truly is very good., Well done for placing. Gutter Cleaning Poole

ds 说:
2023年10月21日 14:40

An impressive share, I with all this onto a colleague who had previously been performing a small analysis for this. Anf the husband in truth bought me breakfast due to the fact I found it for him.. smile. So permit me to reword that: Thnx for that treat! But yeah Thnkx for spending plenty of time go over this, I feel strongly about it and adore reading much more about this topic. When possible, as you become expertise, might you mind updating your site to comprehend details? It can be highly great for me. Massive thumb up just for this text! แทงบอลเงินสด


登录 *


loading captcha image...
(输入验证码)
or Ctrl+Enter