Massively Parallel Procrastination

Kindle 2 users - HEADS UP!

Today, Amazon shipped an update for your Kindle 2 (and for the DX and The Kindle 2 International). It adds PDF support. With reflow.  This is awesome and obsoletes a big part of what I was trying to do with savory.

Now for the bad news. It looks like today's firmware update locks down your device such that it will only take signed firmware updates.  I'm sure some enterprising haxx0r over at mobileread.com will homebrew up an alternate version of the updater that doesn't screw you over. But you might want to leave your wireless turned off if the latest update hasn't already been pushed on you.

Odds and ends

It's been a while since I've blogged about Kindle stuff. I just pushed up the little bit of custom code from my cobbled together Ubuntu/X11 on Kindle hack.  You can find it at http://code.google.com/p/savory/source/browse/trunk/x11-on-kindle/

Making use of that is still a bit of work. You need to install a copy of Jaunty Jackalope (or possibly Karmic) for ARM into a QEmu emulator. Then grab that ext2 filesystem image, copy it over to your kindle (2, DX, etc) and mount it somewhere locally and then chroot into it.  From there, you can use the x.org patch at the google code link above to build a copy of x.org that will actually work on the Kindle's display (and doesn't depend on having a tty).  The various support scripts I used for my various demos are all up there as well.  This isn't nearly as polished as Savory, but, well, it's not really an end-user thing at this point ;)

I don't think I'm going to spend a whole lot more time on this, but you never know.

I've been spending a bunch of my time on this "Perl" thing of late. Apparently, I volunteered myself to get Perl 5.12 out. Now I'm responsible for it or something.

-j

Oops! I did it again - Perl 5.11.1

Milo had been caught red-handed in the act of plundering his countrymen, and, as a result, his stock had never been higher. He proved good as his word when a rawboned major from Minnesota curled his lip in rebellious disavowal and demanded his share of the syndicate Milo kept saying everybody owned. Milo met the challenge by writing the words "A Share" on the nearest scrap of paper and handing it away with a virtuous disdain that won the envy and admiration of almost everyone who knew him. His glory was at a peak, and Colonel Cathcart, who knew and admired his war record, was astonished by the deferential humility with which Milo presented himself at Group Headquarters and made his fantastic appeal for more hazardous assignment.

- Joseph Heller, Catch-22 

 

It gives me great pleasure to announce the release of Perl 5.11.1.

This is the second DEVELOPMENT release in the 5.11.x series leading to a stable release of Perl 5.12.0. You can find a list of high-profile changes in this release in the file "perl5111delta.pod" inside the distribution.

You can (or will shortly be able to) download the 5.11.1 release from:

http://search.cpan.org/~jesse/perl-5.11.1/

The release's SHA1 signatures are:

4eb796d28849ea21466166cea0b580d98163564f  perl-5.11.1.tar.bz2

aa4ca3b0cffa1bbcbcdb09e81c6ece759112ce14  perl-5.11.1.tar.gz

We welcome your feedback on this release. If you discover issues with Perl 5.11.1, please use the 'perlbug' tool included in this distribution to report them. If Perl 5.11.1 works well for you, please use the 'perlthanks' tool included with this distribution to tell the all-volunteer development team how much you appreciate their work.

If you write software in Perl, it is particularly important that you test your software against development releases. While we strive to maintain source compatibility with prior stable versions of Perl wherever possible, it is always possible that a well-intentioned change can have unexpected consequences. If you spot a change in a development version which breaks your code, it's much more likely that we will be able to fix it before the next stable release. If you only test your code against stable releases of Perl, it may not be possible to undo a backwards-incompatible change which breaks your code.

In the release announcement for 5.11.0, I asked readers to test the new version of Perl with their in-house applications and CPAN modules.  Among other things, that testing turned up previously undiscovered issues in a change to Perl's Regular Expression semantics which we were able to defang in time for 5.11.1.

\Notable changes in this release:

• Package declarations can now include a version number. 
• suidperl is no longer available as part of perl. If your code depends on suidperl, you need to find an alternate solution. (This was actually true as of 5.11.0)
• Over the years a number of language constructs and interpreter features have been deprecated and will eventually be removed. As of this release, Perl enables deprecation warnings by default.
• Perl's tests are now aware of (and work around) a bug in Mac OS X 10.6 locales.
• Support for Windows 95, 98, ME and NT4 has officially ended. 

This release represents approximately 3 weeks development since Perl 5.11.0, containing 22,000 lines of changes across 396 files from 26 authors and committers:

Abigail, Alex Vandiver, brian d foy, Chris Williams, Craig A. Berry, David Fifield, David Golden, demerphq, Eric Brine, Geoffrey T. Dairiki, George Greer, H.Merijn Brand, Jan Dubois, Jerry D. Hedden, Jesse Vincent, Josh ben Jore, Max Maischein, Nicholas Clark, Rafael Garcia-Suarez, Simon Schubert, Sisyphus, Smylers, Steve Hay, Steve Peters, Vincent Pit and Yves Orton.

Many of the changes included in this version originated in the CPAN modules included in Perl's core. We're grateful to the entire CPAN community for helping Perl to flourish.

Yves Orton will release Perl 5.11.2 on November 20, 2009.

Leon Brocard will release Perl 5.11.3 on December 20, 2009.

Ricardo Signes will release Perl 5.11.4 on January 20, 2010.

Perl 5.11.0

I played hooky from work for a few days last week to help get Perl 5.11.0 released. I did a little bit of cat-herding, a few hours of copy-editing and some test building. The end result was that we now have an unstable ("blead") release of the next major version of Perl and a plan to keep making them happen.

---

    Whispers of an "evil power" were heard in lines at dairy shops, in streetcars, stores, arguments, kitchens, suburban and long-distance trains, at stations large and small, in dachas and on beaches.  Needless to say, truly mature and cultured people did not tell these stories about an evil power's visit to the capital. In fact, they even made fun of them and tried to talk sense into those who told them. Nevertheless, facts are facts, as they say, and cannot simply be dismissed without explanation: somebody had visited the capital. The charred cinders of Griboyedov alone, and many other things besides, confirmed it.  Cultured people shared the point of view of the investigating team: it was the work of a gang of hypnotists and ventriloquists magnificently skilled in their art.

- M. Bulgakov, The Master and Margarita 

It gives me great pleasure to announce the release of Perl 5.11.0. 

Perl 5.11.0 is a DEVELOPMENT release. We're making it available to you today to make it easy for you to test your software on what will eventually become Perl 5.12.

This release is the result of over two years of development by a global community of developers. You can find a list of high-profile changes in this release in the file "perl5110delta.pod" inside the release.

You can download the 5.11.0 tarball from: http://search.cpan.org/~jesse/perl-5.11.0/

The release's SHA1 signatures are:

 0d436577386c668161e3dad385d233c383bf4c9d  perl-5.11.0.tar.bz2

 3137486cfe00094d1cd9a00e6e61f152f8fdb26e  perl-5.11.0.tar.gz

We welcome your feedback on this release. If you discover issues with Perl 5.11.0, please use the 'perlbug' tool included in this distribution to report them. If Perl 5.11.0 works well for you, please use the 'perlthanks' tool included with this distribution to tell the all-volunteer development team how much you appreciate their work.

If you write software in Perl, it is particularly important that you test your software against development releases. While we strive to maintain source compatibility with prior releases wherever possible, it is always possible that a well-intentioned change can have unexpected consequences. If you spot a change in a development release which breaks your code, it's much more likely that we will be able to fix it before the next stable release. If you only test your code against stable releases of Perl, it may not be possible to undo a backwards-incompatible change which breaks your code.

Today marks a major change in how we'll be releasing development versions of Perl.

Historically, a single individual, the Perl "pumpking" has been personally responsible for all aspects of the Perl development process - ranging from direction setting, dispute resolution and deep hacking to mentoring, patch application and release engineering.

Over the years, we've been blessed with a series of extraordinary leaders. These hackers have eschewed fame, fortune and many nights' sleep for the good of Perl.

To help ensure that we don't burn out our best diplomats and brightest coders, our release process is changing. I have recruited the first few volunteer release managers. Each month, on the 20th, the next release engineer in rotation will cut a new development release.

Today's release of 5.11.0 is a transitional release to test our release machinery and process. The schedule for the near future is as follows:

    2009

    ====

    October 2       -  5.11.0  -   Jesse Vincent

    October 20      -  5.11.1  -   Jesse Vincent

    November 20     -  5.11.2  -   Yves Orton

    December 20     -  5.11.3  -   Leon Brocard

    2010

    ====

    January 20      -  5.11.4  -   Ricardo Signes

If you're interested in volunteering to join the release-engineer rotation, please contact me off-list and I'll add you to our talent pool. It's not a particularly lucrative job - The only perks are your name in perlhist, the chance to choose the epigram for a release announcement and the warm feeling you get from bringing a new version of Perl into the world.

Prophet and SD 0.7 (Cavil) are now available (What I've been up to at work)

Prophet is a lightweight schemaless database designed for peer to peer replication and disconnected operation. Prophet keeps a full copy of your data and (history) on your laptop, desktop or server. Prophet syncs when you want it to, so you can use Prophet-backed applications whether or not you have network.

SD (Simple Defects) is a peer-to-peer issue tracking system built on top of Prophet. In addition to being a full-fledged distributed bug tracker, SD can also bidirectionally sync with your RT, Hiveminder, Trac, GitHub or Google Code issue tracker. 

To learn more about Prophet and SD, visit http://syncwith.us or read the full announcement over on the corporate blog

Things I ought to be blogging about

I just got back from OSCON.  I gave an Ignite talk about my Kindle hacking, a "regular" 45 minute talk about SD, the p2p bug tracker that I work on and a half-day talk on RT.  OSCON, as always, was amazing. I think I only made it to a half-dozen actual sessions - the hallway track kept me that busy.

About a dozen people yelled at me for not doing more to publicize some of the projects I'm involved with. Each and every one of these deserves a whole series of blog posts. Maybe by listing them here, I'll embarass myself into writing a bit more.  

I've had a busy year.  I didn't realize how busy until I wrote this post.  Below are some of the projects I've been involved with / instigated. In almost every case, I've had fantastic collaborators who have helped make the tools a reality. But the projects are all my fault ;)

SD is a peer to peer bug tracker (think git for bugs) that can also sync to RT, Hiveminder, Google Code, GitHub, Trac. Read-only support for syncing with RedMine is available today, with full two-way sync coming soon. Sync plugins are only a few hundred lines of code once you have a CPAN module to talk to an app.  For now, you can check out the talk I gave at OSCON.

K-9 is an opensource email client for Android.  I founded the project when I forked the core android "Email" app to get some needed bugfixes onto user devices quickly. Since then, K-9 has added features and fixes at a pretty rapid pace. Best of all, other folks do a lot of the work :)

Shipwright is a build and distribution system for applications. It has special features designed to tame Perl's CPAN, but it works well for non-Perl apps too.  It lets you build a versioned source archive of your application and all its dependencies. It comes with tools to help you distribute buildable source archives with an app and all its dependencies (down to libc if you swing that way) with a single installation tool.  The resulting binary packages have a bit of magic to make them magically relocatable. Just recently, we also released support for multi-architecture binary distributions.

Changelogger is a tool we've been working on at Best Practical to ease the ordinarily tedious process of creating a human-readable changelog for your software package out of the version control system's commit log.  It lets you open the change categorization and tidying process to a community of developers who can vote on rephrasings and categorization of changelog entries. The tool let Nicholas Clark slice through the 2500 changes included in Perl 5.10.1 in about 7 hours.  Once we get a few more issues cleaned up, we'll be running a changelogger as a public service.

Kindle hacking.  At this point, I have an ubuntu Jaunty Jackalope chroot with working X server, Keyboard and 5-pad.  I need to fix an X server bug, then I'll build a downloadable installer and let the rest of the world start playing around.

I've done a bit of minor javascript hacking to make Twitter's web view more palatable to me: http://fsck.com/~jesse/spyhunter/ and http://fsck.com/~jesse/narrow-twitter.user.js.

At YAPC::US last month, I came up with a pair of 20 line scripts which let me read twitter in Mutt. I get keyboard bindings, proper threading, lightning fast searching AND I can keep it running in screen ;) It's actually just a replacement for /usr/bin/sendmail and some goo to turn tweets into maildir messages.  My plan with this is to clean it up and add RSS feeds, Facebook and a few other activity streams. I may then try to package it up or set it up as a service. The code's all in my github account.

A half-dozen other things I tried out died on the vine: A postgres-based email archival tool, A ground-up rewrite of RT (don't worry, RT4 is alive and well, but that's another post), cleanup of MobiPerl and...well, I've blocked out my memories of the rest of them.

A new definition of wrong

I haven't been blogging about the Kindle for a while...mostly because I haven't hacked on the Kindle in a while.  The release of the DX got me excited about what's possible with a device like the Kindle. I spent a bunch of time trying to get native framebuffer applications working on my Kindle 2. Yesterday, I struck upon an awful, awful idea.

Ubuntu Jaunty has an ARM target.  It's a fairly similar linux to the OS shipped on the Kindle 2 and the Kindle DX. A little bit of fiddling and an NFS export later, I was able to chroot into an Ubuntu environment on my Kindle. 

That was when I discovered that Lab126 have built the Kindle's kernel without CONFIG_VT...and X.org really wants a tty or virtual terminal, but not for any particularly good reason.

This evening, I managed to bludgeon X.org into submission and found myself face to face with everybody's favorite checkerboard background.

After that, it was just a hop, skip and an apt-get until I was watching the mesmerizing transitions of xdaliclock.

(Yes, I know it's incredibly blurry. I haven't managed to get the pinhole camera on my DX calibrated yet)

Clearly this is the killer app for the Kindle just a stepping stone. I still don't have the Keyboard or 5way hooked up and what I do have working is incredibly brittle.  But xpdf (and lots of other stuff) runs unmodified.

Improved PDF rendering for Savory

Now with better default fonts (Google's free Droid fonts) and border cropping.

http://savory.googlecode.com/files/savory-image-2009-04-10.zip

Savory Screenshots

Over the past day, a few readers have asked me what converted PDFs look like on the Kindle 2 and how readable they are. (One of you even said you'll buy a Kindle 2 if I post some screenshots.) And yes, Amazon pays me a referral fee for every Kindle 2 I sell. So far, I've lured two of you in ;)

All the screenshots below have been reduced. Click on them to see full-size versions.

The Savory conversion process

 

 

A scientific paper, converted from PDF

Fit to screen:


Fit to height (yeah, height. It's rotated):

Fit to screen

Full size:

Fit to height:

Playing to the crowd

From the bits of feedback I've gotten on Savory's initial release on Friday, it seems pretty clear that people want to be able to view rendered PDFs that look "like they're supposed to" 

Savory's initial release used a PDF-to-HTML converter that extracts the text from a PDF and turns it into a reflowable ebook.  If your PDFs are generated from text by modern tools. this is great.  Most PDFs aren't.

Over the weekend, I put together a PDF renderer for Savory build on Poppler and the Kindle's built in Manga support (If you have a .zip or .cbz file full of images, the Kindle will display the images like a book)

For now, every PDF is converted twice. Once to text and once to a set of rendered images. So far, it's working pretty well for me. I'd love some feedback on savory-image-beta-2009-04-06.zip from anyone who's game.  Just replace your regular savory image with it and reboot the kindle.