Vincent: Perl 5.16 and Beyond

[ What follows is the prose version of the talk I gave at YAPC::NA, OSCON
and YAPC::EU this summer.  I'll be giving a similar talk at YAPC::Asia. If
you prefer reading big sentence fragments on slides, 
http://www.slideshare.net/obrajesse/perl-516-and-beyond is my slide deck 
from YAPC::EU]

Porters,

Over the last two years, I've been thinking a lot about where Perl 5 is
headed. Well, that's not quite true. I've been thinking a lot about where
Perl 5 is headed since way back in 2005 or 2006 when I was the Perl 6
project manager and it become clear to me that Ponie wasn't likely a horse
I'd want to bet Perl 5 on.

As a community, we've done an absolutely stellar job of getting Perl 5 and
p5p back into a reasonable groove. We know how to do releases. More
individuals have released Perl 5 in the last 18 months as in the first 16
years Perl existed. It shouldn't really surprise anybody that knowing that
work you do will be released soon is a huge motivating factor. Releasing
frequently has dramatically increased Perl 5's rate of change. That's both
good and bad.

I'm thrilled that Perl development is incredibly vibrant. 5.14 is a
dramatically better Perl than 5.10 was, thanks to the work of approximately
260 porters. At the same time, I've made (or failed to make) some design
choices that, in retrospect, I somewhat regret. There are places where Perl
5 is now more baroque than it was before. There have been changes to Perl 5
that have broken backward compatibility where we might have been better
served not doing so. These mistakes are entirely my fault. If I had a
do-over, I'd make different mistakes..er. decisions.

Various members of the community have asked me about my vision for the
future of Perl 5. I was initially very, very hesitant to make any bold
statements about the language, since "we're going to put out a development
release every month and a new stable release every year" felt rather...bold
itself.

Putting out releases is hard work. At this point, it's hard work we have
well in hand.

Much of my thinking about the future of Perl 5 stems from the following
principles:

	* New versions of Perl 5 should not break your existing software
	* Backward compatibility must not stop Perl 5 from evolving

Pay particular attention to "should" and "must" there. It is critically
important that we not alienate the people, communities and companies who
have invested their time and money in Perl 5.  Pulling the rug out from
under them isn't good for them and isn't good for us. Wherever possible, we
need to preserve backward compatibility with earlier versions of Perl 5. At
the same time, it could be argued that _any_ change to Perl 5 breaks
backward compatibility. ("But I was depending on that segfault!") If Perl 5
is going to continue to flourish, we're going to need to be able to change
the language.

* Why we deprecate things

Perl 5 is not Latin. It is a living language, still borrowing liberally
from...just about everything. Sometimes we borrow the wrong
things. Sometimes we borrow things and use them wrong. Sometimes we invent
things and later wish we hadn't. Perl has always been something of a
packrat, but we're in danger of hitting the point of being diagnosed with a
pathological hoarding problem. We need to get better at fixing problems and
moving forward without hurting old code.

* How we handle language changes

We've always had a messy relationship with backward compatibility. Because
we don't have a spec for the language and we have a single implementation,
side-effects bugs and ill-considered accidents have become unchangeable
parts of the language.

For a project of Perl's age, scale and diversity, we've been astonishingly
conservative about how we make incompatible changes. It's not enough. Every
release breaks running code. The more the language evolves, the more legacy
code will break.

Standing still is not an option. Perl's internals, syntax and semantics
have seen some much-needed improvements in the past few years. There are
many additional changes we can't make because they may damage too much
legacy code.

To date, version declarations in code have been intended as a marker for
"minimum acceptable version of perl" -- They've also been used to enable
new language features or semantics changes.  Many _other_ new changes in
Perl are turned on by default without the need for any declaration.
Typically, this is because we don't believe those changes will harm earlier
code, though it's also the case that conditionally enabling certain changes
would be difficult.

Perl 5 doesn't require that a developer declare which version of Perl 5 a
program or library was written to run under. Historically, our default has
been "no declaration means latest, sort of."

That changes from here on in. Code that does not declare a 'use v5.16;' or
later will be presumed to have been written to target v5.14.

If there is no "use v5.xx" line at the top of the code, the runtime should
act as it did on v5.14 without a use v5.14 line.

We'll need a nice compact way to declare what version of Perl you want from
the commandline for one-liners too.

Along with this, there's a change to how we'll be removing deprecated
features from the core.

If a core feature (syntactic or semantic) is removed, a "use v5.xx"
declaration for a earlier version of Perl 5 should re-enable the feature.

This will be difficult. I fully expect people to start throwing pies,
tomatoes or bricks when I say this. I will not require that the
forward-ported implementation of the old feature have performance parity
with the old implementation. In certain circumstances, I _will_ make
exceptions to the mandatory-forward-port rule. Security fixes are one
obvious case. I know we'll run into other cases.

It is my strong preference that features granted R&R (removal and
reinstatement) be implemented as modules, so as not to bloat the runtime
when they're not needed. This isn't a pipe dream.  Classic::Perl already
does this for a few features removed in 5.10 and 5.12;

If it's not possible to reinstate a feature we've removed with existing
APIs, we'll need to look at the cost of removing the feature vs simply
disabling it in the presence of a new-enough v5.xx declaration.

* How quickly can we deprecate things

When I seized power in late 2009, I set us up with a new policy for
deprecation timing. Anything we deprecated needed to warn in version .x
that it would be removed in version .x+1. At the same time, I put us on a
track to ship a new release of Perl 5 each spring. I'm incredibly pleased
to see that multiple vendors have already incorporated Perl 5.12 into their
stable releases

There's a reasonable chance that some vendors on a slightly longer release
cycle may never ship a given major version of Perl, meaning that our
carefully-crafted deprecation warnings will never be seen.

If we manage to implement the R&R policy for deprecated features, we
have little to worry about.  Code that declares "use v5.14" should continue
to function just as it always has, but for code that declares 'use v5.16',
we can make changes without a deprecation cycle.

For cases where we _can't_ implement R&R, I think we need to move to a
two year deprecation cycle, so as to have as minimal an impact as possible
on users who are upgrading.

* Language Modularization

There's a lot of stuff that ships as part of "Perl 5 the distribution"
that's not part of "Perl 5 the language" - Over the years, we've done an
increasingly good job unwinding many dark corners of what we affectionately
refer to as "blead" into CPAN modules. Most of those items have been add-on
scripts and tools that have been part of the distribution but weren't part
of the language.  Most of those modules continue to ship as part of the
distribution and may well do so for many years to come. When I talk about
modularization, I'm not proposing that we stop shipping the newly
modularized code as part of the traditional core distribution.

Over the past few years, a number of people have spent significant effort
to make certain parts of the language runtime a good deal more pluggable. I
expect that we have a good amount of work to do before the core is flexible
enough for us to fully implement this plan.

It's time for us to start extracting parts of what has traditionally been
considered the "language" part of Perl 5 into CPANable modules. To do this
successfully, it is imperative that _nothing_ appear out of the ordinary to
code that expects to use those features. If code doesn't declare use v5.16,
it should still get the 5.14ish environment it would expect.

At the same time, it should be possible for a future Perl to declare that
certain previously-integral features are optional and run-time
loadable/unloadable. There are many situations where Perl is used that
don't need a whole set of the blades on our swiss-army chainsaw.  Forcing
their inclusion makes Perl a less viable candidate for a wide variety of
applications.

Once language features are modularized, it also becomes _possible_ to
maintain and improve them without requiring a full Perl upgrade.

Making this work for some things, like grammar changes, may take herculean
effort but will invariably result in a more robust and flexible Perl.

I don't know what we'll extract or when we'll extract it, but there are a
number of language features that seem like they might make sense to make
pluggable: Formats, SysV IPC functions, Socket IO functions, Unix user
information functions, Unix network information functions and Process and
process group functions. Jesse Luehrs has already built us a first version
of an extraction, modularization and replacement of smartmatch.

It should be clear from this list that I don't intend for us to kill,
banish or exile these functions from the traditional Perl
distribution. They're a useful and important part of our language and our
culture. But they're not all needed or even desired everywhere we'd want
Perl 5 to run. If we can modularize them in a way that doesn't negatively
impact performance, Perl 5's manipulexity gets better without damaging its
whipitupitude. The increased flexibility of the core should actually
improve our whipitupitude at the same time.

* What should ship in the Perl core

When I say "traditional Perl distribution", I'm talking about the
conglomeration of "language" and "toolkit" we currently ship. In 5.001,
that totalled up to 59 packages. In 5.005, it grew to 176.  By 5.008, it
was 338 packages. 5.10 saw 541 packages. 5.12 ballooned to 625
packages. 5.14 was the most modest growth we've had in many versions. It
ships 655 packages. Many of these are, of course, part of a smaller number
of distributions.

The bar for getting a distribution into blead has varied over time. The
"point" of the core distribution has often been up for debate. Is it an SDK
for software written in Perl 5? If so, then we ought to be shipping what we
currently consider to be the best-practice modules for building software in
Perl. Is it "stuff we've always shipped with the Perl core?"  If so, we're
doomed to a life in a house full of stacks of old line-printer paper and
TK50 cartridges.

Right now, what we include in the core distribution is a somewhat eclectic
set of modules. They're not a particularly good SDK. They're not a
particularly good sysadmin toolkit. They do, however, include an excellent
set of modules to bootstrap getting more modules.

During my tenure, the guiding principle for new distributions in the core
has been "does it help us bootstrap CPAN module installation or to test the
core?"

In this day and age, most users get Perl from their operating system vendor
or a third-party packager. Those vendors often look to us to decide what to
ship. And they often ignore us. That's 100% ok. There are vendors who
choose not to ship Perl's documentation as part of their "perl"
package. There are vendors who have considered removing Perl from their
default installation because it's "too big" -- They've talked about
splitting the distribution up according to their own logic or sometimes of
removing it entirely. We've done some amazing work paring down the
installed footprint. So far, I think we've avoided getting the axe.

And then there's the maintenance issue. It takes a lot of work for us to
keep up to date with all the modules we include in the core. We do a pretty
good job, but it takes a lot out of us.

It's time we made some more work for ourselves. We will continue to ship a
distribution of Perl that contains roughly what we've always shipped, but
I'd like it to be our secondary product.

The primary product will be "Perl the language" with the minimal set of
modules needed to test the core, offer all the language features we intend
to offer (Encode, for example) and to bootstrap the installation of new
modules from CPAN. While I intend for us to ship the R&R features in
the core distribution, it shouldn't actually be necessary.

Since I became involved in the Perl community, there have been constant
requests for various "SDKs" for various uses of Perl. The state of the art
in how to do such a thing is better than it was, but isn't really all that
useful at this point. It should be possible to take a "Perl the language"
distribution, drop some modules into a directory, tar/zip it up and hand it
off to your user community. And that's how we should be able to build the
"traditional" Perl distribution.

* Documentation changes

Perl's documentation is. Well, it is many things. Current, well-edited,
complete and completely accurate are four things it is not. We've been
getting a little better at splitting out "introduction for newcomers"
documentation from "canonical reference" documentation.

Our introduction documentation has always been opinionated. The particular
opinions have varied by author, document and phase of the moon. That's
ok. Introductory documentation _should_ be opinionated. Introductory
documentation should recommend "at least one good way to do it" and point
the curious reader who wants to know more to canonical reference
documentation.  Canonical reference documentation should try to actually
spell out how things work with a minimum of opinion.

As we modularize features, we'll need to consider how we reorganize the
docs so that they are maximally useful to users who should neither need nor
want to know about how Perl's internals work.

We need a new document, built from previous perldeltas that succinctly
describes what, other than bug fixes, came or went in each version of Perl
5. Porters should be discouraged from adding or removing a feature without
updating this document (and other documentation).

* Tests

In order to understand how well we're doing on back-compat as we move
forward, we're going to need to be able to run a new Perl against older
versions of the test suite.  One step in that direction would be to be able
to run the test suite against an installed Perl.

* TL;DR

	- New versions of Perl 5 should not break your existing software
	- Backward compatibility must not stop Perl 5 from evolving
	- From 'use v5.16' forward, Perl should start treating 'use v5.x'
	  statements as "try to give me a Perl that looks like v5.x" rather
	  than "give me at least v5.x" 
	- We're awesome at modules. Where possible, we should be
	  modularizing core features. 


Best,

Jesse

$ echo /usr/bin/* /bin/* /usr/sbin/* /sbin/* | wc -w          
4619

$ file /usr/bin/* /bin/* /usr/sbin/* /sbin/* | grep -c shell.*script      
621

$ file /usr/bin/* /bin/* /usr/sbin/* /sbin/* | grep -c perl.*script
420

$ file /usr/bin/* /bin/* /usr/sbin/* /sbin/* | grep -c python.*script
198