This Fortnight on perl5-porters - 28 September-12 October 2008

This Fortnight on perl5-porters - 28 September-12 October 2008
Topics of Interest

$VERSION in Opcode
Y2038 branch
Crash when dying on warning from backticks with vfork() (5.6.1 on Lamp)
Discussing the (im?)possibility of prototype-based polymorphism for 5.12
Small incompatibility between 5.8.8 and 5.10.0
protochk and C++
TODO list for moving to git
Perl_newSVpvf("%lld") is broken
Pathtools QNX regression
git conversion discrepancy
perl@34467
Fedora (Assertion failure on glob assignment)
t/op/regexp.t

TODO of the week

Linker specification files

Patches of Interest

Cross-compile for linux
5.8.x on Interix
Invalid read in regdupe in maint-5.8

New and old bugs from RT

Different behaviour using the -Idir option on the command line and shebang (#6665)
return do { }: take 3 (or 4...) (#38809)
Perl 5.8.8 Build Failures for SunOS 4.1.3 (#59456)
Unicode::UCD::casefold() does not work as documented, nor probably as intended (#58430)
Segfault on weird but plain Perl in eval on 5.10.0 on Windows (#59498)
Memory leak with regex in 5.10.0 (#59516)
Surprising behaviour when $ is left off $last (#59522)
readline returns EBADF at eof in 5.10.0 (#59550)
Error installing a Perl 5.8.8 (#59610)
FOLDCHAR regop not produced for \x, \0, \N{U+....} (#59616)
Save code generated by references in @INC (#59626)
Unicode problem (#58182)
unicore/mktables expects wrong syntax from CaseFolding.txt for Turkish (#58656)
File::Copy does not handle file objects sanely (#59650)
Too late for -CS option (#59652)
SIGSEGV compiling regexp in 5.10.0 (#59654)
Cwd::realpath fails on unsearchable directory (AIX) (#59662)
Segfault when using (?|) in regexp. (#59734)
Documentation issues with 5.8.8/Math/BigInt.pm (#59746)
File::Find behaviour with files among @DIRS (#59750)
Changing effective userid from root to normal user fails on Linux (#59766)
Not OK: perl 5.11.0 +DEVEL +patchaperlup: on i686-linux-64int 2.6.22-1-k7 (UNINSTALLED) (#59776)
Cannot recurse inside (?|) (#59792)
Hash collision? (#59796)
return 0 or die (#59802)
Perl5 Bug Summary

New Core Modules
In Brief

Last week's summary
About this summary

This Fortnight on perl5-porters - 28 September-12 October 2008

"... because a git repository is a collection of assembled changes, not a linear progression, using a commit ID as a way to find out where your code came from doesn't work. This is not git's fault but is an issue with every distributed version control system (SVK tries to pretend otherwise). You could perhaps use a combination of the git repository you pulled from and the last commit ID, but [...] fundamentally the idea is gone. It's a small price to pay." -- Michael G. Schwern, coming to terms with the future.

Topics of Interest

$VERSION in Opcode

The tangent regarding the issue of module version numbers and the techniques to deal with them continued apace over the last fortnight. The core of Matt S. Trout's argument over version.pm is that it injects a method into UNIVERSAL, and this, in an of itself, produces action at a distance, even when version.pm is not at fault. The problem shows up when version.pm is used incorrectly by third-party code, and the resulting errors are very hard to track down.

All that being as it may, Matt appreciates the value of the version comparison code available, and would like to be able to get at it, while leaving UNIVERSAL untouched. Otherwise, it's akin to modifying $[ or $/ at the global level. Tom Christiansen concluded that only the highest level code should be putting things into UNIVERSAL, not by proxy via the use of other modules.

Later on, in a particularly magnanimous gesture, John Peacock announced that he had a patch that would do all that Matt wanted and more, and asked if Matt was satisfied by it.

Y2038 branch

Steve Hay took Michael G. Schwern's Y2038+ code for a spin in the land of Windows, and made a few adjustments to have it play nicely with the compilers there.

Based on Steve's recommendations, Michael produced a series of changes that ended up delivering sane results. In the end, Michael had to lift some code from the FreeBSD project to address the problem of converting strings to long longs. This helped the Microsoft compilers that still had not caught up with the C standard as of the previous millennium.

Later on Michael released a newer patch that included all his latest research on the matter. Steve continued to find a number of rough edges on Windows. H.Merijn Brand thought that it might be worthwhile probing for 64-bit variants of time routines on the platforms that offer them, since that the whole point of the exercise in the first place. And so he did, and said he'd get around to probing for the reentrant versions as well.

Michael then had a patch working on Windows and Cygwin, contingent on a couple of details that needed to be resolved. Some problems turned out to involve the building of miniperl itself, which uses some canned constants before Michael's work is able to work its magic.

Dave Rolsky dropped by to say that he was delighted that all of this work had a positive effect on the innards of Time::Local. On the other hand, it greatly complicated the task of keeping it dual-lifed on CPAN, so much so that he doubted that the new changes would work with older perl releases. Michael showed him how this could be done with some XS goodness.

Crash when dying on warning from backticks with vfork() (5.6.1 on Lamp)

Joshua Juran is the caretaker for Perl on the traditional Mac OS (that is, pre OS/X). Said platform doesn't know how to fork, but it can vfork. In the current state of affairs, asking a program to fork itself with something silly, like /dev/null, leads to some very interesting (read: not useful) results, in that one receives very little information as to what went wrong.

Joshua managed to get as far as showing how a minor change to Perl_do_exec(), Perl_my_popen(), and pp_system() would fix the problem, but he wasn't sure if his internals-fu was strong enough to consider all the ramifications.

Nicholas Clark reasoned that the problem Joshua faced was sufficiently weird (or rather, his platform of choice was weird) that he should write the tests to establish the desired behaviour out on the end of the continuum, and then the rest of us could just follow suit.

Apparently he has done this on a code repository hosted on Sourceforge, but unless I am mistaken they didn't make it to the list.

Discussing the (im?)possibility of prototype-based polymorphism for 5.12

David Nicol responded to a previous thread (which I lack the tuits to track down) with the concept of polymorphism based on prototypes, which would, amongst other things, make the following work:

He promised to investigate the possibility with his Macrame module. In the meantime, Rafa�l Garcia-Suarez reminded David that prototypes in Perl are more for changing the syntax rather that type inferencing.

Small incompatibility between 5.8.8 and 5.10.0

Torsten F�rtsch discovered a problem that had crept into 5.10, which appears to be something to do changes to the implementation of stashes. No-one commented.

protochk and C++

Tony Cook had been puzzled for a while over intermittent smoke failures when using C++ on Debian, and managed to discover that C++'s function overloading mechanism was the cause. He suggested a change to a Configure test, but H.Merijn admitted that it was far outside his scope of expertise and asked for other eyes to look at it and consider the ramifications.

TODO list for moving to git

Nicholas, having endured a painful bisection ordeal under Perforce to track down when something changed, wondered what was holding up the migration to git. His current problem was one of mapping a Perforce revision (a number in a monotonic sequence) to a git checksum.

Mark Mielke wanted to know if there was a page on the web somewhere that recorded the current status of the Perforce-git migration.

Michael G. Schwern offered an interesting parenthetical remark regarding the move from Perforce to git, and why it is the Right Thing. David Golden also noted that git solves the problem of branching at any point in time, hacking, and folding the changes back in a most elegant manner. The price you pay, though, is that it is very hard to point to a specific moment in development (since one only has an unsortable checksum, not a monotonic sequence).

Craig A. Berry, not fully versed in git-speak, was nonetheless able to describe how people would communicate points of interest in the development tree. At the same time, he thought that the barrier to entry to understanding git was much higher than that of Perforce. As it turns out, basic web projections of git repositories already come close to what the All Perl Changes browser provides.

Michael went through the git threads on the list and neatly summarised them on the Perl5 wiki. He noted with satisfaction that there were no showstoppers, just a question of tuits.

Perl_newSVpvf("%lld") is broken

One other item that fell out of the Y2038+ code was that, on 32-bit platforms, printf("%lld\n", 2**32) gives a result of -1. Michael and admitted to being terrified by the code and hoped someone else could dig down through it and figure out where the problem lay.

Pathtools QNX regression

Nicholas Clark looked more closely at Matt Kraai's failures for 5.8.9-tobe on QNX. He determined that it was due to a refactoring of File::Spec::Unix::abs2rel between Pathtools 3.12 and 3.13.

Interestingly, this dredged up an interesting piece of Perl trivia, in that for a short time, the /t flag on a regexp match maintained the taint flag if the target string was tainted. The idea lived a short life.

Nicholas thought that creating a new File::Spec::QNX (and the like), since it would be easier to test and would probably simplify the implementation of catdir. In any event, he wrote a patch and Matt confirmed that it fixed up the problem on QNX 6 (a.k.a QNX Neutrino, or $^O eq 'nto').

git conversion discrepancy

Nicholas Clark uncovered a flaw in Same Vilain's Perforce-git import. Sam said he'd go over the import code to see if he'd missed anything. By and large most of the Perforce branches were bit-for-bit perfect.

perl@34467

Nicholas produced another snapshot of 5.8.9-tobe. Of note were some problems on VMS (for which fixes existed) and some minor problems with Module::Build.

Slaven Rezic tested some 3700 CPAN distributions and listed a dozen or so regressions. Most of them turned out to be minor, but one was an honest to goodness C error, relying on a reallocated SV landing in exactly right place.

Fedora (Assertion failure on glob assignment)

Nicholas Clark wailed and gnashed his teeth at vendors who insist on foisting a perl compiled with -DDEBUGGING upon their users, and pondered tweaking things to make it produce a nasty warning on STDERR during startup.

t/op/regexp.t

Abigail wondered how to deal with a test that causes a segfault during program exit, since even a fresh_perl_is from test.pl will return a correct result, not seeing that $? contains a non-zero value. Michael G. Schwern thought that the expedient measure of testing $? against zero should be sufficient.

TODO of the week

Linker specification files

Some platforms mandate that you provide a list of a shared library's external symbols to the linker, so the core already has the infrastructure in place to do this for generating shared perl libraries. My understanding is that the GNU toolchain can accept an optional linker specification file, and restrict visibility just to symbols declared in that file. It would be good to extend makedef.pl to support this format, and to provide a means within Configure to enable it. This would allow Unix users to test that the export list is correct, and to build a perl that does not pollute the global namespace with private symbols.

Patches of Interest

Cross-compile for linux

Vadim Konovalov continued to deliver his patches for cross-compilation. Alas, few people with commit bits have had the time to review them and enter them into Perforce. There is hope that git may alleviate these sorts of problems.

5.8.x on Interix

Ben Morrow reported on the test failures on Interix for 5.8.9-tobe. Jan Dubois took issue with the skipping of tests, since that might lead the underlying bugs to be swept under the carpet. Ben thought that there was a distinction to be made as to whether the test suite was testing Perl or Perl and its interaction with its host environment (such as bugs in the C library).

Rafa�l hoped that the changes, if they were good enough for 5.8, could also make it into blead. Ben saw no reason why not, he just wanted to help get 5.8.9 out the door at the moment.

Invalid read in regdupe in maint-5.8

Nicholas continued to ponder what Vincent Pit had uncovered, regarding an off-by-one allocation in the regexp compiler.

New and old bugs from RT

Different behaviour using the -Idir option on the command line and shebang (#6665)

Ren�e B�cker wondered why using -I on the shebang line of a Perl program appends the specified directory to @INC, rather than prepending.

Rafa�l Garcia-Suarez explained how and why it was a feature, and Ren�e proposed a patch to make -I always prepend.

return do { }: take 3 (or 4...) (#38809)

sub foo { do { return do { 1; 2 } }; 3 } returns undef. This is unfortunate. Vincent Pit has been chipping away at the problem, trying to solve it. This is his latest offering. He's still not happy with the fix, and suggests that it's either this, or ignore the bug.

Perl 5.8.8 Build Failures for SunOS 4.1.3 (#59456)

Bob Peterson called for help in getting a new Perl running on a very, very long in tooth SunOS platform. Andy Dougherty thought that the last time he had attempted such a task was in 2001, and thought that some of the errors Bob was encountering were due compiler bugs, and that building without -O2 optimisation may alleviate some of the problems.

Nicholas Clark identified some failures as being Mostly Harmless, but others were rather more disconcerting.

Unicode::UCD::casefold() does not work as documented, nor probably as intended (#58430)

Karl Williamson continued to work on Perl's simple-minded implementation of folding case for Unicode characters, the real-world problems that this induces, and a proposal on how to move to something that works whilst retaining backwards compatibility.

Rafa�l asked for some background for the readers following along at home, and Karl responded with a very well written summary of the problem domain.

Segfault on weird but plain Perl in eval on 5.10.0 on Windows (#59498)

Eirik Hanssen and Alexandr Ciornii pinned down some code that failed in Strawberry Perl 5.10.0 and wondered if it was common to all Windows 5.10 builds. Cygwin and Unix builds appeared to be unaffected.

Memory leak with regex in 5.10.0 (#59516)

Robin Hill offered an interesting bug on 5.10 that proceeded to eat all memory. The trigger was a regexp with a quotemeta'ed portion and some single character character classes.

Nicholas Clark noted that while lots of memory gets chewed up, if the program is able to run to completion, every last byte is correctly freed. Dave Mitchell produced a smaller test case to demonstrate the leak... which still cleaned up after itself if things ended in an orderly fashion. Believed (by Nicholas) to be solved.

Surprising behaviour when $ is left off $last (#59522)

Inside a curly brace block, even just for lexical scoping, last will take you out of it without a murmur, even if you write $x < last instead of $x < $last. Unfortunately there's not a whole lot that can be done about it.

readline returns EBADF at eof in 5.10.0 (#59550)

Mark Martinec reported that the readline() function (or a diamond I/O operator <>) returns a EBADF (Bad file descriptor) as its errno status on end-of-file when PERLIO is set to "stdio".

Error installing a Perl 5.8.8 (#59610)

Leandro Gregorio needs some help with an installation on a 64-bit Linux platform.

FOLDCHAR regop not produced for \x, \0, \N{U+....} (#59616)

Karl Williamson is now using the latest version of blead for his work and is pushing out the bugs he encounters in Unicode case folding. His latest item is that specifying characters in hex, octal or named characters defeats the regular expression engine.

Save code generated by references in @INC (#59626)

Shawn Moore wanted to be able to profile code references in @INC via Devel::NYTProf, and found a number of hurdles in his way. He offered a patch that appeared to work, but Nicholas Clark pointed out that it did in fact only appear to work, and making it work would in fact be much more difficult.

Unicode problem (#58182)

Karl Williamson continued on his Unicode trek this fortnight, with a summary of the issues concerning this bug (for which the main point is about how to land it gracefully in 5.10 in a backwards-compatible manner, switchable via a lexical pragma). Another item is what to do about regular expressions which are compiled in one (Unicode-encoding) context and then used in another.

Glenn Linderman and Rafa�l made a few comments, and it appears that Karl has managed to reach a concensus of sorts.

unicore/mktables expects wrong syntax from CaseFolding.txt for Turkish (#58656)

Karl pinned down the bug in Turkic character handling in Unicode. In his words: "it appears that the Unicode folks decided they were doing the wrong thing in 3.1, and changed it in 3.2. Perl is still looking for the 3.1 syntax." That is, they snuck it in without telling anybody. Karl hopes to be able to produce a patch.

File::Copy does not handle file objects sanely (#59650)

Too late for -CS option (#59652)

Either that, or too early, since no-one has the tuits to implement something that would do the right thing. In the meantime, Aristotle Pagaltzis proffered the canonical workaround.

SIGSEGV compiling regexp in 5.10.0 (#59654)

Salvador Fandi�o found a way to make things go boom. But what could that pattern be used to match?

Cwd::realpath fails on unsearchable directory (AIX) (#59662)

John Wiersba showed that a short C program worked where Perl fell over and choked.

Segfault when using (?|) in regexp. (#59734)

Abigail gives the regexp engine indigestion, with a short, sharp perl -wE '";" =~ /(?<a>(?|(?<b>;)))/;' to the ribs.

Documentation issues with 5.8.8/Math/BigInt.pm (#59746)

File::Find behaviour with files among @DIRS (#59750)

J. Flack noted with pleasure that File::Find does more or less the right thing when given a file as a starting point. And in fact, if it didn't, it would be incredibly awkward to make it so from the client side.

Unfortunately, it is completely undocumented, so his/her plea was to have it documented so that it could be relied on, unless perchance it doesn't work at all on some obscure platform.

Changing effective userid from root to normal user fails on Linux (#59766)

Markus Huaraz filed a bug report on Linux regarding the setting of $> preventing perl from reading its own binary. The /proc symbol table entered the picture, but Rafa�l Garcia-Suarez was unable to reproduce the problem.

Not OK: perl 5.11.0 +DEVEL +patchaperlup: on i686-linux-64int 2.6.22-1-k7 (UNINSTALLED) (#59776)

Andreas K�nig reported a smoke failure and suggested the fix (in ExtUtils::MakeMaker).

Cannot recurse inside (?|) (#59792)

Whilst working upon the next release of Regexp::Common, Abigail encountered some flaws in the implementation of (?|) and offered some tests to pin down the problem.

Hash collision? (#59796)

return 0 or die (#59802)

Perl5 Bug Summary

New Core Modules

In Brief

Ren�e B�cker pointed out that there are many patches in RT bug reports, just begging to be applied.

Michael G. Schwern suggested an improved GM/LOCALTIME_MAX/MIN search algorithm. H. Merijn Brand suggested a tweak or two.

Michael G. Schwern discovered that HP's Testdrive being replaced. As a long-time resource for testing VMS issues regarding Perl, it will be greatly missed. The HP engineer that H.Merijn contacted suggested that something could be worked out.

Nicholas Clark wasn't sure if CPAN bug #39403 involved something that needed to be fixed in the core (the ability to overload or intercept an attribute, other than using a source filter).

Nicholas also proposed a wild-assed guess to allow parallel testing on Win32 but Steve Hay reported no joy. Andy Armstrong understood the resulting symptoms sufficiently to volunteer to see if he could figure out why.

Salvador Fandi�o noted that Paul Fenwick was not the first person to whether the prototype for CORE::truncate is wrong.

Nicholas removed the missing type VSTRING in the documentation to ref since it wasn't supposed to be there at all, being an overly optimistic backport from blead.

John O'Rourke ran into some odd behaviour of substr(lc($utf8)) in 5.8.8 that Nicholas had fixed, he thought, thanks to a TPF grant. Since that work had made it to 5.10 and had been backported to the upcoming 5.8.9, he was happy to leave things there.

Nicholas noticed some make distclean failures, in that there were some files left over from a run of the test suite that weren't cleanup up after the target was run. Andy Armstrong began to look at cleaning things up, since he had a nagging suspicion that a number of them might be his fault.

Nicholas saw a Debugger saved lines regression and wondered out loud what was the new officially sanctioned way of noting new subroutines that are defined during an eval. When the right technique is found, a test he wrote should begin to pass.

Last week's summary

About this summary

If you found this summary useful, please consider contributing to the Perl Foundation or attending a YAPC to help support the development of Perl.