Error building phar with compression enabled

[localgod]

I get the following error:

proc_open(): unable to create pipe Too many open files

when I enable the following option in the box.json file:

"compression": "BZ2",

I'm trying to build a phar app with the Cilex framework as the foundation.
I'm running PHP 5.4.27 on OSX 10.9.2

Does this error message happen without the "compression" setting?

http://stackoverflow.com/questions/18744991/symfony2-process-component-unable-to-create-pipe-and-launch-a-new-process

Seems like the gc is still full of bugs (after 10 years). Typical php behavior, don't be afraid... ;-)

@localgod Can you use dev-master to see if you still see this issue?

[localgod]

I'm normally using the phar version from http://box-project.org/
When I try to install dev-master with composer I get this message:

Skipped installation of bin bin/box for package kherge/box: file not found in package

So I can not really test it.

Try:

composer require kherge/box=2.0.x-dev

[kingcrunch]

It seems, that this is still an issue with the current phar. Anything I can do?

PHP Fatal error:  Uncaught exception 'ErrorException' with message 'proc_open(): unable to create pipe Too many open files' in phar:///usr/local/bin/box.phar/src/vendors/symfony/console/Symfony/Component/Console/Application.php:974
Stack trace:
#0 [internal function]: KevinGH\Box\Application->KevinGH\Box\{closure}(2, 'proc_open(): un...', 'phar:///usr/loc...', 974, Array)
#1 phar:///usr/local/bin/box.phar/src/vendors/symfony/console/Symfony/Component/Console/Application.php(974): proc_open('stty -a | grep ...', Array, NULL, NULL, NULL, Array)
#2 phar:///usr/local/bin/box.phar/src/vendors/symfony/console/Symfony/Component/Console/Application.php(784): Symfony\Component\Console\Application->getSttyColumns()
#3 phar:///usr/local/bin/box.phar/src/vendors/symfony/console/Symfony/Component/Console/Application.php(745): Symfony\Component\Console\Application->getTerminalDimensions()
#4 phar:///usr/local/bin/box.phar/src/vendors/symfony/console/Symfony/Component/Console/Application.php(675): Symfony\Component\Console\Application->getTe in phar:///usr/local/bin/box.phar/src/vendors/symfony/console/Symfony/Component/Console/Application.php on line 974

[kherge]

I'm afraid I don't know what else I can do on my end.

I believe that this is a problem with the phar extension itself, which may not be closing open file handles.

@kherge You should reduce the problem to 1-10 lines of code, write a short example and send a bug report (https://bugs.php.net/) or find an existing one with a workaround. I thought the workaround I linked would solve the problem. :S Maybe different bugs can cause the same error message. Good luck!

[lserni]

I have the same problem. The cause is indeed in the phar extension; the "compressFiles" function compresses each file to a temporary one, and when it has finished, it reads them back together again, and then closes them all. Doing so requires N files to be open at once. (You can increase the limit using ulimit() under Unix/Linux, though).

[kherge]

@lserni Thanks for the additional info!

I'll see if I can write a small code example to trigger this issue.

[lserni]

The problem is actually in the "phar.c" file. This is a heavily snipped excerpt. Note that there is one cycle opening as many tmpfiles() as needed, and then another that makes use of them. There seems to be a check that avoids using those handles altogether, though.
An alternative would be to not store entry->cfp, but rather a pair ( offset, length ) with which to seek into a single huge temporary file with fixed cfp. Then all rewind(entry->cfp)'s would be replaced by fseek(hugefp, entry->offset, SEEK_SET), etc.:

newfile = php_stream_fopen_tmpfile();

for (zend_hash_internal_pointer_reset(&phar->manifest);
    zend_hash_has_more_elements(&phar->manifest) == SUCCESS;
    zend_hash_move_forward(&phar->manifest)) {

   ...
   entry->cfp = php_stream_fopen_tmpfile();

}

/* now copy the actual file data to the new phar */
offset = php_stream_tell(newfile);
for (zend_hash_internal_pointer_reset(&phar->manifest);
    zend_hash_has_more_elements(&phar->manifest) == SUCCESS;
    zend_hash_move_forward(&phar->manifest)) {

   if (entry->cfp) {
        file = entry->cfp;
        php_stream_rewind(file);
   }

   ...
   if (entry->cfp) {
        php_stream_close(entry->cfp);
        entry->cfp = NULL;
   }
}

/* finally, close the temp file, rename the original phar,
   move the temp to the old phar, unlink the old phar, and reload it into memory
*/
if (phar->fp && free_fp) {
    php_stream_close(phar->fp);
}

[kherge]

Found an open bug on PHP's bug tracker which pretty much describes what we're experiencing.

Here's a snippet that will reproduce the issue:

<?php

ini_set('memory_limit', -1);

$max_handles = (int) file_get_contents('/proc/sys/fs/file-max');

if (!is_dir('src')) {
    echo "Creating source files...\n";

    mkdir('src');

    $total_dirs = ceil($max_handles / 1000);

    for ($current_dir = 0; $current_dir < $total_dirs; $current_dir++) {
        mkdir("src/$current_dir");

        for ($current_file = 0; $current_file < 1000; $current_file++) {
            touch("src/$current_dir/$current_dir-$current_file.php");
        }
    }
} else {
    echo "Source files already exist, skipping.\n";
}

if (file_exists('bug.phar')) {
    echo "Removing previous phar...\n";

    unlink('bug.phar');
}

echo "Building new phar...\n";

$phar = new Phar('bug.phar');
$phar->buildFromIterator(
    new RecursiveIteratorIterator(
        new RecursiveDirectoryIterator(
            'src',
            FilesystemIterator::SKIP_DOTS
        )
    ),
    'src'
);

echo "Compression files in phar...\n";

$phar = new Phar('bug.phar');
$phar->compressFiles(Phar::GZ);

EDIT: I also updated PHP's bug tracker.

PHP devs do not seem to care about PHP extensions since 2010. Lot of bugs are not resolved since then in the soap and phar extensions (but I guess you can find more extensions easily which suffered the same faith). I don't know whether you can do anything about this. You need some c skills, and fork, fix, pull request, but I guess these options are not available by PHP extensions (I may be wrong).