Downloading Multiple Files with Curl Simultaneously

FREE THOUGHT · FREE SOFTWARE · FREE WORLD

Downloading Multiple Files with Curl Simultaneously

Wouldn’t it be great if you could use php and curl to download multiple files simultaneously using built-in curl functions? You can!


Example Code

Note that this won’t work out of the box or anything, I was passing this function an array containing a bunch of mod_rewrite variables, but you can get an idea of how to use the code.

function curlit_rewritecond($U,$pass){
    global $RPASS,$RSITE;
  $RPASS=$pass;

    $FF_HDR=array("Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5",
  "Accept-Language: en-us,en;q=0.9,de;q=0.8,ja;q=0.8,zh;q=0.7,zh-cn;q=0.6,nl;q=0.5,fr;q=0.5,it;q=0.4,ko;q=0.3,es;q=0.2,ru;q=0.2,pt;q=0.1",
  "Accept-Encoding: gzip,deflate","Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7","Keep-Alive: 300","Connection: keep-alive","Pragma:");

    if($fp = tmpfile()){
        $mh = curl_multi_init();
        foreach ($U as $i => $ur) {
      $url=$RSITE.'?Q='.$ur;
            if (!$url_info = parse_url($url)) die('bad url '.$url);
            $ch[$i] = curl_init($url);
            curl_setopt ($ch[$i], CURLOPT_HEADERFUNCTION, 'aacurlheader');
            curl_setopt ($ch[$i], CURLOPT_HEADER, 1);
            curl_setopt ($ch[$i], CURLOPT_VERBOSE, 0);
            curl_setopt ($ch[$i], CURLOPT_NOBODY, 1);
            curl_setopt ($ch[$i], CURLOPT_CUSTOMREQUEST, 'GET');
            curl_setopt ($ch[$i], CURLOPT_STDERR, $fp);
            curl_setopt ($ch[$i], CURLOPT_FAILONERROR, 0);
            curl_setopt ($ch[$i], CURLOPT_FOLLOWLOCATION, 0);
            curl_setopt ($ch[$i], CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7 (via www.askapache.com)');
            curl_setopt ($ch[$i], CURLOPT_INTERFACE, '208.86.158.195');
            curl_setopt ($ch[$i], CURLOPT_HTTPHEADER, $FF_HDR);
            curl_setopt ($ch[$i], CURLOPT_REFERER, 'http://www.askapache.com');
            curl_setopt ($ch[$i], CURLOPT_ENCODING, 0);
            curl_setopt ($ch[$i], CURLOPT_CONNECTTIMEOUT, 45);
            curl_setopt ($ch[$i], CURLOPT_MAXCONNECTS, 5);
            curl_setopt ($ch[$i], CURLOPT_MAXREDIRS, 0);
            curl_multi_add_handle ($mh,$ch[$i]);
        }
        do { ob_start();$r=curl_multi_exec($mh,$active);$t=ob_get_clean();}
    while($r == CURLM_CALL_MULTI_PERFORM || $active);
    if ($r != CURLM_OK) die("Curl multi read error $r");
        foreach ($U as $i => $url) {
      if (curl_errno($ch[$i])) {echo curl_error($ch[$i])."-".curl_errno($ch[$i]);}
      //else $cch=curl_getinfo($ch[$i]);
            curl_multi_remove_handle($mh,$ch[$i]);
            curl_close($ch[$i]);
        }
        curl_multi_close($mh);
        fclose($fp);
    }
  sleep(1);
  return true;
}

php code example 2

This code will download all the files listed in the $urls array to the folder specified by the $saveto variable.

<?php
 
$urls=array(
'http://f.askapache.com/mp3/12-lessons-for-those-afraid-of-css.mp3',
'http://f.askapache.com/mp3/27-request-methods-for-use-with-apache-and-rewritecond-and-htaccess.mp3',
'http://f.askapache.com/mp3/301-redirect-with-mod_rewrite-or-redirectmatch.mp3',
'http://f.askapache.com/mp3/404-errorpages.mp3',
'http://f.askapache.com/mp3/503-service-temporarily-unavailable.mp3',
'http://f.askapache.com/mp3/adsense-robots.mp3',
'http://f.askapache.com/mp3/alexa-toolbar-firefox.mp3',
'http://f.askapache.com/mp3/allowing-access-from-1-static-ip-and-deny-the-rest.mp3',
'http://f.askapache.com/mp3/apache-authentication-in-htaccess.mp3');
 
$save_to='/home/user/public_html/mp3/';
 
$mh = curl_multi_init();
foreach ($urls as $i => $url) {
    $g=$save_to.basename($url);
    if(!is_file($g)){
        $conn[$i]=curl_init($url);
        $fp[$i]=fopen ($g, "w");
        curl_setopt ($conn[$i], CURLOPT_FILE, $fp[$i]);
        curl_setopt ($conn[$i], CURLOPT_HEADER ,0);
        curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,60);
        curl_multi_add_handle ($mh,$conn[$i]);
    }
}
do {
    $n=curl_multi_exec($mh,$active);
}
while ($active);
foreach ($urls as $i => $url) {
    curl_multi_remove_handle($mh,$conn[$i]);
    curl_close($conn[$i]);
    fclose ($fp[$i]);
}
curl_multi_close($mh);
?>

Curl Multi Functions

curl_multi_init
Returns a new cURL multi handle
curl_multi_add_handle
Add a normal cURL handle to a cURL multi handle
curl_multi_exec
Run the sub-connections of the current cURL handle
curl_multi_close
Close a set of cURL handles
curl_multi_getcontent
Return the content of a cURL handle if CURLOPT_RETURNTRANSFER is set
curl_multi_info_read
Get information about the current transfers
curl_multi_select
Get all the sockets associated with the cURL extension, which can then be “selected”
curl_multi_remove_handle
Remove a multi handle from a set of cURL handles

«
»

Leave your own comment

Reader Comments

  1. Flo ~

    Hi there,

    I’m wondering about the “sleep(1)” in the first example.
    Why did you put it there?

    Thanks in advance,
    Flo.

  2. slance ~

    I’m trying to utilize the php code example 2 to upload various numbers of files based on a users choice. The user chooses files to be downloaded by selecting corresponding check boxes for each file. Once the desired number of check boxes are selected the user submits the form and the files should be downloaded from the server to the users computer.

    Previous users upload the files and the files are save on the server. The users info is inserted as a record in a database along with the file name. This is where the web page get the name of the files.

    I can’t get the php code to the download to users computers can you help?

  3. PHPGURU ~

    @Udegbunam

    I’m missing the point behind this. How does this hack help me a website owner? Forgive my ignorance.

    If you have to ask, not sure why you came to this page.

  4. pretatma gandhi ~

    Hoooo haaa

    @001

  5. tom ~

    I cant get it working, it just creates all files 0kb. Any ideas?

  6. Udegbunam Chukwudi ~

    I’m missing the point behind this. How does this hack help me a website owner? Forgive my ignorance.

  7. Tommy ~

    Does this automatically select a disk drive to save to? You know, like C: or D: or E: in the computer?

  8. bonaventure ~

    I would add

    curl_setopt($conn[$i], CURLOPT_VERBOSE, 1);
    

    My array had more than 20 connections and I didn’t realize that my wholesaler limited me to 20 concurrent sessions. I figured out the problem with on activated the verbose feature.
    Thanks for your script. It works great!

  9. Josh Fraser ~

    Thanks for sharing this example. I made some modifications so that you can process each request as soon as it completes. It makes things a lot faster when you’re dealing with a large number of requests:

  10. noora ~

    can i use this script to client side downloading

  11. maleos ~

    Thank you this script is very helpful and work fine with me.

  12. Sotland ~

    Is there that much use for using curl? The downloading would be TOO FAST, and the remote host will block you. Better to stick to file_get_contents with a delay between files, or go to spider hell.

  13. lokesh yadav ~

    what type of change we need to done in php.ini files to use this method beacause i used this but this not works.

  14. Raghu Veer ~

    this code works fine. I had tested with different files of different file sizes :)

Go for it!


It's very simple - you read the protocol and write the code. -Bill Joy

HTML | DCMI | GRDDL | XOXO | XDMP | XFN | DOM | XML | XHTML 1.1 Strict | CSS 2.1 | W3C

↑ TOPExcept where otherwise noted, content on this site is licensed under a Creative Commons Attribution 3.0 License, just credit with a link.
This site is not supported or endorsed by The Apache Software Foundation (ASF). All software and documentation produced by The ASF is licensed. "Apache" is a trademark of The ASF. NCSA HTTPd.
UNIX ® is a registered Trademark of The Open Group. POSIX ® is a registered Trademark of The IEEE.

Site Map | Contact Webmaster | Glossary | License and Disclaimer | Terms of Service