FREE THOUGHT · FREE SOFTWARE · FREE WORLD

Home  »  PHP  »  Downloading Multiple Files with Curl Simultaneously

by 5 comments

Wouldn't it be great if you could use php and curl to download multiple files simultaneously using built-in curl functions? You can!



Example Code

Note that this won't work out of the box or anything, I was passing this function an array containing a bunch of mod_rewrite variables, but you can get an idea of how to use the code.

function curlit_rewritecond($U,$pass){
    global $RPASS,$RSITE;
  $RPASS=$pass;
 
    $FF_HDR=array("Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5",
  "Accept-Language: en-us,en;q=0.9,de;q=0.8,ja;q=0.8,zh;q=0.7,zh-cn;q=0.6,nl;q=0.5,fr;q=0.5,it;q=0.4,ko;q=0.3,es;q=0.2,ru;q=0.2,pt;q=0.1",
  "Accept-Encoding: gzip,deflate","Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7","Keep-Alive: 300","Connection: keep-alive","Pragma:");
 
    if($fp = tmpfile()){
        $mh = curl_multi_init();
        foreach ($U as $i => $ur) {
      $url=$RSITE.'?Q='.$ur;
            if (!$url_info = parse_url($url)) die('bad url '.$url);
            $ch[$i] = curl_init($url);
            curl_setopt ($ch[$i], CURLOPT_HEADERFUNCTION, 'aacurlheader');
            curl_setopt ($ch[$i], CURLOPT_HEADER, 1);
            curl_setopt ($ch[$i], CURLOPT_VERBOSE, 0);
            curl_setopt ($ch[$i], CURLOPT_NOBODY, 1);
            curl_setopt ($ch[$i], CURLOPT_CUSTOMREQUEST, 'GET');
            curl_setopt ($ch[$i], CURLOPT_STDERR, $fp);
            curl_setopt ($ch[$i], CURLOPT_FAILONERROR, 0);
            curl_setopt ($ch[$i], CURLOPT_FOLLOWLOCATION, 0);
            curl_setopt ($ch[$i], CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7 (via www.askapache.com)');
            curl_setopt ($ch[$i], CURLOPT_INTERFACE, '208.86.158.195');
            curl_setopt ($ch[$i], CURLOPT_HTTPHEADER, $FF_HDR);
            curl_setopt ($ch[$i], CURLOPT_REFERER, 'http://www.askapache.com');
            curl_setopt ($ch[$i], CURLOPT_ENCODING, 0);
            curl_setopt ($ch[$i], CURLOPT_CONNECTTIMEOUT, 45);
            curl_setopt ($ch[$i], CURLOPT_MAXCONNECTS, 5);
            curl_setopt ($ch[$i], CURLOPT_MAXREDIRS, 0);
            curl_multi_add_handle ($mh,$ch[$i]);
        }
        do { ob_start();$r=curl_multi_exec($mh,$active);$t=ob_get_clean();}
    while($r == CURLM_CALL_MULTI_PERFORM || $active);
    if ($r != CURLM_OK) die("Curl multi read error $r");
        foreach ($U as $i => $url) {
      if (curl_errno($ch[$i])) {echo curl_error($ch[$i])."-".curl_errno($ch[$i]);}
      //else $cch=curl_getinfo($ch[$i]);
            curl_multi_remove_handle($mh,$ch[$i]);
            curl_close($ch[$i]);
        }
        curl_multi_close($mh);
        fclose($fp);
    }
  sleep(1);
  return true;
}

php code example 2

This code will download all the files listed in the $urls array to the folder specified by the $saveto variable.

<?php
 
$urls=array(
'http://f.askapache.com/mp3/12-lessons-for-those-afraid-of-css.mp3',
'http://f.askapache.com/mp3/27-request-methods-for-use-with-apache-and-rewritecond-and-htaccess.mp3',
'http://f.askapache.com/mp3/301-redirect-with-mod_rewrite-or-redirectmatch.mp3',
'http://f.askapache.com/mp3/404-errorpages.mp3',
'http://f.askapache.com/mp3/503-service-temporarily-unavailable.mp3',
'http://f.askapache.com/mp3/adsense-robots.mp3',
'http://f.askapache.com/mp3/alexa-toolbar-firefox.mp3',
'http://f.askapache.com/mp3/allowing-access-from-1-static-ip-and-deny-the-rest.mp3',
'http://f.askapache.com/mp3/apache-authentication-in-htaccess.mp3');
 
$save_to='/web/user/public_html/mp3/';
 
$mh = curl_multi_init();
foreach ($urls as $i => $url) {
    $g=$save_to.basename($url);
    if(!is_file($g)){
        $conn[$i]=curl_init($url);
        $fp[$i]=fopen ($g, "w");
        curl_setopt ($conn[$i], CURLOPT_FILE, $fp[$i]);
        curl_setopt ($conn[$i], CURLOPT_HEADER ,0);
        curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,60);
        curl_multi_add_handle ($mh,$conn[$i]);
    }
}
do {
    $n=curl_multi_exec($mh,$active);
}
while ($active);
foreach ($urls as $i => $url) {
    curl_multi_remove_handle($mh,$conn[$i]);
    curl_close($conn[$i]);
    fclose ($fp[$i]);
}
curl_multi_close($mh);
?>

Curl Multi Functions

curl_multi_init
Returns a new cURL multi handle
curl_multi_add_handle
Add a normal cURL handle to a cURL multi handle
curl_multi_exec
Run the sub-connections of the current cURL handle
curl_multi_close
Close a set of cURL handles
curl_multi_getcontent
Return the content of a cURL handle if CURLOPT_RETURNTRANSFER is set
curl_multi_info_read
Get information about the current transfers
curl_multi_select
Get all the sockets associated with the cURL extension, which can then be "selected"
curl_multi_remove_handle
Remove a multi handle from a set of cURL handles

Tags

September 5th, 2007

Comments Welcome

  • http://www.securitywonks.net Raghu Veer

    this code works fine. I had tested with different files of different file sizes :)

  • Dirk Hooghe

    I am searching a way to give the user the possibility to download multiple files (midi files) on their hard disk. The second script I use to do this but it is only working when I run it from my local computer (local apache server). When I execute it from my website it won't work.
    I used

    $save_to='c:webmidi';

    to go to the local drive. Has anybody a solution?

  • omi

    if the browser is closed downloading is also stop so how do we use shell_exec command so that if the browser is closed it does not effect my downloading?

  • Hemant Arora

    Works fine for files with binary data. But, I was unable to upload PHP files through this script, as when I did, the files uploaded were either 0kb in size or had some php error as content. This is happening because the script executes the PHP code inside each file I transfer and saves the output as content of downloaded file. Please advise a solution. Thanks a lot.

  • http://www.eservices-india.com/ eservicesindia

    how to improve cache or performance in curl

Popular Articles
My Online Tools

Related Articles
Newest Posts
Twitter


  • @askapache · Jul 21
    Magic spells for sending thoughts across time? Books
  • @askapache · Jul 20
    TV is just a relic of the previous generation. We just don't know it yet.
  • @askapache · Jul 20
    I will never go back on my ideals, no matter the cost. I'll never let the economic vultures steal my dreams. I'd rather give up the ghost
  • @askapache · Jul 18
    I don't want a better seat, I want control of the engine
  • @askapache · Jul 14
    No matter how good u r, there will always be someone 2x, 5x, 100x better. This is true for me and everyone. No direction but forward.
  • @askapache · Jul 12
    Heads up, I'll DDoS the f out of askapache next week, to see how resilient it really is :) - will try to overflow disk, net, and ip stack
  • RUN GCC! This is a typical shirt I wear, from the  t.co/46LYbFr4k2  shop. A clerk at the LQ recognized it!  t.co/jjmT0dkCPu 
  • Merlin the Magician  t.co/iMmRbanUi4 
  • ROGUE CODE - Latest novel from @markrussinovich  t.co/apkn0LoPIt 
  • RTFM - surprisingly very helpful and way more comprehensive than it looks! @redteamfieldman #pwnAllTheThings  t.co/xiaJ5g0aC9 
  • Dear Hacker - Letters to the Editor of 2600, from Emmanuel Goldstein  t.co/JCfLab7FAJ 
  • The Mythical Man-Month - Essays on Software Engineering, by Frederick P. Brooks, Jr.  t.co/ilWN5GHElr 
  • "where wizards stay up late" - The Origins of the Internet. Favorite book detailing the birth of the net and IMPs  t.co/gY9VTGJgZz 
  • ZERO DAY - read before Trojan horse  t.co/pPMLGDJv8P 
  • Trojan Horse, a novel!  t.co/Hf8EtYaZVa 
  • The Hacker Playbook - very nice high level overview of attacks  t.co/lHwNVWi61u 
  • Clean Code - A Handbook of Agile Software Craftsmanship  t.co/hnJX0x1qIc 
  • Secrets of the JavaScript Ninja - By my absolute favorite JS hacker John Resig!  t.co/tZ42ljmcCl 
  • Hacking Exposed 7: Network Security Secrets & SolutionsMy all time favorite, basic but thorough and accurate.  t.co/jycW0RDVtZ 
  • Empty words will be no surrogate for cold resolve. Pain is nothing.  t.co/qXjpRxbjCw 

Hacking and Hackers

The use of "hacker" to mean "security breaker" is a confusion on the part of the mass media. We hackers refuse to recognize that meaning, and continue using the word to mean someone who loves to program, someone who enjoys playful cleverness, or the combination of the two. See my article, On Hacking.
-- Richard M. Stallman






[hide]

It's very simple - you read the protocol and write the code. -Bill Joy

Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution 3.0 License, just credit with a link.
This site is not supported or endorsed by The Apache Software Foundation (ASF). All software and documentation produced by The ASF is licensed. "Apache" is a trademark of The ASF. NCSA HTTPd.
UNIX ® is a registered Trademark of The Open Group. POSIX ® is a registered Trademark of The IEEE.

| Google+ | askapache

Site Map | Contact Webmaster | License and Disclaimer | Terms of Service

↑ TOPMain