FREE THOUGHT · FREE SOFTWARE · FREE WORLD

Home  »  PHP  »  Downloading Multiple Files with Curl Simultaneously

by 5 comments

Wouldn't it be great if you could use php and curl to download multiple files simultaneously using built-in curl functions? You can!



Example Code

Note that this won't work out of the box or anything, I was passing this function an array containing a bunch of mod_rewrite variables, but you can get an idea of how to use the code.

function curlit_rewritecond($U,$pass){
    global $RPASS,$RSITE;
  $RPASS=$pass;
 
    $FF_HDR=array("Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5",
  "Accept-Language: en-us,en;q=0.9,de;q=0.8,ja;q=0.8,zh;q=0.7,zh-cn;q=0.6,nl;q=0.5,fr;q=0.5,it;q=0.4,ko;q=0.3,es;q=0.2,ru;q=0.2,pt;q=0.1",
  "Accept-Encoding: gzip,deflate","Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7","Keep-Alive: 300","Connection: keep-alive","Pragma:");
 
    if($fp = tmpfile()){
        $mh = curl_multi_init();
        foreach ($U as $i => $ur) {
      $url=$RSITE.'?Q='.$ur;
            if (!$url_info = parse_url($url)) die('bad url '.$url);
            $ch[$i] = curl_init($url);
            curl_setopt ($ch[$i], CURLOPT_HEADERFUNCTION, 'aacurlheader');
            curl_setopt ($ch[$i], CURLOPT_HEADER, 1);
            curl_setopt ($ch[$i], CURLOPT_VERBOSE, 0);
            curl_setopt ($ch[$i], CURLOPT_NOBODY, 1);
            curl_setopt ($ch[$i], CURLOPT_CUSTOMREQUEST, 'GET');
            curl_setopt ($ch[$i], CURLOPT_STDERR, $fp);
            curl_setopt ($ch[$i], CURLOPT_FAILONERROR, 0);
            curl_setopt ($ch[$i], CURLOPT_FOLLOWLOCATION, 0);
            curl_setopt ($ch[$i], CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7 (via www.askapache.com)');
            curl_setopt ($ch[$i], CURLOPT_INTERFACE, '208.86.158.195');
            curl_setopt ($ch[$i], CURLOPT_HTTPHEADER, $FF_HDR);
            curl_setopt ($ch[$i], CURLOPT_REFERER, 'http://www.askapache.com');
            curl_setopt ($ch[$i], CURLOPT_ENCODING, 0);
            curl_setopt ($ch[$i], CURLOPT_CONNECTTIMEOUT, 45);
            curl_setopt ($ch[$i], CURLOPT_MAXCONNECTS, 5);
            curl_setopt ($ch[$i], CURLOPT_MAXREDIRS, 0);
            curl_multi_add_handle ($mh,$ch[$i]);
        }
        do { ob_start();$r=curl_multi_exec($mh,$active);$t=ob_get_clean();}
    while($r == CURLM_CALL_MULTI_PERFORM || $active);
    if ($r != CURLM_OK) die("Curl multi read error $r");
        foreach ($U as $i => $url) {
      if (curl_errno($ch[$i])) {echo curl_error($ch[$i])."-".curl_errno($ch[$i]);}
      //else $cch=curl_getinfo($ch[$i]);
            curl_multi_remove_handle($mh,$ch[$i]);
            curl_close($ch[$i]);
        }
        curl_multi_close($mh);
        fclose($fp);
    }
  sleep(1);
  return true;
}

php code example 2

This code will download all the files listed in the $urls array to the folder specified by the $saveto variable.

<?php
 
$urls=array(
'http://f.askapache.com/mp3/12-lessons-for-those-afraid-of-css.mp3',
'http://f.askapache.com/mp3/27-request-methods-for-use-with-apache-and-rewritecond-and-htaccess.mp3',
'http://f.askapache.com/mp3/301-redirect-with-mod_rewrite-or-redirectmatch.mp3',
'http://f.askapache.com/mp3/404-errorpages.mp3',
'http://f.askapache.com/mp3/503-service-temporarily-unavailable.mp3',
'http://f.askapache.com/mp3/adsense-robots.mp3',
'http://f.askapache.com/mp3/alexa-toolbar-firefox.mp3',
'http://f.askapache.com/mp3/allowing-access-from-1-static-ip-and-deny-the-rest.mp3',
'http://f.askapache.com/mp3/apache-authentication-in-htaccess.mp3');
 
$save_to='/web/user/public_html/mp3/';
 
$mh = curl_multi_init();
foreach ($urls as $i => $url) {
    $g=$save_to.basename($url);
    if(!is_file($g)){
        $conn[$i]=curl_init($url);
        $fp[$i]=fopen ($g, "w");
        curl_setopt ($conn[$i], CURLOPT_FILE, $fp[$i]);
        curl_setopt ($conn[$i], CURLOPT_HEADER ,0);
        curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,60);
        curl_multi_add_handle ($mh,$conn[$i]);
    }
}
do {
    $n=curl_multi_exec($mh,$active);
}
while ($active);
foreach ($urls as $i => $url) {
    curl_multi_remove_handle($mh,$conn[$i]);
    curl_close($conn[$i]);
    fclose ($fp[$i]);
}
curl_multi_close($mh);
?>

Curl Multi Functions

curl_multi_init
Returns a new cURL multi handle
curl_multi_add_handle
Add a normal cURL handle to a cURL multi handle
curl_multi_exec
Run the sub-connections of the current cURL handle
curl_multi_close
Close a set of cURL handles
curl_multi_getcontent
Return the content of a cURL handle if CURLOPT_RETURNTRANSFER is set
curl_multi_info_read
Get information about the current transfers
curl_multi_select
Get all the sockets associated with the cURL extension, which can then be "selected"
curl_multi_remove_handle
Remove a multi handle from a set of cURL handles

Tags

September 5th, 2007

Comments Welcome

  • http://www.securitywonks.net Raghu Veer

    this code works fine. I had tested with different files of different file sizes :)

  • Dirk Hooghe

    I am searching a way to give the user the possibility to download multiple files (midi files) on their hard disk. The second script I use to do this but it is only working when I run it from my local computer (local apache server). When I execute it from my website it won't work.
    I used

    $save_to='c:webmidi';

    to go to the local drive. Has anybody a solution?

  • omi

    if the browser is closed downloading is also stop so how do we use shell_exec command so that if the browser is closed it does not effect my downloading?

  • Hemant Arora

    Works fine for files with binary data. But, I was unable to upload PHP files through this script, as when I did, the files uploaded were either 0kb in size or had some php error as content. This is happening because the script executes the PHP code inside each file I transfer and saves the output as content of downloaded file. Please advise a solution. Thanks a lot.

  • http://www.eservices-india.com/ eservicesindia

    how to improve cache or performance in curl

My Online Tools
WordPress Sites

My Picks

Related Articles
Newest Posts
Twitter

  • @askapache · Apr 22
    Within our small sphere of the universe we can look at our past, but only our future is within our control
  • @askapache · Apr 16
    RT @commandlinefu:  t.co/htRbYF8SFf  - Print ASCII Character Chart (AskApache) #echo #column #ascii #for #octal #askapache #chart
  • @askapache · Apr 11
    Angel headed hipsters burning for the ancient heavenly connection to the starry dynamo in the machinery of the night - Ginsberg
  • @askapache · Apr 11
    If you haven't experienced a hacker war on a free shell network like SDF, you haven't lived!
  • @askapache · Apr 10
    @kovshenin @wpekadotcom @CloudFlare Maybe it's an April fools joke that finally propagated
  • @askapache · Apr 10
    @kovshenin @wpekadotcom @CloudFlare wait this is a joke right?
  • ZERO DAY - read before Trojan horse  t.co/pPMLGDJv8P 
  • Trojan Horse, a novel!  t.co/Hf8EtYaZVa 
  • The Hacker Playbook - very nice high level overview of attacks  t.co/lHwNVWi61u 
  • Clean Code - A Handbook of Agile Software Craftsmanship  t.co/hnJX0x1qIc 
  • Secrets of the JavaScript Ninja - By my absolute favorite JS hacker John Resig!  t.co/tZ42ljmcCl 
  • Hacking Exposed 7: Network Security Secrets & SolutionsMy all time favorite, basic but thorough and accurate.  t.co/jycW0RDVtZ 
  • Empty words will be no surrogate for cold resolve. Pain is nothing.  t.co/qXjpRxbjCw 
  • REVERSING: Secrets of Reverse Engineering  t.co/GaWo29lWWG 
  • NEUROMANCER  t.co/3OoknUcb5Z 
  • "The Shockwave Rider", by John Brunner (1975 hacker sci-fi)  t.co/ZW56HVUefW 
  • The Rootkit ARSENAL - Escape and Evasion in the Dark Corners of the System  t.co/1FzX6bHgsQ 
  • "We Are Anonymous - Inside the Hacker World of LulzSec, Anonymous, and the Global Cyber Insurgency" better be good!  t.co/GL0cFNiUOq 
  • THE IDEA FACTORY Bell Labs  t.co/FyVhgNwwT5 
  • The Datacenter as a Computer -- Urs Holzle  t.co/M5WIYs1OVg 

Friends and Recommends
Hacking and Hackers

The use of "hacker" to mean "security breaker" is a confusion on the part of the mass media. We hackers refuse to recognize that meaning, and continue using the word to mean someone who loves to program, someone who enjoys playful cleverness, or the combination of the two. See my article, On Hacking.
-- Richard M. Stallman






[hide]

It's very simple - you read the protocol and write the code. -Bill Joy

Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution 3.0 License, just credit with a link.
This site is not supported or endorsed by The Apache Software Foundation (ASF). All software and documentation produced by The ASF is licensed. "Apache" is a trademark of The ASF. NCSA HTTPd.
UNIX ® is a registered Trademark of The Open Group. POSIX ® is a registered Trademark of The IEEE.

| Google+ | askapache

Site Map | Contact Webmaster | License and Disclaimer | Terms of Service

↑ TOPMain