FREE THOUGHT · FREE SOFTWARE · FREE WORLD

Home  »  PHP  »  Downloading Multiple Files with Curl Simultaneously

by 5 comments

Wouldn't it be great if you could use php and curl to download multiple files simultaneously using built-in curl functions? You can!



Example Code

Note that this won't work out of the box or anything, I was passing this function an array containing a bunch of mod_rewrite variables, but you can get an idea of how to use the code.

function curlit_rewritecond($U,$pass){
    global $RPASS,$RSITE;
	$RPASS=$pass;

    $FF_HDR=array("Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5",
	"Accept-Language: en-us,en;q=0.9,de;q=0.8,ja;q=0.8,zh;q=0.7,zh-cn;q=0.6,nl;q=0.5,fr;q=0.5,it;q=0.4,ko;q=0.3,es;q=0.2,ru;q=0.2,pt;q=0.1",
	"Accept-Encoding: gzip,deflate","Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7","Keep-Alive: 300","Connection: keep-alive","Pragma:");

		if($fp = tmpfile()){
        $mh = curl_multi_init();
        foreach ($U as $i => $ur) {
			$url=$RSITE.'?Q='.$ur;
            if (!$url_info = parse_url($url)) die('bad url '.$url);
            $ch[$i] = curl_init($url);
            curl_setopt ($ch[$i], CURLOPT_HEADERFUNCTION, 'aacurlheader');
            curl_setopt ($ch[$i], CURLOPT_HEADER, 1);
            curl_setopt ($ch[$i], CURLOPT_VERBOSE, 0);
            curl_setopt ($ch[$i], CURLOPT_NOBODY, 1);
            curl_setopt ($ch[$i], CURLOPT_CUSTOMREQUEST, 'GET');
            curl_setopt ($ch[$i], CURLOPT_STDERR, $fp);
            curl_setopt ($ch[$i], CURLOPT_FAILONERROR, 0);
            curl_setopt ($ch[$i], CURLOPT_FOLLOWLOCATION, 0);
            curl_setopt ($ch[$i], CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7 (via www.askapache.com)');
            curl_setopt ($ch[$i], CURLOPT_INTERFACE, '208.86.158.195');
            curl_setopt ($ch[$i], CURLOPT_HTTPHEADER, $FF_HDR);
            curl_setopt ($ch[$i], CURLOPT_REFERER, 'http://www.askapache.com');
            curl_setopt ($ch[$i], CURLOPT_ENCODING, 0);
            curl_setopt ($ch[$i], CURLOPT_CONNECTTIMEOUT, 45);
            curl_setopt ($ch[$i], CURLOPT_MAXCONNECTS, 5);
            curl_setopt ($ch[$i], CURLOPT_MAXREDIRS, 0);
            curl_multi_add_handle ($mh,$ch[$i]);
        }
        do { ob_start();$r=curl_multi_exec($mh,$active);$t=ob_get_clean();}
		while($r == CURLM_CALL_MULTI_PERFORM || $active);
		if ($r != CURLM_OK) die("Curl multi read error $r");
        foreach ($U as $i => $url) {
			if (curl_errno($ch[$i])) {echo curl_error($ch[$i])."-".curl_errno($ch[$i]);}
			//else $cch=curl_getinfo($ch[$i]);
            curl_multi_remove_handle($mh,$ch[$i]);
            curl_close($ch[$i]);
        }
        curl_multi_close($mh);
        fclose($fp);
		}
	sleep(1);
	return true;
}

php code example 2

This code will download all the files listed in the $urls array to the folder specified by the $saveto variable.

 $url) {
    $g=$save_to.basename($url);
    if(!is_file($g)){
        $conn[$i]=curl_init($url);
        $fp[$i]=fopen ($g, "w");
        curl_setopt ($conn[$i], CURLOPT_FILE, $fp[$i]);
        curl_setopt ($conn[$i], CURLOPT_HEADER ,0);
        curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,60);
        curl_multi_add_handle ($mh,$conn[$i]);
    }
}
do {
    $n=curl_multi_exec($mh,$active);
}
while ($active);
foreach ($urls as $i => $url) {
    curl_multi_remove_handle($mh,$conn[$i]);
    curl_close($conn[$i]);
    fclose ($fp[$i]);
}
curl_multi_close($mh);
?>

Curl Multi Functions

curl_multi_init
Returns a new cURL multi handle
curl_multi_add_handle
Add a normal cURL handle to a cURL multi handle
curl_multi_exec
Run the sub-connections of the current cURL handle
curl_multi_close
Close a set of cURL handles
curl_multi_getcontent
Return the content of a cURL handle if CURLOPT_RETURNTRANSFER is set
curl_multi_info_read
Get information about the current transfers
curl_multi_select
Get all the sockets associated with the cURL extension, which can then be "selected"
curl_multi_remove_handle
Remove a multi handle from a set of cURL handles

Tags

September 5th, 2007

Comments Welcome

  • Raghu Veer

    this code works fine. I had tested with different files of different file sizes :)

  • Dirk Hooghe

    I am searching a way to give the user the possibility to download multiple files (midi files) on their hard disk. The second script I use to do this but it is only working when I run it from my local computer (local apache server). When I execute it from my website it won't work.
    I used

    $save_to='c:webmidi';

    to go to the local drive. Has anybody a solution?

  • omi

    if the browser is closed downloading is also stop so how do we use shell_exec command so that if the browser is closed it does not effect my downloading?

  • Hemant Arora

    Works fine for files with binary data. But, I was unable to upload PHP files through this script, as when I did, the files uploaded were either 0kb in size or had some php error as content. This is happening because the script executes the PHP code inside each file I transfer and saves the output as content of downloaded file. Please advise a solution. Thanks a lot.

  • eservicesindia

    how to improve cache or performance in curl


Related Articles


My Online Tools
Popular Articles


Hacking and Hackers

The use of "hacker" to mean "security breaker" is a confusion on the part of the mass media. We hackers refuse to recognize that meaning, and continue using the word to mean someone who loves to program, someone who enjoys playful cleverness, or the combination of the two. See my article, On Hacking.
-- Richard M. Stallman









[hide]

It's very simple - you read the protocol and write the code. -Bill Joy

Except where otherwise noted, content on this site is licensed under a Creative Commons Attribution 3.0 License, just credit with a link.
This site is not supported or endorsed by The Apache Software Foundation (ASF). All software and documentation produced by The ASF is licensed. "Apache" is a trademark of The ASF. NCSA HTTPd.
UNIX ® is a registered Trademark of The Open Group. POSIX ® is a registered Trademark of The IEEE.

+Askapache | askapache

Site Map | Contact Webmaster | License and Disclaimer | Terms of Service

↑ TOPMain