« AskApache Search Engine VerifySpeed Tips: Turn On Compression »
Downloading Multiple Files with Curl Simultaneously
Wouldn’t it be great if you could use php and curl to download multiple files simultaneously using built-in curl functions? You can!
Note that this won’t work out of the box or anything, I was passing this function an array containing a bunch of mod_rewrite variables, but you can get an idea of how to use the code.
function curlit_rewritecond($U,$pass){
global $RPASS,$RSITE;
$RPASS=$pass;
$FF_HDR=array("Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5",
"Accept-Language: en-us,en;q=0.9,de;q=0.8,ja;q=0.8,zh;q=0.7,zh-cn;q=0.6,nl;q=0.5,fr;q=0.5,it;q=0.4,ko;q=0.3,es;q=0.2,ru;q=0.2,pt;q=0.1",
"Accept-Encoding: gzip,deflate","Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7","Keep-Alive: 300","Connection: keep-alive","Pragma:");
if($fp = tmpfile()){
$mh = curl_multi_init();
foreach ($U as $i => $ur) {
$url=$RSITE.'?Q='.$ur;
if (!$url_info = parse_url($url)) die('bad url '.$url);
$ch[$i] = curl_init($url);
curl_setopt ($ch[$i], CURLOPT_HEADERFUNCTION, 'aacurlheader');
curl_setopt ($ch[$i], CURLOPT_HEADER, 1);
curl_setopt ($ch[$i], CURLOPT_VERBOSE, 0);
curl_setopt ($ch[$i], CURLOPT_NOBODY, 1);
curl_setopt ($ch[$i], CURLOPT_CUSTOMREQUEST, 'GET');
curl_setopt ($ch[$i], CURLOPT_STDERR, $fp);
curl_setopt ($ch[$i], CURLOPT_FAILONERROR, 0);
curl_setopt ($ch[$i], CURLOPT_FOLLOWLOCATION, 0);
curl_setopt ($ch[$i], CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7 (via www.askapache.com)');
curl_setopt ($ch[$i], CURLOPT_INTERFACE, '208.86.158.195');
curl_setopt ($ch[$i], CURLOPT_HTTPHEADER, $FF_HDR);
curl_setopt ($ch[$i], CURLOPT_REFERER, 'http://www.askapache.com');
curl_setopt ($ch[$i], CURLOPT_ENCODING, 0);
curl_setopt ($ch[$i], CURLOPT_CONNECTTIMEOUT, 45);
curl_setopt ($ch[$i], CURLOPT_MAXCONNECTS, 5);
curl_setopt ($ch[$i], CURLOPT_MAXREDIRS, 0);
curl_multi_add_handle ($mh,$ch[$i]);
}
do { ob_start();$r=curl_multi_exec($mh,$active);$t=ob_get_clean();}
while($r == CURLM_CALL_MULTI_PERFORM || $active);
if ($r != CURLM_OK) die("Curl multi read error $r");
foreach ($U as $i => $url) {
if (curl_errno($ch[$i])) {echo curl_error($ch[$i])."-".curl_errno($ch[$i]);}
//else $cch=curl_getinfo($ch[$i]);
curl_multi_remove_handle($mh,$ch[$i]);
curl_close($ch[$i]);
}
curl_multi_close($mh);
fclose($fp);
}
sleep(1);
return true;
}
This code will download all the files listed in the $urls array to the folder specified by the $saveto variable.
<?php
$urls=array(
'http://f.askapache.com/mp3/12-lessons-for-those-afraid-of-css.mp3',
'http://f.askapache.com/mp3/27-request-methods-for-use-with-apache-and-rewritecond-and-htaccess.mp3',
'http://f.askapache.com/mp3/301-redirect-with-mod_rewrite-or-redirectmatch.mp3',
'http://f.askapache.com/mp3/404-errorpages.mp3',
'http://f.askapache.com/mp3/503-service-temporarily-unavailable.mp3',
'http://f.askapache.com/mp3/adsense-robots.mp3',
'http://f.askapache.com/mp3/alexa-toolbar-firefox.mp3',
'http://f.askapache.com/mp3/allowing-access-from-1-static-ip-and-deny-the-rest.mp3',
'http://f.askapache.com/mp3/apache-authentication-in-htaccess.mp3');
$save_to='/home/user/public_html/mp3/';
$mh = curl_multi_init();
foreach ($urls as $i => $url) {
$g=$save_to.basename($url);
if(!is_file($g)){
$conn[$i]=curl_init($url);
$fp[$i]=fopen ($g, "w");
curl_setopt ($conn[$i], CURLOPT_FILE, $fp[$i]);
curl_setopt ($conn[$i], CURLOPT_HEADER ,0);
curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,60);
curl_multi_add_handle ($mh,$conn[$i]);
}
}
do {
$n=curl_multi_exec($mh,$active);
}
while ($active);
foreach ($urls as $i => $url) {
curl_multi_remove_handle($mh,$conn[$i]);
curl_close($conn[$i]);
fclose ($fp[$i]);
}
curl_multi_close($mh);
?>
« AskApache Search Engine Verify
Speed Tips: Turn On Compression »
The love of liberty is the love of others; the love of power is the love of ourselves.
-- William Hazlitt
Please consider donating to support active development of the free software and articles here.![]()
The power of the Web is in its universality. Access by everyone regardless of disability is an essential aspect. Tim Berners-Lee
I cant get it working, it just creates all files 0kb. Any ideas?
I’m missing the point behind this. How does this hack help me a website owner? Forgive my ignorance.
Does this automatically select a disk drive to save to? You know, like C: or D: or E: in the computer?
I would add
curl_setopt($conn[$i], CURLOPT_VERBOSE, 1);
My array had more than 20 connections and I didn’t realize that my wholesaler limited me to 20 concurrent sessions. I figured out the problem with on activated the verbose feature.
Thanks for your script. It works great!
Thanks for sharing this example. I made some modifications so that you can process each request as soon as it completes. It makes things a lot faster when you’re dealing with a large number of requests:
can i use this script to client side downloading
Thank you this script is very helpful and work fine with me.
Is there that much use for using curl? The downloading would be TOO FAST, and the remote host will block you. Better to stick to file_get_contents with a delay between files, or go to spider hell.
what type of change we need to done in php.ini files to use this method beacause i used this but this not works.
this code works fine. I had tested with different files of different file sizes :)
It's very simple - you read the protocol and write the code. -Bill Joy
HTML | DCMI | GRDDL | XOXO | XDMP | XFN | DOM | XML | XHTML 1.1 Strict | CSS 2.1 | W3C | TLDP | WAI | DISA | ICSI | GIAC | SANS RR | GHOST | DEFCON | NIST | DHS CYBER | NIST | Phrack | GDB | IEEE
↑ TOPExcept where otherwise noted, content on this site is licensed under a Creative Commons Attribution 3.0 License, just credit with a link.
This site is not supported or endorsed by The Apache Software Foundation (ASF). All software and documentation produced by The ASF is licensed. "Apache" is a trademark of The ASF. HTTPD based on NCSA HTTPd
Hoooo haaa
@001