Downloading Multiple Files with Curl Simultaneously

FREE THOUGHT · FREE SOFTWARE · FREE WORLD

Downloading Multiple Files with Curl Simultaneously

Wouldn’t it be great if you could use php and curl to download multiple files simultaneously using built-in curl functions? You can!


php code example

This code will download all the files listed in the $urls array to the folder specified by the $saveto variable.

<?php
 
$urls=array(
'http://f.askapache.com/mp3/12-lessons-for-those-afraid-of-css.mp3',
'http://f.askapache.com/mp3/27-request-methods-for-use-with-apache-and-rewritecond-and-htaccess.mp3',
'http://f.askapache.com/mp3/301-redirect-with-mod_rewrite-or-redirectmatch.mp3',
'http://f.askapache.com/mp3/404-errorpages.mp3',
'http://f.askapache.com/mp3/503-service-temporarily-unavailable.mp3',
'http://f.askapache.com/mp3/adsense-robots.mp3',
'http://f.askapache.com/mp3/alexa-toolbar-firefox.mp3',
'http://f.askapache.com/mp3/allowing-access-from-1-static-ip-and-deny-the-rest.mp3',
'http://f.askapache.com/mp3/apache-authentication-in-htaccess.mp3');
 
$save_to='/home/user/htdocs/mp3/';
 
$mh = curl_multi_init();
foreach ($urls as $i => $url) {
    $g=$save_to.basename($url);
    if(!is_file($g)){
        $conn[$i]=curl_init($url);
        $fp[$i]=fopen ($g, "w");
        curl_setopt ($conn[$i], CURLOPT_FILE, $fp[$i]);
        curl_setopt ($conn[$i], CURLOPT_HEADER ,0);
        curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,60);
        curl_multi_add_handle ($mh,$conn[$i]);
    }
}
do {
    $n=curl_multi_exec($mh,$active);
}
while ($active);
foreach ($urls as $i => $url) {
    curl_multi_remove_handle($mh,$conn[$i]);
    curl_close($conn[$i]);
    fclose ($fp[$i]);
}
curl_multi_close($mh);
?>

Curl Multi Functions

curl_multi_init
Returns a new cURL multi handle
curl_multi_add_handle
Add a normal cURL handle to a cURL multi handle
curl_multi_exec
Run the sub-connections of the current cURL handle
curl_multi_close
Close a set of cURL handles
curl_multi_getcontent
Return the content of a cURL handle if CURLOPT_RETURNTRANSFER is set
curl_multi_info_read
Get information about the current transfers
curl_multi_select
Get all the sockets associated with the cURL extension, which can then be “selected”
curl_multi_remove_handle
Remove a multi handle from a set of cURL handles

«
»

Skip to Comments

Add Your Opinion

Reader Comments

  1. tom ~

    I cant get it working, it just creates all files 0kb. Any ideas?

  2. Udegbunam Chukwudi ~

    I’m missing the point behind this. How does this hack help me a website owner? Forgive my ignorance.

  3. Tommy ~

    Does this automatically select a disk drive to save to? You know, like C: or D: or E: in the computer?

  4. bonaventure ~

    I would add

    curl_setopt($conn[$i], CURLOPT_VERBOSE, 1);

    My array had more than 20 connections and I didn’t realize that my wholesaler limited me to 20 concurrent sessions. I figured out the problem with on activated the verbose feature.
    Thanks for your script. It works great!

  5. Josh Fraser ~

    Thanks for sharing this example. I made some modifications so that you can process each request as soon as it completes. It makes things a lot faster when you’re dealing with a large number of requests:

  6. noora ~

    can i use this script to client side downloading

  7. maleos ~

    Thank you this script is very helpful and work fine with me.

  8. Sotland ~

    Is there that much use for using curl? The downloading would be TOO FAST, and the remote host will block you. Better to stick to file_get_contents with a delay between files, or go to spider hell.

  9. lokesh yadav ~

    what type of change we need to done in php.ini files to use this method beacause i used this but this not works.

  10. Raghu Veer ~

    this code works fine. I had tested with different files of different file sizes :)


It's very simple - you read the protocol and write the code. -Bill Joy

HTML | DCMI | GRDDL | XOXO | XDMP | XFN | DOM | XML | XHTML 1.1 Strict | CSS 2.1 | W3C | TLDP | WAI | DISA | ICSI | GIAC | SANS RR | GHOST | DEFCON | NIST | DHS CYBER | NIST | .:: Phrack Magazine ::.

↑ TOPExcept where otherwise noted, content on this site is licensed under a Creative Commons Attribution 3.0 License, just credit with a link.
This site is not supported or endorsed by The Apache Software Foundation (ASF). All software and documentation produced by The ASF is licensed. "Apache" is a trademark of The ASF. HTTPD based on NCSA HTTPd

Site Map | Contact Webmaster | Email AskApache | Glossary | License and Disclaimer | Terms of Service