New Lolipower download script

And this time it’s in perl!
I rewrote the script from scratch to see how much i knew Perl, and this is what I worked out.
It’s base on WWW::Mechanize and HTML::TreeBuilder::XPath respectively for managing the cookies and for searching in the html tree.

Using it is fairly simple.

lolipower Title

and it will download all the episodes of the anime in the local folder.
With respect to the ruby version you can find in these pages the title can be partial, the script will find the complete title. If the title matches more then one anime then a list of anime that maches will be printed and the download aborted.

There are various options, especially 2 that I added to put the script in a cronjob to get mailed when a new episode comes out and automatically download it at night time.

lolipower --dest TitleFolder --new-ep Title

The --dest options tells the script where to save the files instead of the current directory, while --new-ep suppress every output except for new downloaded episodes, and will write in the mail even if the anime is completed.

One another nice feature is that it doesn’t check anymore each file’s md5 hash but only for freshly downloaded files. The behavior can be hoverwer forced with the --md5 flag.

Aside from the options --starep and --endep that do exactly what you can think of there is the --search command. It will search the title in the list of animes without downloading anything.

There is another cute option to allow serial download of more then one anime! The --conf option.
It accepts file with # as comment line and structured like

# comment line
Title=Folder
Title=Folder

where Title is the anime title (or a match for a unique anime) and Folder is a absolute path to the location where to store the episodes.

And from version 1.1 there is support for out of order episodes on the website, –startep and –endep works a lot better and the pages on the server are loaded less times.

The script can be found here

Download anime from lolipower.org

This little ruby script can download a whole anime serie from lolipower.org, when the anime page respects the “standards”. It verifies the MD5 checksum of downloads and it avoids to download already downloaded (checksum checked) or duplicated episodes.
If there are duplicate episodes the mkv format one will be preferred. You can also choose ,with command line arguments, the first and last episode to download.

You can find the link to the project in the code page or just here
git://github.com/alka/down-lolipower.git

4chan

I have written for testing purpose a script to download images that would have been posted on the website 4chan.org. I tried to let it go for 1 day.. I’ve got 2.7GB of data for 16661 images (the number tell me something about of what i have done).

Here’s the code

#! /usr/bin/ruby
# License: GPLv3 or newer

urls=['http://img.4chan.org/b/1.html',
    " ... other 4chan boards you want ..." ]

require 'rubygems'
require 'net/http'
require 'thread'

def download(url)
    uri = URI.parse(url)
    Net::HTTP.start(uri.host) { |h|
        page = h.get(uri.path)
        data = page.body.split(/(href=|\n)/).delete_if {
                               |x| !(x =~ /\"http/)}
        data = data.map {|x| x.gsub(/^\"/,'')}
        data = data.map { |x| x.gsub(/\".*/,'')}
        data = data.delete_if {|x| !(x =~ /(png|jpg|gif)$/)}
        data.uniq.each {|x|
            img = x.gsub(/.*\//,'')
            if !File.exist?(img) then
                puts "downloading #{img}"
                resp = h.get(URI.parse(x).path)
                open(img,"wb") { |f|
                    f.write(resp.body)
                }
            end
        }
        h.close
    }
end

threads = []
s = Mutex.new
urls.each { |x|
    Thread.start{
        s.synchronize{
            threads << Thread.current
        }
        download(x)}
    }

while (threads.length != urls.length) do
    Thread.pass
    sleep(5)
end

threads.each{ |x|
    begin
        x.join
    rescue Exception => e
    end }
puts "finished"

Ruby rtorrent controller

Finnaly i had time to work on the rtorrent controller.
it works, that’s all i can say everyone will have to test it and fix the various values to setup the network for each own best performances.

There are many costants that may be configured to solve problems you may encounter with the configuration. Everything should be clear in the comments but i’ll try to add  more comments anyway when i have time.

With the configuration in the distribution (made for myself for my 2M adsl) i can have my sister doing video chats and playing online without any lag problem and when everything is finihed rtorrent autmatically will raise his own bandwidth limits.

There is currently a drawback. Rtorrent versions prior of 0.8.1 have a memory leak regarding xmlrpc-rtorrent communications. This means that rtorrent is going to eat all your ram. I am using the program monit to monitor my own servers and it’s  just trivial with it to keep rtorrent in a sane memory usare buy restarting it everytime it goes too far. Otherwise even a cronjob will do the trick restarting rtorrent every 1 or 2 days.

You can find the link to the project in the code page or just here
git://github.com/mellon85/ruby-rtorrent.git