Rtorrent web interface

May 25, 2009

Me and alka wrote a web application in ruby which serves as an interface
to rtorrent.
It has been written in ruby using the ramaze and sequel frameworks. It
is currently tested on debian stable with mostly apt-get installed gems,
for more info check the README file included (the only one external to
the distribution should be the xmlrpc gem which is on another post in
this blog).
It supports authentication and accesses rtorrent through the unix socket
interface, allowing to run the server as a user process.
An internal copy of the information from rtorrent is stored in a
in-memory sqlite database (maybe it can be totaly avoided, but that was how we planned it in the beggining) to avoid to access too much times to rtorrent
along with the caching of pages in ramaze.

Currently may lack some feature, but it’s doing it’s job, the main
missing feature is the user interface. If someone can write a
CSS and maybe do some cleanup in the interface any help would be
appreciated.

As usual in the code page everything can be found or directly here
git://github.com/mellon85/rtorrent-wrb.git


rTorrent 0.8.4 and ruby-controller 0.2

January 4, 2009

Yesterday i switched to rtorrent 0.8.4 (debian build from experimental).

Rtorrent after some little adjustment in the configuration file has started working. I just had some problems with the torrent which i hadn’t completed.

Except these two little problems it just works, along with the ruby controller too (without any modification).
In the meanwhile I have added some little fixes to the controller and a new feature. 
Now the available upload bandwidth is decreased by a fraction of the actual rtorrent download speed.
This is done to prevent even further to rtorrent to make itself slower flooding the upload bandwidth with ack packets.


Ruby rtorrent controller

September 27, 2008

Finnaly i had time to work on the rtorrent controller.
it works, that’s all i can say everyone will have to test it and fix the various values to setup the network for each own best performances.

There are many costants that may be configured to solve problems you may encounter with the configuration. Everything should be clear in the comments but i’ll try to add  more comments anyway when i have time.

With the configuration in the distribution (made for myself for my 2M adsl) i can have my sister doing video chats and playing online without any lag problem and when everything is finihed rtorrent autmatically will raise his own bandwidth limits.

There is currently a drawback. Rtorrent versions prior of 0.8.1 have a memory leak regarding xmlrpc-rtorrent communications. This means that rtorrent is going to eat all your ram. I am using the program monit to monitor my own servers and it’s  just trivial with it to keep rtorrent in a sane memory usare buy restarting it everytime it goes too far. Otherwise even a cronjob will do the trick restarting rtorrent every 1 or 2 days.

You can find the link to the project in the code page or just here
git://github.com/mellon85/ruby-rtorrent.git


Ruby xml-talking to rtorrent

June 23, 2008

I haven’t had very much time to write some ruby code to communicate with rtorrent through the local domain socket. But i got something that starts to work.

The main problem is that Ruby XMLRPC library is unfriendly with user with special needs. There is no way to get the xml data, send it the way i want and get it back and let the library parse it!
To solve this i had to modify the library, with a patch that when is ready i’ll subit to the ruby developers.
I have added a class that represents a socket factory! the easy way to solve the problem. In less then 30 lines it just works. There is a new method to initialize the Client class too, and even the hashed (new3) method will work if the field socket is specified.

  class SocketFactory
      def issue() # returns a new socket to be used with the library
          raise "This class must be subclassed"
      end
  end
  def new4( socket )
      self.new(nil,nil,nil,nil,nil,nil,nil,nil,nil,socket)
  end
  alias new_with_socket new4

After these modifications to the initialization code i had to modify the private method do_rpc adding this at the beginning of the method

      data = nil
      if @socket != nil then # using user socket
          # must do the same things, send the same things
          # but to the socket! not to a http/https server
          sock = @socket.issue()
          if sock.write(request) != request.length then
              raise "Not all the data has been sent"
          end
          # read the whole data until the socket is closed
          data = sock.read()
          # after any answer the socket is closed
          return data # return data to the parser
      else

and an end just after the return statement of the original method. There is an additional check to verify that the socket object is a subclass of the SocketFactory class.

I wrote a simple socket factory to test the communications with rtorrent itself in the real world. I had to write to it using the SCGI protocol, which obviously is supported in ruby, but it doesn’t support local domain socket, for no real reasons, since it is indistinguishable from any other kind of socket, but that was the statement in rdoc

One thing that SCGI doesn‘t support is using UNIX Domain sockets in addition to TCP/IP sockets. This isn‘t really needed, but it is handy in a shared hosting situation where you don‘t want others connecting to your processes or if you have to request open ports. Sorry, no UNIX Domain sockets in SCGI.

It’s crazy… so i wrote a simple scgi wrapper. which is incredibly simple

    def self.wrap( content, uri, method="POST" )
        null=""
        header = "CONTENT_LENGTH #{content.length} SCGI#{null}1
                  REQUEST_METHOD #{method} REQUEST_URI #{uri} "
        return "#{header.length}:#{header},#{content}"
    end

There are some missing chars (many “\0″), in the string header and the null string are just “\0″, but can’t be shown here as in any my previous post, the files in the git repository are ok.

To use this wrapper i have to write a socket factory that will create sockets that will wrap the xmldata in a scgi header.
So i wrote a class SCGIFactory, which is still incomplete (no error checking) but is working.

    def issue()
        if @method != nil
            SCGIWrappedSocket.new(UNIXSocket.new(@path),@uri,@method)
        else
            SCGIWrappedSocket.new(UNIXSocket.new(@path),@uri)
        end
    end
    def write(x)
        @sock.write(SCGI.wrap(x,@uri,@method))
    end 

    def read()
        data = @sock.read()
        # receiving an html response (very dumb parsing)
        # divide in 2
        # 1 -> status + headers
        # 2 -> data
        return data.split("\r\n\r\n").last
    end

As you can see the read method need many enhancements. I need a way to parse http data from a string in any default library, but no default library do this. If anybody knows hot to do that, please tell me.

I wrote this test

require 'xmlrpc/client'
require 'SCGIFactory'

f=SCGIFactory::Unix.new("/rtorrent.sock","/RPC2")
c=XMLRPC::Client.new_with_socket(f)
puts "#{c.call("system.listMethods")}"

And I can get the whole list of the supported methods directly from the program itself

All the code is available in a git repository, different from the ruby upnp one.. Git repository can be found in the code page

[Continued by: XMLRPC sockets changes]


UPnP – Ruby Integration

June 8, 2008

[EDIT:This has been released as gem. check the code page]

At the moment i’m trying to create a ruby module to integrate some upnp functionality in the language.

I would like to do so to enhance the experience with rtorrent, a really nerdish torrent client. Ok, it uses curses, as interface, but it is just great, never crashed in 6 months and had a tiny memory footprint.
The idea is to connect to the local router and gather information about the port mappings and the data currently flowing in the local network and trough estimation change the upload rate.
Useful to use the bandwidth when it is unused, as long as i pay it i want to use it!  Here is how I have built such a module on Mac OS X 10.5.3 with the apple’s version of ruby (1.8). To avoid to write the code i’ll use the miniupnp (1.0) library and the utility swig (1.3.35)
The first is a upnp library which interacts with the IGDs (Internet Gateway Device) and the latter is an wrapper builder that will write all the glue code (C \iff Ruby) in my place.

Get miniupnp to compile

The first thing is to get the library to compile on my mac. The distribution makefile has some minor flaws, I have modified it to build the dynamic library with this patch makefile-osx.patch
--- a/Makefile
+++ b/Makefile
 HEADERS = miniupnpc.h miniwget.h upnpcommands.h igd_desc_parse.h \
 upnpreplyparse.h upnperrors.h
 LIBRARY = libminiupnpc.a
-SHAREDLIBRARY = libminiupnpc.so
+SHAREDLIBRARY = libminiupnpc.dylib
 SONAME = $(SHAREDLIBRARY).$(APIVERSION)
 EXECUTABLES = upnpc-static upnpc-shared \
 testminixml minixmlvalid testupnpreplyparse
@@ -83,7 +83,7 @@ $(LIBRARY):	$(LIBOBJS)
 $(AR) crs $@ $?

 $(SHAREDLIBRARY):	$(LIBOBJS)
-	$(CC) -shared -Wl,-soname,$(SONAME) -o $@ $^
+	$(CC) -dynamiclib -o $@ $^

 upnpc-static:	upnpc.o $(LIBRARY)
 $(CC) -o $@ $^
As the build completes everything is almost done. Another patch has to be applied to avoid a swig warning for a memory leak (miniupnp.g-memleak.patch).
--- a/miniupnpc.h
+++ b/miniupnpc.h
@@ -16,7 +16,7 @@ extern "C" {
 #endif

 /* Structures definitions : */
-struct UPNParg { const char * elt; const char * val; };
+struct UPNParg { char * elt; char * val; };

 int simpleUPnPcommand(int, const char *, const char *,
 const char *, struct UPNParg *,

Get Glue Code

Then we have to write a swig interface file, describing what should be put in the ruby module writing this in the file upnp.i
%module miniupnp
%{
#include "miniupnpc.h"
%}

%include "cpointer.i"
%pointer_functions(unsigned int, uintp);

%import declspec.h
%include miniupnpc.h
%include upnpcommands.h
%include igd_desc_parse.h

Make the Module

This will create a module that will contain all the functions from miniupnpc.h, upnpcommands.h and igd_desc_parse.h and a struct i have added to ease the use of the library. Swig is then called

swig -ruby upnp.i
gcc -fPIC upnp_wrap.c -c -o upnp_wrap.o -I/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/universal-darwin9.0/

Generating the file upnp_wrap.c glue code which must be compiled with the ruby includes. From this point there is a decision to take, if the miniupnp library is to be linked to the module as an external shared library or be included in it as static code.
To link it dynamically

gcc -fPIC -bundle -undefined dynamic_lookup upnp_wrap.o -o miniupnp.bundle -lruby -lminiupnpc -L<miniupnpc install path>

And to link it statically

gcc -fPIC -bundle -undefined dynamic_lookup upnp_wrap.o -o miniupnp.bundle -lruby miniwget.o minixml.o igd_desc_parse.o minisoap.o miniupnpc.o upnpreplyparse.o upnpcommands.o minissdpc.o upnperrors.o

Test it

Now is time to use the newly created module. This can be done in 2 ways too. Installing it in the ruby library path or configuring the library path.


# system install
cp miniupnp.bundle /Library/Ruby/Site/1.8/
# setup path, bundle_path is where the
# bundle is (for instance $PWD)
export RUBYLIB=bundle_path

Here there is a little script to test that everything is working
# The feature name begins with a lowercase letter...
require 'miniupnp'

# max time to wait for the upnp devices
MAX_WAIT_TIME = 1000

# discover upnp
list = Miniupnp.upnpDiscover(MAX_WAIT_TIME,nil,nil)
if list == nil then
    puts "No UPNP device found"
    exit 1
end
puts "URL : #{list.descURL}"

# get valid internet gateway device
urls  = Miniupnp::UPNPUrls.new
datas = Miniupnp::IGDdatas.new
lan = ""*16
r = Miniupnp.UPNP_GetValidIGD(list,urls,datas,lan,16)
case r
when 0
    puts "No IGD found"
    exit 2
when 1
    puts "A valid connected IGD has been found"
when 2
    puts "A valid IGD has been found but it is reported as not connected"
when 3
    puts "An UPnP device has been found but was not recognized as an IGD"
    exit 3
end

puts "Client LAN IP: #{lan}"
ext=""*16
r = Miniupnp.UPNP_GetExternalIPAddress(urls.controlURL,datas.servicetype,ext)
if r == 0 then
    puts "External IP: #{ext}"
else
    puts "Error while retriving the external ip address"
end

# print urls
puts "*** urls"
puts "controlURL: #{urls.controlURL}"
puts "ipcondescURL: #{urls.ipcondescURL}"
puts "controlURL_CIF: #{urls.controlURL_CIF}"

# print datas
puts "*** datas"
puts "cureltname: #{datas.cureltname}"
puts "urlbase: #{datas.urlbase}"
puts "level: #{datas.level}"
puts "state: #{datas.state}"
puts "controlurl_CIF: #{datas.controlurl_CIF}"
puts "eventsuburl_CIF: #{datas.eventsuburl_CIF}"
puts "scpdurl_CIF: #{datas.scpdurl_CIF}"
puts "servicetype_CIF: #{datas.servicetype_CIF}"
puts "devicetype_CIF: #{datas.devicetype_CIF}"
puts "controlurl: #{datas.controlurl}"
puts "eventsuburl: #{datas.eventsuburl}"
puts "scpdurl: #{datas.scpdurl}"
puts "servicetype: #{datas.servicetype}"
puts "devicetype: #{datas.devicetype}"

puts "Get maximum bandwith"
up=Miniupnp.new_uintp()
down=Miniupnp.new_uintp()
Miniupnp.UPNP_GetLinkLayerMaxBitRates(urls.controlURL_CIF,datas.servicetype_CIF,down,up)
puts "up: #{Miniupnp.uintp_value(up)} down: #{Miniupnp.uintp_value(down)}"
Miniupnp.delete_uintp(up)
Miniupnp.delete_uintp(down)

# release memory
Miniupnp.freeUPNPDevlist(list)
Miniupnp.FreeUPNPUrls(urls)
I can’t get wordpress to display correctly the content of lan and ext variables, it should be “” repeated 16 times. The result should be something like this
URL : http://192.168.0.1:49152/gateway.xml
A valid connected IGD has been found
Client LAN IP: 192.168.0.100
External IP: ***.***.***.243
*** urls
controlURL: http://192.168.0.1:49152/upnp/control/WANIPConnection
ipcondescURL: http://192.168.0.1:49152/ipcfg.xml
controlURL_CIF: http://192.168.0.1:49152/upnp/control/
WANCommonInterfaceConfig
*** datas
cureltname: presentationURL
urlbase: http://192.168.0.1:49152
level: 0
state: 3
controlurl_CIF: /upnp/control/WANCommonInterfaceConfig
eventsuburl_CIF: /upnp/event/WANCommonInterfaceConfig
scpdurl_CIF: /cmnicfg.xml
servicetype_CIF: urn:schemas-upnp-org:service:WANCommonInterfaceConfig:1
devicetype_CIF: urn:schemas-upnp-org:device:WANDevice:1wayDevice:1
controlurl: /upnp/control/WANIPConnection
eventsuburl: /upnp/event/WANIPConnection
scpdurl: /ipcfg.xml
servicetype: urn:schemas-upnp-org:service:WANIPConnection:1
devicetype: urn:schemas-upnp-org:device:WANConnectionDevice:1

Maximum bandwith
up: 352000
down: 2464000

This is just a start, I’ll make more testing and i’ll make sure it works. There may be some more things to add to upnp.i

Full code can be found in the code page
Edit 15-06-08: Module name changed to miniupnp from upnp

[Continued by: UPnP Wrapper]
[EDIT:This has been released as gem. check the code page]