Archive for June, 2008

XMLRPCS gem published

I have just published on rubyforge the gem for my xmlrpc modifications. It’s called xmlrpcs and contains the class discussed in the previous post.

xmlrpcs Rubyforge homepage
Online RDoc

[Edit] consider the gem licensed under BSD-3 Clause license

The changes are that now it has full rdoc comments and the path to include it is ‘xmlrpc/xmlrpcs’ I would like to relase my upnp binding as a gem too, but that will come later. To install it just do gem install xmlrpcs and it’s done.

 

, , , , ,

Leave a comment

Ruby XMLRPC sockets changes

[Continues: Ruby xml-talking to rtorrent]

I have rewritten how i changed the socket in XMLRPC are modified in a more Ruby style way.

A new class, in XMLRPC module, called ClientS (in a file called xmlrpcs.rb). It is a subclass of the Client class to ease the implementation of other kind of trasport system if the Net::HTTP based one doesn’t fit your needs.

require 'xmlrpc/client'

class XMLRPC::ClientS < XMLRPC::Client
        def initialize(info)
            @info = info
        end
        
        private
       
        # create new socket
        def new_socket(info,async)
            raise "Must be subclassed"
        end

        # write xmlrpc request in the previously created socket
        def write_request(socket,request)
          if socket.write(request) != request.length then
              raise "Not all the data has been sent"
          end
        end

        # read response from the socket
        def read_response(socket)
            socket.read()
        end

        # do_rpc working with custom sockets
        def do_rpc( request, async )
          sock = new_socket(@info,async)
          write_request(sock,request)
          return read_response(sock)
        end
    end
end

This makes it very simple to extend the class by just subclassing ClientS or just redefining the function new_socket.
To avoid to close the socket at each request, which may be not the right choice for every code, you can redefine the do_rpc
The info field is used to pass some arguments to create the socket, for instance

require 'xmlrpcs'
require 'socket'

class XMLClient < XMLRPC::ClientS
    def new_socket( info, async )
        UNIXSocket.new(info)
    end
end

c=XMLClient.new("/path/to/sock")
c.call(..)

The code can be found in the code page

[Continued by: XMLRPCs gem published]

, , , , ,

2 Comments

Ruby xml-talking to rtorrent

I haven’t had very much time to write some ruby code to communicate with rtorrent through the local domain socket. But i got something that starts to work.

The main problem is that Ruby XMLRPC library is unfriendly with user with special needs. There is no way to get the xml data, send it the way i want and get it back and let the library parse it!
To solve this i had to modify the library, with a patch that when is ready i’ll subit to the ruby developers.
I have added a class that represents a socket factory! the easy way to solve the problem. In less then 30 lines it just works. There is a new method to initialize the Client class too, and even the hashed (new3) method will work if the field socket is specified.

  class SocketFactory
      def issue() # returns a new socket to be used with the library
          raise "This class must be subclassed"
      end
  end
  def new4( socket )
      self.new(nil,nil,nil,nil,nil,nil,nil,nil,nil,socket)
  end
  alias new_with_socket new4

After these modifications to the initialization code i had to modify the private method do_rpc adding this at the beginning of the method

      data = nil
      if @socket != nil then # using user socket
          # must do the same things, send the same things
          # but to the socket! not to a http/https server
          sock = @socket.issue()
          if sock.write(request) != request.length then
              raise "Not all the data has been sent"
          end
          # read the whole data until the socket is closed
          data = sock.read()
          # after any answer the socket is closed
          return data # return data to the parser
      else

and an end just after the return statement of the original method. There is an additional check to verify that the socket object is a subclass of the SocketFactory class.

I wrote a simple socket factory to test the communications with rtorrent itself in the real world. I had to write to it using the SCGI protocol, which obviously is supported in ruby, but it doesn’t support local domain socket, for no real reasons, since it is indistinguishable from any other kind of socket, but that was the statement in rdoc

One thing that SCGI doesn‘t support is using UNIX Domain sockets in addition to TCP/IP sockets. This isn‘t really needed, but it is handy in a shared hosting situation where you don‘t want others connecting to your processes or if you have to request open ports. Sorry, no UNIX Domain sockets in SCGI.

It’s crazy… so i wrote a simple scgi wrapper. which is incredibly simple

    def self.wrap( content, uri, method="POST" )
        null=""
        header = "CONTENT_LENGTH #{content.length} SCGI#{null}1
                  REQUEST_METHOD #{method} REQUEST_URI #{uri} "
        return "#{header.length}:#{header},#{content}"
    end

There are some missing chars (many “\0”), in the string header and the null string are just “\0”, but can’t be shown here as in any my previous post, the files in the git repository are ok.

To use this wrapper i have to write a socket factory that will create sockets that will wrap the xmldata in a scgi header.
So i wrote a class SCGIFactory, which is still incomplete (no error checking) but is working.

    def issue()
        if @method != nil
            SCGIWrappedSocket.new(UNIXSocket.new(@path),@uri,@method)
        else
            SCGIWrappedSocket.new(UNIXSocket.new(@path),@uri)
        end
    end
    def write(x)
        @sock.write(SCGI.wrap(x,@uri,@method))
    end 

    def read()
        data = @sock.read()
        # receiving an html response (very dumb parsing)
        # divide in 2
        # 1 -> status + headers
        # 2 -> data
        return data.split("\r\n\r\n").last
    end

As you can see the read method need many enhancements. I need a way to parse http data from a string in any default library, but no default library do this. If anybody knows hot to do that, please tell me.

I wrote this test

require 'xmlrpc/client'
require 'SCGIFactory'

f=SCGIFactory::Unix.new("/rtorrent.sock","/RPC2")
c=XMLRPC::Client.new_with_socket(f)
puts "#{c.call("system.listMethods")}"

And I can get the whole list of the supported methods directly from the program itself

All the code is available in a git repository, different from the ruby upnp one.. Git repository can be found in the code page

[Continued by: XMLRPC sockets changes]

, , , , , ,

2 Comments

UPnP – Ruby Wrapper

[EDIT:This has been released as gem. check the code page]

[Continues: UPnP Integration]

After some testing i wrote an UPnP class to wrap the UPNP library for a more friendly and usable interface.
The wrapper class can be found on the same code repository.

I have added a nice feature too, adding the discovery of UPnP devices as a thread on object creation, to let it probe the network without bothering the class user. But this lead me into troubles… the thread that was going to probe the network made ruby wait until it is completed, even if it has to wait on a socket… digging for the threading implementation of ruby i finally got where the problem was.
Ruby uses green thread, cooperative user threads… that was the problem.. every thread has to tell which other thread is the next. But what happens if it remains blocked on a system call? it simply makes the whole ruby machine to wait … fortunately ruby 2.0 should solve this problem, requiring that the threads must be operative system threads.

 

The Initializer allows you to choose how much time should the implementation wait for an UPnP answer from the network, and if the network probing thread must be launched. Remember that the object must be initialized with a discoverIGD call, except if you let the thread start.
Initialization functions

  • initialize(max_wait=1000,autodiscover=true)
  • discoverIGD()

IP functions

  • getLanIP()
  • getRouterIP()
  • getExternalIP()

Router info

  • getStatusInfo()
  • getConnectionTypeInfo()

Traffic info

  • getTotalBytesSent()
  • getTotalBytesReceived()
  • getTotalPacketsSent()
  • getTotalPacketsReceived()
  • getMaxLinkBitrates()

Mappings

  • getPortMappings()
  • getPortMapping(nport,proto)
  • addPortMapping(nport,lport,client,desc,proto)
  • deletePortMapping(nport,proto)

The port numbers may be passed as integer or string, and they are always returned as integer (lib miniupnp works only with strings). All the errors are managed through exceptions.
In fact there is a nice thing about ruby threading, the exceptions that occour in a thread are received by the thread that calls join on the thread. Due to this, the first method you’ll call (except for your ip and router ip functions), if the autosearch feature is enabled, will return the exception, if any, from the detect thread.
Uh, i was just about to forget this. nport is the net port, and lport is the lan port.
This is a simple example that will print each mapping in the router. The call to getPortMappings will wait until the discover thread has finished searching a router.

require 'upnp'
u = UPNP.new()
u.getPortMappings.each {|m| puts m.to_s}

 

 [EDIT:This has been released as gem. check the code page]

, ,

1 Comment

Convert Man pages to PDF

Always wanted to watch a man page in a pdf file? It is clearly needed sometimes, especially on mac, since it is the easiest way i know to print a man page or to read it not from a terminal.

The problem is solved easily for both Mac OS and linux. I had to add control for <code>pstopdf</code> too, since on mac it is called that way and has different parameters (but you may have <code>ps2pdf</code> installed with macports anyway).

#! /bin/bash

if [ $# -eq 1 ] ; then
    to_pdf=$(which ps2pdf)
    if [ -z "$to_pdf" ] ; then
        to_pdf=$(which pstopdf)
    fi

    name="$1"
    case "$to_pdf"  in
        *pstopdf) man -t "$name" | "$to_pdf" -i -o "$fname.pdf" ;;
        *ps2pdf)  man -t "$name" | "$to_pdf" - "$name.pdf" ;;
        *)        man -t "$name" > "$fname.ps"
    esac
    exit $?
fi
echo "Wrong number of parameters"
exit 1

Just call it like man2pdf gcc and you’ll get gcc.pdf

, , , , ,

1 Comment

DeMonad

At my second year of university i made a project for the “Programming Languages” course. It consisted in a simple program written in Haskell to eliminate a Monad from the code.
It exactly rewrites another piece of haskell code containing the definition of one single monad, and rewrites it without it.
The monad must be user defined, it is no possible to remove the IO monad for instance.
A current limitation is the monad must be defined in the file to rewrite, not that is a big limitation thought.

But an example is better then words…
This code uses a list as state, and gives the user 2 functions, insertList to insert a list, and getElement to get an element from the list.

data Test a = Test (TestS -> (a,TestS))

instance Monad Test where
    return a      = Test (\s -> (a,s))
    Test f >>= gm = Test (\s0 -> let (x,s1) = f s0
                                     (Test g) = gm x
                                   in g s1)

insertList l = Test (\s0 -> ((),l))
getElement n = Test (\(l) -> (l!!n,l))

runTest :: TestS -> Test a -> a
runTest s0 (Test c) = fst $ c s0

testcode = do h <- getElement 5
              p <- getElement 4
              t <- getElement 2
              return p

test = runTest [0..] testcode

Then, after passing it to the DeMonadizer

ghc --make Main.hs DeMonad.hs
./Make test.hs

the code looks like this (it’s ugly, i know)

module Test where

type TestS = [Int]

data Test a = Test (TestS -> (a, TestS))
insertList l = Test (\ s0 -> ((), l))
getElement n = Test (\ (l) -> (l !! n, l))

runTest :: TestS -> Test a -> a
runTest s0 (Test c) = fst $ c s0
testcode
  = (\ (Test f) ->
       Test
         (\ s0 ->
            let (x, s1) = f s0
                (Test g)
                  = (\ h ->
                       (\ (Test f) ->
                          Test
                            (\ s0 ->
                               let (x, s1) = f s0
                                   (Test g)
                                     = (\ p ->
                                          (\ (Test f) ->
                                             Test
                                               (\ s0 ->
                                                  let (x, s1) = f s0
                                                      (Test g) = (\ t -> Test (\ s -> (p, s))) x
                                                    in g s1))
                                            (getElement 2))
                                         x
                                 in g s1))
                         (getElement 4))
                      x
              in g s1))
      (getElement 5)
test = runTest [0 ..] testcode

As you can see the monad definition is gone, and with it also the do syntax.

This solution allowed some haskell code to run as curry code!
It is very similar in syntax, but it is a funcional-logic programming language, and as such it has non-determinism (and everybody should love the magic of =:=). The code is rather long, since it must provide the language with actions for each of his syntactical elements, it can be found at his code page.

, , , , , ,

Leave a comment

Parallelized Make

Since the introduction of dual core machines the number of mono core machines is declined and almost no computer of that kind are sold at all. Having such a machine is clearly a big advantage, but for many things you must tell programs that you are on a multi-cpu machines to make them use all the power. So i wrote a very little and simple script to replace the default make.

Just place this script in a path before the standard make. I made a folder ~/.bin in my home directory and modified the PATH accordingly export PATH=~/.bin:$PATH


#!/bin/sh

# verbosity check
if [ "$VERBOSE" = "yes" ] ; then
	verbose=yes
fi

# Get the number of cores
if [ -z $NCPU ] ; then
    tmp=$(sysctl hw.ncpu)
    NCPU=${tmp#hw.ncpu: }
	if [ "$VERBOSE" = "yes" ] ; then
		echo "Parallelization enabled with $NCPU threads"
	fi
fi

j=$(($NCPU*5/2))
if [ ! -z $NOPMAKE ] ; then
    echo "Parallelization Disabled" >&2
    /usr/bin/make $@
    exit $?
fi

# Start the compilation in parallel
if [ ! -z "$DISTCC_HOSTS" ] ; then
    if [ -z "$HOST_COUNT" ] ; then
        export HOST_COUNT=$(echo $DISTCC_HOSTS | wc -w)
    fi
    /usr/bin/make CC='distcc /usr/bin/gcc' CXX='distcc /usr/bin/g++' -j$((j+$HOST_COUNT)) $@
else
    /usr/bin/make -j$j $@
fi

To make it work on linux the sysctl command must be replaced with a count of the lines containing “processor” in the file /proc/cpuinfo. Something like grep processor /proc/cpuinfo | wc -l

, , ,

Leave a comment