Sunday, 5 July 2015

Building FFMPEG and VLC from sources on Debian Jessie

Debian does not come with ffmpeg. Instead, it comes with libav, which is a fork from ffmpeg. But there are situations when you find yourself in need of ffmpeg. In this case, you have to install it yourself. And it's far easier than you may think.

Start by upgrading your packages and installing build-essential:
sudo apt-get update
sudo apt-get install build-essential

Download ffmpeg and vlc:
mkdir -p $HOME/sources/software
cd $HOME/sources/software
wget http://ffmpeg.org/releases/ffmpeg-2.7.1.tar.gz
wget http://get.videolan.org/vlc/2.2.0/vlc-2.2.0.tar.xz

Install packages ffmpeg depends on during compilation:
sudo apt-get install libmp3lame-dev libvorbis-dev libtheora-dev \
    libspeex-dev yasm pkg-config libfaac-dev libopenjpeg-dev \
    libx264-dev libass-dev

Compile and install ffmpeg:
tar xpf ffmpeg-2.7.1.tar.gz
cd ffmpeg-2.7.1/
./configure --enable-gpl --enable-postproc --enable-swscale \
    --enable-avfilter --enable-libmp3lame --enable-libvorbis \
    --enable-libtheora --enable-libx264 --enable-libspeex \
    --enable-shared --enable-pthreads --enable-libopenjpeg \
    --enable-libfaac --enable-nonfree --enable-libass
make
sudo make install
sudo /sbin/ldconfig

Install packages vlc depends on during compilation:
sudo apt-get build-dep vlc

Compile and install vlc:

./configure --prefix=/usr/local --with-ffmpeg-tree=/usr/local \
  --enable-x11 --enable-xvideo --disable-gtk \
  --enable-sdl --enable-ffmpeg --with-ffmpeg-mp3lame \
  --enable-mad --enable-libdvbpsi --enable-a52 --enable-dts \
  --enable-libmpeg2 --enable-dvdnav --enable-faad \
  --enable-vorbis --enable-ogg --enable-theora --enable-faac\
  --enable-mkv --enable-freetype --enable-fribidi \
  --enable-speex --enable-flac --enable-livedotcom \
  --with-livedotcom-tree=/usr/lib/live --enable-caca \
  --enable-skins --enable-skins2 --enable-alsa --disable-kde\
  --disable-qt --enable-wxwindows --enable-ncurses \
  --enable-release
make
sudo make install

Profit! I told you that it would be easy!

Remember that now you have 2 versions of vlc installed in your system. Despite they apparently are the same version, the version which comes with Debian was still a RC (release candidate) at the time it was built. The version we installed is an official release of vlc, not a release candidate.

More than that, the version of vlc we installed employs ffmpeg under the hood from /usr/local, whilst the version which comes with Debian employs libav.



Friday, 4 April 2014

IT Consulting Business


A chronicle about IT consulting business, based on real life, sort of.


Let me present you a story. I hope you enjoy.

First of all, lets create a mental picture of a fridge which rockets itself to the Moon.

Our sales people proposed that apparatus to a client... and the client bought!
We don't have such apparatus yet ... and nobody knows how to do that ...  but we need to deliver that in 6 months, anyway.

OK. This is what we sold to a client. So, lets make that happen!

OK. Lets hire a bunch of very well paid crazy techie guys who must accomplish the difficult task of creating such apparatus in a matter of months. Well, despite very capable and working 12-15 hours a day, these crazy techie guys failed to create such apparatus. Not really a surprise, to be honest. Anyway, despite of the failure...  they were able to create something which looks like a rocket, something which is more or less rectangular and ... better than anyone expected: it is painted in white!

Eventually such apparatus is barely capable of reaching the Moon... which is amazing! ... somewhat unexpected!

Our sales guys are satisfied enough and more surprising than that: the client is satisfied too! In particular, the client loved the colour it is painted.

OK. Supplier (us) and client agrees that such apparatus must regularly make travels to the Moon.


Initially things did not work very well, so we've
decided to keep 3 of those very well paid crazy techie guys. They fixed lots of things, fortunately!  After 3 or 4 more months working 15 hours a day, the apparatus works much, much better. Only 5 passengers died and we covered all funeral costs, obviously. I think we are just lucky!... hundreds could have died! OK. I guess we can now kick these techie guys outside of the door. Wow! they are so expensive!...

And that's it. Once the product is delivered the supplier (us) is only interested on getting the annual support fees paid, which must obviously cover all maintenance costs and obviously must produce maximum profits. Profit is all that matters, you know.

The supplier (us) is not interested anymore on making such apparatus to be a real fridge which rockets itself to the Moon because... well... the client accepted the apparatus the way it is, whatever it is and whatever you call it. All that matters is that they need to keep one low paid techie guy to renew the white paint every time it faints.

In order to increase profits a bit more, someone had the bright idea of replacing the last remaining techie guy by a remote worker, who works from home and costs less.


This remote worker apparently feels plenty happy of being able of working from home, very close to his own fridge, which never rocketed itself to the Moon.

And this is the end of the story.

Friday, 21 March 2014

Configuring Postfix for relaying on Debian using GMAIL

This article explains how Postfix can be installed and configured to route emails to an external SMTP server, in particular, using GMAIL
 

Configuration


Run the script below as root.

#!/bin/bash

apt-get install postfix sasl2-bin bsd-mailx -y

##################################
# Choose:
#   * 'Satelite system'
#   * leave relayhost blank
##################################

cd /etc/postfix

cp -p main.cf main.cf.ORIGINAL

cat << EOD >> main.cf
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_mechanism_filter = plain, login
smtp_sasl_security_options = noanonymous
EOD

# Substitute server, username and password below by your own settings
SERVER=smtp.gmail.com
USERNAME=your.username@gmail.com
PASSWORD=your.password

cat << EOD > sasl_passwd
${SERVER} ${USERNAME}:${PASSWORD}
EOD

chmod 400 sasl_passwd

postmap /etc/postfix/sasl_passwd

/etc/init.d/postfix restart

Testing your configuration


As a regular user, try something like this:

$ echo 'It works!' | mailx -s test me@mydomain.com


Credits: Setup postfix to relay outbound mail using sasl

If you found this article useful, it will be much appreciated if you create a link to this article somewhere in your website.

Friday, 28 February 2014

Recovering from HTTP errors using URL Handlers

This article shows how URL handers, defined by urllib2, can be employed in practice in order to circumvent troubles we usually find when we write robots for collecting information from the Internet.

First things first (and usually a source of confusion): There are two sister libraries in Python which address retrieval of information from URLs; they are: urllib and urllib2. Conceptually, urllib2 works as a derived class of urllib. Just conceptually, because the actual implementation does not employ classes as a conventional object oriented paradigm would dictate.

If you are seeking detailed documentation about these libraries, I'm afraid to inform that your only choice is spending a couple of hours studying the source code of urllib.py and urllib2.py.

Setting an User Agent


OK. Now that you have the full documentation at hand, we can start. The first thing our robot needs to do is hiding its presence from the server side. One simple measure is employing an innocent user agent. We need to define a class derived from urllib2.BaseHandler which is responsible for setting the user agent before a request is sent to the server side. This is shown below:

import urllib2

class UserAgentProcessor(urllib2.BaseHandler):
    """A handler to add a custom UA string to urllib2 requests
    """
    def __init__(self, uastring):
        self.handler_order = 100
        self.ua = uastring

    def http_request(self, request):
        request.add_header("User-Agent", self.ua)
        return request

    https_request = http_request


(credits: This code was shamelessly copied from this article by Andrew Rowls)

Handling HTTP ERROR 404 (Not Found)


There are other things we need to do, such as throttling our requests, otherwise the server side will easily guess that there's a robot on our side sending dozens of requests per second. But throttling is a subject that I'm not going to cover here. You can later create your throttling handler, after you get better acquainted with some techniques covered in this article.

Some webservers are really busy, which may cause failures to our requests. Other webservers deliberatly reject requests given certain circumstances, for example: the server side may detect that we are sending dozens of requests per second and may decide to punish us for 10 minutes. Again we are back to the subject of throttling, which we are not going to cover here. But let's address this sort of issue partially, which may be of practical use in a majority of situations.

Let's say the webserver responds HTTP ERROR 404 (Not Found) eventually (or even regularly), even when the resource is existing in reality. We just need to be a little skeptic and send another request after waiting a couple of seconds. Eventually we need to be far more skeptic (or a little stubborn, if you will) and send several additional requests, before we become sure enough that the resource is actually and truly non-existent.

What we need to do is basically stamp requests so that we will have means to determine whether a request needs to be sent again to the server side, eventually waiting some time before that. Also, requests to different webservers may require different parameters for number of retries and for the delay to be employed. See below how we implemented this things:


import urllib2

class HTTPNotFoundHandler(urllib2.BaseHandler):
    """A handler which retries access to resources when 404 (NotFound) is received
    """

    handler_order = 600 # before HTTPDigestAuthHandler and ProxyDigestAuthHandler

    def __init__(self, retries=5, delay=2):
        self.retries = int(retries)
        self.delay   = float(delay)
        assert(self.retries >= 1)
        assert(self.delay >= 0.0)

    def http_request(self, req):
        if hasattr(req, 'headers') and 'Error_404' in req.headers:
            Error_404 = req.headers['Error_404']
            assert(int(Error_404['retries']) >= 1)
            assert(float(Error_404['delay']) >= 0.0)
        return req

    def http_error_404(self, req, fp, code, msg, headers):
        if hasattr(req, 'headers') and 'Error_404' in req.headers:
            Error_404 = req.headers['Error_404']
        else:
            Error_404 = dict()
            Error_404['delay']   = self.delay
            Error_404['retries'] = self.retries

        count   = Error_404['count'] if 'count' in Error_404 else 1
        retries = Error_404['retries']
        delay   = Error_404['delay']
        if count == retries:
            raise urllib2.HTTPError(req.get_full_url(),
                                    code,
                                    msg,
                                    headers,
                                    fp)
        else:
            # Don't close the fp until we are sure that

            # we won't use it with HTTPError.
            fp.read()
            fp.close()
            # sleep a little while
            from time import sleep
            sleep(delay)
            # send another request
            Error_404['count'] = count + 1
            req.add_header('Error_404', Error_404)
            return self.parent.open(req)

    https_error_404 = http_error_404


Now, let's add two utility functions:

def install_opener(opener=None):
    import urllib2
    if opener is None:
        urllib2.install_opener(build_opener())
    else:
        urllib2.install_opener(opener)
    return urllib2

def build_opener(

                  user_agent='Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0',
                  http_404_retries=3,
                  http_404_delay=2.0):
    return urllib2.build_opener(
        UserAgentProcessor(user_agent),
        HTTPNotFoundHandler(http_404_retries, http_404_delay) )


Just put all the code you see in this up to this point into a file, say: api.py.

Test cases


Now, let's  create some test cases for it, using pytest. First thing consists on creating the conftest.py file, like shown below:

from __future__ import print_function

from pytest import fixture

@fixture
def opener():
    from mypackage.api import api
    return api.build_opener()

@fixture
def urllib2(opener):
    from mypackage.api import api
    return api.install_opener(opener)


If you are not acquainted to pytest, a very brief explanation of the code above is that we are defining functions opener and urllib2 which we will later employ as parameters to other functions. In a nutshell, pytest replaces the parameter by a call to the special functions (marked by @fixture) we have defined.

Now, let's create a file for test cases called test_urllib.py, like shown below:

import pytest

class TestOpeners(object):

    def xtest_build_opener(self, opener):
        pass

    def xtest_existing(self, urllib2):
        url = 'http://google.com'
        f = urllib2.urlopen(url)
        assert(f.code == 200)

    def xtest_existing_but_faulty(self, urllib2):
        url = 'http://biz.yahoo.com/p/'
        f = urllib2.urlopen(url)
        assert(f.code == 200)

    def xtest_non_existing(self, urllib2):
        from urllib2 import HTTPError
        url = 'http://google.com/this_url_does_not_exist'
        with pytest.raises(HTTPError):
            f = urllib2.urlopen(url)

    def test_non_existing_with_header(self, urllib2):
        from urllib2 import HTTPError
        url = 'http://google.com/this_url_does_not_exist'
        req = urllib2.Request(url, headers = {
            'Error_404'  : { 'retries': 5,
                             'delay'  : 2.0 }})
        with pytest.raises(HTTPError):
            f = urllib2.urlopen(req)

    def test_wrong_header_retries_1(self, urllib2):
        from urllib2 import HTTPError
        url = 'http://google.com'
        req = urllib2.Request(url, headers = {
            'Error_404' : { 'retries': 'rubbish',
                            'delay'  : 2.0 }})
        with pytest.raises(ValueError):
            f = urllib2.urlopen(req)

    def test_wrong_header_retries_2(self, urllib2):
        from urllib2 import HTTPError
        url = 'http://google.com/this_url_does_not_exist'
        req = urllib2.Request(url, headers = {
            'Error_404' : { 'retries': 0,
                            'delay'  : 2.0 }})
        with pytest.raises(AssertionError):
            f = urllib2.urlopen(req)


Conclusion

You can have better and more robust control of requests without even touching your application code by installing a custom opener to urllib2.

Thursday, 13 February 2014

Strong type checking in Python? What the hell is this !!???

This article describes a Python annotation which combines documentation with type checking in order to help Python developers to gain better understanding and control of the code, whilst allowing them to catch mistakes on the spot, as soon as they occur.

Being a Java developer previously but extradited to Python by my own choice, I sometimes feel some nostalgy from the old times, when the Java compiler used to tell me all sorts of stupidities I used to do.

In the Python world no one is stupid obviously, except probably me who many times find myself passing wrong types of arguments by accident or by pure stupidity, in case you accept the hypothesis that there's any difference between the two situations.

When you are coding your own stuff, chances are that you know very well what is going on. In general, you have the entire bloody API alive and kicking inside your head. But when you are learning some third party software, in particular large frameworks, chances are that your code is called by something you don't understand very well, which decides to pass arguments to your code which you do not have a clue what they are about.


Documentation

Documentation is a good way of sorting out this difficulty. Up-to-date documentation, in particular, is the sort of thing I feel extremely happy when I have chance to find one. My mood is being constantly crunched these days, if you understand what I mean.

Outdated documentation is not only useless but also undesirable. Possibly for this reason some (or many?) people prefer no documentation at all, since absence of information is better than misinformation, they defend.

It's very difficult to keep documentation up-to-date, unless you are forced somehow to do so. Maybe at gun point?


Strong type checking

I'm not in the quest of convincing anyone that strong type checking is good or useful or desirable.  Like everything in life, there are pros and cons.

On the other hand, I'd like to present a couple of benefits which keep strong type checking in my wishlist:

* I'd like to have the ability to stop the application as soon as a wrong type is received by a function or returned by a function to its caller. Stop early, catch mistakes easily, immediately, on spot.

* I'd like to identify and document argument types being passed by frameworks to my code, easily, quickly, effectively, without having to turn the Internet upside down every time I'm interested to learn what the hell argument x is about.


Introducing sphinx_typesafe

Doing a bit of research, I found an interesting library called IcanHasTypeCheck (or ICHTC for short), which I ended up rewriting almost from scratch during the last revision and I've renamed it to sphinx_typesafe.

Let me explain the idea:

In the docstring of a function or method, you employ Sphinx-style documentation patterns in order to tell types associated to variables.

If your documentation is pristine, the number of arguments in the documentation match the number of arguments in the function or method definition.

If your logic is pristine, the types of arguments you documented match the types of arguments actually passed to the function or method at runtime, or returned by the function or method to the caller, at runtime.

You just need to add an annotation @typesafe before the function or method, and sphinx_typesafe checks if the documentation matches the definition.

If you don't have a clue about the type of an argument, simply guess some unlikely type, say: None. Then run the application and sphinx_typesafe will interrupt the execution of it and report that the actual type does not match None. The next step is obviously substitute None by the actual type.


Benefits

A small example tells more than several paragraphs.
Imagine that you see some code like this:

    import math
   
    def d(p1, p2):
        x = p1.x - p2.x
        y = p1.y - p2.y
        return math.sqrt(x*x + y*y)



Imagine that you had type information about it, like this:


    import math
    from sphinx_typesafe import typesafe

    @typesafe
    def d(p1, p2):
        """
        :type p1: shapes.Point
        :type p2: shapes.Point
        :rtype  : float
        """
        x = p1.x - p2.x
        y = p1.y - p2.y
        return math.sqrt(x*x + y*y)


Now you are able to understand what this code is about, quickly!.
In particular, you are able to tell what it is the domain of types this code is intended to operate on.

When you run this code, if this function receives a shapes.Square instead of a shape.Point, it would stop immediately. Notice that, eventually, a shape.Square may have components x and y which would make the function return wrong results silently. Imagine your test cases catching this situation!

So, I hope I demonstrated the two benefits I was interested on.



Missing Features


Polymorphism

Sometimes I would like to tell that an argument can be a file but also a str. At the moment I can say that the argument can be types.NotImplementedType meaning "any type". But I would like something more precise, like this:

    :type f: [file, str]
   
This is not difficult to implement, actually, but we are not there yet.


Non intrusive

I would like to have a non intrusive way to turn on type checking and a very cheap way of turning off type checking, if possible without any code change.

Thinking more about use cases, I guess that type checking is very useful when you are developing and, in particular, when you are running your test suite. You are probably not interested on having the overhead of type checking on production code which was theoretically exhaustively tested.

Long story short, I would like to integrate sphinx_typesafe with pytest, so that an automatic decoration of functions and methods would happen automagically and without any code change.

If pytest finds a docstring which happens to contain a Sphinx-style type specification on it, @typesafe is applied to the function or method. That would be really nice! You could also run your code in production without type checking since type checking was never turned on in the first place.

The idea looks to be great, but my ignorance on pytest internals and my limited time prevents me of going ahead. Maybe in future!


Python3 support

The sources of sphinx_typesafe itself are ready for Python3, but sphinx_typesafe does not handle properly your sources written in Python3 yet. It's not difficult to implement, actually: it's just a matter of adjusting one function, but we are not there yet. Maybe you feel compelled to contribute?


More Information

https://pypi.python.org/pypi/sphinx_typesafe


Credits

Thank Klaas for inspiration and his IcanHasTypeCheck (or ICHTC for short).

Tuesday, 10 December 2013

Money in Brazil is Real

Nice! You are going to Brazil to see the World Cup or the Olympics or maybe just for enjoying 6,000km of sunny wonderful coast?

In this case you have heard that money in Brazil is Real. Yeah... this is the name of the currency: Real. Its symbol is BRL, which means Brazilian Real.

In Brazilian Portuguese (yes: in Brazil we speak Portuguese, not Spanish!), the word real has two meanings:

* like real in English
* like royal in English

So, now you know that Brazilians, despite far from monarchy for almost 2 centuries now, they have a currency which is royal. Yes, royal !

I know, I know... English speakers make jokes with Brazilian money, which is real, not imaginary, isn't it? .... LOL ... not a problem... Brazilians are easy going and we love jokes, any kind of joke, lots of them, politically correct or not, racially correct of not, sexually correct or not, religiously correct or not... because this is how the world is: with all sorts of things which make this world. So, regardless of your political orientation, sexual orientation, your race or religion or anything it might be... it's time for a joke.

By the way, prepare your mood for Brazilians, in particular in Rio. Cariocas (plural of Carioca: people who have born in Rio) are known for their quick thinking when there's a situation which might lead to a joke. You may miss a joke, I may miss a joke, but Cariocas never miss one. So, do not be offended if you suddenly become victim of a joke... better learn quickly how to make others victims of your jokes too. :)

Living in England I know that many jokes are not acceptable here. To be more realistic, Brits make lots of jokes too, about everything ... but only in the privacy of their homes, where they are sure that they will not be prosecuted for being naughty with someone else's race, religion or sexual orientation.

Things are different in Brazil. Bullying is national culture, all over the place. Or still national culture, to be more realistic. Unfortunately, British mood and life style is slowly contaminating Brazilians. Unfortunately, bullying is becoming something not acceptable anymore. Maybe globalization is responsible for this. This is pretty sad and absolutely unacceptable!

You may find the previous paragraph nasty. But be sure, it's your fault and only your fault. This is your mindset, which is shamefully narrow and only tuned to your own culture for decades, not being able of seeing anything outside your small island. When you go to another country, you have to adapt yourself to the other culture's mindset. It's that simple! :)

Well, anyway... despite that humour in Brazil is not as good as it was one decade ago, it's still very far for Brazilians a day when only jokes about the weather would be socially acceptable.

Sorry ... I couldn't resist to this joke... LOL

Sunday, 8 December 2013

Using TypeTokens to retrieve generic parameters

Note: I've recovered this article from http://archive.org.
The original article is  presented here mostly untouched.



Super Type Tokens, also known by Type-safe Heterogenerous Container (or simply THC) is very well described in article by Neal Gafter, who explains how Super Type Tokens can be used in order to retrieve Run Time Type Information (RTTI) which would be erased otherwise, in a process known as type erasure.


Overview

There are circumstances where you'd like to have a class which behaves in different ways depending on generic parameters.

Contrary to what is widely accepted, type erasure can be avoided, which means that the callee has ability to know which generic parameters were employed during the call.

For example, imagine a List which would not rely on Java Collections Framework but on array of primitive types, because performance would be much better than JCF classes. So, you'd like to tell List that it should internally allocate an array of ints or an array of doubles, depending on a generic parameter you specify. Something like this:
List<Integer> myList = new PrimitiveList<Integer>()
... would be backed by int[] whilst
List<Double> myList = new PrimitiveList<Double>()
... would be backed by a double[].

The problem: Type Erasure

When Generics was implemented in Java5, it was decided that this feature would be offered by javac (the Java compiler) and only very minimum changes would be implemented in other components of the architecture. The big benefit of this decision is that the implementation of this feature was relatively simple and imposed only relatively minimum risk to existing Java applications, guaranteeing compatibility and stability of the Java ecosystem as a whole.

The problem with this implementation is that only javac knows about generic types you specified in your source code. This knowledge exists only at compilation time. At run time, your callee class has no clue what generic parameters were employed during the call. It happens because information relative to generic types is lost in a process known as type erasure, which basically means that javac does not put type information it has at compilation time in the bytecode, which ultimately means that your running application does not know anything about type information you've defined in your source code.

Confused? Well ... it basically means that the code below is not possible:
class MyClass<T> {
    private final T o;

    public MyClass() {
        this.o = new T();
    }
}
... because at run time MyClass does not actually know anything about the type of generic parameter T. In spite javac at compile time is able to perform syntax and semantics validation of your source code, at run time all information regarding generic type T is thoroughly lost.

Actually, the previous statement may not be 100% correct under certain circumstances. This is what we will see in the next topic.

How type erasure can be avoided

When Generics was implemented in Java5, the type system was reviewed and long story short, information about generic types can be made available at run time under specific circumstances. This is a very important concept to our discussion here:
Generic types are available to anonymous classes.

Anonymous classes

Let's debate a little bit what an anonymous class is and also what it is not.
Let's suppose we are instantiating MyClass like this:
MyClass<Double> myInstance = new MyClass<Double>() {
        //
        // Some code here
        //
        // In this block we are adding functionality to MyClass
        //
};

We are actually creating an instance of MyClass, but also we are adding some additional code to it, which is enclosed by curly braces.

What it means is that we are creating an object myInstance of an anonymous class of MyClass. It does not mean that MyClass is itself an anonymous class! MyClass is definitely not anonymous because you have declared it somewhere else, correct?

In the snippet of code above we are using something which is an extended thing made from our original definition of MyClass plus some more logic. This extended thing is actually the anonymous class we are talking about. In other words, the myInstance.class was never declared, which means it is anonymous.

How javac handles anonymous classes

When javac finds an anonymous class, it creates data structures in the bytecode (which are available at run time) which holds the actual generic type parameters employed during the call. So, we have another very important concept here:
The Java compiler employs type erasure when objects are instantiated
except when objects are instantiated from anonymous classes.

In other words, our aforementioned MyClass do not know any type information when it is called like this
MyClass<Double> myClass = new MyClass<Double>();
but it does know generic type information when it is called like this:
MyClass<Double> myClass = new MyClass<Double>() { /* something here */ };

In order to obtain generic type information at run time, you do have to change the call, in order to employ an anonymous class made of your original class and not your original class directly. In the next topic we will cover what needs to be done in your implementation of MyClass in order to retrieve generic type information, but a very specific concept is that it will not work unless you call an anonymous class of your defined class. So:
MyClass<Double> myClass1 = new MyClass<Double>();     // type erasure DOES happen
MyClass<Double> myClass2 = new MyClass<Double>() { }; // type erasure DOES NOT happen!

Notice that you only need to have an anonymous class. If you don't have any additional logic to be added if you don't need anything additional. Like you see when object myClass2 was created, there's an anonymous block which is absolutely empty, in this example.

Classical solution

Let's review what we are interested here: we are interested on generic types, which are types. Observe that types are ultimately class definitions. So, we would like to give our class MyClass<T> the ability to know that its T generic parameter is actually a T.class.

In our classical solution described here it can be done very easily simply passing what we need during the call. This is something like this:
MyClass<Double> myClass = new MyClass<Double>(Double.class);

Observe that this is not a very good solution because you have to tell Double three times: (1) when you define the type, (2) when you pass the generic parameter and (3) when you pass the formal parameter Double.class. It looks too verbose and too repetitive, isn't it?

Anyway, this is what the great majority of developers do. They simply tell that Double is generic parameter and then they tell Double.class just after as a formal parameter during the call. In spite it works, the code does not look optimal and it even may lead to bugs later when your application becomes bigger and you start to refactor things, etc.

More flexible solution

We already visited a classical solution for the problem of type erasure and we had already seen how an anonymous call can be done. Now we need to understand how generic types can be retrieved at run time without having to pass Double so many times as we did in our classical solution.

Going straight to the point, let's define an skeleton of our MyClass which does the job we need. Joining some ideas from the classical solution and using some incantation offered by a class called TypeTokenTree. Below we explain the general concept:
import org.jquantlib.lang.reflect.TypeTokenTree;

public class MyClass<T> {

    private final Class<?> typeT;

    public MyClass(final Class<?> typeT) {
        this.typeT = typeT;
        init();
    }

    public MyClass() {
        this.typeT = new TypeTokenTree(this.getClass()).getElement(0);
        init();
    }

    private init() {
        // perform initializations here
    }
}

The code above allows you to call MyClass employing 2 different strategies:
MyClass<Double> myClass1 = new MyClass<Double>(Double.class); // classical solution
MyClass<Double> myClass2 = new MyClass<Double>() { };         // only sorcerers do this

Notice that object myClass1 employs the classical solution we described, which is what the great majority of developers do. The object myClass2 was created using the incantation explained in this article and we will explain it better below.

Digging the solution

Class TypeTokenTree is a helper class which returns the Class of the n-th generic parameter. In the line
this.typeT = new TypeTokenTree(this.getClass()).getElement(0);

We are building an instance of TypeTokenTree, passing the actual class of the current instance and asking for the 0-th generic type parameter.

Please observe what we've written in bold: the actual class of the current instance may be or may not be MyClass. Got it? Observe that the actual class of the current instance will not be MyClass if you employed an anonymous call. In this case, i.e: when you have an anonymous call, javac generates code which keeps generic type information available in the bytecode. Notice that:
TypeTokenTree fails when a non-anonymous call is done!

This is OK. Actually, there's no way to be anything different from that!. It's application's responsibility to recover from such situation.

In the references section below you can find links to class TypeTokenTree and another class it depends on: TypeToken. These files are implemented as part of JQuantLib and contain code which is specific to JQuantLib and may not be convenient for everyone. For this reason, we can see below modified versions of these classes which aims to be independent of JQuantLib and aims to explain in detail how the aforementioned incantation works.

First of all, you need to have a look at method getGenericSuperclass from the JDK. This method is basically the root of the incantation and it basically traverses data structures created in the bytecode by javac. These data structures provide type information regarding the generic types you employed. In general, getGenericSuperclass returns null, which means that the current instance belongs to a non-anonymous class. In the rare circumstances you employ anonymous classes, getGenericSuperclass will return something different of null. And this is how we do this magic.

When getGenericSuperclass does not return null, you have opportunity to traverse the data structure javac created in the bytecode and you can discover what was available at compile time (finally!), effectively getting rid of type erasure.
static public Type getType(final Class<?> klass, final int pos) {
    // obtain anonymous, if any, class for 'this' instance
    final Type superclass = klass.getGenericSuperclass();

    // test if an anonymous class was employed during the call
    if ( !(superclass instanceof Class) ) {
        throw new RuntimeException("This instance should belong to an anonymous class");
    }

    // obtain RTTI of all generic parameters
    final Type[] types = ((ParameterizedType) superclass).getActualTypeArguments();

    // test if enough generic parameters were passed
    if ( pos < types.length ) {
        throw RuntimeException(String.format(
           "Could not find generic parameter %d because only %d parameters were passed",
              pos, types.length));
    }

    // return the type descriptor of the requested generic parameter
    return types[pos];
}

Pros and cons

The big benefit of employing Type Tokens is that the code becomes less redundant, I mean:
MyClass<Double> myClass = new MyClass<Double>() { };
... is absolutely enough. You don't need anything like this:
MyClass<Double> myClass = new MyClass<Double>(Double.class);
On the other hand, the code also becomes obcure, because failing to remember to add the anonymous block will end up on an exception thrown by class TypeToken.
MyClass<Double> myClass = new MyClass<Double>() { }; // succeeds
MyClass<Double> myClass = new MyClass<Double>();     // TypeTokenTree throws an Exception

The point is: this technique is not widely advertised and most developers never heard that this could be done. If you are sharing your code with your peers, contributors or clients, chances are that you will have to spend some time explaining the magic the code does. In general, developers forget to make the call properly, which leads to failures at runtime, as just explained above.

There's also a small performance penalty imposed when TypeToken is called, once this information may be available at compile time and javac can simply write it down straight away when you call
MyClass<Double> myClass = new MyClass<Double>(Double.class);

Test Cases

OK. Now you visited the theory, you'd like to see how this thing really works. Below you can find some test cases which exercise classes TypeToken and TypeTokenTree'. These test cases cover some varied scenarios and they should be enough to illustrate how the techniques explained here can be used in the real world.


References





If you found this article useful, it will be much appreciated if you create a link to this article somewhere in your website.

Thanks

Richard Gomes 20:16, 3 January 2011 (GMT) [Date of the original article ]