Das DasIch

GSoC: Benchmark suite is merged

2011-06-02T00:51:00.000-07:00

I've started working on the coding part of the project, so far the PyPy and CPython benchmarks have been merged.

I've also done a couple of small commits such as fixing typos, eliminating unnecessary whitespace etc.

The next part of the project will be implementing the configurable tool which downloads and builds interpreters based on a given configuration.

So far I'm able to load and validate a simple configuration, which will be a simple Python file as this is the easiest way to start.

I've a very long weekend ahead, from today to monday which will give me a lot of time to work on the project.

Accepted to GSoC

2011-04-26T06:52:00.000-07:00

Yesterday the projects which were accepted to GSoC have been announced. Among them are several interesting projects under the PSF umbrella, including mine.

During GSoC I will create a benchmark suite (based on existing ones) with "real world" benchmarks which can be easily used for every Python interpreter. Up until now each interpreter more or less rolled his own suite of benchmarks of varying quality. This makes comparisons unnecessarily hard and binds resources better used elsewhere.

Furthermore I will create an application which is able to download and build interpreters and execute the benchmarks with them using a simple configuration. Up until now such an application does not exist and e.g. http://speed.pypy.org compares released and current(from trunk) PyPy versions with other released CPython versions. As nice as that is, being able to compare the most current versions of various implementations is clearly favorable.

Once that work is completed I will port the benchmark suite to Python 3.x, as several benchmarks have dependencies that do not support 3.x, yet, I will not be able to port the entire suite, however it will be at least in start when it comes to benchmarks for 3.x.

I'm currently compiling a list with information on available benchmarks (what and how does it test) so that people unfamiliar with them can achieve an easy overview, once that is finished I will send E-Mails to the CPython, PyPy, IronPython, Jython and Cython mailing lists with the benchmarks I propose and asking for other benchmarks or changes to my proposed list.

Further information on my project will be published here and on Twitter as soon as possible.

Writing CLI Applications in Python: A Rant

2011-04-04T12:08:00.000-07:00

A couple of weeks ago I searched for a way to better organize my music. I have several GBs of music all more or less properly tagged and organized however I wanted to be able to reorganize it, change metadata, search and add covers and add new music easily. The existing applications are either horribly confusing or simply don't provide the features I want, so like every programmer In decided I could do better and started a project.

As I usually write web applications, libraries or do small researchy projects to learn stuff I researched a bit concerning the tools I need. I needed stuff for configuration, something to deal with input and output from and to the CLI.

The first thing I noticed is that there is absolutely no solution to handle configuration. I want something that handles multiple hierarchical configuration files, preserves comments in them even when the configuration changes and at best supports more formats than just INI, choosing the proper parser based on the file extension.

A search for that on pypi shows several packages, several of them don't have a description, those that do have one don't necessarily have documentation and those that have it, tend to lack it and provide no way to contribute to the project or to report bugs. For all intents and purposes those projects don't exist.

Trying to figure out how to handle configuration I took a look at the mercurial source code (another project which needs a more obvious link to a source code browser), I learned that I never want to do that again, at least when it comes to that part of the source. Oh, I nearly forgot, apparently configuration is best handled on your own which is what mercurial is doing.

The next thing I considered was handling CLI arguments for which there are two widely used solutions optparse and argparse. Optparse is the older one and is probably used by more people than argparse so I decided to look at that first, it has no way to handle commands or arguments so I deemed it unusable for my purposes.

On first glance argparse seems to be almost identical to optparse, that is because the developers wanted to preserve "backwards-compatibility", at some point they recognized that this doesn't work and changed the API making it a merge of optparse and "something else". argparse handles arguments and provides commands however that latter is rather awkward to use.

You can't just create a command and add it to the parser, no you have to call `.add_subparsers` on the parser which does not add multiple subparsers as one might think, it returns a special object with a single `.add_parser` method which adds a subparser to the parser `.add_subparsers` was called on. I have no idea why you have to do that and as I value my sanity I probably really don't want to know but something tells me that nobody sane involved ever gave the API design any consideration.

As it is really just designed as a parser `argparse.Parser.parse_args` always returns a flat data structure which does not provide information about commands invoked, which would certainly be helpful to call the appropriate function implementing that command. The documented solution for this problem is to add a default function `func` to every subparser (and yes you can specify "defaults" independent of options or arguments on a subparser which are actually not default at all because they are never changed) and call the function, which ends up as `func` in the result, with the result.

I realize things are almost always more difficult than they appear to be and there are probably good reasons for the decisions which have been made but surely there have to be better solutions to this problem.

User input is a somewhat ugly thing, all that parsing and validating dealing with those idiots calling themselves users is not really pleasant so I was hopeful that at least output on a terminal can be considered a solved problem. It is not.

If you want to write a paragraph of text, wrapped to the width of the terminal, to stdout, you have to get the width of the terminal in platform dependent ways via iocntl and fcntl on linux (I guess you have to wrap the Windows API with ctypes); luckily at least textwrap is already in the stdlib.

You have to implement progress bars and coloring yourself unless you want to have dependencies for all of these things.

Also don't forget that a simple print statement may cause problems as soon as you don't have an ASCII decoding and that even if you decode to the proper stdout encoding if possible user input might not be encode-able (umlauts to ASCII) and that you therefore may have to transliterate unless you are willing to just replace and ignore these errors but I'm sure everyone of you does this carefully everywhere.

The fact that there are no solutions to these problems is a really big WTF and makes writing CLI applications a pain in the ass which really shouldn't be the case.

Documentation Directory Structure

2011-02-06T08:14:00.000-08:00

If you are using Sphinx for documentation thinking about the directory structure is important because changing it later gets painful. The builders replicate the directory structure in the source directory in the build directory so if you want to change it you will have to make redirects for the HTML documentation.

In my experience the following structure is rather flexible:

docs/user
Documentation directed at users such as tutorials, guides etc.

docs/api
API documentation mostly generated with autodoc.

docs/development
Documentation directed at developers of your project.

docs/additional
Use this for your change log, license etc.

Privacy vs. Acceptance

2011-01-30T04:54:00.000-08:00

In the last couple of months I've seen a lot of debate going on about privacy. Some people complain about the fact that privacy seems to disappear others see it as a positive evolution of cultures and then there are all those people who either don't care or simply go along with it.

I see why people like privacy, exhibitionism is not a common trait and people fear the judgement of others. Knowledge gives you power, if the balance between the knowledge others have about and you have about them shifts to either side disadvantages arise. Nevertheless I think there is something very important to gain by giving up privacy: acceptance of others or tolerance towards them.

The internet gives us the possibility to re-evaluate what society looks like, it provides us with a picture of an accuracy we have never seen before in the world where the media, which provided a mirror of society, filtered the information from the picture.

Ethics and morals will adjust to that new picture, what is considered normal and what not will switch.

The difference between what is considered normal and the people, we live and communicate with, the people we meet, might stay the same but by changing the view on what is normal acceptance will change for the better.

I believe that by providing a broader picture of society, a more accurate picture of society acceptance will be broader, people will not have to fear judgement of others (as much) and therefore are encouraged to develop themselves into what they want to be.

I believe in an open world, what do you believe in?

New at Pocoo

2010-09-03T09:42:00.001-07:00

Just a short message for those who haven't heard, yet. I'm now a member of Pocoo, the umbrella project for Werkzeug, Jinja2, Sphinx and a lot of other very awesome projects :)

A summary of the Google Summer of Code

2010-08-19T01:48:00.000-07:00

The Google Summer of Code is nearly over and it's time to present some results. First of all every project we made can be considered successful, the team was great and I hope to be able to work with the others in the future to achieve new great things. However let's get to the point: As some of you already now Sphinx has now Python 3.x support in trunk which means that with the next non-bugfix release you will be able to use Sphinx with Python 3.x.

In separate branches which are hopefully merged soon into trunk we have i18n support, this allows you to build gettext message catalogs which contain ids as comments which you can use to identify messages which have been changed in the documentation. Another great achievement is websupport, this allows you to create web applications using Sphinx with server-side search, comments on paragraphs and code blocks and proposals to change the documentation from users.

My contribution to i18n and websupport has been an AST based merging algorithm which allows you to easiely track changes across multiple builds of the documentation. This makes it possible to identify changes to the documentation so that we don't have to delete all the comments with every documentation rebuild.

Making Sphinx faster

2010-08-14T00:16:00.000-07:00

I've recently spend a lot of time thinking about programming languages, it's something I'm very interested in and creating my own is an item on my todo list. One topic that comes up if you think about that is parallelization so to get my mind of sphinx-web-support for a while I looked at Sphinx to see how easily I could use it in Sphinx. Before you get too excited these are just a couple of thoughts on my part, there could be something I'm still missing, now read on.

Sphinx' Design

In order to implement this feature we have to look at the design of Sphinx. The design is more or less simple, we have an Application which set's everything up and can be used to run the build process. The build process is handled by the Environment, the environment parses every document in the source directory, creates a doctree(AST) transforms is as necessary and uses the information from the doctree to populate the index, after that the doctree is stored in build/doctrees/document.doctree. Once every document has been processed the environment invokes a Builder, the builder loads each doctree, modifies it if necessary and passes it on to the Writer which creates the code for each doctree we store in the build directory under the name of the builder.

The Problem

Currently the environment does actually a lot more than it should in my opinion, the index is kept global in the environment, as well as everything we need to know about the current document. This makes it impossible to simply parallelize the process of parsing and building process because there is too much shared state.

The Solution

The obvious solution and the better design is to keep the data associated with a specific document in an object I call DocumentContext, this context is used to store the necessary information for a document as well as information we get from the document which is relevant for the Environment. After parsing, transforming and processing each document we use the context and put the relevant information in the environment.

This way the Environment is immutable from a parser perspective and we can easily use parallelization to make the entire build process a lot faster than it is currently.

Another Problem and another Solution: Backwards compatibility

Changing Sphinx in the way I propose will probably break some extensions, it will definitely break the existing domains. Personally I don't really care about this issue because I think software has to evolve and constantly change over time in order to make it in the long run.

However I know that a lot of people do care so I propose something a lot of people know from web applications, context locals, basically they are proxies which point to the objects in the current context, which is either a process, a thread or even a simpler concept based on coroutines. Using those the current API could be kept at least partially and we could deprecate it first before removing it at some point in the future.

Hey, what you are doing?

2010-08-10T04:02:00.000-07:00

Those of you who follow the discussions on the IRC channel and Twitter already know, we have Python 3.x Support now in Trunk, so sphinx-py3k can be considered a success. However what else is going on?

One of the problems with both i18n and websupport is that we need a way to identify parts of documentation across multiple builds. A simple example is a document, it has multiple paragraphs and we want store comments for each paragraph, we need to keep track of the paragraph even if the document changes or the paragraph itself does. If we don't we have to throw away all the comments for every build as we don't know where we have to put them or if they still apply in case a paragraph has been removed entirely.

Especially identifying a changed paragraph is a bit complicated and required a bit of research on my side however I have a solution which should work mostly, it doesn't pass all the test I came up with however I hope to be able to finish my work soon, so I can talk to birkenfeld about merging my branch with trunk so it can be used in web-support and i18n.

You can take a look at the code in the bitbucket repo, if you want to keep updated about the most recent developments I suggest visiting #pocoo on freenode and/or following me on Twitter. These would also be the right places to ask me questions about the project or to simply ask me about the current status.

163 tests passed

2010-07-11T05:01:00.000-07:00

The most important milestone of sphinx-py3k has been reached by passing the complete test suite with Python 2 and Python 3. Now it should be possible to use Sphinx with Python 3 without encountering any problems.

However nothing is ever perfect so I encourage everybody to test sphinx-py3k and report problems should they occur, so that they can be fixed.

Current git branch in a zsh prompt

2010-06-26T14:39:00.000-07:00

I've looked for hours to find a way to just display the current git branch in the prompt of my zsh shell and I've come to this more or less understandable solution which should work with any common VCS available:

autoload -U colors && colors
autoload -Uz vcs_info

zstyle ':vcs_info:*:prompt:*' formats "$VCSPROMPT" "[%b]"

precmd() {
vcs_info 'prompt'

if [ -n vcs_info_msg_0_ ]; then
RPROMPT="${vcs_info_msg_1_}"
else
RPROMPT=""
fi
}

PROMPT=$'%F{green}%~#%f '

The result is that the current working directory is shown on the left side and the name of the branch in braces on the right. I could probably configure it in a way which shows me a lot more information but all I want is a little reminder about the branch I'm currently in.

The current state of sphinx-py3k

2010-06-18T03:23:00.000-07:00

I have not made a report in quite some time. I know I am supposed to make weekly reports but up until now I have not found anything really worth reporting at least since the last report. So what happend that you can find here one now?

I fixed every error in the test suite, which were about 55, now there are only 7 failures left. They might be even harder to fix then a usual error but I do not think this is going to be much of a problem.

What is a real problem are doctests. Currently there is no way of converting those automatically and even if there was there are certain problems like the change in the results. The representation of the string types has changed and some functions in the math module returns ints instead of floats were applicable and if you look around you can probably find more examples. A converter cannot deal with these changes easiely. I have some ideas on the problem but I will have to talk to my mentor about a solution.

GSoC: Scripts are (mostly) working

2010-05-16T18:40:00.000-07:00

Looks like another week or so has ended which means a status report by me. Instead of porting Sphinx itself I took my time to port the scripts which are used to check the coding style and a couple of other things.

This means that there is no difference between developing Sphinx with 2.x or 3.x anymore as long as you ignore the errors and failures the test suite gives you when using 3.x.

There still are some problems with the reindent script which is in use. I got the latest one from the CPython svn, adjusted it a bit so it can be converted with 2to3 but for some reason it behaves differently with 3.x.

P.S.: You might have noticed that I changed the blog title, that is because I plan on publishing more non GSoC related posts and I think opening another blog is not worth it.

Progress on the 3.x Port

2010-05-08T13:21:00.000-07:00

Since the beginning of the project I was able to remvove every deprecation warning which occurs when running the Sphinx test suite with python -3. Also it is now possible, using distribute, to install it using Python 3. However there are still several problems which do not cause any deprecation warnings and are not fixed by 2to3. So it is not usable for now.

The remaining problems are usually problems with handling strings and files. Even when used correctly in 2.x the changes made with 3.x are causing several problems.

I did nearly nothing during the last week in terms of coding so I will do a lot of work this weekend and make sure to have more time in the future.

Hashable or Unhashable: That is the question!

2010-04-30T03:04:00.000-07:00

Python 3 has changed a lot of things, one of them being that if a class implements __eq__ the __hash__ method will not be inherited. This behavior seems weird at first but if you take a closer look at the topic it shows you a great problem.

Most Python developers do not really think about the design and behavior of an object when they create it, they have a class with an __init__ method, a couple of attributes and methods and may be a __repr__. If they implement other special methods they are usually __getattr__, __*item etc. but given an arbitary class you created could you tell me if it is supposed to be hashable or not and if it is supposed be hashable if it actually is?

Usually you do not care about __hash__ at all, you inherit from object which implements it and on CPython it returns the id of an object on other implementations it may be the same, it may be something different altogether but it is unique to the object.

This result of this is that everytime you create a type and you do not implement __hash__, objects of that type will be hashable even if it is not supposed to be. The worst case is that if you implement __eq__ but not __hash__, equal objects do not behave like equal objects when used as keys in dictionaries, if you put them into a set or if you do something else which relies on hashes as a way to check if objects are equal.

This results in two rules you should always stick to:
1. If you implement __eq__ also implement __hash__.
2. If your object is mutable, set __hash__ to None in order to make it unhashable, this way your object has the same behavior you expect from dictionaries or lists.

If you write new code stick to the rules and you are good to go in your old code make sure to implement those methods and you make it one step easier to port it to Python 3.

Congratulations! Your proposal has been accepted.

2010-04-27T08:48:00.000-07:00

Yesterday Google released the list of accepted projects for GSoC, my project is one of them. In the next months I will be one of the three students working on Sphinx, my part will be porting Sphinx to Python 3.x and the integration of the web application which was developed during the last year.

Currently we are in the "Community Bonding Period" which means basically we are all idling in #pocoo to get in touch with the community and get an idea of the development process until May 24 which will be the day everybody starts coding. However as Sphinx is not such a big project at least in terms of the number of contributors, I think that spending so much time on "bonding" is a waste of time so I will probably start earlier.

So that's it for now. I will try to give you as much information as possible through blog posts in the future so everyone can easiely follow the progress.