How to develop

This chapter is dedicated to developers, and will guide you through code organization, design choices, etc. This is not a tutorial to python, nor to git. It will provide pointers and explanation, but will not teach you how to program.

Ingredients

libreant is coded in python2.7. Its main components are an elasticsearch db, a Fsdb and a web interface based on Flask.

Details about libraries

Elasticsearch is a big beast. It has a lot of features and it can be scaring. We can suggest this elasticsearch guide. The python library for elasticsearch, elasticsearch-py, is quite simple to use, and has a nice documentation.

Fsdb is a quite simple “file database”: the main idea behind it is that it is a content-addressable storage. The address is simply the sha1sum of the content.

Flask is a “web microframework for python”. It’s not a big and complete solution like django, so you’ll probably get familiar with it quite soon.

Installation

Using virtualenv

We will assume that you are familiar with virtualenvs. If you are not, please get familiar!

Inside a clean virtualenv, run

python setup.py develop

You are now ready to develop. And you’ll find two tools inside your $PATH: webant and libreant-manage. The first is a webserver that will run the web interface of libreant, while the second is a command-line tool to do basic operations with libreant: exporting/importing items, searching, etc.

Using Vagrant

Download, setup and run the virtual machine:

vagrant up

You will then find in /liberant the installation of liberant, you can login to the vagrant box with:

vagrant ssh

Code design

This section is devoted to get a better understanding on why the code is like it is, the principles that guides us, and things like that.

Design choices

few assumptions about data
We try to be very generic about the items that libreant will store. We do not adhere to any standard about book catalogation, nor metadata organization, nor nothing like that. We leave the libraries free to set metadata how they prefer. There is only one mandatory field in items, which is language. The reason it is this way, is that it’s important to know the language of the metadata in order for full-text search to work properly. There are also two somewhat-special fields: title and actors; they are not required, but are sometimes used in the code (being too much agnostic is soo difficult!)
no big framework
we try to avoid huge frameworks like django or similar stuff. This is both a precise need, and a matter of taste. First of all, libreant uses many different storage resources (elasticsearch, fsdb, and this list will probably grow), so most frameworks will not fit our case. But it’s also because we want to avoid that the code is “locked” in a framework and therefore difficult to fork.

File organization

setup.py is the file that defines how libreant is installed, how are packages built, etc. The most common reason you could care about it, is if you need to add some dependency to libreant.

libreantdb

libreantdb/ is a package containing an abstraction over elasticsearch. Again: this is elasticsearch-only, and completely unaware of any other storage, or the logic of libreant itself.

webant

webant/ is a package; you could think that it only contains web-specific logic, but this is not the case. Instead, all that is not in libreantdb is in webant, which is surely a bit counterintuitive.

The web application (defined in webant.py) “contains” a Blueprint called agherant. Agherant is the part of libreant that cares about “aggregating” multiple nodes in one single search engine. We believe that agherant is an important component, and if we really want to make libreant a distributed network, it should be very reusable. That’s why agherant is a blueprint: it should be reusable easily.

manage.py is what will be installed as libreant-manage: a simple command-line manager for lot of libreant operations. libreant-manage is meant to be a tool for developers (reproduce scenarios easily) and sysadmins (batch operations, debug), surely not for librarians! This program is actually based on flask-script, so you may wonder why we use flask for something that is not web related at all; the point is that we use flask as an application framework more than a web framework.

templates/ is… well, it contains templates. They are written with jinja templating language. The render_template function

documentation

Documentation is kept on doc/source/ and is comprised of .rst files. The syntax used is restructuredText. Don’t forget to update documentation when you change something!

API

You can read API

Coding style

PEP8 must be used in all the code.

Docstrings are used for autogenerating api documentation, so please don’t forget to provide clear, detailed explanation of what the module/class/function does, how to use it, when is it useful, etc. If you want to be really nice, consider using restructured-text directives to improve the structure of the documentation: they’re fun to use.

We care a lot about documentation, so please don’t leave documentation out-of-date. If you change the parameters that a function is accepting, please document it. If you are making changes to the end user’s experience, please fix the user manual.

Never put “binary” files in the source. With ‘binary’, we also mean “any files that could be obtained programmatically, instead of being included”. This is, for example, the case of .mo.

Testing

Unit tests are important both as a way of avoding regressions and as a way to document how something behaves. If your code is testable, you should test it. Yes, even if its behaviour might seem obvious. If the code you are writing is not easy to test, you should think of making it more easy to test. We use nose suite to manage tests, you can run all the tests and read coverage summary by typing:

python setup.py test
We usually follow these simple steps to add new tests:
  • create a directory named test inside the package you want to test
  • create a file in this folder test/test_sometestgroupname.py
  • write test functions inside this file

We prefer not to have one big file, instead we usually group tests in different file with a representative name. You can see a full testing example in the preset package.

Note

if you are testing a new package remember to add the new package name in cover-package directive under [nosetests] section in /setup.cfg file.

Contributing

Like libreant? You can help!

We have a bugtracker, and you are welcome to pick tasks from there :) We use it also for discussions. Our most typical way of proposing patches is to open a pull request on github; if, for whatever reason, you are not comfortable with that, you can just contact us by email and send a patch, or give a link to your git repository.