[PureOS] Laniakea Python version is now online!

[Matthias Klumpp]

Hello!
Since last week, all of PureOS infrastructure is now switched to the
latest, mostly Python-based version of Laniakea[1].

## What is Laniakea?
Laniakea is a suite of tools and adapters which originated in the
Tanglu Debian derivative. Its purpose is to bring together data and
configuration from the multitude of separate tools that are required
to run a Debian derivative, as well as adding missing functionality in
an integrated way. Tools which are wrapped by Laniakea include
Britney, dose3, germinator, dak and more.
Due to Laniakea's database being the authoritative source of
information for a derivative, there is always only one place to change
configuration, individual configuration for the wrapped tools is
automatically generated from database information.
Also, the centralization of data in a database and well-defined
directory structure allows Laniakea modules to cross-reference
information and make decisions based on data from other tools, e.g.
the debcheck information is used to determine whether a package is
actually buildable in the distribution.
Using the database, all information Laniakea knows about is accessible
(currently read-only) via modern web interfaces.
Laniakea also contains a global worker pool to execute arbitrary
long-running tasks. These tasks currently are image builds and package
builds. All tasks are run on separate machines in systemd-nspawn
containers via debspawn[2], for build tasks static-analysis reports
are generated in the Firehose XML format[3].
Laniakea also contains a message-bus based on ZeroMQ and JSON to let
modules running on other machines know about changes and react to
events when running in a more isolated environment. This also allows
notifications via email or Matrix.

## What changed now?
Laniakea previously was written entirely in the D programming
language. Since the project was a fun experiment in Tanglu, using the
language also wasn't much of an issue. With our new requirements in
PureOS and limited manpower though, it became apparent that the choice
of D was hindering progress in this case, simply due of D's much
smaller ecosystem of existing tools and libraries. Time spent at
improving the database connection code in D could be better spent at
implementing a new feature we need.

So, Laniakea was mostly ported to Python 3.6+ with some code still
remaining in D. Since D has great bindings to Python, writing a module
to run existing D code from Python was very easy. This also allows us
to reuse a lot of the more complex logic in Laniakea without changes
(and the chance to introduce new bugs). Porting progress can be
followed via this document[4].

Since Monday last week, PureOS' infrastructure is now running on the
Python-ified version of Laniakea, removing the extra maintenance
burden of developing two versions of the same software in parallel.

## How does the change affect me?
Ideally, a user or developer of PureOS should not see any changes. The
new implementation should work just as the previous one.
The web UI on master.pureos.net looks slightly different, but that is
it. Since the new Laniakea database has no indices yet, certain
queries could take a bit longer.

One notable change is the absence of software.pureos.net - this module
could not be easily ported and needs to be rewritten entirely. I am
working on that, and we will definitely get it back, better than
before.

The Laniakea message piblishing code also has not been ported yet, so
modules can not communicate with each other via ZeroMQ. This should
not ceuase any issues, but when it is finally implemented, we will get
a new Matrix bot to notify about archive changes immediately.

## What's next?
### Create an integration test framework for Laniakea
We need better testing, or rather, any testing for some Laniakea
modules and larger pieces of the codebase. Since a large chunk of
Laniakea is interfacing with existing code or with its own modules, we
need some way to do intergration testing properly.

### More unit testing!
The Python code doesn't have many tests written yet, so this needs to
be added as well.

### Bring back software.po.n (webswview)
The web application just needs to be rewritten. This is a work in
progress at time.

### Reimplement Laniakea messaging
The Lighthouse module of Laniakea needs to get its message publication
support back. I intend to make use of signedjson[5] for message
signing this time. When implementing this feature, the very crude
current Lighthouse implementation also needs a complete overhaul.
Additionally, we can write a Matrix bot for Laniakea again at this stage.

### Better dak integration
The Debian Archive Kit (dak) and Laniakea basically exist alongside
each other, not interacting all that much. That should be changed to
make dak and Laniakea talk to each other better to reduce the reliance
on cronjobs.

### Better performance
There is a lot of potential for performance improvements in Laniakea,
most notably in the module that imports archive data into the
database.
Making these improvements will allow for many nice things, like faster
package processing.

### Package for Debian
Laniakea should be available from withing Debian. This is a long-term
goal as it requires at least some stability of the database schemas
and more and better testing. Also, the JavaScript dependencies of the
web modules need to be sorted out properly.
Ultimately though, at some point you can just apt install Laniakea to
create your own Debian derivative.

## What can I do?
Contribute! Either by writing code for Laniakea (documentation is
currently updated to reflect the Python changes) or filing bugs. If
you see anything weird happening in PureOS infrastructure that hasn't
happened before, please file a bug in the PureOS bugtracker[6] or
against Laniakea directly if you know it is to blame already.

Happy hacking!
    Matthias

[1]: https://github.com/lkorigin/laniakea
[2]: https://github.com/lkorigin/debspawn
[3]: https://github.com/fedora-static-analysis/firehose
[4]: https://github.com/lkorigin/laniakea/blob/master/docs/porting-status.md
[5]: https://github.com/matrix-org/python-signedjson
[6]: https://tracker.pureos.net/