May 2019 - PureOS-project - Purism Mailing Lists

"PureOS changes" notifications
by Chris Lamb 14 Jun '19

14 Jun '19

Hi -project, Many years ago I created an IRC bot for Debian that would live on its own channel and simply notify on new uploads, bugs and similar changes in that distribution: https://chris-lamb.co.uk/projects/debian-devel-changes-bot … it is now rather popular and, indeed, many people have noted to me in the interim years not only how motivating it was to be exposed to others' work like that without having to explicitly or actively look for it but it informs & keeps them up-to-date on the project in a wider, less-technical, sense. I found it also a valuable learning tool — if I don't understand a bug report that scrolls past I sometimes click it and invariably end up learning something I didn't previously know. § In this light, I created the following ticket early last year to get this rolling for PureOS: https://tracker.pureos.net/T375 … however, movement on this appears to have stalled or otherwise is lower on others' priority lists than I believe it should be. The basic idea would be to have a separate "dev/pureos/changes" (or similarly named) Matrix room that would have this info spammed into it. Discussions would remain on dev/pureos or, of course, on issue tracker, etc. The bug itself unfortunately is a little ambiguous about what we would like to include in the notifications (tracker bugs vs uploads vs Git commits...), but I firmly and adamently believe that getting *anything* at the moment would be great start so I wasn't being overly specific by design. (The task is also being tracked in parallel on the sysadmin tasks tracker too, adding somewhat to the confusion.) Note that this would also help relieve some developers of feeling like they are the only ones who know about certain areas, preventing further potential "siloing" of knowledge. Just to underline, whilst it might appear to be a "wishlist" feature, I believe it will provide some always-needeed social cohesion and connection within the PureOS development team! As geeks we often criminally underrated such things.. we are humans too! Any thoughts? In particular, can we get some strong +1's from you all (especially Jeremiah) to take back to the ticket and hopefully parley that into getting this reprioritised? Best wishes, -- Chris Lamb https://puri.sm

2 2

Updates to tracker.pureos.net
by Jeremiah C. Foster 25 May '19

25 May '19

Hello, Today (Friday May 24th) there has been an update to the machine that runs our Phabricator instance. It has had the OS updated to Stretch (Debian 9.9). There also has been the installation of service files to start phabricator on boot. I think it might be nice to update phabricator as well the version running now is kinda old: https://tracker.pureos.net/config/version/ I'm happy to upgrade phabricator if it's simply a case of 'git pull' and then launching the apps but perhaps Matthias you know of a right way to do this? Cheers, Jeremiah

2 1

Certificate renewal and tracker.pureos.net going offline this weekend
by Jeremiah C. Foster 21 May '19

21 May '19

Hi, There are some expired certs and some general DNS cruft that admins are going to be addressing in the coming week on the tracker.pureos.net site. I mention this just as a heads-up. I hope to come back with more info once it is firmed up. Regards, Jeremiah

1 0

Re: [PureOS] Laniakea Python version is now online!
by Matthias Klumpp 21 May '19

21 May '19

Am Mo., 20. Mai 2019 um 19:44 Uhr schrieb Chris Lamb <chris.lamb(a)puri.sm>: > > Matthias Klumpp wrote: > > > > > Since last week, all of PureOS infrastructure is now switched to the > > > > latest, mostly Python-based version of Laniakea[1]. > > > > > > This is great news. Congratulations Matthias! > > Indeed, congratulations and thank you for posting about it here too. > My apologies for this late reply getting lost between various cracks > in my inbox. :( Don't worry, there is no expiry date on replies ;-) > > > If so, perhaps we use a separate Laniakea channel for Matrix so as > > > not to flood the PureOS channel. > > I completely +1 the idea of a #pureos-changes channel. > > I use this pattern in many projects; having that "soft" visibility is > extremely helpful for all sorts of reasons (both technical and non- > technical) in terms of encouraging more of a team feel, avoiding any > feeling of being isolated from others. Indeed, I even wrote the > #debian-devel- changes bot back in 2008 for these very reasons and it > has been remarkably successful in these regards. Yes, we had this in Tanglu as well and it was really useful :-) > Can I thus resubmit my request for this to be re-prioritised? :) It's definitely a high priority, but giving people access to easy information about the archive is even more important I would say. That being said, software.pureos.net is available again now, so you can view build logs easily again. Next I will also re-add the debcheck and migration information to the package details pages, as that's important as well for diagnosing problems. Then maybe also refine the pages to sort by suite as well, and also improve the application/component search a bit (as that's the neat thing our users may be interested to use, and it's easy to do). Than I would actually be moving on to reinstate the messaging system between Laniakea components, however I don't feel at ease doing that without a proper testing framework in place, so that also has a slightly higher priority. But then, the next thing definitely is the message bus, Matrix bot and possibly Fedmsg bridge. > Oh, one adjunct question; how can we contribute/learn more about this > codebase? Whilst justifiable for many reasons, as it is on Github it > definitely feels a little separate from the rest of the PureOS code. > Can we ameliorate this somehow? I don't have any concrete suggestions > at this time, alas. On why it's on a different platform: That is simply because the project started outside of PureOS before I joined Purism, and was for a while shared between two derivatives. Nowadays I actually prefer it to be not as tightly coupled to the PureOS project, because I hope that as soon as it's mature enough more derivatives may consider using it. That's very often easier when the infrastructure for a project is at a more neutral place. The README file of the project as well as some PureOS wiki pages definitely tell people who want to know who was responsible for moving the project forward though. On contributing: There is outdated documentation on https://lkorigin.github.io/ which I should update soon to reflect what the Python version has to offer (and to elaborate on some points). The code is also documented, but generating some API documentation from it would also be helpful, I guess. I also gave a talk about the project at Debconf, but that's probably not a good starting point. The talk slides however may be useful, I'd have to modernize them as well though, just like the rest of the documentation. To build the project itself, one can use - rather unusual for a Python project - Meson. Meson/ninja will build the binary parts and then install all pieces into the right location, then one can use the `lk-admin` tool to initialize the database and configure the system. Currently, to make Laniakea really useful, one needs to combine it with either dak or reprepro for archive management. I'll certainly answer any questions anyone trying to understand any of the components may have, and hopefully also write a nice quickstart guide. The problem to get all the tasks done is - as always - time ;-) Cheers, Matthias

2 1

Laniakea Python version is now online!
by Matthias Klumpp 20 May '19

20 May '19

Hello! Since last week, all of PureOS infrastructure is now switched to the latest, mostly Python-based version of Laniakea[1]. ## What is Laniakea? Laniakea is a suite of tools and adapters which originated in the Tanglu Debian derivative. Its purpose is to bring together data and configuration from the multitude of separate tools that are required to run a Debian derivative, as well as adding missing functionality in an integrated way. Tools which are wrapped by Laniakea include Britney, dose3, germinator, dak and more. Due to Laniakea's database being the authoritative source of information for a derivative, there is always only one place to change configuration, individual configuration for the wrapped tools is automatically generated from database information. Also, the centralization of data in a database and well-defined directory structure allows Laniakea modules to cross-reference information and make decisions based on data from other tools, e.g. the debcheck information is used to determine whether a package is actually buildable in the distribution. Using the database, all information Laniakea knows about is accessible (currently read-only) via modern web interfaces. Laniakea also contains a global worker pool to execute arbitrary long-running tasks. These tasks currently are image builds and package builds. All tasks are run on separate machines in systemd-nspawn containers via debspawn[2], for build tasks static-analysis reports are generated in the Firehose XML format[3]. Laniakea also contains a message-bus based on ZeroMQ and JSON to let modules running on other machines know about changes and react to events when running in a more isolated environment. This also allows notifications via email or Matrix. ## What changed now? Laniakea previously was written entirely in the D programming language. Since the project was a fun experiment in Tanglu, using the language also wasn't much of an issue. With our new requirements in PureOS and limited manpower though, it became apparent that the choice of D was hindering progress in this case, simply due of D's much smaller ecosystem of existing tools and libraries. Time spent at improving the database connection code in D could be better spent at implementing a new feature we need. So, Laniakea was mostly ported to Python 3.6+ with some code still remaining in D. Since D has great bindings to Python, writing a module to run existing D code from Python was very easy. This also allows us to reuse a lot of the more complex logic in Laniakea without changes (and the chance to introduce new bugs). Porting progress can be followed via this document[4]. Since Monday last week, PureOS' infrastructure is now running on the Python-ified version of Laniakea, removing the extra maintenance burden of developing two versions of the same software in parallel. ## How does the change affect me? Ideally, a user or developer of PureOS should not see any changes. The new implementation should work just as the previous one. The web UI on master.pureos.net looks slightly different, but that is it. Since the new Laniakea database has no indices yet, certain queries could take a bit longer. One notable change is the absence of software.pureos.net - this module could not be easily ported and needs to be rewritten entirely. I am working on that, and we will definitely get it back, better than before. The Laniakea message piblishing code also has not been ported yet, so modules can not communicate with each other via ZeroMQ. This should not ceuase any issues, but when it is finally implemented, we will get a new Matrix bot to notify about archive changes immediately. ## What's next? ### Create an integration test framework for Laniakea We need better testing, or rather, any testing for some Laniakea modules and larger pieces of the codebase. Since a large chunk of Laniakea is interfacing with existing code or with its own modules, we need some way to do intergration testing properly. ### More unit testing! The Python code doesn't have many tests written yet, so this needs to be added as well. ### Bring back software.po.n (webswview) The web application just needs to be rewritten. This is a work in progress at time. ### Reimplement Laniakea messaging The Lighthouse module of Laniakea needs to get its message publication support back. I intend to make use of signedjson[5] for message signing this time. When implementing this feature, the very crude current Lighthouse implementation also needs a complete overhaul. Additionally, we can write a Matrix bot for Laniakea again at this stage. ### Better dak integration The Debian Archive Kit (dak) and Laniakea basically exist alongside each other, not interacting all that much. That should be changed to make dak and Laniakea talk to each other better to reduce the reliance on cronjobs. ### Better performance There is a lot of potential for performance improvements in Laniakea, most notably in the module that imports archive data into the database. Making these improvements will allow for many nice things, like faster package processing. ### Package for Debian Laniakea should be available from withing Debian. This is a long-term goal as it requires at least some stability of the database schemas and more and better testing. Also, the JavaScript dependencies of the web modules need to be sorted out properly. Ultimately though, at some point you can just apt install Laniakea to create your own Debian derivative. ## What can I do? Contribute! Either by writing code for Laniakea (documentation is currently updated to reflect the Python changes) or filing bugs. If you see anything weird happening in PureOS infrastructure that hasn't happened before, please file a bug in the PureOS bugtracker[6] or against Laniakea directly if you know it is to blame already. Happy hacking! Matthias [1]: https://github.com/lkorigin/laniakea [2]: https://github.com/lkorigin/debspawn [3]: https://github.com/fedora-static-analysis/firehose [4]: https://github.com/lkorigin/laniakea/blob/master/docs/porting-status.md [5]: https://github.com/matrix-org/python-signedjson [6]: https://tracker.pureos.net/

3 3

Updates on reproducible builds in PureOS
by Jeremiah C. Foster 18 May '19

18 May '19

Hi, I've received and email recently from the folks at the FSF about PureOS reproducible builds. When we were at LibrePlanet was asked if the FSF would be kind enough to 'verify' our reproducible builds and they replied via mail saying; > Yes, we are interested in doing this. Generally, we will want > the build and verification process to be publicly documented > and mostly automated for an initial setup To which I replied, here is how we build our ISOs; https://tracker.pureos.net/w/development/howto_build/ I've set up debspawn to build in ISO from cron as well and I'll automated diffoscope to create and html report for us and the FSF to review. I also have created this document to explain a bit about reproducible builds in PureOS. This document pertains to PureOS and it refers to upstream source to find out more about the upstream project. https://tracker.pureos.net/w/development/reproducible_isos/ Currently the work being done is around automation; - having image building being done by cron - running diffoscope The ISOs that get built today are around 1.5 gigs which makes me wonder if they're too large to put in Debian's Jenkins machine lest they fill up disk. Happy to hear feedback on all the above. Regards, Jeremiah

2 1

Please assume all issues require tracking unless told otherwise
by Jonas Smedegaard 02 May '19

02 May '19

[ posting again, to correct non-lists address this time ] Hi all, There seems to be confusion about which issues need tracking and which are adequate to just discuss casually on Matrix. If in doubt, please read the topic line of the matrix room you're in. PureOS matrix room currently offers this guidance: > Create new issue at https://tracker.pureos.net/ (or locate and add to > existing issue) for all the things unless explicitly told it needs no > tracking or followup Let's please help each other remember to track all that needs tracking. - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private _______________________________________________ Pureos-project mailing list Pureos-project(a)lists.puri.sm https://lists.puri.sm/listinfo/pureos-project

1 0

Please assume all issues require tracking unless told otherwise
by Jonas Smedegaard 02 May '19

02 May '19

Hi all, There seems to be confusion about which issues need tracking and which are adequate to just discuss casually on Matrix. If in doubt, please read the topic line of the matrix room you're in. PureOS matrix room currently offers this guidance: > Create new issue at https://tracker.pureos.net/ (or locate and add to > existing issue) for all the things unless explicitly told it needs no > tracking or followup Let's please help each other remember to track all that needs tracking. - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private

1 0