[PureOS] PureOS Phone images

[Matthias Klumpp]

Hi, sorry for the delay...

Am Fr., 28. Aug. 2020 um 09:19 Uhr schrieb Guido Günther
<guido.gunther at puri.sm>:
>
> Hi Jeremiah,
> thanks for moving things over to a more persistent media!
>
> On Thu, Aug 27, 2020 at 10:29:04PM -0400, Jeremiah C. Foster wrote:
> > Hi,
> >
> > [Feel free to reply on-list, I just wanted to make sure everyone saw
> > this.]
> >
> >
> > [Please find attached an invite to a mumble call at 16:00 CEST
> > tomorrow.]
> >
> > Welcome to a thread about PureOS images for our Librem 5 phone.
> >
> > Let's start with Guido who knows a thing or two about the Librem 5;
> >
> > "I'd like to (slowly) start filling in enough of byzantium so we can
> > build images for the phone. First candidates would be a kernel and
> > flash-kernel. We currently keep the sources at
> > https://source.puri.sm/Librem5/flash-kernel and
> > https://source.puri.sm/Librem5/linux-next - i just wanted to check that
> > there aren't any objections when the number of packages with patches
> > increases quite a bit."
> >
> > Matthias writes;
> > "First of all, thank you for communicating this early :-) I have no
> > fundamental objections to adding it, you would just need to target
> > "byzantium" or "landing".
> >
> > - Any chance that the kernel changes would break the desktop flavor of
> > PureOS? Or would that be a completely separate kernel package and not
> > merged with the desktop kernel?
>
> It's a separate kernel package at the moment. I don't intend to switch
> to Debian's kernel package until we also do that in Debian itself.
> If you look at page 25 here
>
>   https://git.sigxcpu.org/cgit/talks/2020-debconf-mobile/plain/talk.pdf
>
> you see that the diff is still a bit off but that will be improving
> a lot with 5.10. Also i think we'll run newer major kernel versions
> past bullseye release for some time.
>
> > - You'd need to make sure phone things work on amd64 as well as on
> > arm64 in case they have a convergent base and are used on both form
> > factors. And 3) the PureOS team is tiny so we can't maintain lots and
>
> That was the plan we discusses a while back with Jeremiah. Lots of
> stuff will be adaptive with GNOME 3.38 and for GTK Sadiq worked on
> making that possible:
>
> https://source.puri.sm/Librem5/gtk/-/merge_requests/18
> https://source.puri.sm/Librem5/gtk/-/issues/23

Sounds great!

> > lots of PureOS<->Debian delta. But with the phone team's help I think
> > that's absolutely doable there's also secondary questions, like whether
> > it makes sense to just move shared sources to the pureos group(s) in
> > Gitlab also, I'll have to go through with improving Laniakea's sync
> > overview page to be less cluttered - at the moment it's quite hard to
> > know what needs merging where ;-) (that's actually the thing I am
> > working on at the moment, since that also gets rid of the last bit of D
> > code and makes deploying Laniakea easier).
> >
> > - Laniakea can theoretically also build packages from Git repos -
> > you'll need a list of repos and signed tags of a certain format in
> > them, LK will then verify the tag signature and send the repo to a
> > worker to create a source package out of that's then auto-uploaded.
> > It's quite neat in concept...
> >
> > ....in practice I would need to get rid of the manual list and actually
> > verify signatures (at the moment anything would be built, so that's
> > really not deployable)
>
> That would be cool since i'd like to phase out the route via arm02
> (which allows us to do that for the phone right now) as soon as
> possible - that's also one of the reasons we don't want to allow
> sloppy builds for byzantium.

That would remove the annoying "where is my package now?" question -
because currently arm02 occasionally doesn't upload a thing, or
uploads a thing twice and we have to find out where the package was
lost (it occasionally hanging in NEW also isn't amazing, of course).

> Is there a public bug list for PureOS Infra/Laneakia that once could
> follow to learn what's planned, has been discussed?

There's a bunch of bugs on the Tracker, and ancient meeting notes I
discussed with Zlatan a while back - so the honest answer would
probably be: No, nothing comprehensive exists. Since the Laniakea
project is at Github (as an independent location) I could create some
issues there to track the individual missing features. Or
alternatively on the Tracker (whcih does have the superior bug
tracking and also knows about issue interdependencies).

> > Anyway, that's $future_stuff, let's work on getting the phone pieces
> > into byzantium easily first :-)"
> >
> > Jeremiah writes;
> > - I would like to understand more about the increase in kernel patches;
> > how will they be maintained? What are the expectations of the Librem 5
> > team, is there going to be a transfer of responsibility for the patches
> > from the Librem 5 team to PureOS team?
>
> Kernel packages are separate - kernel maintenance will be the same as
> for amber.
>
> > Our resources are currently thinly stretched across our CI / CD, QA,
> > and support teams, which means adding devices is a heavy lift that
> > needs co-ordination. We don't really have a process for additional
> > devices yet either and there needs to be some thought as we grow given
> > that our Librem 5 traffic might significantly increase load and
> > bandwidth on existing infrastructure. Do we have any data on;
> >
> > - Currently Librem 5 CI (I believe we run Munin on the ARM 02 server
> > no? Do we have output from that which might show load, disk, CPU, etc?
>
> Yes. It's all rather boring load and I/O wise on arm02 since we build very
> few release packages. Bigger packages (gtk, mesa, kernel) take some disk
> but that's not different from what happens for amd64/arm64 right now.

We are building some insane stuff occasionally on basil as part of the
wider Debian archive. An Eigen3 build almost killed the builder once
^^
So I am not that worried about capacity (yet...), we can see how it
goes and expand as needed. Laniakea isn't built to handle CI tasks
well - it possibly could do CI-ish things, but there are other tools
which are much better suited for building every single commit. CD
however is what it should do well.

> > - How about the current instance of Laniakea? Are we running Munin? If
> > not, do we have data on the server from a capacity planning standpoint?
> >
> > - I'd like to know a bit more about the plan - do we intend to use your
> > new process to build from git repos with signed tags Matthias? Or are
> > we going to continue to build on ARM 02 and then upload to Laniakea?
>
> For the 'mass switch' it would be very good to build from git via
> Laneakia so save maintenance and rather use arm02 as another builder
> for Laneakia (but that obviously needs Matthias involvement).

Yes, I will need to invest a lot of time into writing the missing code
(having a second person to review the concept or even the code for
(security) issues would also be helpful).
Ideally I'll write a summary of this.
The scheme in short form is (as currently intended):
* User tags something in Git with a GPG-signed tag
* Git hook sends message to Lighthouse (Laniakea's message relay and
job server) notifying about that fact
* GPG signature of the Git tag is verified against a list of allowed
users (GPG signature needs to be valid, user needs to have permission
to make releases for the particular package, ...)
* On failure, error message is generated and passed back to Lighthouse
and also recorded in the database => Our Matrix bot can yell at the
person who made the mistake, and we can show the invalid tag on a
monitoring web UI
* On success: A job is created with the Git Hash and git reposotory as data
* Laniakea Worker takes job from Lighthouse and builds a source+binary
package & uploads that to the archive (provided the Git repo has a
commit with the hash that the job data contained. Communication
between workers and the master server is encrypted/signed, so you
shouldn't be able to hijack it)
* Once the source package has entered the archive, jobs will be
created for the remaining missing architectures
* Binary-only packages are built & uploaded as usual

> Regarding building from git let me add that we have done some
> prototyping to be able to restrict packages to certain maintainers which
> might be useful when we have more app contributors. I think taking the
> requirements for that into account (without blocking progress) would be
> good.

So far I didn't really take into account that non-project-members or
non-Purism people may use this system. I'm a bit paranoid by that, if
we do something like that their packages should end up in a 3rd-party
separate repository and the builder pool used for the external code
should probably be separated from the one that builds the main PureOS
packages (just in case someone builds malicious stuff that breaks out
of the container or VM somehow, not all of PureOS is compromised).

> That said the 'where to apps go' question is one of the more interesting ones for
> byzantium from my PoV since we side stepped that on amber by just using
> amber-phone and requiring a review from a team member. This raises the
> following questions (not necessarily for today's meeting since that
> should (in my opinion) focus on getting convergent base in byzantium):
>
> - which packaging formats do non-core apps (not necessarily maintained
>   by a PureOs/L5 team member) use? We use deb now. The current ones
>   are all debs https://source.puri.sm/librem5-apps and we might
>   want to continue to support that.
> - Where do uploads go? So will there be a byzantium-apps,
>   byzantium-apps-staging and maybe the same with `-sloppy` or `-preview`
>   for not yet perfect apps? (orthogonal to the packaging format)
>   or should that be more like Ubuntu's PPAs (which i'm not a fan of
>   for this purpose)
> - In case of Debian package: how are we preventing people to upload
>   a patched glibc there?
> - How (and by whom) are permissions managed? (somewhat orthogonal to the
>   packaging format)
> - How is security support for these handled? (at least removal should be
>   triggered on known CVEs) (orthogonal to the packaging format)
> - what is the policy for being allowed to upload an app, how is that
>   being enforced. For librem5-apps we've been basically following
>   DFSG and l5 dev policy (https://storage.puri.sm/librem5/l5-policy/html/)

All very good questions. I am not a huge fan of externally generated
.deb packages, as running them is equal to providing someone else with
full root access to your device. We could sanitize external packaging
(e.g. disallow any maintainer scripts that aren't generated by
debhelper) but that would still be kind of risky. Limiting the allowed
package names should be absolutely possible though - we still couldn't
prevent people from conflicting with a security fix package to
compromise a system and do other nasty stuff though.
I think in the long run, using something like Flatpak would be a safer
bet, but in the short run using dpkg may be easier to implement
(Laniakea has crude Flatpak support, but that's more a prototype than
actually usable at the moment).

Cheers,
    Matthias

-- 
I welcome VSRE emails. See http://vsre.info/