On Fri, 2020-08-28 at 09:18 +0200, Guido Günther wrote:

Hi Jeremiah, thanks for moving things over to a more persistent media!

<smiling emoticon here>

On Thu, Aug 27, 2020 at 10:29:04PM -0400, Jeremiah C. Foster wrote:

...

Welcome to a thread about PureOS images for our Librem 5 phone.

Let's start with Guido who knows a thing or two about the Librem 5;

"I'd like to (slowly) start filling in enough of byzantium so we can build images for the phone. First candidates would be a kernel and flash-kernel. We currently keep the sources at https://source.puri.sm/Librem5/flash-kernel and https://source.puri.sm/Librem5/linux-next - i just wanted to check that there aren't any objections when the number of packages with patches increases quite a bit."

Matthias writes; "First of all, thank you for communicating this early :-) I have no fundamental objections to adding it, you would just need to target "byzantium" or "landing".

- Any chance that the kernel changes would break the desktop flavor of PureOS? Or would that be a completely separate kernel package and not merged with the desktop kernel?

It's a separate kernel package at the moment. I don't intend to switch to Debian's kernel package until we also do that in Debian itself. If you look at page 25 here https://git.sigxcpu.org/cgit/talks/2020-debconf-mobile/plain/talk.pdf

you see that the diff is still a bit off but that will be improving a lot with 5.10. Also i think we'll run newer major kernel versions past bullseye release for some time.

So we'll need to maintain kernel packages until 5.10 is released?

...

- You'd need to make sure phone things work on amd64 as well as on arm64 in case they have a convergent base and are used on both form factors. And 3) the PureOS team is tiny so we can't maintain lots and

That was the plan we discusses a while back with Jeremiah. Lots of stuff will be adaptive with GNOME 3.38 and for GTK Sadiq worked on making that possible:

https://source.puri.sm/Librem5/gtk/-/merge_requests/18 https://source.puri.sm/Librem5/gtk/-/issues/23

...

lots of PureOS<->Debian delta. But with the phone team's help I think that's absolutely doable there's also secondary questions, like whether it makes sense to just move shared sources to the pureos group(s) in Gitlab also,

Regarding the merge to Gitlab -- we'll also need to do a "Respects Your Freedom" application at the FSF. For this we'll need the kernel sources, firmware, and documentation all in one place likely. It is a lot of work to review a device like ours since it holds a lot more software than a USB key for example.

...

I'll have to go through with improving Laniakea's sync overview page to be less cluttered - at the moment it's quite hard to know what needs merging where ;-) (that's actually the thing I am working on at the moment, since that also gets rid of the last bit of D code and makes deploying Laniakea easier).

- Laniakea can theoretically also build packages from Git repos - you'll need a list of repos and signed tags of a certain format in them, LK will then verify the tag signature and send the repo to a worker to create a source package out of that's then auto-uploaded. It's quite neat in concept...

....in practice I would need to get rid of the manual list and actually verify signatures (at the moment anything would be built, so that's really not deployable)

That would be cool since i'd like to phase out the route via arm02 (which allows us to do that for the phone right now) as soon as possible - that's also one of the reasons we don't want to allow sloppy builds for byzantium.

Is there a public bug list for PureOS Infra/Laneakia that once could follow to learn what's planned, has been discussed?

Sort of; https://source.puri.sm/pureos/infra You'll find issues for Laniakea here: https://source.puri.sm/pureos/infra/laniakea/-/issues Matthias *may* prefer issues go here: https://github.com/lkorigin/laniakea/issues.

...

Anyway, that's $future_stuff, let's work on getting the phone pieces into byzantium easily first :-)"

Jeremiah writes; - I would like to understand more about the increase in kernel patches; how will they be maintained? What are the expectations of the Librem 5 team, is there going to be a transfer of responsibility for the patches from the Librem 5 team to PureOS team?

Kernel packages are separate - kernel maintenance will be the same as for amber.

Okay, that sounds good. Then we'll need to ensure those who're working on the kernel can upload new packages to Laniakea. I think this is as simple as providing the gpg key of each person who needs to upload to Matthias.

...

Our resources are currently thinly stretched across our CI / CD, QA, and support teams, which means adding devices is a heavy lift that needs co-ordination. We don't really have a process for additional devices yet either and there needs to be some thought as we grow given that our Librem 5 traffic might significantly increase load and bandwidth on existing infrastructure. Do we have any data on;

- Currently Librem 5 CI (I believe we run Munin on the ARM 02 server no? Do we have output from that which might show load, disk, CPU, etc?

Yes. It's all rather boring load and I/O wise on arm02 since we build very few release packages. Bigger packages (gtk, mesa, kernel) take some disk but that's not different from what happens for amd64/arm64 right now.

+1

...

- How about the current instance of Laniakea? Are we running Munin? If not, do we have data on the server from a capacity planning standpoint?

- I'd like to know a bit more about the plan - do we intend to use your new process to build from git repos with signed tags Matthias? Or are we going to continue to build on ARM 02 and then upload to Laniakea?

For the 'mass switch' it would be very good to build from git via Laneakia so save maintenance and rather use arm02 as another builder for Laneakia (but that obviously needs Matthias involvement).

Agreed.

Regarding building from git let me add that we have done some prototyping to be able to restrict packages to certain maintainers which might be useful when we have more app contributors. I think taking the requirements for that into account (without blocking progress) would be good.

+1

That said the 'where to apps go' question is one of the more interesting ones for byzantium from my PoV since we side stepped that on amber by just using amber-phone and requiring a review from a team member. This raises the following questions (not necessarily for today's meeting since that should (in my opinion) focus on getting convergent base in byzantium):

- which packaging formats do non-core apps (not necessarily maintained by a PureOs/L5 team member) use? We use deb now. The current ones are all debs https://source.puri.sm/librem5-apps and we might want to continue to support that.

+1 Currently core and non-core apps are all debs. The PureOS Store when it comes into existence will have flatpaks but I think that Matthias might have flatpak functionality in Laniakea as well.

- Where do uploads go? So will there be a byzantium-apps, byzantium-apps-staging and maybe the same with `-sloppy` or `- preview` for not yet perfect apps? (orthogonal to the packaging format) or should that be more like Ubuntu's PPAs (which i'm not a fan of for this purpose)

- In case of Debian package: how are we preventing people to upload a patched glibc there?

You have to be in the PureOS keyring which Matthias maintains.

- How (and by whom) are permissions managed? (somewhat orthogonal to the packaging format)

See above.

- How is security support for these handled? (at least removal should be triggered on known CVEs) (orthogonal to the packaging format)

A review of Debian's process indicates we might be able to lift their process into ours and use their tool. There would still be the need for a team however for anything that is not in Debian.

- what is the policy for being allowed to upload an app, how is that being enforced. For librem5-apps we've been basically following DFSG and l5 dev policy ( https://storage.puri.sm/librem5/l5-policy/html/)

I can't view that URL.

Since we have same apps at https://source.puri.sm/librem5-apps we could likely use one to testdrive this.

+1 Regards, Jeremiah

On Wed, 2020-09-02 at 14:12 +0200, Guido Günther wrote:

...

...

...

I'll have to go through with improving Laniakea's sync overview page to be less cluttered - at the moment it's quite hard to know what needs merging where ;-) (that's actually the thing I am working on at the moment, since that also gets rid of the last bit of D code and makes deploying Laniakea easier). - Laniakea can theoretically also build packages from Git repos - you'll need a list of repos and signed tags of a certain format in them,

For clarity - do you mean a "git tag" or another type of tag? I think using git tags and signed-off-by lines is standard practice, certainly in kernel development, and I strongly feel we should adopt that.

...

...

...

LK will then verify the tag signature

Sorry to be pedantic but it helps me to understand. LK will verify the git tag and the gpg sig? I can see how verification might be done with the gpg sig, e.g. check the keyring, but the git tag might be harder to "verify".

...

...

...

and send the repo to a worker to create a source package out of that's then auto- uploaded. It's quite neat in concept... ....in practice I would need to get rid of the manual list and actually verify signatures (at the moment anything would be built, so that's really not deployable)

So we have an opportunity to define the policy and process before we put it into code? That sounds like something that is worth discussing now before there are tons of packages coming into the repo. :-)

...

...

That would be cool since i'd like to phase out the route via arm02 (which allows us to do that for the phone right now) as soon as possible - that's also one of the reasons we don't want to allow sloppy builds for byzantium. Is there a public bug list for PureOS Infra/Laneakia that once could follow to learn what's planned, has been discussed?

<snip>

So just to be sure, we *don't* use tracker.pureos.org for that? I was under the impression we'll use that from our discusson on Friday.

It's my preferred solution to use tracker.pureos.org to track issues in Laniakea *and* the various packages that are coming into PureOS. This way we can separate concerns (software development issues in Gitlab, maintenance and delivery issues in Tracker.) If we have consensus I'm strongly in favor of encouraging this as policy. Regards, Jeremiah

Hi, sorry for the delay... Am Fr., 28. Aug. 2020 um 09:19 Uhr schrieb Guido Günther <guido.gunther@puri.sm>:

Hi Jeremiah, thanks for moving things over to a more persistent media!

On Thu, Aug 27, 2020 at 10:29:04PM -0400, Jeremiah C. Foster wrote:

...

Hi,

[Feel free to reply on-list, I just wanted to make sure everyone saw this.]

[Please find attached an invite to a mumble call at 16:00 CEST tomorrow.]

Welcome to a thread about PureOS images for our Librem 5 phone.

Let's start with Guido who knows a thing or two about the Librem 5;

"I'd like to (slowly) start filling in enough of byzantium so we can build images for the phone. First candidates would be a kernel and flash-kernel. We currently keep the sources at https://source.puri.sm/Librem5/flash-kernel and https://source.puri.sm/Librem5/linux-next - i just wanted to check that there aren't any objections when the number of packages with patches increases quite a bit."

Matthias writes; "First of all, thank you for communicating this early :-) I have no fundamental objections to adding it, you would just need to target "byzantium" or "landing".

- Any chance that the kernel changes would break the desktop flavor of PureOS? Or would that be a completely separate kernel package and not merged with the desktop kernel?

It's a separate kernel package at the moment. I don't intend to switch to Debian's kernel package until we also do that in Debian itself. If you look at page 25 here

https://git.sigxcpu.org/cgit/talks/2020-debconf-mobile/plain/talk.pdf

you see that the diff is still a bit off but that will be improving a lot with 5.10. Also i think we'll run newer major kernel versions past bullseye release for some time.

...

- You'd need to make sure phone things work on amd64 as well as on arm64 in case they have a convergent base and are used on both form factors. And 3) the PureOS team is tiny so we can't maintain lots and

That was the plan we discusses a while back with Jeremiah. Lots of stuff will be adaptive with GNOME 3.38 and for GTK Sadiq worked on making that possible:

https://source.puri.sm/Librem5/gtk/-/merge_requests/18 https://source.puri.sm/Librem5/gtk/-/issues/23

Sounds great!

...

lots of PureOS<->Debian delta. But with the phone team's help I think that's absolutely doable there's also secondary questions, like whether it makes sense to just move shared sources to the pureos group(s) in Gitlab also, I'll have to go through with improving Laniakea's sync overview page to be less cluttered - at the moment it's quite hard to know what needs merging where ;-) (that's actually the thing I am working on at the moment, since that also gets rid of the last bit of D code and makes deploying Laniakea easier).

- Laniakea can theoretically also build packages from Git repos - you'll need a list of repos and signed tags of a certain format in them, LK will then verify the tag signature and send the repo to a worker to create a source package out of that's then auto-uploaded. It's quite neat in concept...

....in practice I would need to get rid of the manual list and actually verify signatures (at the moment anything would be built, so that's really not deployable)

That would be cool since i'd like to phase out the route via arm02 (which allows us to do that for the phone right now) as soon as possible - that's also one of the reasons we don't want to allow sloppy builds for byzantium.

That would remove the annoying "where is my package now?" question - because currently arm02 occasionally doesn't upload a thing, or uploads a thing twice and we have to find out where the package was lost (it occasionally hanging in NEW also isn't amazing, of course).

Is there a public bug list for PureOS Infra/Laneakia that once could follow to learn what's planned, has been discussed?

There's a bunch of bugs on the Tracker, and ancient meeting notes I discussed with Zlatan a while back - so the honest answer would probably be: No, nothing comprehensive exists. Since the Laniakea project is at Github (as an independent location) I could create some issues there to track the individual missing features. Or alternatively on the Tracker (whcih does have the superior bug tracking and also knows about issue interdependencies).

...

Anyway, that's $future_stuff, let's work on getting the phone pieces into byzantium easily first :-)"

Jeremiah writes; - I would like to understand more about the increase in kernel patches; how will they be maintained? What are the expectations of the Librem 5 team, is there going to be a transfer of responsibility for the patches from the Librem 5 team to PureOS team?

Kernel packages are separate - kernel maintenance will be the same as for amber.

...

Our resources are currently thinly stretched across our CI / CD, QA, and support teams, which means adding devices is a heavy lift that needs co-ordination. We don't really have a process for additional devices yet either and there needs to be some thought as we grow given that our Librem 5 traffic might significantly increase load and bandwidth on existing infrastructure. Do we have any data on;

- Currently Librem 5 CI (I believe we run Munin on the ARM 02 server no? Do we have output from that which might show load, disk, CPU, etc?

Yes. It's all rather boring load and I/O wise on arm02 since we build very few release packages. Bigger packages (gtk, mesa, kernel) take some disk but that's not different from what happens for amd64/arm64 right now.

We are building some insane stuff occasionally on basil as part of the wider Debian archive. An Eigen3 build almost killed the builder once ^^ So I am not that worried about capacity (yet...), we can see how it goes and expand as needed. Laniakea isn't built to handle CI tasks well - it possibly could do CI-ish things, but there are other tools which are much better suited for building every single commit. CD however is what it should do well.

...

- How about the current instance of Laniakea? Are we running Munin? If not, do we have data on the server from a capacity planning standpoint?

- I'd like to know a bit more about the plan - do we intend to use your new process to build from git repos with signed tags Matthias? Or are we going to continue to build on ARM 02 and then upload to Laniakea?

For the 'mass switch' it would be very good to build from git via Laneakia so save maintenance and rather use arm02 as another builder for Laneakia (but that obviously needs Matthias involvement).

Yes, I will need to invest a lot of time into writing the missing code (having a second person to review the concept or even the code for (security) issues would also be helpful). Ideally I'll write a summary of this. The scheme in short form is (as currently intended): * User tags something in Git with a GPG-signed tag * Git hook sends message to Lighthouse (Laniakea's message relay and job server) notifying about that fact * GPG signature of the Git tag is verified against a list of allowed users (GPG signature needs to be valid, user needs to have permission to make releases for the particular package, ...) * On failure, error message is generated and passed back to Lighthouse and also recorded in the database => Our Matrix bot can yell at the person who made the mistake, and we can show the invalid tag on a monitoring web UI * On success: A job is created with the Git Hash and git reposotory as data * Laniakea Worker takes job from Lighthouse and builds a source+binary package & uploads that to the archive (provided the Git repo has a commit with the hash that the job data contained. Communication between workers and the master server is encrypted/signed, so you shouldn't be able to hijack it) * Once the source package has entered the archive, jobs will be created for the remaining missing architectures * Binary-only packages are built & uploaded as usual

Regarding building from git let me add that we have done some prototyping to be able to restrict packages to certain maintainers which might be useful when we have more app contributors. I think taking the requirements for that into account (without blocking progress) would be good.

So far I didn't really take into account that non-project-members or non-Purism people may use this system. I'm a bit paranoid by that, if we do something like that their packages should end up in a 3rd-party separate repository and the builder pool used for the external code should probably be separated from the one that builds the main PureOS packages (just in case someone builds malicious stuff that breaks out of the container or VM somehow, not all of PureOS is compromised).

That said the 'where to apps go' question is one of the more interesting ones for byzantium from my PoV since we side stepped that on amber by just using amber-phone and requiring a review from a team member. This raises the following questions (not necessarily for today's meeting since that should (in my opinion) focus on getting convergent base in byzantium):

- which packaging formats do non-core apps (not necessarily maintained by a PureOs/L5 team member) use? We use deb now. The current ones are all debs https://source.puri.sm/librem5-apps and we might want to continue to support that. - Where do uploads go? So will there be a byzantium-apps, byzantium-apps-staging and maybe the same with `-sloppy` or `-preview` for not yet perfect apps? (orthogonal to the packaging format) or should that be more like Ubuntu's PPAs (which i'm not a fan of for this purpose) - In case of Debian package: how are we preventing people to upload a patched glibc there? - How (and by whom) are permissions managed? (somewhat orthogonal to the packaging format) - How is security support for these handled? (at least removal should be triggered on known CVEs) (orthogonal to the packaging format) - what is the policy for being allowed to upload an app, how is that being enforced. For librem5-apps we've been basically following DFSG and l5 dev policy (https://storage.puri.sm/librem5/l5-policy/html/)

All very good questions. I am not a huge fan of externally generated .deb packages, as running them is equal to providing someone else with full root access to your device. We could sanitize external packaging (e.g. disallow any maintainer scripts that aren't generated by debhelper) but that would still be kind of risky. Limiting the allowed package names should be absolutely possible though - we still couldn't prevent people from conflicting with a security fix package to compromise a system and do other nasty stuff though. I think in the long run, using something like Flatpak would be a safer bet, but in the short run using dpkg may be easier to implement (Laniakea has crude Flatpak support, but that's more a prototype than actually usable at the moment). Cheers, Matthias -- I welcome VSRE emails. See http://vsre.info/