Hi!
There's been some discussion amongst various folks that our communication culture can be suboptimal at times. I thought I would try and encourage greater use of our mailing list to compliment our chat channel and this irregular (but hopefully frequent) email is a collection of "bits" in the spirit of a Debian's "Bits from . . . "
In PureOS we have a number of current topics which I'll enumerate here;
1. Reproducible builds 2. Supporting Pureboot and Purism hardware 3. Continuous delivery of PureOS 4. PureOS store (and application discovery)
These are high level topics which are organized in no particular priority. I thought that I'd try to have updates in each of these areas and to tie in the work being done in other parts of Purism that may have an impact on those topics expanding the topic list where necessary.
In regard to reproducible builds, Chris Lamb and I have been meeting regularly to discuss PureOS builds and their reproducibility. We've managed to create a automated scan of PureOS builds via diffoscope;
http://45.33.86.90/pureos-8.0-images.html
That page is created automatically each day from two builds done on that host. I believe that we'll be able to say that the PureOS ISO is reproducible when the builds are byte-for-byte identical. We hold some documentation here;
https://tracker.pureos.net/w/development/reproducible_isos/
The next topic I'd like to talk about is the delta between PureOS and Debian. I mention this because I feel it fits into the Continuous Delivery topic above and is something that I've been talking to Matthias and Gunther about, albeit breifly. The reason to understand the delta between PureOS and Debian is to be able to calculate roughly the effort needed to move our currently Phone OS from Debian to PureOS. Hopefully being able to specify what is different between the two distros will enable us to create a PureOS image that matches the versions and/or the functionality of the work being done in the L5 team. I'll send a separate email regarding this topic as it is important and we may need to gather a number of people to participate.
Work is ongoing on the PureOS Store. Rodolfo has been working recently on adaptive design and it looks good. More work to be done of course. I'm working on getting a round trip for Animatch and other flatpaks, like Lollypop. At this stage I'm looking to export the built flatpak to the server and see if I can't use the "install" button to install it on the devkit. More updates soon.
We may have an additional colleague if the org chart is any indication, I think that will be a welcome addition. I'll let everyone know more as soon as I know more.
Well, just a medium length email this time, if they get too long they tend to get a bit boring. There's still lots to write about but hopefullly another "Bits from PureOS" will be coming soon to an inbox near you.
Cheers,
Jeremiah
Dear Jeremiah,
In regard to reproducible builds, Chris Lamb and I have been meeting regularly to discuss PureOS builds and their reproducibility.
[…]
I believe that we'll be able to say that the PureOS ISO is reproducible when the builds are byte-for-byte identical.
That is absolutely correct, but I would go further in that I would underline that is no other definition of "reproducible" from our point of view.
One remaining "policy" question here is what value we use for the SOURCE_DATE_EPOCH environment variable:
https://reproducible-builds.org/docs/source-date-epoch/
As in, a build of the PureOS issue needs to have a single, fixed, date associated with it. This can be as simple as the date of the relevant Git commit or tag that was built from but we need to decide on where it comes from, one way or another.
Then in our test framework or if anyone else wished to reproduce the same .ISO themselves, we would export that very SOURCE_DATE_EPOCH value to the environment in all builds of that specific version.
Various tools and utilities that I have already patched upstream to detect & use that value and it would make a bunch of stuff immediately reproducible. For example, casper which is — at the time of writing — is introducing variances between the builds. It would use that value in its metadata rather than the current date/time etc.
Well, just a medium length email this time, if they get too long they tend to get a bit boring.
Beethoven's second-most famous letter contains the phrase: "only a few words today and with a pencil" … so you are in good company in sending shorter messages. What I'm really trying to say here is please keep sending these mails.
[…] encourage greater use of our mailing list to compliment our chat channel and this irregular (but hopefully frequent) email is a collection of "bits" in the spirit of a Debian's "Bits from . . . "
Huzzah! Indeed, this "Debian" thing you refer to sounds like they have a bunch of good ideas. How can I find out more about it? ;)
Best wishes,
On Thu, 2019-07-04 at 18:00 -0300, Chris Lamb wrote:
Dear Jeremiah,
In regard to reproducible builds, Chris Lamb and I have been meeting regularly to discuss PureOS builds and their reproducibility.
[…]
I believe that we'll be able to say that the PureOS ISO is reproducible when the builds are byte-for-byte identical.
That is absolutely correct, but I would go further in that I would underline that is no other definition of "reproducible" from our point of view.
+1
One remaining "policy" question here is what value we use for the SOURCE_DATE_EPOCH environment variable:
https://reproducible-builds.org/docs/source-date-epoch/
As in, a build of the PureOS issue needs to have a single, fixed, date associated with it. This can be as simple as the date of the relevant Git commit or tag that was built from but we need to decide on where it comes from, one way or another.
While I want to implement this as soon as possible, if not sooner, I want to confer with Matthias about where this ought to be set. I'm currently using debspawn but that wraps a container around Debian's live-build I believe. The build script recommended in the PureOS documentation requires cloning make-live, so perhaps the latest git commit can be used for source date epoch? Line 24 in the URL below;
https://source.puri.sm/pureos/infra/make-live/blob/master/auto/config
The question otherwise is where do I put SOURCE_DATE_EPOCH? I assume in some makefile as a variable.
Then in our test framework or if anyone else wished to reproduce the same .ISO themselves, we would export that very SOURCE_DATE_EPOCH value to the environment in all builds of that specific version.
Various tools and utilities that I have already patched upstream to detect & use that value and it would make a bunch of stuff immediately reproducible. For example, casper which is — at the time of writing — is introducing variances between the builds. It would use that value in its metadata rather than the current date/time etc.
This makes me suspicious that I can just export SOURCE_DATE_EPOCH in a top level shell file and everything'll work?
Well, just a medium length email this time, if they get too long they tend to get a bit boring.
Beethoven's second-most famous letter contains the phrase: "only a few words today and with a pencil" … so you are in good company in sending shorter messages. What I'm really trying to say here is please keep sending these mails.
Will do!
[…] encourage greater use of our mailing list to compliment our chat channel and this irregular (but hopefully frequent) email is a collection of "bits" in the spirit of a Debian's "Bits from . . . "
Huzzah! Indeed, this "Debian" thing you refer to sounds like they have a bunch of good ideas. How can I find out more about it? ;)
It's complex. It's anarchic. You have to be technical. But it returns on your investment of time and energy a thousand fold - in bug reports.
Best,
Jeremiah
Hi Jeremiah,
One remaining "policy" question here is what value we use for the SOURCE_DATE_EPOCH environment variable
[…]
I'm currently using debspawn but that wraps a container around Debian's live-build I believe. The build script recommended in the PureOS documentation requires cloning make-live, so perhaps the latest git commit can be used for source date epoch? Line 24 in the URL below;
https://source.puri.sm/pureos/infra/make-live/blob/master/auto/config
I fear you might be slightly misunderstanding this environment variable; it shouldn't be hardcoded into some repo and nor should it depend on whether we are using debspawn, etc. Rather, it should reflect the repo or the "state of the world" itself.
Thus using the date from the HEAD commit of:
https://source.puri.sm/pureos/infra/make-live
… seems to be a good start for now. We can always change it later, so we need not spend many brain cycles beyond this on this topic.
The question otherwise is where do I put SOURCE_DATE_EPOCH? I assume in some makefile as a variable.
You should export it in your build script/cron that is doing the two builds, or otherwise ensure that it is exported to the build process. Everything "downstream" from there in the build process should not care or really even know about it.
I believe:
$ git --git-dir=/path/to/your/pureos/infra/make-live/.git log -1 --pretty=%ct
… will return the value we would use.
Huzzah! Indeed, this "Debian" thing you refer to sounds like they have a bunch of good ideas. How can I find out more about it? ;)
It's complex. It's anarchic. You have to be technical. But it returns on your investment of time and energy a thousand fold - in bug reports.
Oh? Nah, I think I'll give it a miss…
Best wishes,
On Fri, 2019-07-05 at 16:01 -0300, Chris Lamb wrote:
Hi Jeremiah,
One remaining "policy" question here is what value we use for the SOURCE_DATE_EPOCH environment variable
[…]
I'm currently using debspawn but that wraps a container around Debian's live-build I believe. The build script recommended in the PureOS documentation requires cloning make-live, so perhaps the latest git commit can be used for source date epoch? Line 24 in the URL below;
https://source.puri.sm/pureos/infra/make-live/blob/master/auto/config
I fear you might be slightly misunderstanding this environment variable;
That is likely the case. ;-)
it shouldn't be hardcoded into some repo
I didn't mean it should be hardcoded into a repo, I was rather trying to find a relevant timestamp that might serve as the source of the environment variable. I thought that using something like a git commit might show the "state of the world" in a relevant fashion, i.e. the last commit to a given repo.
and nor should it depend on whether we are using debspawn, etc.
I was musing about whether and how environment variables get passed along across various boundaries (container, ssh).
Rather, it should reflect the repo or the "state of the world" itself.
I'm uncertain as to exactly what "state of the world" means in this context.
Thus using the date from the HEAD commit of:
https://source.puri.sm/pureos/infra/make-live
… seems to be a good start for now. We can always change it later, so we need not spend many brain cycles beyond this on this topic.
Sounds great. I'll reuse this. :-)
The question otherwise is where do I put SOURCE_DATE_EPOCH? I assume in some makefile as a variable.
You should export it in your build script/cron that is doing the two builds, or otherwise ensure that it is exported to the build process.
Excellent, I'll start there and test to see if it is seen by the build process.
Everything "downstream" from there in the build process should not care or really even know about it.
Understood. Everything downstream should just "consume" that environment variable.
I believe:
$ git --git-dir=/path/to/your/pureos/infra/make-live/.git log -1 --pretty=%ct
… will return the value we would use.
Thank you.
Huzzah! Indeed, this "Debian" thing you refer to sounds like they have a bunch of good ideas. How can I find out more about it? ;)
It's complex. It's anarchic. You have to be technical. But it returns on your investment of time and energy a thousand fold - in bug reports.
Oh? Nah, I think I'll give it a miss…
Really? I think they could use someone like you. :D
Hi Jeremiah,
I was musing about whether and how environment variables get passed along across various boundaries (container, ssh).
Indeed. If anything, this is what you may encounter problems with, particularly as containers are designed to be, well, not influenced by their outside environment...
Obvious workarounds (if you need them) that I end up using all the time here are shell "tricks" like explicitly exporting stuff with env(1):
- start-container build-script.sh + start-container env SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}" build-script.sh
... and throwing a whole bunch of "echo SDE=$SOURCE_DATE_EPOCH" around is probably the best way to find which barrier is not being crossed.
Best wishes,
On Tue, 2019-07-09 at 16:20 -0300, Chris Lamb wrote:
Hi Jeremiah,
I was musing about whether and how environment variables get passed along across various boundaries (container, ssh).
Indeed. If anything, this is what you may encounter problems with, particularly as containers are designed to be, well, not influenced by their outside environment...
I sent a merge request to Matthias for this: https://source.puri.sm/-/ide/project/pureos/infra/make-live/merge_requests/1...
- Jeremiah
Obvious workarounds (if you need them) that I end up using all the time here are shell "tricks" like explicitly exporting stuff with env(1):
- start-container build-script.sh
- start-container env SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
build-script.sh
... and throwing a whole bunch of "echo SDE=$SOURCE_DATE_EPOCH" around is probably the best way to find which barrier is not being crossed.
Best wishes,
On Thu, 2019-07-11 at 20:34 -0400, Jeremiah C. Foster wrote:
On Tue, 2019-07-09 at 16:20 -0300, Chris Lamb wrote:
Hi Jeremiah,
I was musing about whether and how environment variables get passed along across various boundaries (container, ssh).
Indeed. If anything, this is what you may encounter problems with, particularly as containers are designed to be, well, not influenced by their outside environment...
I sent a merge request to Matthias for this: https://source.puri.sm/-/ide/project/pureos/infra/make-live/merge_requests/1...
And then, upon further reflection, I merged my own merge request. -_- I know it is ridiculous, but I wanted to test the change and my debspawn runners are going to be fired off shortly and I pull from Gitlab so . . .
- Jeremiah
- Jeremiah
Obvious workarounds (if you need them) that I end up using all the time here are shell "tricks" like explicitly exporting stuff with env(1):
- start-container build-script.sh
- start-container env SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
build-script.sh
... and throwing a whole bunch of "echo SDE=$SOURCE_DATE_EPOCH" around is probably the best way to find which barrier is not being crossed.
Best wishes,
PureOS-project mailing list PureOS-project@lists.puri.sm https://lists.puri.sm/listinfo/pureos-project
Am Fr., 12. Juli 2019 um 02:38 Uhr schrieb Jeremiah C. Foster jeremiah.foster@puri.sm:
On Thu, 2019-07-11 at 20:34 -0400, Jeremiah C. Foster wrote:
On Tue, 2019-07-09 at 16:20 -0300, Chris Lamb wrote:
Hi Jeremiah,
I was musing about whether and how environment variables get passed along across various boundaries (container, ssh).
Indeed. If anything, this is what you may encounter problems with, particularly as containers are designed to be, well, not influenced by their outside environment...
I sent a merge request to Matthias for this: https://source.puri.sm/-/ide/project/pureos/infra/make-live/merge_requests/1...
And then, upon further reflection, I merged my own merge request. -_- I know it is ridiculous, but I wanted to test the change and my debspawn runners are going to be fired off shortly and I pull from Gitlab so . . .
That's fine - I generally think merging trivial changes without much review isn't bad :-)
Cheers, Matthias
On Fri, 2019-07-12 at 10:25 +0200, Matthias Klumpp wrote:
Am Fr., 12. Juli 2019 um 02:38 Uhr schrieb Jeremiah C. Foster jeremiah.foster@puri.sm:
On Thu, 2019-07-11 at 20:34 -0400, Jeremiah C. Foster wrote:
On Tue, 2019-07-09 at 16:20 -0300, Chris Lamb wrote:
Hi Jeremiah,
I was musing about whether and how environment variables get passed along across various boundaries (container, ssh).
Indeed. If anything, this is what you may encounter problems with, particularly as containers are designed to be, well, not influenced by their outside environment...
I sent a merge request to Matthias for this: https://source.puri.sm/-/ide/project/pureos/infra/make-live/merge_requests/1...
And then, upon further reflection, I merged my own merge request. -_- I know it is ridiculous, but I wanted to test the change and my debspawn runners are going to be fired off shortly and I pull from Gitlab so . . .
That's fine - I generally think merging trivial changes without much review isn't bad :-)
That's good to hear. I realize that merging without code review is generally a bad practice and I will certainly try to avoid it in the future, my impatience got the best of me this time. :-/
Cheers,
Jeremiah
-
Chris Lamb -
Jeremiah C. Foster -
Matthias Klumpp