Hi,
I impatiently added the SOURCE_DATE_EPOCH to the version of make-live that we use to build PureOS images. It seems to have made a difference though not completely eliminated our blockers to reproducibility; http://45.33.86.90/pureos-8.0-images.html
While the byte-for-byte difference now is much smaller, and the date is consistent across builds in the ISO at least, other parts of the build are getting the date from the file system I believe.
For example, under the section casper/filesystem.squashfs[0.] unsqaushfs -s {} says;
2 Creation·or·last·append·time·Fri·Jul·12·07:12:11·2019 2 Creation·or·last·append·time·Fri·Jul·12·05:11:57·2019
The leads me to believe that I need to ensure that SOURCE_DATE_EPOCH is being seen when the squashfs file system is written using Lamby's debug method of printing "echo SDE=$SOURCE_DATE_EPOCH" in various places to determine if it's being seen in each build context.
After that I think the next step, or even a step to take anyway, is to run strip-nondeterminism over the resulting ISOs[1.]
0. http://45.33.86.90/pureos-8.0-images.html#casper-filesystem.squashfs 1. https://packages.debian.org/sid/strip-nondeterminism
Feedback, corrections most welcome.
Regards,
Jeremiah
Hi Jeremiah,
I impatiently added the SOURCE_DATE_EPOCH to the version of make-live that we use to build PureOS images. It seems to have made a difference
[…]
The leads me to believe that I need to ensure that SOURCE_DATE_EPOCH is being seen when the squashfs file system is written
This is somewhat of non-sequitur to my ears — if it made any difference it surely is being seen in the "last mile" and thus relevant build context? (Regardless, sprinkling in a few debug statements makes sense just as a reassurance so I would go ahead and do this anyway.)
After that I think the next step, or even a step to take anyway, is to run strip-nondeterminism over the resulting ISOs[1.]
Alas, we would part ways here in that I would be against running this. I doubt that would have the desired effect and nor would be a clean and sustainable route going forward.
To elaborate, strip-nondeterminism is not a clairvoyant magic wand for cleaning stuff up, it's more of a surgical strike against quite- specific problems that are currently out-of-scope for me to fix in upstream toolchains right now. That is to say, in an ideal world the tool would be deprecated and removed. Give us time. :)
Just glancing at the diff in order that you can get another "go" at this before the start of the weekend: my gut feel is that we are building with versions of tools that do not have various patches designed to make various outputs reproducible. (Just as use one relevant example, I have made patches to mksquashfs and casper that may not be present in the versions you are using.)
This may (or may not…) be in addition to the aforementioned SOURCE_DATE_EPOCH not being present in the right contexts....
Best wishes,
On Fri, 2019-07-12 at 13:11 -0300, Chris Lamb wrote:
Hi Jeremiah,
I impatiently added the SOURCE_DATE_EPOCH to the version of make- live that we use to build PureOS images. It seems to have made a difference
[…]
The leads me to believe that I need to ensure that SOURCE_DATE_EPOCH is being seen when the squashfs file system is written
This is somewhat of non-sequitur to my ears — if it made any difference it surely is being seen in the "last mile" and thus relevant build context? (Regardless, sprinkling in a few debug statements makes sense just as a reassurance so I would go ahead and do this anyway.)
After that I think the next step, or even a step to take anyway, is to run strip-nondeterminism over the resulting ISOs[1.]
Alas, we would part ways here in that I would be against running this. I doubt that would have the desired effect and nor would be a clean and sustainable route going forward.
To elaborate, strip-nondeterminism is not a clairvoyant magic wand for cleaning stuff up, it's more of a surgical strike against quite- specific problems that are currently out-of-scope for me to fix in upstream toolchains right now. That is to say, in an ideal world the tool would be deprecated and removed. Give us time. :)
Okay, I'll avoid it. :-) I did a naive run and it produced no discernable effects so the wand was out of magic.
Just glancing at the diff in order that you can get another "go" at this before the start of the weekend: my gut feel is that we are building with versions of tools that do not have various patches designed to make various outputs reproducible. (Just as use one relevant example, I have made patches to mksquashfs and casper that may not be present in the versions you are using.)
I'm on stretch on the build machine, I'll upgrade to buster. :-)
This may (or may not…) be in addition to the aforementioned SOURCE_DATE_EPOCH not being present in the right contexts....
We shall find out.
Cheers,
Jeremiah
Jeremiah Foster wrote:
To elaborate, strip-nondeterminism is not a clairvoyant magic wand for cleaning stuff up
[…]
Okay, I'll avoid it. :-) I did a naive run and it produced no discernable effects so the wand was out of magic.
Patches welcome. ;-)
Best wishes,
-
Chris Lamb -
Jeremiah C. Foster