multi-stage build

[flaviut]

Using a multi-stage build like this has the advantage of not invalidating caches for dependencies that don't change.

However, I don't think that it's a good idea to have this code in a different repo. I think it only makes the whole CI process more unnecessarily complicated.

Testing: the build passes, I have not tried running the AppImage. I would not consider this ready to review on anything but the highest level; does this general approach look interesting?

Specific stages can be pulled out into their own tags: docker build --target openscad-appimage-x86_64-base -t openscad/appimage-x86_64-base . for example will tag the base image.

[flaviut]

@kintel can you take a look and let me know what you think about the high-level idea of using multi-stage builds here?

Also, do you have any opposition to moving these docker builds into the main openscad repo instead of having them in a separate place?

[flaviut]

If this was built as part of the main openscad repo, I would change this to

Suggested change

	RUN curl -L "https://github.com/openscad/openscad/archive/${OPENSCAD_COMMIT}.tar.gz" | tar -xz --strip-components=1 && \
	bash ./scripts/uni-get-dependencies.sh && \
	apt-get clean && \
	rm -rf /var/lib/apt/lists/*
	ADD ./scripts/uni-get-dependencies.sh ./scripts/check-dependencies.sh

that means that openscad-appimage-x86_64-base would only be rebuilt if either of those two files changed.

[kintel]

I'm not very familiar with the Docker setup, so I need some time to catch up.
In terms of migrating dockerfiles to the main repo, @t-paul may have some opinions.

[t-paul]

I'm not a huge fan of putting all into the main repo, but lets see how that goes. Right now I'm not sure what the direction things are supposed to go in general. A slightly outdated overview is in https://github.com/openscad/openscad/wiki/CI-Setup

So to go forward how about:

1. move the AppImage Docker into the main repo, but have the directory structure such that other Dockerfiles can be added easily
2. define the To-Be build/CI setup covering the reshuffling of things and the new release build stuff done recently
3. do all the changes defined in 2) and finish the automatic release setup

I'll try to draft something in https://github.com/openscad/openscad/wiki/CI-Setup-(new) based on my current understanding (I may have missed some recent changes, so it will need a couple of iterations).

[flaviut]

That sounds about right. My thinking is for the full release process to be in a single CI workflow that runs end-to-end on a regular basis.

Two services, which means removing OBS and Concourse from the picture:

• Github Actions: would stay as-is, responsible for running dev builds and tests
• CircleCI: responsible for building releases and release infra (like docker base images)

Overall workflow as I'd like to see it:

graph TD
    publish-appimage-base-aarch64 --> publish-appimage-base-manifest
    publish-appimage-base-x86_64 --> publish-appimage-base-manifest
    publish-appimage-base-manifest --> appimage-x86_64
    publish-appimage-base-manifest --> appimage-aarch64

    publish-wasm-base --> openscad-wasm-node
    publish-wasm-base --> openscad-wasm-web

    publish-mxe-base-x86_64 --> openscad-windows-x86_64
    publish-mxe-base-aarch64 --> openscad-windows-aarch64

    openscad-macos

    publish-openscad-aarch64 --> publish-openscad-manifest
    publish-openscad-x86_64 --> publish-openscad-manifest
    create-tarball --> validate-tarball-build

Loading

Notes:

1. I think trying to build from the tarball is more trouble than it's worth. The only difference between git and tarball are the version.txt and commit.txt, and we only really need a single linux build there, with no artifacts, to validate that they work.
2. Since there's no current windows-aarch64 release, I think that's worth punting until a future release.

[kintel]

Re. tarball - I think it's worth the effort. We already need to validate the tarball, and it feels cleaner to build actual releases from the actual source package. Ideally, the result shouldn't differ, but it's just too possible that something may go wrong in the future (git dependency sneaking in, source packaging not picking up the right files, submodule issues). ..and the trouble is a pretty small amount of CircleCI config.