Platform Spec is ambiguous about usage of `extender`

[c0d1ngm0nk3y]

We want to adapt our platform to make use of extensions, but the usage of the extender is not that clear.

If using extender, the platform MUST execute extender in either or both of: the build environment, the run environment

In addition with this comment, it seems like the extender must be called from the build image when called with --build and from the run image when called with --run. But the either or both in the spec is a bit confusing.

• If so - what is the background for calling the extender (run) in the context of the run image? Both detect and generate are already executed and "only" executing kaniko is still missing and the image should not make any difference there, right?

Background of this question is that it would complicate things quite a bit, e.g.:

• Calling it from within the creator, see issue
• Implementing a platform from within a container, e.g. cnbBuild

The usage of the extender is not clearly specified, imho.

• extender (build) MUST be called regardless of any extension present since it includes the build
• extender (run) MUST NOT be called if there is no run extension detected. At least judging by the reference implementation. Wouldn't it be easier if the extender (run) can always be called and might do nothing?

[c0d1ngm0nk3y]

related comment:

buildpacks/lifecycle#1155 (comment)
buildpacks/lifecycle#1155 (comment)

[c0d1ngm0nk3y]

@natalieparellano Thanks for the explanation. That helps to understand the background.

But this causes 2 questions:

• This only works if the platform can start containers. Our platform runs in a container, so currently, we are using creator. Would it be possible to enhance the lifecycle that the platform can decide if this specific kaniko feature is used or not? It seems like an optimisation, right?

• Naively, we switch in a poc from using creator to call the phases individually. So that we were able to test extensions. But with the mentioned details for extender (run), we are not spec conform. We called the extender (run) from the builder, but the resulting run image not only had the extension applied, but was based on the run NOT the build image. Shouldn't that not have worked following your explanation?

I think the extender is still considered optional. But, we should make it clear in the spec that the creator does not call the extender.

What does optional mean in this context? So calling the creator is no longer spec conform, but you can call if you know there are no extensions?

[natalieparellano]

Would it be possible to enhance the lifecycle that the platform can decide if this specific kaniko feature is used or not?

Maybe, and in fact we have talked about it in the past. This would require some discussion. I think the main reasons not to do it would be security (potentially exposing registry credentials to buildpack/extension code) and complexity.

We called the extender (run) from the builder, but the resulting run image not only had the extension applied,

The quoted spec line should probably read something like:

If using extender, the platform MUST execute extender in either or both of: the build environment (if extending the build-time base image), the run environment (if extending the runtime base image)

So calling the creator is no longer spec conform, but you can call if you know there are no extensions?

Right, you can only use the creator if there are no extensions. Potentially, we can make this "if there are no extensions that extend the runtime base image" i.e., extending the build-time base image should be possible (we just didn't implement it yet). The creator is defined as

Running creator SHALL be equivalent to running detector, analyzer, restorer, builder and exporter in order with identical inputs where they are accepted, with the following exceptions.

Which is true, so in that sense the creator is spec compliant.

[loewenstein]

• We called the extender (run) from the builder, but the resulting run image not only had the extension applied, but was based on the run NOT the build image. Shouldn't that not have worked following your explanation?

The most interesting question imho is that it apparently still works - but shouldn't, if I understand your explanation correctly.

[natalieparellano]

it apparently still works

I think it would depend on the differences between the build and run images - see for example buildpacks/lifecycle#1285

It MAY work, but it is not guaranteed to work 😅

[c0d1ngm0nk3y]

It MAY work, but it is not guaranteed to work 😅

We digged a little deeper to better understand what is going on. So it kind of worked in our case since the packages of the run image are a subset of the packages of the build image. But even with that, it is not guaranteed to work (e.g. installing something that is already present on the build image).

And also we realized starting the extender from within the same container would expose /layers and /workspace to be changed.

So we came to the conclusion that we won't be able to support extensions in our platform since we are already running in a container.

@natalieparellano What are the future plans for extensions? As long as there are optional, we are still spec conform. Will extension support be mandatory for platforms in the future?

[natalieparellano]

@c0d1ngm0nk3y this probably warrants further discussion, but I think there is a way around this, albeit one that requires further work.

Even if extensions are never "required", it could lead to a bifurcation of the ecosystem if some builders require extensions for order resolution to succeed. Ultimately, extensions are a way to satisfy build plan requires from buildpacks, and if there are no extensions (or running extend isn't possible) then those requires must be satisfied by pre-installing them in the base image.

In theory, one could write an extension that introspects the packages on the run image during detect and outputs provides with the run.Dockerfile itself being a no-op. There are two problems here:

• Such an extension would need to be run (during detect) in the context of the run image, whereas today it is run in the context of the build image (this is harder to solve)
• The lifecycle itself would need to determine that all run.Dockerfiles are no-ops and signal to the platform that it should skip run image extension entirely (this is easy to solve)

The missing link here is that base images today have no way of declaring the requirements that they satisfy, ever since we got rid of mixins. Mixins had their own problems (the stack ID and coupling of build & run images together introduced a bunch of unnecessary complication) but we probably need some way for base images to statically declare the packages they contain, so the lifecycle can read that off of the run image and provide it to extensions during detect.

All that said, I wonder how many folks are planning to use run image extension? Most of the use cases I have heard involve some combination of build image extension and run image switching. It might be worth conducting a survey of our major buildpack providers to find out what is planned. Perhaps we could do away with run image extension entirely (or make it a very limited special case) and save ourselves a bunch of trouble.

[dmikusa]

All that said, I wonder how many folks are planning to use run image extension?

I haven't yet used it for anything, but I would like to.

Most of the use cases I have heard involve some combination of build image extension and run image switching.

This works nicely for problems where you have a limited set of things to switch between in the run image, like say three different Java major version branches.

There are cases, for example, PHP extensions that require C-shared libraries to run, where there are too many permutations. Using a buildpack extension to dynamically add those at runtime allows you to be more fine-grained about what gets pulled into the run image. Don't need PHP extension XYZ, ok we don't install those packages. Need PHP extension ABC, ok we'll install its required shared libraries. You could in theory do this with runtime image switching, but it would be a nightmare of different combinations and in practice, I don't think you'd ever get beyond having a few different extension bundles.

[natalieparellano]

we probably need some way for base images to statically declare the packages they contain

Perhaps an SBOM in one of the standardized formats supported by the buildpack? We could error if we cannot find an SBOM for the run image (we could look in the registry using tag-based discover, etc or the platform could provide it)

[loewenstein]

All that said, I wonder how many folks are planning to use run image extension? Most of the use cases I have heard involve some combination of build image extension and run image switching. It might be worth conducting a survey of our major buildpack providers to find out what is planned. Perhaps we could do away with run image extension entirely (or make it a very limited special case) and save ourselves a bunch of trouble.

Mainly, I see a case for statically optimised run images, as in knowing that Java has less native dependencies that let's say the Node engine, there could be different run images per language / runtime.
This can grow pretty quickly and get out of hands with just too many combinations to solve statically.

I could see the need to dynamically optimise run images like @dmikusa mentions, which would right now not be possible with our Piper/Jenkins based platform. Luckily, I haven't seen the need yet for Java, Node or Python - but if this comes up we will have a problem indeed.

Independently, a declarative way for run images to participate as a provider in the build plan definitely makes sense and I like the idea to use a standard format like the sbom variants that are anyway already supported in the CNV ecosystem.