Revert "fix: handle queries on non-existing table gracefully"

[taimoorzaeem]

This reverts commit 390ba19.

Continues #4468. Solves #4613

[steve-chavez]

Helping with the test on #5121

[steve-chavez]

Looking at #3602, the solution for this should be in line with the new "schema cache as best effort", so I don't think we can look at the schema cache for table existence in every case.

So for #3602, the tables should only be looked at when they're part of resource embedding. If it's just a /tbl?select=..&id=eq.1 then that should not look into the schema cache and just go directly into the db, then map the pg error to 404.

[steve-chavez]

For #3697 it should be a similar logic, just let the request fail and go to the db if the insert doesn't use the more advanced features that require the schema cache.

[wolfgangwalther]

So for #3602, the tables should only be looked at when they're part of resource embedding. If it's just a /tbl?select=..&id=eq.1 then that should not look into the schema cache and just go directly into the db, then map the pg error to 404.

Can we really avoid looking at table/columns names in general (without resource embedding)? I don't think so, don't we need this to support domain representations as well?

[taimoorzaeem]

Can we really avoid looking at table/columns names in general (without resource embedding)? I don't think so, don't we need this to support domain representations as well?

Yes, I think we can avoid them in general. That's how it was before #3869, no?

If the table or column doesn't exist we'll get the error from pg, but if they do, then the domain representation is applied automatically by postgrest, right?

[wolfgangwalther]

If the table or column doesn't exist we'll get the error from pg, but if they do, then the domain representation is applied automatically by postgrest, right?

So, you are saying that:

• You create a new table with a domain representation, but don't reload the schema cache, yet.
• You make a first request and get a response without the domain representation applied.
• You reload the schema cache.
• You make a new request with a new response with the domain representation applied.

Right?

You will get two different responses for the same GET request, depending on whether the schema cache was reloaded or not. There was never an intention to return the values without the representation applied.

I don't think we should do this.

[taimoorzaeem]

• You create a new table with a domain representation, but don't reload the schema cache, yet.
• You make a first request and get a response without the domain representation applied.
• You reload the schema cache.
• You make a new request with a new response with the domain representation applied.

Right?

Ah, right right..

There was never an intention to return the values without the representation applied.

I see. So, it looks like schema cache reload is necessary to correctly support all table queries? If that's the case, then this dynamic table creation wouldn't work at all, meaning we can't revert this.

[steve-chavez]

If we cannot do this because of domain representations, maybe we should restrict their application somehow? Like they're only applied on some new Prefer value?

Because for #4613 (comment), all the huge amount of tables are on the same schema. So this cannot be solved by splitting the cache load somehow, like by schema.

[steve-chavez]

You will get two different responses for the same GET request, depending on whether the schema cache was reloaded or not. There was never an intention to return the values without the representation applied.

The above should be fine now that we do schema cache as "best effort", is the job of the developer to ensure responses are maintained and no breaking changes happen. We discussed this somewhere on #4873.

Edit: now that we state this on docs, this can be a fix.

[wolfgangwalther]

The above should be fine now that we do schema cache as "best effort", is the job of the developer to ensure responses are maintained and no breaking changes happen. We discussed this somewhere on #4873.

No, this is a different case. The case we discussed was "the API used to return A, but should return B now" - if the schema cache is stale and still returns A, that's fine as "best effort" / "it's a cache!".

But the case here is, that the developer intends to return A first, then B - but because of the schema cache, the API suddenly returns C - something entirely else.

This is unacceptable.

[steve-chavez]

Any ideas how to fix #4613 then? Given that we know all tables are on the same schema?

• You make a first request and get a response without the domain representation applied.
• You reload the schema cache.
• You make a new request with a new response with the domain representation applied.

That doesn't look any different from changing a column's type.

@mkleczek Perhaps you have some input here?

[mkleczek]

Any ideas how to fix #4613 then? Given that we know all tables are on the same schema?

• You make a first request and get a response without the domain representation applied.
• You reload the schema cache.
• You make a new request with a new response with the domain representation applied.

That doesn't look any different from changing a column's type.

@mkleczek Perhaps you have some input here?

Quick thought: would it be possible to have "load single table (possibly with its relationships)" functionality? I guess it should be fairly quick comparing to full schema reload.

[wolfgangwalther]

That doesn't look any different from changing a column's type.

No, if you change a column's type both the old response and the new response have been valid at some point in time. The best effort schema cache then just boils down to a delay in delivering the new correct response.

This is fundamentally different to responding with something incorrect.

[wolfgangwalther]

Quick thought: would it be possible to have "load single table (possibly with its relationships)" functionality? I guess it should be fairly quick comparing to full schema reload.

In principle, this is what I have been suggesting over various issues now: partial schema cache loads, on demand etc.

The problem with relationships is, that we always need to parse all view definitions to find these relationships, so making this single case fast can be problematic in certain cases.

[mkleczek]

Quick thought: would it be possible to have "load single table (possibly with its relationships)" functionality? I guess it should be fairly quick comparing to full schema reload.

In principle, this is what I have been suggesting over various issues now: partial schema cache loads, on demand etc.

The problem with relationships is, that we always need to parse all view definitions to find these relationships, so making this single case fast can be problematic in certain cases.

Hmm... I wonder if we can filter out unrelated views using pg_depend before parsing definitions.

[wolfgangwalther]

Hmm... I wonder if we can filter out unrelated views using pg_depend before parsing definitions.

It's been a while that I worked on that json transformation query for view definitions, but IIRC I did try that. It ultimately ended up being slower, because the recursive part of the query would have to include the json transformation - whereas the current version does all the json transformations at once and then adds the recursiveness later. Now.. this might be very different for the single table case, so this could actually work.

[steve-chavez]

this is what I have been suggesting over various issues now: partial schema cache loads, on demand etc.

On-demand means not having a full schema cache load done at startup but instead only doing it per-request?

Edit: In theory that could solve this issue, but then it would make the response time of PostgREST too variable (also considering #3212). It would be painful to debug, it's better to pay the price at startup. So I think it's a no-go.

[steve-chavez]

On #2144 (comment), there was the idea of db-fk-embedding = true proposed by @wolfgangwalther. Setting it to false disables relationship detection.

Given that for the case of #4613 there are no relationships, perhaps that could speed up scache load enough and solve it.

[wolfgangwalther]

On-demand means not having a full schema cache load done at startup but instead only doing it per-request?

It depends. It could be a full replacement, doing everything on-demand. It could be the old model, but additionally requesting schema information on the request for tables that are not in the cache. Tons of options.

Edit: In theory that could solve this issue, but then it would make the response time of PostgREST too variable (also considering #3212). It would be painful to debug, it's better to pay the price at startup. So I think it's a no-go.

I mean... do we actually know how much the performance price is of doing schema detection stuff query specific without any schema cache at all?

Without knowing it for real, we can't really tell whether it's worth it. Considering the enormous amount of complexity and challenges with the schema cache, I do wonder whether we should try to remove it entirely and see how much performance that costs. Sure - it will be slower. But surely this additional complexity for all parties is not worth every % of improvement. It still might make sense, because the improvement is huge - or it might not. We can't say without numbers, I think.