Converts Protocol Buffers schemas (.proto: proto2, proto3 and editions) into
AdHoc protocol descriptions (.cs).
The package becomes the namespace, a message a pack, a service an interface in a connection, an rpc an RPC method, a
streaming rpc becomes an actor. What the .proto says about the values is kept as AdHoc says it: the varint
its author chose for a field becomes [X] or [A], a Timestamp becomes a DateTime, a wrapper becomes T?,
a oneof becomes a pack of optional alternatives. What describes protobuf's own wire - field numbers, packing -
is left behind: AdHoc lays out its own wire.
The converter is one C# program with no dependencies: it has its own reader of the protobuf language, runs locally and sends nothing anywhere. It is the converter that used to be built into AdHocAgent and the code generator, now a project of its own and a member of the family of converters: it shares their conventions (CONVENTIONS.md) and references nothing outside this folder.
- Protocol Buffers, the language: proto3, proto2, editions; the grammar: proto3, proto2, edition 2023
- The well-known types: https://protobuf.dev/reference/protobuf/google.protobuf/
- gRPC, the four kinds of a method: https://grpc.io/docs/what-is-grpc/core-concepts/
- The reference implementation of the language,
protoc: https://github.com/protocolbuffers/protobuf - AdHoc protocol, the manual: https://github.com/AdHoc-Protocol/AdHoc-protocol
Where the samples come from (fetch-samples.sh downloads them, the upstream layout is kept so that imports
resolve). samples/sources.txt gives the original page of every file, and the header of
every description links the originals it was made from:
Folder of samples/ |
Source |
|---|---|
grpc/ |
gRPC examples, the health checking protocol and server reflection: https://github.com/grpc/grpc/tree/master/examples/protos · https://github.com/grpc/grpc-proto |
protobuf/ |
protobuf's own schema, its test schemas and its example: https://github.com/protocolbuffers/protobuf |
googleapis/ |
common types of Google APIs (status, error details, date, money), ByteStream, HTTP annotations: https://github.com/googleapis/googleapis |
etcd/ |
the etcd v3 API: https://github.com/etcd-io/etcd/tree/main/api |
containerd/ |
containerd services (containers, events): https://github.com/containerd/containerd/tree/main/api |
opentelemetry/ |
OpenTelemetry (OTLP): traces, metrics, logs and the services that export them: https://github.com/open-telemetry/opentelemetry-proto |
kubernetes/ |
the Container Runtime Interface, between the kubelet and a container runtime: https://github.com/kubernetes/cri-api |
arrow/ |
Apache Arrow Flight: https://github.com/apache/arrow/blob/main/format/Flight.proto |
cloudevents/ |
the CloudEvents format: https://github.com/cloudevents/spec |
route_guide.proto of the gRPC examples (111 lines) and the AdHoc/route_guide.cs it generates (193 lines),
both abridged here. Full files: the original
and the result.
Protocol Buffers
service RouteGuide {
rpc GetFeature(Point) returns (Feature) {}
rpc ListFeatures(Rectangle) returns (stream Feature) {}
rpc RecordRoute(stream Point) returns (RouteSummary) {}
rpc RouteChat(stream RouteNote) returns (stream RouteNote) {}
}
message Point {
int32 latitude = 1;
int32 longitude = 2;
}
message Feature {
// The name of the feature.
string name = 1;
// The point where the feature is detected.
Point location = 2;
}AdHoc protocol description
namespace routeguide { // the package
public interface route_guide { // the file
public class Point {
[X] int latitude; // `int32` is a varint: [X] spends the saving on both signs
[X] int longitude;
}
public class Feature {
/// <summary>The name of the feature.</summary>
string name;
/// <summary>The point where the feature is detected.</summary>
Point location;
}
interface ClientServerConnection : Connects<Client, Server> {
public interface RouteGuide { // the service
(L____________, Feature) GetFeature(Point req);
public interface ListFeatures : Actor { // the answer is a stream
int MaxActiveInstances => UNLIMITED;
[L____________<Return_ListFeatures, Rectangle>]
struct Call_ListFeatures { }
[____________r<Feature>] // any number of items, the state stays
[____________R<End, EndOfStream>] // no more of them
struct Return_ListFeatures { }
}
public interface RouteChat : Actor { // a stream both ways
int MaxActiveInstances => UNLIMITED;
[l____________<RouteNote>]
[____________r<RouteNote>]
[L____________<Drain_RouteChat, EndOfStream>] // the caller has no more to say
struct Call_RouteChat { }
[____________r<RouteNote>]
[____________R<End, EndOfStream>]
struct Drain_RouteChat { }
}
}
}
}
}A plain call became an RPC method: the Client (Left) calls, the Server answers. A streaming call became an actor whose states say who may send what and when the call is over.
gRPC knows the first method as /routeguide.RouteGuide/GetFeature. The description says the same without writing
that path anywhere: the package is its namespace, the service is an interface in the connection, the method is
declared in it. A hierarchy of names is carried over as a hierarchy of declarations, never as a text beside them.
The samples are schemas that run in production. Each row leads to the original schema in its repository and to the description generated from it; all of them pass the agent's check.
| Project | Schema | Description | What to look at |
|---|---|---|---|
| OpenTelemetry (OTLP) | trace_service.proto, metrics_service.proto, logs_service.proto | trace_service.cs, metrics_service.cs, logs_service.cs | a service of one package that carries messages of three others; the signals themselves are in trace.cs, metrics.cs, logs.cs |
| Kubernetes CRI | api.proto | api.cs | 2 398 lines, two services, 36 RPC methods and 7 streams between the kubelet and a container runtime |
| etcd v3 | rpc.proto | rpc.cs | six services, 38 RPC methods, Watch and LeaseKeepAlive as two-way streams, types of two more packages |
| Apache Arrow Flight | Flight.proto | Flight.cs | seven of ten methods stream, three of them both ways |
| containerd | containers.proto, events.proto | containers.cs, events.cs | calls that answer with Empty as one-way calls, a subscription as a stream |
| gRPC | route_guide.proto, health.proto, reflection.proto | route_guide.cs, health.cs, reflection.cs | the four kinds of a method; a oneof in a request |
| CloudEvents | cloudevents.proto | cloudevents.cs | an envelope: a map of attribute values that are a oneof, a oneof of payloads |
| Google APIs | bytestream.proto, error_details.proto | bytestream.cs, error_details.cs | streams of bytes; Duration |
| Protocol Buffers | descriptor.proto, unittest.proto | descriptor.cs, unittest.cs | proto2 and an edition: groups, extensions, defaults, every scalar type |
OpenTelemetry's trace export, the way every collector receives spans:
package opentelemetry.proto.collector.trace.v1;
import "opentelemetry/proto/trace/v1/trace.proto";
service TraceService {
rpc Export(ExportTraceServiceRequest) returns (ExportTraceServiceResponse) {}
}
message ExportTraceServiceRequest {
repeated opentelemetry.proto.trace.v1.ResourceSpans resource_spans = 1;
}namespace opentelemetry.proto.collector.trace.v1 { // the package of the file
public interface trace_service {
public class ExportTraceServiceRequest {
opentelemetry.proto.trace.v1.ResourceSpans[,,] resource_spans;
}
public struct opentelemetry { // the packages of what the file imports
public struct proto {
public struct trace {
public struct v1 {
public class ResourceSpans { ... }
public class Span { ... }
}
}
public struct resource { public struct v1 { public class Resource { ... } } }
public struct common { public struct v1 { public class KeyValue { ... } } }
}
}
interface ClientServerConnection : Connects<Client, Server> {
public interface TraceService {
(L____________, ExportTraceServiceResponse) Export(ExportTraceServiceRequest req);
}
}
}
}The file is 77 lines and declares one method; the description is 711 lines, because what the method carries - spans, resources, attribute values - is declared in three imported packages and is written out with it, each type where its package puts it.
| Path | Contents |
|---|---|
src/Protobuf2AdHoc.cs |
The converter: files and imports, name resolution, the name pass, the emitter |
src/ProtoReader.cs |
The reader of the protobuf language: a lexer and a recursive-descent parser |
src/Originals.cs |
Reads sources.txt: the original page of an input file, for the header of its description |
src/adhoc/AdHocNames.cs |
Local copy of AdHocAgent's keyword list and its brush rename rule |
Protobuf2AdHoc.csproj |
.NET 10 console project; compiles src/ only; no packages |
fetch-samples.sh, samples/ |
40 real .proto files, the script that downloads them and samples/sources.txt, the original page of each |
build.sh |
Builds the converter into out/ and regenerates AdHoc/ from samples/ |
AdHoc/*.cs |
Generated descriptions, one per .proto that has something to send, kept for review |
validate.sh |
AdHocAgent parse-only check of a folder of descriptions |
tests/ |
Synthetic schemas - every construct once, every edge case, the editions syntax - with their descriptions |
CONVENTIONS.md |
The rules every converter of the family follows |
For each <name>.proto the converter writes <name>.cs (namespace <package>, interface <name>). Its header
names the file and the imported files it takes types from, each with the page of its original when a
sources.txt in a folder above lists it (fetch-samples.sh writes one for the samples; a path ending with /
there stands for every file under that folder). Then, in order:
- Packs Inventory (Dashboard) - one
<see cref='…'/>line per pack that travels on its own, pre-tagged with the services that carry it, so branches can route by tag (KeepDoc). Noidis pinned: pack ids are AdHoc's own and AdHocAgent assigns them. - Notes on what only the developer can decide:
longagainstlongJS, where[X]/[A]do not fit,bytesas a value against a conduit. _DefaultMaxLengthOfwith permissive caps (65 535), and what the converter declares itself when the file needs it:NoArg(protobuf'sEmpty),EndOfStream,DurationMs.- Packs - the messages and enums of the file in the order they are written, then the types of imported
files that these are typed with: those of the same package beside them, those of another package in
containers named after it (
google.rpc.Status). - Hosts -
ServerandClient; both request all six language generators. - The connection -
ClientServerConnection : Connects<Client, Server>with one interface per service, and the stateExchangefor the messages no method carries. - Attribute declarations - the custom attribute classes used above. AdHoc turns them into constants of the generated code; it does not read them.
Protobuf has several types for one range of values, and which one the author picked is a statement about the values: a varint is chosen where small numbers are the common ones, a fixed width where they are not.
| Protobuf | AdHoc | Notes |
|---|---|---|
int32, sint32 |
[X] int |
a varint: [X] gives small values of either sign few bytes |
int64, sint64 |
[X] long |
|
uint32 |
[A] uint |
an unsigned varint: values cluster at the low end |
uint64 |
[A] ulong |
|
fixed32, fixed64 |
uint, ulong |
fixed width: no attribute |
sfixed32, sfixed64 |
int, long |
|
float, double, bool, string |
the same | |
bytes |
Binary[,,] |
a value held whole; a BLOB is better declared [S(N)] File, the file says so where it opens |
A varint loses on values that are systematically large - an identifier, a hash, a coordinate scaled by 1e7: there the attribute is to be dropped by hand, and the generated file says so where it opens.
| Protobuf | AdHoc | Notes |
|---|---|---|
message |
public class |
nested messages and enums stay nested |
package of the file |
the namespace of the description |
a package is what a namespace is called in protobuf; a file without one gets org.protobuf |
| a type of another package | nested containers named after that package: public struct google { public struct rpc { ... } } |
referred to as protobuf does it, google.rpc.Status |
repeated T |
T[,,] |
a list whose length travels with it; repeated bytes is Binary[,,][,,] |
map<K, V> |
Map<K, V> |
the varint of a key or a value as [Key: X], [Val: A] |
map<bool, V> |
two fields, <name>_false and <name>_true |
a map keyed by a boolean holds two values at most: two fields say it plainly |
optional (proto2, proto3) |
T? |
|
required, a proto3 field without a label |
T |
|
| an edition field without a label | T?, or T when features.field_presence is IMPLICIT or LEGACY_REQUIRED |
an edition gives a field presence by default |
oneof name { … } |
a nested pack Name of optional alternatives and one field Name name; |
what a oneof is on the wire; that exactly one alternative is set is stated in the doc |
group (proto2) |
a nested pack and a field of it | the field is named as protobuf names it: the name of the group in lower case |
extend Message { … } |
the fields are added to that message, under a comment that says where they were declared | an extension is a field of the message it extends; extend of an options message declares an option and is not carried |
enum |
public enum, values as they are |
an enum with one value: a struct of constants, a field of it is [X] int |
reserved, extensions |
a comment at the place | |
| field number | nothing | it is protobuf's wire: AdHoc sends fields in the order they are declared, and the order of the .proto is kept |
| a comment above a declaration or after it on its line | the doc comment | a comment separated by a blank line is not attached, as in protoc |
| Protobuf | AdHoc | Notes |
|---|---|---|
google.protobuf.Timestamp |
DateTime |
|
google.protobuf.Duration |
DurationMs, an alias of AdHoc's Duration in steps of 1 ms |
an AdHoc duration is not negative and not finer than a millisecond |
google.protobuf.Empty |
NoArg as a request or a reply; bool as a field |
an empty pack is AdHoc's cheapest signal; as a field its presence is all it says |
DoubleValue, FloatValue, BoolValue, StringValue, BytesValue |
double?, float?, bool?, string?, Binary[,,]? |
a wrapper exists to make a scalar optional |
Int32Value, Int64Value, UInt32Value, UInt64Value |
[X] int?, [X] long?, [A] uint?, [A] ulong? |
|
google.protobuf.Any |
Binary[,,] |
a serialized message and the URL of its type; noted at the field |
google.protobuf.FieldMask |
string[,,] |
|
google.protobuf.Struct, Value, ListValue |
packs written out as struct.proto declares them |
a JSON value as a pack of optional alternatives |
| a message with no fields, as the type of a field | bool |
noted at the field |
They are recognised by what a name resolves to, not by how it is written: a local message Value is that
message, google.protobuf.Value is the well-known one. The files of google/protobuf that declare them need
not be on disk: protoc carries them inside itself, and so does the converter.
| Protobuf | AdHoc | Notes |
|---|---|---|
service |
an interface in the connection | groups the methods as the document does |
rpc M(Req) returns (Res) |
(L____________, Res) M(Req req); |
Left = Client calls, Right = Server answers |
rpc M(Empty) returns (Res) |
(L____________, Res) M(NoArg _); |
|
rpc M(Req) returns (Empty) |
a one-way call: [l____________<Req>] struct M { } in the actor OneWay of the service |
nothing comes back, so nothing is waited for |
rpc M(Req) returns (stream Res) |
interface M : Actor: the items go by [____________r<Res>] while the actor stays in its state, EndOfStream ends the call |
|
rpc M(stream Req) returns (Res) |
interface M : Actor: the items go by [l____________<Req>], EndOfStream passes the turn, the reply ends the call |
|
rpc M(stream Req) returns (stream Res) |
interface M : Actor with two states: both send, then the caller says EndOfStream and the answerer finishes |
|
the path gRPC knows a method by, /package.Service/Method |
where the method is declared: the namespace, the interface of the service, the method | no attribute repeats it; a name that had to change says in its doc what the .proto calls it |
| a message of the file that no method carries and no field is typed with | the state Exchange: [_____lr_____<(A, B)>] |
it travels on its own, either way; without services these are the root messages of the file |
A call that answers with Empty is taken for a notification. If the caller has to know that the call was done,
write it as a method that answers with the empty pack: (L____________, NoArg) M(Req req);.
| Protobuf | AdHoc | Notes |
|---|---|---|
deprecated = true |
[Obsolete] |
on a value of an enum: a line of its doc |
default = … (proto2) |
[Default(…)] |
metadata: AdHoc sends what the field holds |
an option the authors declared, (google.api.http) = { get: "/v1/items" } |
[Option(@"(google.api.http)", @"{ get: ""/v1/items"" }")] |
on the file, a message, a field, a oneof, a method; on a service and on a value of an enum: a line of the doc |
java_package, go_package, optimize_for, packed, json_name, ctype, jstype, lazy, features.* … |
nothing | they steer other code generators or protobuf's own wire |
A name comes from the document and cannot always stay as it is. A description cannot hold:
- a keyword of any target language as a name - handled the way the agent does it (
type→Type); - a field and a nested type with one name, a member named like its pack;
- sibling types that differ only by case (
Client/client): their generated files collide on Windows; - a type with the name of a type of an enclosing scope: inside, the name would mean the nested one (a nested
Statusnext to a top-levelStatus); - a type called like something the description itself refers to (
File,Set,Duration,DateTime); - a type of another package named like a type of the file: inside its container the name would mean the nested one;
- a method called like its service (
service Watch { rpc Watch … }).
Names are settled in one pass, top-down, when every file is read. Of two names in conflict the deeper one
changes, so the outer, more visible name keeps the spelling of the document. It changes the way the agent
renames a keyword: its lowercase letters go upper case one after another until the conflict is gone
(Status → STatus). Only names that differ by nothing but case cannot be told apart that way and get a number
(client2). References are resolved before that pass and kept as objects, so a rename reaches every use.
Three more rules, each for a name that is not the document's own:
- a segment of the package that is a keyword of C# starts with a capital in the namespace (
event→Event); - the pack of a oneof that wants a name a nested message has says what it is:
ChoiceOneof; - what the converter declares itself yields to the document: a file with a message
Clientgets the hostClientHost, one with a messageNoArggetsNoArgs.
A reference is written as short as C# resolves it: by the simple name from inside the message a type is nested
in, by its path (Outer.Inner) from elsewhere.
One description is written per .proto file. It holds what the file declares and, of the files it imports,
the types its own declarations are typed with - each with the top-level message it is nested in, and whatever
that one needs in turn. A type of the same package stands beside the file's own, a type of another package in
containers named after that package. So every description stands alone, and a type shared by two files is in both.
An import is looked for, in this order: in the import roots given with -I; in the folders above the importing
file, nearest first (one of them is the root the import was written from); by the end of its path under the
input folder, when exactly one file has it; among the well-known files of google/protobuf, which the
converter carries. An import that is not found is reported. If a type it declares is needed, the field is
carried as raw bytes with a note at the field; if it only declares options - google/api/annotations.proto
does - nothing is lost.
A file that declares no message and no service - enums only, or options only - has nothing to send and gets no description: its enums are written out with the files that use them.
- Protobuf's wire format: field numbers,
packed, groups as a wire type,message_set_wire_format. None of it is needed: AdHoc lays out its own wire. - Options of code generators (
java_package,go_package,csharp_namespace,optimize_for…). - Negative and sub-millisecond durations:
google.protobuf.Durationhas nanoseconds and a sign, an AdHocDurationhas neither. google.protobuf.Anyis opaque bytes: which message it holds is told by its URL, at run time.- Declarations of options (
extend google.protobuf.FieldOptions { … }): the options applied are kept as[Option], their declarations are not data. - That exactly one alternative of a oneof is set: AdHoc leaves it to the code that fills the pack.
- Whether a call answered with
Emptywas done: it becomes a one-way call (see Services). - Early end of a two-way stream by the answerer: in the generated actor the caller ends its stream first.
- Collection bounds. Protobuf declares none.
_DefaultMaxLengthOfis set to 65 535 so nothing is cut silently at AdHoc's default of 255; tighten it and put[D]on the fields whose real bound is known. - String constants over 1000 characters (a long value of an option) are cut at 1000 characters, and a
comment above says so; the
.protohas the whole text. - Services of imported files are not written out: they belong to the description of their own file.
- A comment separated from its declaration by a blank line (a licence header, a banner of a section).
Everything dropped at a specific place leaves a comment at that place.
.NET SDK 10 is required. There are no packages to restore.
./fetch-samples.sh # download the upstream .proto files into samples/
./build.sh # build into out/ and convert samples/ → AdHoc/
./validate.sh AdHoc # AdHocAgent parse-only check of every generated file (nothing is uploaded)Convert your own schemas:
dotnet out/Protobuf2AdHoc.dll path/to/service.proto # → ./AdHoc/service.cs
dotnet out/Protobuf2AdHoc.dll path/to/protos path/to/out # every .proto under the folder
dotnet out/Protobuf2AdHoc.dll api/v1/service.proto out -I third_party # an extra root for importsCLI contract: <input file or folder> [output folder] [-I <import root>]...; the output folder defaults to
<cwd>/AdHoc; one .cs per .proto, named after the file - two files of one name in different folders take
their folders into the name (v1_service.cs, v2_service.cs). Imports that are not found and types that
are not declared are printed. The exit code is 2 when an input could not be read or nothing was converted.
validate.sh runs AdHocAgent with ADHOC_PARSE_ONLY=1 on a copy of every .cs (the agent rewrites the file in
place) and prints OK when the agent exits with 0 and reports no ERR / WRN. It looks for AdHocAgent.exe on
PATH, then in a sibling build of the agent; set AGENT=/path/to/AdHocAgent.exe to point it elsewhere.
Result for the shipped samples - 37 of 37 descriptions OK; three files have nothing to send and get no description. Of the 123 methods of the samples 91 are RPC methods, 29 are streaming actors and 3 are one-way calls.
trace_service.proto is 77 lines and its description 711: the one method it declares carries a tree of
messages from three imported files, and they are written out with it.
tests/ holds synthetic schemas, all OK; run them after changing the converter:
dotnet out/Protobuf2AdHoc.dll tests tests/AdHoc && ./validate.sh tests/AdHoc| Schema | What it is for |
|---|---|
coverage.proto |
every construct the converter maps, once: all scalar types, repeated, maps, a oneof, the well-known types, options, the four kinds of a method, a message no method carries |
edge-cases.proto |
every name conflict the converter settles and every shape it has to rewrite: messages called File, Client, Server, NoArg, Duration; Client next to client; a field named like its message and like its type; a nested Status next to a top-level one and next to an imported one; groups; extensions, one named like a field of its target; an enum with one value; allow_alias; a map keyed by a boolean; a default with bytes that are not text; an import that is not found; a oneof that wants the name of a nested message; a service named like a message; a method named like its service; a stream of empty messages |
editions.proto |
the editions syntax: presence said by a feature, names reserved without quotes |
imported/money.proto |
what edge-cases.proto imports: another package with a type of the same name |