feat: add hetzner cloud provider

[Thunderbottom]

adds a provider for hetzner cloud that does:

• filter instances by label selector and datacenter location
• fetch public/private IP address based on provided address type
• checks for associated floating IPs in case of blocked public/private IP addresses
• accepts API token as environment variable: HCLOUD_TOKEN

Enhancement over: #159
Closes: #73

[hashicorp-cla]

All committers have signed the CLA.

[VoyTechnology]

LGTM!

[svenstaro]

Could we get this merged?

[laukaichung]

While waiting for it to be merged, is there any way for now to automatically get a list of private IPs for the retry_join field without hardcoding a list of IPs? I'm planning to run a consul cluster on Hetzner Cloud with private network. It seems that GetPrivateIPs from go-sockaddr, e,g retry_join={{ GetPrivateIPs }} would work but I''m not sure how to use it in the config.

[dnephin]

I don't think there is any support for that yet, but hashicorp/consul#9100 is a proposal that should make it possible.

[vitamindit]

While waiting for it to be merged, is there any way for now to automatically get a list of private IPs for the retry_join field without hardcoding a list of IPs? ...

I'm waiting for it too :) ... here is my approach:

You get private IP addresses once the server is assigned to a network or subnet.

Then you should use e.g. the hcloud server-info API. This will return the available servers including their private IP addresses for available interfaces.

From this server-info response you can then construct the list of desired private cluster IP addresses.

[KevinGimbel]

What's blocking this PR from being merged? I'm currently evaluating Consul for work and I'd love to use/try the auto-discovery feature as part of this PoC.

[gardion]

Please merge, it would massively help adopting Consul for our platform.

[tomschlenkhoff]

Any help needed? Any blockers left? If not please (with sugar on top) merge, would be a boost for Consul/Hetzner adoption in Germany/EU.

[Thunderbottom]

I believe there's no blockers, although the repo hasn't been updated since last year. So I'm not sure if it'll be merged any time soon.

[ronaldburns]

This is a feature we need for our infrastructure. It doesn't appear like it's going to be committed anytime soon.
We've forked this and made a fix or two.

go-discover
consul

If you'd like to immediately use this implementation, binaries have been added for the latest releases.

Consul 1.11.3 (hcloud)

Consul 1.11.4 (hcloud)

Consul 1.12.0-beta1 (hcloud)

[devnet-io]

What are the blockers on getting this merged? This will be very useful for my org. Happy to assist with what is needed.

[KrakenMare]

Echoing the above. It would be fantastic to see this merged or insight into what is needed for a merge.

[pugnacity]

any news here?

[marco-m]

Hello @dnephin, @riddhi89, @alvin-huang,
we were almost there with #159 :-), then after a while @Thunderbottom dared to summon the skeleton (just past-Halloween pun) and did this second attempt.
I understand very well that you are fully loaded, but if you could spare a moment to review this PR many of us would be happy. Thanks again!
@Thunderbottom would you have time to resolve the conflicts, to ease the work of the HashiCorp reviewers? Merci!

[acaloiaro]

FYI @Thunderbottom I can't imagine you're enthusiastic to do more work on this PR since it hasn't been re-reviewed yet, but in case you are -- a merge conflict has occurred.

[Thunderbottom]

Hi, haven't gotten the time to check GitHub that often anymore. I have fixed all the issues and rebased to the latest master. let me know if anything else needs to be done.

[svenstaro]

@riddhi89 Any chance we could get this merged?

[uni0nist]

Any updates?

[uni0nist]

Hello! @alvin-huang any chance you could review this PR? Or do you have somebody else from HashiCorp to suggest? I really want to see Hetzner Cloud in go-discovery so that Сonsul and Nomad could use auto-join. Thanks!

[acaloiaro]

@Neyury If you're desperate to use @Thunderbottom's patch, you can do what I did and fork nomad and add a replace to its go.mod as I've done here: hashicorp/nomad@332c3e4

Then my server.server_join stanza looks like:

  server_join {
    retry_join = [ "provider=hcloud label_selector=nomad-server apiToken=${hetzner_compute_access_token}" ]
  }

Note: You'll want to label your nomad server compute nodes with nomad-server to use the above.

I've been running a hetzner nomad cluster this way for over 6 months. I do recommend pinning your fork to a tagged version of nomad, however.

I have a 1.4.x nomad build with patched go-discover here: https://github.com/acaloiaro/nomad/releases/tag/v1.4.x

You really should not use binaries from unknown strangers on the internet. But feel free to use it in a dev/test environment if you feel that it's sufficiently isolated from the rest of your infrastructure

[tagirb]

Hi Yuriy, thanks for taking care of this feature, also very interested in it. I'd suggest involving @LKaemmerling from Hetzner for this.

[apricote]

Julian here from the @hetznercloud Integrations team. This looks fine regarding the usage of hcloud-go, I can not speak about go-discover.

Happy to help if you got any questions.

[acaloiaro]

Hashicorp are "semi-actively moving our repos to https://github.com/hashicorp/go-netaddr".

I emailed a member of their team to learn about the status of this repository. Those are the words of one individual, but you can see from the project activity that the project is not under heavy development.

Anyone waiting for this or other features to land in go-discover should probably not hold their breath about them being added to go-discover. It sounds like this project has become too brittle and difficult to maintain, motivating the migration to go-netaddr. An understandable move given this project's dependencies.

With that said, I don't see how go-netaddr replaces go-discover's cloud host-aware functionality. go-netaddr seems like a nice complement to, but not replacement for go-discover.

[kaspergrubbe]

Hello from 2024, I also hit this in my own setup, and would have loved for go-discover to support hcloud.

[tomgroenwoldt]

Hello from 2025. Is this still considered to be merged?

My current workaround for anyone interested:

# /etc/consul.d/consul.hcl
...
retry_join = [ XXX ]
...

Then, cloud-init runs hcloud to find out about all servers with a given label and substitutes the XXX with their IPs:

# Extract IPs of labeled servers from table and separate them by commas.
output=$(hcloud server list -o noheader -o columns=private_net -l nomad-server=true | sed 's/ *(<your_network_name>)//g' | sed 's/^/"/;s/$/"/' | tr '\n' ',' | sed 's/,$//')
sed -i "s/XXX/$output/" /etc/consul.d/consul.hcl

This leads to:

# /etc/consul.d/consul.hcl
...
retry_join = [ "10.0.0.3","10.0.0.2","10.0.0.4","10.0.0.5" ]
...

While this works for now I'd really like to use the auto discover feature of consul. Thanks for the work!

[kaspergrubbe]

Now that we're sharing workarounds, I am adding the IP of my servers to a DNS A-record:

resource "aws_route53_record" "consulglue" {
  zone_id = ""
  name    = "consulglue.example.com"
  type    = "A"
  ttl     = 300
  records = compact([for x in flatten(cluster-servers.devices[*].addresses) : can(regex(":", x)) ? "" : x])
}

And I use that in Consul like:

  "retry_join": ["consulglue.example.com"]

But it is isn't a great solution as it introduces race-conditions, the cloud-init solution is very clever :)

[acaloiaro]

FYI I refactored Nomad's retry_join to support go-netaddrs in 2023 (hashicorp/nomad#18745) since I think it's clear that go-discover is now in maintenance mode. Through go-netaddrs, Nomad now supports auto-discovery for many more providers.

That work should probably be ported over to consul in favor of trying to make go-discover work for any new or updated use cases.

The following comment of mine earlier in this thread was wrong

With that said, I don't see how go-netaddr replaces go-discover's cloud host-aware functionality. go-netaddr seems like a nice complement to, but not replacement for go-discover.

Go-netaddrs should succeed go-discover, as it's not feasible to add an unbounded number of depenencies to go-discover in order to support new providers and/or use cases.