Improve makecode.py to make it more robust

[mjuenema]

There have been some recent issues with the tools/makecode.py script where it fails to build Python modules for some providers.

• Add ESXI Provider Add ESXI Provider #145
• Hashicorp providers appear to have moved to a different repo breaking codegen Hashicorp providers appear to have moved to a different repo breaking codegen #147
• Update providers.yml Update providers.yml #144

I wanted to start a discussion around the question of how to improve tools/makecode.py more robust.

The first problem with the script is that it does not actually implement a Go language parser for processing the Terraform
provider source code. Instead, makecode.py uses regular expression to extract the required data from the Go source files. This is rather brittle as it requires the Go code to be formatted in a specific way. It may be possible to work-around the problem by converting the Go source code into a "standardised" format using a source code formatter like GoFmt

The second problem is that makecode.py makes certain assumptions about the directory structure of the Terraform provider modules and the location of the provider.go file.

[starhawking]

Looking at how they build the docs at https://github.com/hashicorp/terraform-plugin-docs there's a terraform providers schema -json command. This seems like it would be the canonical source to generate things from, but might be kinda a pain to actually get the providers. Bonus would be that it also includes things types, so it'd be possible to do something like add python type annotations for all the providers. It also looks like you can hit the terraform registry api, so they might expose an endpoint that could be used to collect that information. https://www.terraform.io/docs/internals/provider-registry-protocol.html

[ilons]

Great initiative to break this out (and nice to see you being active)!

I've actually started some small work to improve the state of the script already. While I do think it would make sense to make a more robust solution, I also think that we could get a way better result with small changes to the existing solution.

One problem not mentioned here is that some providers choose to restructure their code, and especially breaking some parts out to different files.

[mjuenema]

For the record, I looked into @starhawking's suggestion about the terraform providers schema -json command.

Using the sample Terraform configuration below...

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.27"
    }
  }
}

...running terraform providers schema -json results in 130K lines (when pretty-printed) of JSON output covering all(?) of the AWS provider.

{
  "format_version": "0.1",
  "provider_schemas": {
    "registry.terraform.io/hashicorp/aws": {
      "provider": {
        "version": 0,
        "block": {
[sic]
        "aws_instance": {
          "version": 1,
          "block": {
            "attributes": {
              "ami": {
                "type": "string",
                "description_kind": "plain",
                "required": true
              },
              "arn": {
                "type": "string",
                "description_kind": "plain",
                "computed": true
              },
              "associate_public_ip_address": {
                "type": "bool",
                "description_kind": "plain",
                "optional": true,
                "computed": true
              },
              "availability_zone": {
                "type": "string",
                "description_kind": "plain",
                "optional": true,
                "computed": true
              },
}

This option could make the current approach of running regular expressions on Go source code obsolete. You can also do type checking and automatically generate documentation. I can see Terrascript 0.10 on the horizon.

I am going to check out the Terraform Provider Registry next. It would be great if one could get a list of providers through the API.

[mjuenema]

It is possible to get a list of providers from the Terraform Provider Registry as shown below. One just has to iterate over the offset

$ curl 'https://registry.terraform.io/v1/providers' | jq
{
  "meta": {
    "limit": 15,
    "current_offset": 0,
    "next_offset": 15,
    "next_url": "/v1/providers?offset=15"
  },
  "providers": [
    {
      "id": "hashicorp/helm/2.0.2",
      "owner": "",
      "namespace": "hashicorp",
      "name": "helm",
      "alias": null,
      "version": "2.0.2",
      "tag": "v2.0.2",
      "description": "terraform-provider-helm",
      "source": "https://github.com/hashicorp/terraform-provider-helm",
      "published_at": "2021-01-18T20:49:49Z",
      "downloads": 10731244,
      "tier": "official"
    },
    {
      "id": "hashicorp/tls/3.1.0",
      "owner": "",
      "namespace": "hashicorp",
      "name": "tls",
      "alias": null,
      "version": "3.1.0",
      "tag": "v3.1.0",
      "description": "terraform-provider-tls",
      "source": "https://github.com/hashicorp/terraform-provider-tls",
      "published_at": "2021-02-19T17:58:57Z",
      "downloads": 10537953,
      "tier": "official"
    },

[ilons]

Now, THAT looks interesting!

[mjuenema]

It looks like as if we are agreeing on that this is the way forward.

I am happy to start work on rewriting makecode.py or do you want to do it? We should avoid working on this at the same time.

The first step would probably be limited to creating the same output as the current implementation of the script. Type checking, type annotations and auto-generated documentation can then be added later.

Any thoughts?

[ilons]

Seems like a reasonable aproach. I won't do more on the makecode.py for now (after the current PR). And also have very limited time to do anything during the weekend.
Would happily assist with PR / implementing specific parts (as long as it can be done in small chunks).

[mjuenema]

So I am progressing nicely with re-writing makecode.py until I realised that I opened another can of worms: There can be multiple providers of the same name, but from different "sources".

For exmple, search for 'null' in the Terrafform Registry and there are four providers of that name (https://registry.terraform.io/search/providers?q=null). That's why Terraform introduced namespaces, which we would have to mirror accordingly. Ouch!

Fortunately only a small number of providers are affected so we may get away without namespaces for now.

[mjuenema]

So you would end up with something like import terrascript.provider.hashicorp.aws.

[ilons]

Great work!

While this is not a big change in itself, it would still break backwards compatibility.
I think it make sense to aim for a version 1.0 release including this, the change to how variables are interpolated and also if we have anything else.

[ilons]

I obviously failed to include what sparked my comment initially:
I absolutely think we should namespace the providers, unless that is a big thing to do.

[mjuenema]

I am working on makecode.py. See this comment for current progress.