Introduce support for step-types

[sandrogattuso]

This PR introduces the ability to manage step-types.
@palson-cf I'm not 100% sure this is the best way to handle this type of configuration in particular for the versioning bit.
I've documented some of the limitations in the docs folder, however, here a summary of the main items:

• The read function always get the latest published version. I couldn't find a way to dynamically extract the version specified in the step_types_yaml so that the get can extract the correct version.
• Destroy action removes the step-type completely. This is the behaviour that I would expect, but I have to admit that is questionable if we should do that or just remove the version specified in the yaml

EDIT: added a second type of resource to manage all the versions of a step from a single resource. I had to add some business logic to handle certain behaviours of the API (maybe it's me misinterpreting how it works, appreciate inputs on this)

[vadimgusev-codefresh]

Hi @sandrogattuso! I've took a look at your PR and played a little bit with it.
Sad to say, but the way it currently implemented breaks Terraform's CRUD flow.

For example:

1. Define in Terraform step with name myorg/foo, terraform apply
2. Rename it in Terraform to myorg/foo-2, terraform plan
3. Terraform intends to modify single object, terraform apply
4. Codefresh will now have 2 steps: myorg/foo and myorg/foo-2
5. From that point Terraform's state diverges from reality

I've verified it on your latest commit.

Seems like resource with a single attribute in form of YAML Step spec (which is from Terraform's point of view is just a wall of text) is not enough to properly track resource's lifecycle.

[sandrogattuso]

Hi @sandrogattuso! I've took a look at your PR and played a little bit with it.
Sad to say, but the way it currently implemented breaks Terraform's CRUD flow.

For example:

1. Define in Terraform step with name myorg/foo, terraform apply
2. Rename it in Terraform to myorg/foo-2, terraform plan
3. Terraform intends to modify single object, terraform apply
4. Codefresh will now have 2 steps: myorg/foo and myorg/foo-2
5. From that point Terraform's state diverges from reality

I've verified it on your latest commit.

Seems like resource with a single attribute in form of YAML Step spec (which is from Terraform's point of view is just a wall of text) is not enough to properly track resource's lifecycle.

@vadimgusev-codefresh thanks for the feedback, I think we have 2 options here:

1. I can add the attribute name to resource_step_types.go similarly to resource_step_types_versions.go and have the ForceNew set on it. This would address the scenario you described.
2. I can remove completely the implementation of resource_step_types.go and keep resource_step_types_versions.go as only option.

From a user perspective I think the resource with a single attribute has very little use so I would actually go for the second option, but happy to hear your point of view and then I can make the changes accordingly.

[vadimgusev-codefresh]

@sandrogattuso I agree with you on all points:

1. Single attribute resource is hard to use.
2. Having two implementations of the same resource is confusing.

Since versioned implementation effectively supersedes resource_step_types.go I would go with leaving only one.

And since this one is much bigger that previous (that broke quite easily), would you mind writing some minimal tests for it?

[sandrogattuso]

@vadimgusev-codefresh I've finished the clean up and added some test cases. Let me know if you would like to have further changes

[vadimgusev-codefresh]

@sandrogattuso I've checked out your latest changes. Good job on the tests. Also I was not able to break TF state like in previous implementation.

My only nitpick is: please clean up old implementation of data source and docs. Otherwise LGTM, will merge after cleanup.

[sandrogattuso]

@vadimgusev-codefresh Removed the doc for the old implementation of resource.
For the data source part, the 2 implementations are actually covering different use cases:

• step_types ==> Query of a specific version to retrieve the step definition
• step_types_versions ==> Query a step type to retrieve the list of published versions (no yaml definition returned)

I thought it could be a useful functionality to be able to get the list of versions available, but if you think is redundant or confusing I'll remove it.

I've also updated the docs for the second one since the description was not reflecting what the datasource actually does.

[vadimgusev-codefresh]

@sandrogattuso cannot we just query a full list of versions with yaml definitions so data would match the resource's structure? Or would it be inconvenient to navigate such structure?

[sandrogattuso]

@vadimgusev-codefresh I've pushed a version with the datasource matching the format of the resource. In general, seems a bit more complicated to navigate the structure (added an example in the README).
Not the most user friendly approach from a consumption perspective but, it's something that you have manage whenever you have an attribute configured as typeSet (there are plenty of similar cases in popular providers like AWS and Google).
Let me know which implementation you prefer

[vadimgusev-codefresh]

@sandrogattuso thanks for your work! LGTM, merging.