9.2 KiB

Raw Blame History

Add new resources

First, you need to understand how driftctl scan works. Here you'll find a global overview of the steps that compose the scan:

Then, you'll find below a more detailed flow of how we handle the enumeration and the fetching of resource's details from the remote:

Defining the resource

First step would be to add a file called pkg/resource/<providername>/<resourcetype>.go. This file will define a string constant that will be the resource type identifier in driftctl.

Optionally, if your resource is to be supported by driftctl experimental deep mode, you can add a function that will be applied to this resource at creation. This allows to prevent useless diffs to be displayed. You can also add metadata to fields so that they are compared or displayed differently.

For example this defines the aws_iam_role resource:

const AwsIamRoleResourceType = "aws_iam_role"

func initAwsIAMRoleMetaData(resourceSchemaRepository resource.SchemaRepositoryInterface) {
	// assume_role_policy drifts will be displayed as json
	resourceSchemaRepository.UpdateSchema(AwsIamRoleResourceType, map[string]func(attributeSchema *resource.AttributeSchema){
		"assume_role_policy": func(attributeSchema *resource.AttributeSchema) {
			attributeSchema.JsonString = true
		},
	})
	// force_detach_policies should not be compared so it will be removed before the comparison
	resourceSchemaRepository.SetNormalizeFunc(AwsIamRoleResourceType, func(res *resource.Resource) {
		val := res.Attrs
		val.SafeDelete([]string{"force_detach_policies"})
	})
}

When it's done you'll have to add this function to the metadata initialisation located in pkg/resource/<providername>/metadatas.go:

func InitResourcesMetadata(resourceSchemaRepository resource.SchemaRepositoryInterface) {
    initAwsAmiMetaData(resourceSchemaRepository)
}

In order for you new resource to be supported by our terraform state reader you should add it in pkg/resource/resource_types.go inside the supportedTypes slice.

var supportedTypes = map[string]struct{}{
    "aws_ami":                               {},
}

All resources inside driftctl are resource.Resource structs. All the other attributes are represented inside a map[string]interface

Repository, Enumerator and DetailsFetcher

Then you will have to implement two interfaces:

Repositories are the way we decided to hide direct calls to SDK and pagination logic. It's a common abstraction pattern for data retrieval.
remote.common.Enumerator is used to enumerate resources. It will call the cloud provider SDK to get the list of resources. For some resource it could make other call to enrich the resource with additional attributes when driftctl is used in deep mode
remote.common.DetailsFetcher is used to retrieve resource's details. It makes a call to Terraform provider ReadResource. This implementation is optional and is only needed if your resource type is to be supported by experimental deep mode. Please also note that it exists a generic implementation called remote.common.GenericDetailsFetcher that can be used with most resource types.

Repository

This will be the component that hides all the logic linked to your provider SDK. All providers have different ways to implement pagination or to name function in their API.

Here we will name all listing functions ListAll<ResourceTypeName>.

For AWS we decided to split repositories using the Amazon logic. So you'll find repositories for EC2, S3 and so on. Some provider does not have this grouping logic. Keep in mind that like all our file/struct repositories should not be too big.

For our GitHub implementation the number of listing functions was not that heavy, so we created a unique repository for everything:

type GithubRepository interface {
	ListRepositories() ([]string, error)
	ListTeams() ([]Team, error)
	ListMembership() ([]string, error)
	ListTeamMemberships() ([]string, error)
	ListBranchProtection() ([]string, error)
}

type githubRepository struct {
	client GithubGraphQLClient
	ctx    context.Context
	config githubConfig
	cache  cache.Cache
}

func NewGithubRepository(config githubConfig, c cache.Cache) *githubRepository {
	ctx := context.Background()
	ts := oauth2.StaticTokenSource(
		&oauth2.Token{AccessToken: config.Token},
	)
	oauthClient := oauth2.NewClient(ctx, ts)

	repo := &githubRepository{
		client: githubv4.NewClient(oauthClient),
		ctx:    context.Background(),
		config: config,
		cache:  c,
	}

	return repo
}

As you can see, this contains the logic to create the GitHub client (it might be created outside the repository if it makes sense to share it between multiple repositories). driftctl, sometimes, needs to retrieve the list of resources more than once, so we cache each request to avoid unnecessary call.

Enumerator

Enumerators can be found in pkg/remote/<providername>/<type>_enumerator.go. It will call the cloud provider SDK to get the list of resources.

Note that at this point, resources should not be entirely fetched and most of them will have empty attributes (e.g. only their id and type). Most of the resource returned by enumerator have empty attributes: they only represent type and terraform id.

There are exceptions to this:

Sometimes, you will need more information about resources for them to be fetched in the DetailsFetcher. For those cases, you will add specific attributes to the map of data.
For complex cases (e.g. middlewares) where you would need driftctl to run as expected in deep and non-deep mode, you would need to enumerate resources as well as to fetch manually specific attributes, using the remote SDK, before adding them to the map of data.

You can use an already implemented Enumerator as example.

For example, to implement aws_instance resource you will need to add a ListAllInstances() function to repository.EC2Repository.

Bear in mind it will be called by the Enumerator to retrieve the list of instances.

Enumerator constructor could use these arguments:

an instance of Repository that you will use to retrieve information about the resource
the global resource factory that should always be used to create a new resource.Resource

Enumerator then needs to implement:

SupportedType() resource.ResourceType that will return the constant you defined in the type file
Enumerate() ([]*resource.Resource, error) that will return the list of resources

type EC2InstanceEnumerator struct {
	repository repository.EC2Repository
	factory    resource.ResourceFactory
}

func NewEC2InstanceEnumerator(repo repository.EC2Repository, factory resource.ResourceFactory) *EC2InstanceEnumerator {
	return &EC2InstanceEnumerator{
		repository: repo,
		factory:    factory,
	}
}

func (e *EC2InstanceEnumerator) SupportedType() resource.ResourceType {
	return aws.AwsInstanceResourceType
}

func (e *EC2InstanceEnumerator) Enumerate() ([]*resource.Resource, error) {
	instances, err := e.repository.ListAllInstances()
	if err != nil {
		return nil, remoteerror.NewResourceListingError(err, string(e.SupportedType()))
	}

	results := make([]*resource.Resource, len(instances))

	for _, instance := range instances {
		results = append(
			results,
			e.factory.CreateAbstractResource(
				string(e.SupportedType()),
				*instance.InstanceId,
				map[string]interface{}{},
			),
		)
	}

	return results, err
}

As you can see, listing errors are treated in a particular way. Instead of failing and stopping the scan they will be handled, and an alert will be created. So please don't forget to wrap these errors inside a NewResourceListingError. For some provider error handling is not that coherent, so you might need to check in pkg/remote/resource_enumeration_error_handler.go and add a new case for your error. You should test enumerator behavior when you do not have permission to enumerate resources. In the snippet above, ListAllInstances may return an AccessDenied error that should be handled.

Once the enumerator is written you have to add it to the remote initialization located in pkg/remote/<providername>/init.go:

    remoteLibrary.AddEnumerator(NewEC2InstanceEnumerator(s3Repository, factory))

DetailsFetcher

DetailsFetchers are only used by driftctl experimental deep mode.

This is the component that call Terraform provider to retrieve all attributes for each resource. We do not want to reimplement what has already been done in each Terraform provider. Thus, you should not call the remote SDK there.

If common.GenericDetailsFetcher satisfies your needs you should always prefer using it instead of implementing a custom DetailsFetcher in a new struct.

The DetailsFetcher should also be added to pkg/remote/<providername>/init.go even if you use the generic version:

    remoteLibrary.AddDetailsFetcher(aws.AwsEbsVolumeResourceType, common.NewGenericDetailsFetcher(aws.AwsEbsVolumeResourceType, provider, deserializer))

Don't forget to add unit tests after adding a new resource.

You can find example of functional tests in pkg/remote/<type>_scanner_test.go.

You should also add acceptance tests if you think it makes sense. They are located next to the resource definition described in the first step.

More information about adding tests can be found in testing documentation

9.2 KiB Raw Blame History