Wording fixes

main
Elie 2021-08-24 17:07:34 +02:00
parent ad14dcd66e
commit 5e6d4df63c
No known key found for this signature in database
GPG Key ID: 399AF69092C727B6
8 changed files with 115 additions and 103 deletions

View File

@ -16,5 +16,5 @@ Resource listing is done using cloud providers SDK. Resource details retrieval i
- `Remote` is a representation of a cloud provider
- `Resource` is an abstract representation of a cloud provider resource (e.g. S3 bucket, EC2 instance, etc ...)
- `Enumerator` is used to list resources from a given type on a given remote and return a resource list, it should exist only one Enumerator per resource
- `DetailFetcher` is used to retrieve details for resources from a given type, this is an optional layer and is used only in deep mode.
- `Enumerator` is used to list resources of a given type from a given remote and return a resource list, it should exist only one Enumerator per resource
- `DetailsFetcher` is used to retrieve resource's details of a given type, this is an optional layer and is used only in deep mode.

Binary file not shown.

Before

Width:  |  Height:  |  Size: 46 KiB

After

Width:  |  Height:  |  Size: 46 KiB

View File

@ -1,26 +1,26 @@
@startuml
hnote across: Retrieve Resources
Driftctl -> IACSupplier: Resource()
IACSupplier --> Driftctl: stateResources []Resource
Driftctl -> RemoteSupplier: Resource()
RemoteSupplier --> Driftctl: remoteResources []Resource
driftctl -> IACSupplier: Resource()
IACSupplier --> driftctl: stateResources []Resource
driftctl -> RemoteSupplier: Resource()
RemoteSupplier --> driftctl: remoteResources []Resource
hnote across: Run Middlewares
Driftctl --> Driftctl: List Middlewares
loop on every middleware
Driftctl -> Middleware: Execute(remoteResources, stateResources)
Middleware --> Driftctl: remoteResources, stateResources []Resource
driftctl --> driftctl: List Middlewares
loop on each middleware
driftctl -> Middleware: Execute(remoteResources, stateResources)
Middleware --> driftctl: remoteResources, stateResources []Resource
end
hnote across: Run Filters
Driftctl -> FilterEngine: Run(remoteResources)
FilterEngine --> Driftctl: remoteResources []Resource
Driftctl -> FilterEngine: Run(stateResources)
FilterEngine --> Driftctl: stateResources []Resource
driftctl -> FilterEngine: Run(remoteResources)
FilterEngine --> driftctl: remoteResources []Resource
driftctl -> FilterEngine: Run(stateResources)
FilterEngine --> driftctl: stateResources []Resource
hnote across: Analyze
Driftctl -> Analyzer: Analyze(remoteResources, stateResources)
Analyzer --> Driftctl: Analyze
driftctl -> Analyzer: Analyze(remoteResources, stateResources)
Analyzer --> driftctl: Analyze
@enduml

Binary file not shown.

Before

Width:  |  Height:  |  Size: 56 KiB

After

Width:  |  Height:  |  Size: 57 KiB

View File

@ -1,16 +1,16 @@
@startuml
Driftctl -> Scanner: Resource()
driftctl -> Scanner: Resource()
hnote across: Enumeration phase
Scanner -> Scanner: List Enumerators
loop
Scanner -> Enumerator: Enumerate()
Enumerator -> RemoteSDK: List resource
Enumerator -> RemoteSDK: List resources
RemoteSDK --> Enumerator: []remoteRes
alt optionaly retrive resource needed attributes
Enumerator -> RemoteSDK: Retreive needed\nattributes
alt optionally retrieve resource needed attributes
Enumerator -> RemoteSDK: Retrieve needed attributes
RemoteSDK --> Enumerator: Attrs
end
Enumerator --> Scanner: []Resource with\nlimited attributes
Enumerator --> Scanner: []Resource with limited attributes
end
alt if deep mode enabled
hnote across: Details fetching phase
@ -23,5 +23,5 @@ Deserializer -> DetailsFetcher: Resource
DetailsFetcher -> Scanner: Resource with\nfull attributes
end
end
Scanner --> Driftctl: []Resource
Scanner --> driftctl: []Resource
@enduml

View File

@ -1,19 +1,23 @@
# Add a new remote provider
A remote provider in Driftctl is a cloud provider like AWS, Github, GCP or Azure.
Current architecture allows to add a new provider in a few step.
A remote provider in driftctl represents a cloud provider like AWS, GitHub, GCP or Azure.
Our current architecture allows to add a new provider in a few steps.
## Declaring the new remote provider
First you need to create a new directory in `pkg/remote/<provider name>`. It will sit next to already implemented one like `pkg/remote/aws`.
Inside this directory you will create a `init.go`. First thing to do will be to define the remote name constant:
Inside this directory, you will create a `init.go` file in which you will define the remote name constant:
```go
const RemoteAWSTerraform = "aws+tf"
```
`+tf` mean that we use terraform to retrieve details of resources, in the future maybe it will exist other way to read resource details.
`+tf` means that we use Terraform to retrieve resource's details, in the future, we may add other ways to read those details.
You will then create a function to initialize the provider and all resource's enumerators. The best way to do it would be to copy the function signature from another provider:
You will then create a function to init the provider and all the future resource enumerator. The best way to do it would be to copy the function signature from an other provider:
```go
func Init(
// Version required by the user
@ -22,37 +26,37 @@ func Init(
alerter *alerter.Alerter,
// Library that contains all providers
providerLibrary *terraform.ProviderLibrary,
// Library that contains the enumerators and details fetcher for each supported resources
// Library that contains enumerators and details fetchers for each supported resources
remoteLibrary *common.RemoteLibrary,
// progress display
// Progress displayer
progress output.Progress,
// Repository for all resource schema
// Repository for all resource schemas
resourceSchemaRepository *resource.SchemaRepository,
// Factory used to create driftctl resource
factory resource.ResourceFactory,
// Drifctl config directory (in which terraform provider is downloaded)
// driftctl configuration directory (where Terraform provider is downloaded)
configDir string) error {
// Define the default version of terraform provider to be used. When the user does not require a specific one
// You need to define the default version of the Terraform provider when the user does not specify one
if version == "" {
version = "3.19.0"
}
// This is this actual terraform provider creation
// Creation of the Terraform provider
provider, err := NewAWSTerraformProvider(version, progress, configDir)
if err != nil {
return err
}
// And then initialisation
// And then initialization
err = provider.Init()
if err != nil {
return err
}
// You'll need to create a new cache that will be use to cache fetched resources lists
// You'll need to create a new cache that will be used to cache fetched lists of resources
repositoryCache := cache.New(100)
// Deserializer is used to convert cty value return by terraform provider to driftctl Resource
// Deserializer is used to convert cty value returned by Terraform provider to driftctl Resource
deserializer := resource.NewDeserializer(factory)
// Adding the provider to the library
@ -60,10 +64,11 @@ func Init(
}
```
When it's done you'll create a `provider.go` file to contains your terraform provider representation. Again you should looks at other implementation :
Once done, you'll create a `provider.go` file to contain your Terraform provider representation. Again you should look at other implementation:
```go
// Define your actual provider representation, It is required to compose with terraform.TerraformProvider and to have a name and a version
// Please note that the name should match the real terraform provider name.
// Define your actual provider representation, it is required to compose with terraform.TerraformProvider, a name and a version
// Please note that the name should match the real Terraform provider name.
type AWSTerraformProvider struct {
*terraform.TerraformProvider
session *session.Session
@ -77,7 +82,7 @@ func NewAWSTerraformProvider(version string, progress output.Progress, configDir
version: version,
name: "aws",
}
// Use terraformproviderinstaller to retreive the provider if needed
// Use Terraform ProviderInstaller to retrieve the provider if needed
installer, err := tf.NewProviderInstaller(tf.ProviderConfig{
Key: p.name,
Version: version,
@ -90,7 +95,7 @@ func NewAWSTerraformProvider(version string, progress output.Progress, configDir
SharedConfigState: session.SharedConfigEnable,
}))
// Config is dependant on the teraform provider needs.
// ProviderConfig is dependent on the Terraform provider needs.
tfProvider, err := terraform.NewTerraformProvider(installer, terraform.TerraformProviderConfig{
Name: p.name,
DefaultAlias: *p.session.Config.Region,
@ -117,7 +122,7 @@ func (p *AWSTerraformProvider) Version() string {
}
```
The config returned in `GetProviderConfig` should be annotated with `cty` tags to be passed to the provider.
The configuration returned in `GetProviderConfig` should be annotated with `cty` tags to be passed to the provider.
```go
type githubConfig struct {
@ -127,15 +132,17 @@ type githubConfig struct {
}
```
You are now almost done. You'll need to make driftctl aware of this provider. Thus, the in `pkg/remote/remote.go` file, add your new constant in `supportedRemotes`:
You are now almost done. You'll need to make driftctl aware of this provider so in `pkg/remote/remote.go` add your new constant in `supportedRemotes`:
```go
var supportedRemotes = []string{
aws.RemoteAWSTerraform,
github.RemoteGithubTerraform,
}
```
And don't forget to modify the Activate function to be able to activate your new provider. You'll need to add a new case in the switch:
Don't forget to modify the Activate function. You'll need to add a new case in the switch statement:
```go
func Activate(remote, version string, alerter *alerter.Alerter,
providerLibrary *terraform.ProviderLibrary,
@ -157,18 +164,21 @@ func Activate(remote, version string, alerter *alerter.Alerter,
Your provider is now set up!
## Prepare Driftctl to support new resources
## Prepare driftctl to support new resources
Each new resource of the newly added provider will be located in `pkg/resource/<provider name>` directory. You need to create the latter and the `metadatas.go` file inside it.
New resource for the just added provider will be located in `pkg/resource/<provider name>`. You should create this directory and the `metadata.go` file.
Inside this file add a new function:
```go
func InitResourcesMetadata(resourceSchemaRepository resource.SchemaRepositoryInterface) {
}
```
And add a call to it in the `remote/<provider>/init.go` you created at first step.
Then, add a call to this function in the `remote/<provider>/init.go` file you created in the first step.
You also need to create a test schema for upcoming tests.
You also need to create a test schema for upcoming test.
Please use `TestCreateNewSchema` located in `test/schemas/schemas_test.go` to generate a schema file that will be used for the mocked provider.
Everything is now ready, you should [start adding new resources](new-resource.md)!

View File

@ -1,21 +1,24 @@
# Add new resources
First you need to understand how driftctl scan works. Here you'll find a global overview of the step that compose the scan:
First, you need to understand how `driftctl scan` works. Here you'll find a global overview of the steps that compose the scan:
![Diagram](media/generalflow.png)
And here you'll see a more detailed flow of the retrieving resource sequence from remote:
Then, you'll find below a more detailed flow of how we handle the enumeration and the fetching of resource's details from the remote:
![Diagram](media/resource.png)
## Defining the resource
First step would be to add a file under `pkg/resource/<providername>/resourcetype.go`.
This file will define a const string that will be the resource type identifier in driftctl.
Optionally, if your resource is to be supported by driftctl experimental deep mode, you can add a function that will be
applied to this resource when it's created. This allows to prevent useless diff to be displayed.
You can also add some metadata to fields so they are compared or displayed differently.
First step would be to add a file called `pkg/resource/<providername>/<resourcetype>.go`.
This file will define a string constant that will be the resource type identifier in driftctl.
Optionally, if your resource is to be supported by driftctl experimental deep mode, you can add a function that will be applied to this resource at creation.
This allows to prevent useless diffs to be displayed.
You can also add metadata to fields so that they are compared or displayed differently.
For example this defines the `aws_iam_role` resource:
```go
const AwsIamRoleResourceType = "aws_iam_role"
@ -35,6 +38,7 @@ func initAwsIAMRoleMetaData(resourceSchemaRepository resource.SchemaRepositoryIn
```
When it's done you'll have to add this function to the metadata initialisation located in `pkg/resource/<providername>/metadatas.go`:
```go
func InitResourcesMetadata(resourceSchemaRepository resource.SchemaRepositoryInterface) {
initAwsAmiMetaData(resourceSchemaRepository)
@ -42,6 +46,7 @@ func InitResourcesMetadata(resourceSchemaRepository resource.SchemaRepositoryInt
```
In order for you new resource to be supported by our terraform state reader you should add it in `pkg/resource/resource_types.go` inside the `supportedTypes` slice.
```go
var supportedTypes = map[string]struct{}{
"aws_ami": {},
@ -56,24 +61,24 @@ All the other attributes are represented inside a `map[string]interface`
Then you will have to implement two interfaces:
- Repositories are the way we decided to hide direct calls to sdk and pagination logic. It's a common abstraction pattern for data retrieval.
- `remote.comon.Enumerator` is used to read resources list. It will call the cloud provider SDK to get the list of resources.
- Repositories are the way we decided to hide direct calls to SDK and pagination logic. It's a common abstraction pattern for data retrieval.
- `remote.common.Enumerator` is used to enumerate resources. It will call the cloud provider SDK to get the list of resources.
For some resource it could make other call to enrich the resource with additional attributes when driftctl is used in deep mode
- `remote.comon.DetailsFetcher` is used to retrieve resource details. It makes a call to terraform provider `ReadResource`.
- `remote.common.DetailsFetcher` is used to retrieve resource's details. It makes a call to Terraform provider `ReadResource`.
This implementation is optional and is only needed if your resource type is to be supported by experimental deep mode.
Please also note that it exists a generic implementation as `remote.common.GenericDetailsFetcher` that can be used with most resource type.
Please also note that it exists a generic implementation called `remote.common.GenericDetailsFetcher` that can be used with most resource types.
### Repository
This will be the component that hide all the logic linked to your provider sdk. All provider have different way to implement pagination or to name function in their api.
Here we will name all listing function `ListAll<ResourceTypeName>`.
This will be the component that hides all the logic linked to your provider SDK. All providers have different ways to implement pagination or to name function in their API.
For aws we decided to split repositories using the amazon logic. So you'll find repositories for EC2, S3 and so on.
Here we will name all listing functions `ListAll<ResourceTypeName>`.
For AWS we decided to split repositories using the Amazon logic. So you'll find repositories for EC2, S3 and so on.
Some provider does not have this grouping logic. Keep in mind that like all our file/struct repositories should not be too big.
So it might be useful to create a grouping logic.
For our Github implementation the number of listing function was not that heavy so we created a unique repository for everything:
For our GitHub implementation the number of listing functions was not that heavy, so we created a unique repository for everything:
```go
type GithubRepository interface {
@ -109,37 +114,33 @@ func NewGithubRepository(config githubConfig, c cache.Cache) *githubRepository {
}
```
So as you can see this contains the logic to create the github client (it might be created outside the repository if it
makes sense to share it between multiple repositories). It also get a cache so every request is cached.
Driftctl sometimes needs to retrieve list of resources more than once, so we cache every request to avoid unnecessary call.
As you can see, this contains the logic to create the GitHub client (it might be created outside the repository if it makes sense to share it between multiple repositories).
driftctl, sometimes, needs to retrieve the list of resources more than once, so we cache each request to avoid unnecessary call.
### Enumerator
This is used to build a resources list. Enumerators can be found in `pkg/remote/<providername>/<type>_enumerator.go`. It will call the cloud provider SDK to get the list of resources.
Enumerators can be found in `pkg/remote/<providername>/<type>_enumerator.go`. It will call the cloud provider SDK to get the list of resources.
Note that at this point resources should not be entirely fetched.
Note that at this point, resources should not be entirely fetched and most of them will have empty attributes (e.g. only their id and type).
Most of the resource returned by enumerator have empty attributes: they only represent type and terraform id.
**There are exception to this**:
- Sometime, you will need some more information about resources to retrieve them using the provider they should be added to the resource attribute maps.
- For some more complex cases, middleware needs more information that the id and type and in order to make classic run of driftctl coherent with a run with deep mode activated,
these informations should be fetched manually by the enumerator using the remote sdk.
**There are exceptions to this**:
- Sometimes, you will need more information about resources for them to be fetched in the `DetailsFetcher`. For those cases, you will add specific attributes to the map of data.
- For complex cases (e.g. middlewares) where you would need driftctl to run as expected in deep and non-deep mode, you would need to enumerate resources as well as to fetch manually specific attributes, using the remote SDK, before adding them to the map of data.
Note that we use the classic repository to hide calls to the provider sdk.
You will probably need to at least add a listing function to list you new resource.
You can use an already implemented Enumerator as example.
You should use an already implemented Enumerator as example.
For example, to implement `aws_instance` resource you will need to add a `ListAllInstances()` function to `repository.EC2Repository`.
For example when implementing ec2_instance resource you will need to add a ListAllInstances() function to `repository.EC2Repository`.
It will be called by the enumerator to retrieve the instances list.
Bear in mind it will be called by the Enumerator to retrieve the list of instances.
Enumerator constructor could use these arguments:
- an instance of `Repository` that you will use to retrieved information about the resource
- an instance of `Repository` that you will use to retrieve information about the resource
- the global resource factory that should always be used to create a new `resource.Resource`
Enumerator then need to implement:
- `SupportedType() resource.ResourceType` that will return the constant you defined in the type file at first step
- `Enumerate() ([]*resource.Resource, error)` that will return the resource listing.
Enumerator then needs to implement:
- `SupportedType() resource.ResourceType` that will return the constant you defined in the type file
- `Enumerate() ([]*resource.Resource, error)` that will return the list of resources
```go
type EC2InstanceEnumerator struct {
@ -181,35 +182,36 @@ func (e *EC2InstanceEnumerator) Enumerate() ([]*resource.Resource, error) {
}
```
As you can see, listing error are treated in a particular way. Instead of failing and stopping the scan they will be handled, and an alert will be created.
As you can see, listing errors are treated in a particular way. Instead of failing and stopping the scan they will be handled, and an alert will be created.
So please don't forget to wrap these errors inside a `NewResourceListingError`.
For some provider error handling is not that coherent, so you might need to check in `pkg/remote/resource_enumeration_error_handler.go` and add a new case for your error.
You should test enumerator behavior when you do not have permission to enumerate resource, in the snippet above, `ListAllInstances` may return an `AccessDenied` error or so that should be handled.
You should test enumerator behavior when you do not have permission to enumerate resources. In the snippet above, `ListAllInstances` may return an `AccessDenied` error that should be handled.
Once the enumerator is written you have to add it to the remote initialization located in `pkg/remote/<providername>/init.go`:
Once the enumerator is written you have to add it to the remote init located in `pkg/remote/<providername>/init.go` :
```go
remoteLibrary.AddEnumerator(NewEC2InstanceEnumerator(s3Repository, factory))
```
### DetailsFetcher
DetailsFetcher are only used by driftctl experimental deep mode.
DetailsFetchers are only used by driftctl experimental deep mode.
This is the component that call terraform provider to retrieve the full attribute for each resource.
We do not want to reimplement what has already been done in every terraform provider, so you should not call the remote sdk there.
This is the component that call Terraform provider to retrieve all attributes for each resource.
We do not want to reimplement what has already been done in each Terraform provider. Thus, you should not call the remote SDK there.
If `common.GenericDetailsFetcher` satisfy your needs you should always prefer using it instead of implementing a custom `DetailsFetcher` in a new struct.
If `common.GenericDetailsFetcher` satisfies your needs you should always prefer using it instead of implementing a custom `DetailsFetcher` in a new struct.
The `DetailsFetcher` should also be added to `pkg/remote/<providername>/init.go` even if you use the generic version:
```go
remoteLibrary.AddDetailsFetcher(aws.AwsEbsVolumeResourceType, common.NewGenericDetailsFetcher(aws.AwsEbsVolumeResourceType, provider, deserializer))
```
***Don't forget to add unit tests after adding a new resource.***
You can find example of "functional" tests in pkg/remote/<type>_scanner_test.go
You can find example of **functional tests** in `pkg/remote/<type>_scanner_test.go`.
You should also add acceptance tests if you think it makes sense, they are located next to the resource definition described at first step.
You should also add **acceptance tests** if you think it makes sense. They are located next to the resource definition described in the first step.
More information about test can be found in [testing documentation](testing.md)
More information about adding tests can be found in [testing documentation](testing.md)

View File

@ -2,19 +2,19 @@
driftctl uses **unit tests**, **functional tests** and **acceptance tests**.
- A **unit test** is a very scoped test that test only a very precise part of code
- A **unit test** tests only a very specific part of code
- Pros:
- Very quick to develop, run and maintain
- Cons:
- Does not ensure that we do not break integration with other part of the code
- A **functional test** cover a larger part of the code than unit tests, but it mock external dependencies
- A **functional test** covers a larger part of the code than unit tests, but it mocks external dependencies
- Pros:
- Ensure that multiples components works well together
- Ensures that multiple components work well together
- Still quick to develop and run
- Cons:
- Mocking every external dependencies can be complicated
- Can be complicated to maintain since it is not scoped to a precise part of the code
- An **acceptance test** or **integration test** are the closest of the end-user behavior.
- Can be complicated to maintain since it is not scoped to a specific part of the code
- An **acceptance test** or **integration test** is the closest of the end-user behavior
- Pros:
- Very close to a real product usage
- Can cover regressions very efficiently