268 lines
9.7 KiB
Markdown
268 lines
9.7 KiB
Markdown
# Testing
|
|
|
|
driftctl uses both **unit test** and **acceptance test**.
|
|
Acceptance test are not required, but at least a good unit test coverage is required for a PR to be merged.
|
|
This documentation section's goal is about how we manage our test suite in driftctl.
|
|
|
|
driftctl uses gotestsum to wrap `go test`, you can install required tools to run test with `make install-tools`
|
|
|
|
To run unit test simply run
|
|
```shell script
|
|
make install-tools
|
|
make test
|
|
```
|
|
|
|
Before the test suite starts, we run `golangci-lint`.
|
|
If there are any linter issues, you have to fix them first.
|
|
|
|
For the driftctl team, code coverage is very important as it helps show which part of your code is not covered.
|
|
We kindly ask you to check your coverage to ensure every important part of your code is tested.
|
|
We do not expect 100% coverage for every line of code, but at least every critical part of your code should be covered.
|
|
For example, we don't care about covering `NewStruct()` constructors if there is no big logic inside.
|
|
Remember, a covered code does not mean that every condition is tested and asserted, so be careful to test the right things.
|
|
A bug can still happen in a covered part of your code.
|
|
|
|
We use the golden file pattern to assert on results. Golden files could be updated with `-update flag`.
|
|
For example, I've made some modifications to s3 bucket policy, I could update golden files with the following command:
|
|
|
|
```shell script
|
|
go test ./pkg/remote/aws/ --update s3_bucket_policy_no_policy
|
|
```
|
|
|
|
⚠️ Beware that updating golden files may call external services.
|
|
In the example above, as we are using mocked AWS responses in json golden files, you should have to configure proper resources on AWS side before running an update.
|
|
For convenience, we try to put, as much as possible, terraform files used to generate golden files in test folders.
|
|
|
|
**A quick way to get started is to copy/paste an existing test and adapt it to your needs.**
|
|
|
|
## Unit testing
|
|
|
|
Unit testing should not use any external dependency, so we mock every call to the cloud provider's SDK (see below for more details on mocking)
|
|
|
|
### Mocks
|
|
|
|
In driftctl unit test suite, every call to the cloud provider's SDK should be mocked.
|
|
We use mocks generated by mockery in our tests.
|
|
See below each step to create a mock for a new AWS service (e.g. EC2).
|
|
|
|
1. Create a mock interface in `test/aws/ec2.go`
|
|
```go
|
|
package aws
|
|
|
|
import (
|
|
"github.com/aws/aws-sdk-go/service/ec2/ec2iface"
|
|
)
|
|
|
|
type FakeEC2 interface {
|
|
ec2iface.EC2API
|
|
}
|
|
```
|
|
2. Use mockery to generate a full mocked struct `mockery --name FakeS3 --dir ./test/aws`
|
|
3. Mock a response in your test (list IAM users for example)
|
|
|
|
```go
|
|
client := mocks.FakeIAM{}
|
|
client.On("ListUsersPages",
|
|
&iam.ListUsersInput{},
|
|
mock.MatchedBy(func(callback func(res *iam.ListUsersOutput, lastPage bool) bool) bool {
|
|
callback(&iam.ListUsersOutput{Users: []*iam.User{
|
|
{
|
|
UserName: aws.String("test-driftctl"),
|
|
},
|
|
{
|
|
UserName: aws.String("test-driftctl2"),
|
|
},
|
|
}}, true)
|
|
return true
|
|
})
|
|
).Once().Return(nil)
|
|
```
|
|
|
|
⚠️ If you have several mocks on the same method, the "mock" library will evaluate code in your `MatchedBy` multiple times even if the first parameter does not match.
|
|
It means your callback will always be called, this is an unwanted behaviour most of the time!
|
|
A workaround is to manage flags but this is an ugly solution, here is an example using a boolean flag:
|
|
|
|
```go
|
|
mockCalled := false
|
|
client := mocks.FakeIAM{}
|
|
client.On("ListUsersPages",
|
|
&iam.ListUsersInput{},
|
|
mock.MatchedBy(func(callback func(res *iam.ListUsersOutput, lastPage bool) bool) bool {
|
|
if mockCalled {
|
|
return false
|
|
}
|
|
callback(&iam.ListUsersOutput{Users: []*iam.User{
|
|
{
|
|
UserName: aws.String("test-driftctl"),
|
|
},
|
|
{
|
|
UserName: aws.String("test-driftctl2"),
|
|
},
|
|
}}, true)
|
|
mockCalled = true
|
|
return true
|
|
})
|
|
).Once().Return(nil)
|
|
```
|
|
|
|
🙏 We are still looking for a better way to handle this, contributions are welcome.
|
|
|
|
References:
|
|
- https://github.com/stretchr/testify/issues/504
|
|
- https://github.com/stretchr/testify/issues/1017
|
|
|
|
## Acceptance testing
|
|
|
|
driftctl provides a kind of acceptance test framework (`test/acceptance`) to help you run those tests.
|
|
The goal here is to apply some terraform code, and then run a series of **Check**.
|
|
A **Check** consists of running driftctl and checking for results using json output.
|
|
driftctl uses assertion struct to help you check output results. See below for more details.
|
|
|
|
Each acceptance test should be prefixed by `TestAcc_` and should be run using env var `DRIFTCTL_ACC=true`.
|
|
|
|
```shell script
|
|
DRIFTCTL_ACC=true go test -run=TestAcc_ ./pkg/resource/aws/aws_instance_test.go
|
|
```
|
|
|
|
### Credentials
|
|
|
|
Acceptance tests need credentials to perform real world action on cloud providers:
|
|
- Read/write access are required to perform terraform action
|
|
- Read only access is required for driftctl execution
|
|
|
|
Recommended way to run acc tests is to use two distinct credentials:
|
|
one for terraform related actions, and one for driftctl scan.
|
|
|
|
In our acceptance tests, we may need read/write permissions during specific contexts
|
|
(e.g. terraform init, apply, destroy)or lifecycle (PreExec and PostExec).
|
|
If needed, you can override environment variables in those contexts by adding `ACC_` prefix on env variables.
|
|
|
|
#### AWS
|
|
|
|
You can use `ACC_AWS_PROFILE` to override AWS named profile used for terraform operations.
|
|
|
|
```shell script
|
|
ACC_AWS_PROFILE=read-write-profile AWS_PROFILE=read-only-profile DRIFTCTL_ACC=true go test -run=TestAcc_ ./pkg/resource/aws/aws_instance_test.go
|
|
```
|
|
|
|
In the example below, the `driftctl` AWS profile must have read/write permissions and will be used
|
|
for both terraform operations and driftctl run.
|
|
|
|
This is **not** the recommended way to run tests as it may hide permissions issues.
|
|
|
|
```shell script
|
|
AWS_PROFILE=driftctl DRIFTCTL_ACC=true go test -run=TestAcc_ ./pkg/resource/aws/aws_instance_test.go
|
|
```
|
|
|
|
### Workflow
|
|
|
|
- **`OnStart`** You may run some code before everything
|
|
- **terraform apply**
|
|
- For each declared check loop
|
|
- **`PreExec`**
|
|
- **driftctl scan**
|
|
- **check results**
|
|
- **`PostExec`**
|
|
- **`OnEnd`**
|
|
- **terraform destroy**
|
|
|
|
⚠️ **driftctl tests handle terraform resources removal, but it is up to you to remove potential unmanaged resources added in `PreExec` step !**
|
|
|
|
### Example
|
|
|
|
The following test runs terraform to create an EC2 instance.
|
|
Then, we add a new tag (ENV: production) to the instance.
|
|
Finally, we check the drift.
|
|
|
|
```go
|
|
func TestAcc_AwsInstance_WithBlockDevices(t *testing.T) {
|
|
var mutatedInstanceId string
|
|
acceptance.Run(t, acceptance.AccTestCase{
|
|
// This path should contain terraform files
|
|
Path: "./testdata/acc/aws_instance",
|
|
// Pass args to driftctl execution
|
|
// DO NOT PASS --output flag as it is handled automatically by test runner
|
|
// You may use a .driftignore file in your test directory or use filters to limit driftctl scope
|
|
// Try to be minimalist as possible as test will be easier to maintain over time
|
|
Args: []string{"scan"}, // TODO add filter to limit scan scope to aws_instances
|
|
Checks: []acceptance.AccCheck{
|
|
{ // First check does not have any PreExec or PostExec
|
|
Check: func(result *acceptance.ScanResult, stdout string, err error) {
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
// Assert that no drift are detected
|
|
result.AssertDriftCountTotal(0)
|
|
// We could assert on analysis object directly
|
|
// Below we check for infra strictly in sync, beware that this check should fail
|
|
// if you run your acceptance test on a messy cloud provider state (existing dangling resources for example)
|
|
// without using filter or driftignore
|
|
//
|
|
// Note that the result struct is composed of analysis result AND assertion library
|
|
// You could use result.Equal() directly for example
|
|
result.True(result.Analysis.IsSync())
|
|
},
|
|
},
|
|
{
|
|
// In this PreExec, we retrieve the created instance ID and add a new tag
|
|
// using AWS SDK
|
|
// We store the instance ID in a var to assert on it after driftctl run
|
|
PreExec: func() {
|
|
client := ec2.New(awsutils.Session())
|
|
response, err := client.DescribeInstances(&ec2.DescribeInstancesInput{
|
|
Filters: []*ec2.Filter{
|
|
{
|
|
Name: aws.String("instance-state-name"),
|
|
Values: []*string{
|
|
aws.String("running"),
|
|
},
|
|
},
|
|
{
|
|
Name: aws.String("tag:Name"),
|
|
Values: []*string{
|
|
aws.String("test_instance_1"),
|
|
},
|
|
},
|
|
},
|
|
})
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
if len(response.Reservations[0].Instances) != 1 {
|
|
t.Fatal("Error, unexpected number of instances found, manual check required")
|
|
}
|
|
mutatedInstanceId = *response.Reservations[0].Instances[0].InstanceId
|
|
_, _ = client.CreateTags(&ec2.CreateTagsInput{
|
|
Resources: []*string{&mutatedInstanceId},
|
|
Tags: []*ec2.Tag{
|
|
{
|
|
Key: aws.String("Env"),
|
|
Value: aws.String("Production"),
|
|
},
|
|
},
|
|
})
|
|
},
|
|
// Check that driftctl detected a drift on manually modified instances
|
|
Check: func(result *acceptance.ScanResult, stdout string, err error) {
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
result.AssertResourceHasDrift(
|
|
mutatedInstanceId,
|
|
awsresources.AwsInstanceResourceType,
|
|
analyser.Change{
|
|
Change: diff.Change{
|
|
Type: diff.CREATE,
|
|
Path: []string{"Tags", "Env"},
|
|
From: nil,
|
|
To: "Production",
|
|
},
|
|
},
|
|
)
|
|
},
|
|
},
|
|
},
|
|
})
|
|
}
|
|
```
|