driftctl/doc/contributing/tests.md

# Testing

driftctl uses both **unit test** and **acceptance test**.
Acceptance test are not required, but at least a good unit test coverage is required for a PR to be merged.
This documentation section's goal is about how we manage our test suite in driftctl.

driftctl uses gotestsum to wrap `go test`, you can install required tools to run test with `make install-tools`

To run unit test simply run
```shell script
make install-tools
make test
```

Before the test suite starts, we run `golangci-lint`.
If there are any linter issues, you have to fix them first.

For the driftctl team, code coverage is very important as it helps show which part of your code is not covered.
We kindly ask you to check your coverage to ensure every important part of your code is tested.
We do not expect 100% coverage for every line of code, but at least every critical part of your code should be covered.
For example, we don't care about covering `NewStruct()` constructors if there is no big logic inside.
Remember, a covered code does not mean that every condition is tested and asserted, so be careful to test the right things.
A bug can still happen in a covered part of your code.

We use the golden file pattern to assert on results. Golden files could be updated with `-update flag`.
For example, I've made some modifications to s3 bucket policy, I could update golden files with the following command:

```shell script
go test ./pkg/remote/aws/ --update s3_bucket_policy_no_policy
```

⚠️ Beware that updating golden files may call external services.
In the example above, as we are using mocked AWS responses in json golden files, you should have to configure proper resources on AWS side before running an update.
For convenience, we try to put, as much as possible, terraform files used to generate golden files in test folders.

**A quick way to get started is to copy/paste an existing test and adapt it to your needs.**

## Unit testing

Unit testing should not use any external dependency, so we mock every call to the cloud provider's SDK (see below for more details on mocking)

### Mocks

In driftctl unit test suite, every call to the cloud provider's SDK should be mocked.
We use mocks generated by mockery in our tests.
See below each step to create a mock for a new AWS service (e.g. EC2).

1. Create a mock interface in `test/aws/ec2.go`
```go
package aws

import (
	"github.com/aws/aws-sdk-go/service/ec2/ec2iface"
)

type FakeEC2 interface {
	ec2iface.EC2API
}
```
2. Use mockery to generate a full mocked struct `mockery --name FakeS3 --dir ./test/aws`
3. Mock a response in your test (list IAM users for example)

```go
client := mocks.FakeIAM{}
client.On("ListUsersPages",
    &iam.ListUsersInput{},
    mock.MatchedBy(func(callback func(res *iam.ListUsersOutput, lastPage bool) bool) bool {
        callback(&iam.ListUsersOutput{Users: []*iam.User{
            {
                UserName: aws.String("test-driftctl"),
            },
            {
                UserName: aws.String("test-driftctl2"),
            },
        }}, true)
        return true
    })
).Once().Return(nil)
```

⚠️ If you have several mocks on the same method, the "mock" library will evaluate code in your `MatchedBy` multiple times even if the first parameter does not match.
It means your callback will always be called, this is an unwanted behaviour most of the time!
A workaround is to manage flags but this is an ugly solution, here is an example using a boolean flag:

```go
mockCalled := false
client := mocks.FakeIAM{}
client.On("ListUsersPages",
    &iam.ListUsersInput{},
    mock.MatchedBy(func(callback func(res *iam.ListUsersOutput, lastPage bool) bool) bool {
        if mockCalled {
            return false
        }
        callback(&iam.ListUsersOutput{Users: []*iam.User{
            {
                UserName: aws.String("test-driftctl"),
            },
            {
                UserName: aws.String("test-driftctl2"),
            },
        }}, true)
        mockCalled = true
        return true
    })
).Once().Return(nil)
```

🙏 We are still looking for a better way to handle this, contributions are welcome.

References:
- https://github.com/stretchr/testify/issues/504
- https://github.com/stretchr/testify/issues/1017

## Acceptance testing

driftctl provides a kind of acceptance test framework (`test/acceptance`) to help you run those tests.
The goal here is to apply some terraform code, and then run a series of **Check**.
A **Check** consists of running driftctl and checking for results using json output.
driftctl uses assertion struct to help you check output results. See below for more details.

Each acceptance test should be prefixed by `TestAcc_` and should be run using env var `DRIFTCTL_ACC=true`.

```shell script
DRIFTCTL_ACC=true go test -run=TestAcc_ ./pkg/resource/aws/aws_instance_test.go
```

### Credentials

Acceptance tests need credentials to perform real world action on cloud providers:
- Read/write access are required to perform terraform action
- Read only access is required for driftctl execution

Recommended way to run acc tests is to use two distinct credentials:
one for terraform related actions, and one for driftctl scan.

In our acceptance tests, we may need read/write permissions during specific contexts
(e.g. terraform init, apply, destroy)or lifecycle (PreExec and PostExec).
If needed, you can override environment variables in those contexts by adding `ACC_` prefix on env variables.

#### AWS

You can use `ACC_AWS_PROFILE` to override AWS named profile used for terraform operations.

```shell script
ACC_AWS_PROFILE=read-write-profile AWS_PROFILE=read-only-profile DRIFTCTL_ACC=true go test -run=TestAcc_ ./pkg/resource/aws/aws_instance_test.go
```

In the example below, the `driftctl` AWS profile must have read/write permissions and will be used
for both terraform operations and driftctl run.

This is **not** the recommended way to run tests as it may hide permissions issues.

```shell script
AWS_PROFILE=driftctl DRIFTCTL_ACC=true go test -run=TestAcc_ ./pkg/resource/aws/aws_instance_test.go
```

### Workflow

- **`OnStart`** You may run some code before everything
- **terraform apply**
- For each declared check loop
  - **`PreExec`**
  - **driftctl scan**
  - **check results**
  - **`PostExec`**
- **`OnEnd`**
- **terraform destroy**

⚠️ **driftctl tests handle terraform resources removal, but it is up to you to remove potential unmanaged resources added in `PreExec` step !**

### Example

The following test runs terraform to create an EC2 instance.
Then, we add a new tag (ENV: production) to the instance.
Finally, we check the drift.

```go
func TestAcc_AwsInstance_WithBlockDevices(t *testing.T) {
	var mutatedInstanceId string
	acceptance.Run(t, acceptance.AccTestCase{
		// This path should contain terraform files
		Path: "./testdata/acc/aws_instance",
		// Pass args to driftctl execution
		// DO NOT PASS --output flag as it is handled automatically by test runner
		// You may use a .driftignore file in your test directory or use filters to limit driftctl scope
		// Try to be minimalist as possible as test will be easier to maintain over time
		Args: []string{"scan"}, // TODO add filter to limit scan scope to aws_instances
		Checks: []acceptance.AccCheck{
			{ // First check does not have any PreExec or PostExec
				Check: func(result *acceptance.ScanResult, stdout string, err error) {
					if err != nil {
						t.Fatal(err)
					}
					// Assert that no drift are detected
					result.AssertDriftCountTotal(0)
					// We could assert on analysis object directly
					// Below we check for infra strictly in sync, beware that this check should fail
					// if you run your acceptance test on a messy cloud provider state (existing dangling resources for example)
					// without using filter or driftignore
					//
					// Note that the result struct is composed of analysis result AND assertion library
					// You could use result.Equal() directly for example
                    result.True(result.Analysis.IsSync())
				},
			},
			{
				// In this PreExec, we retrieve the created instance ID and add a new tag
				// using AWS SDK
				// We store the instance ID in a var to assert on it after driftctl run
				PreExec: func() {
					client := ec2.New(awsutils.Session())
					response, err := client.DescribeInstances(&ec2.DescribeInstancesInput{
						Filters: []*ec2.Filter{
							{
								Name: aws.String("instance-state-name"),
								Values: []*string{
									aws.String("running"),
								},
							},
							{
								Name: aws.String("tag:Name"),
								Values: []*string{
									aws.String("test_instance_1"),
								},
							},
						},
					})
					if err != nil {
						t.Fatal(err)
					}
					if len(response.Reservations[0].Instances) != 1 {
						t.Fatal("Error, unexpected number of instances found, manual check required")
					}
					mutatedInstanceId = *response.Reservations[0].Instances[0].InstanceId
					_, _ = client.CreateTags(&ec2.CreateTagsInput{
						Resources: []*string{&mutatedInstanceId},
						Tags: []*ec2.Tag{
							{
								Key:   aws.String("Env"),
								Value: aws.String("Production"),
							},
						},
					})
				},
				// Check that driftctl detected a drift on manually modified instances
				Check: func(result *acceptance.ScanResult, stdout string, err error) {
					if err != nil {
						t.Fatal(err)
					}
					result.AssertResourceHasDrift(
						mutatedInstanceId,
						awsresources.AwsInstanceResourceType,
                        analyser.Change{
                            Change: diff.Change{
                                Type: diff.CREATE,
                                Path: []string{"Tags", "Env"},
                                From: nil,
                                To:   "Production",
                            },
                        },
					)
				},
			},
		},
	})
}
```