Skip to content

AWS integration feature #624

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
henkberendsen wants to merge 30 commits into
base: main
Choose a base branch
from
Open

AWS integration feature #624

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
b4c77aa
AWS synchronisation button (#8)
henkberendsen Feb 24, 2023
4562dc1
Add django-tinymce and django-bleach, boto3 and moto dependencies
1058274 Feb 24, 2023
5ffe375
Add logger and replace prints with logs
1058274 Feb 26, 2023
7786fcb
Add function to create AWS organization
1058274 Feb 26, 2023
471f67b
Add unit tests for creating AWS organization
1058274 Feb 26, 2023
cddde7e
Added logger setlevel (#20)
Jer111 Mar 3, 2023
18f7046
Db sync (#16)
Jer111 Mar 14, 2023
88adc62
Db sync (#25)
mitchellboes Mar 14, 2023
76095bd
Added function to generate which users have to be invited after the s…
mitchellboes Mar 14, 2023
5990950
Create and attach SCP policies (#29)
1058274 Apr 4, 2023
0ccdc03
12 moto helper (#36)
Jer111 Apr 11, 2023
47cfcf7
Add checks for edge cases between AWS and Giphouse databases (#37)
mitchellboes Apr 13, 2023
a25799f
Extraction of AWS data
flam123 Apr 14, 2023
5d5225a
AWS synchronization pipeline (and integration bug fixes) (#42)
1058274 Apr 22, 2023
1a0fb69
Dedicated module for AWS helper data structures (#47)
1058274 Apr 25, 2023
2674dd2
44 class for handling all aws api calls (#50)
henkberendsen May 9, 2023
8b2be91
Merge #41 (check double iteration names, members in correct iteration…
mitchellboes May 9, 2023
5b75d24
Refactor pipeline preconditions (#54)
1058274 May 10, 2023
5a5bace
Refactor creating course OU and attaching policy (#57)
1058274 May 11, 2023
df8d8f8
Remove unnecessary moto_client variables in test_awsapitalker.py (#56)
henkberendsen May 12, 2023
dd77a9d
Refactor generate synchronization list and extract AWS tree
Jer111 May 16, 2023
9b6a427
Refactor create and move accounts function
flam123 May 23, 2023
188b52b
62 policy id and tag fields on frontend panel (#65)
Jer111 May 30, 2023
7d06c1e
Refactor AWSSync pipeline (#67)
Jer111 Jun 2, 2023
77280eb
changes for resolving security (#68)
Fiflakos Jun 6, 2023
3055e48
Documentation AWS integration feature (#70)
1058274 Jun 6, 2023
429f64c
Finishing touches (#71)
henkberendsen Jun 6, 2023
8fdfe90
Documentation AWS configuration (#72)
mitchellboes Jun 9, 2023
e511157
Fix migrations
henkberendsen Jun 10, 2023
e619ebb
Apply bug fix which was accidentally reverted in a previous commit
henkberendsen Jun 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 104 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ This is the code for the website of [GiPHouse](http://giphouse.nl/) powered by [

## Table of Contents
- [GiPHouse website](#giphouse-website)
- [Table of Contents](#table-of-contents)
- [Features](#features)
- [Authentication and Users](#authentication-and-users)
- [GitHub OAuth](#github-oauth)
Expand All @@ -16,15 +17,18 @@ This is the code for the website of [GiPHouse](http://giphouse.nl/) powered by [
- [Questionnaires](#questionnaires)
- [Room Reservations](#room-reservations)
- [Course, Project and Static Information](#course-project-and-static-information)
- [Projects and Repositories](#projects-and-repositories)
- [Projects, Repositories and AWS](#projects-repositories-and-aws)
- [GitHub Synchronization](#github-synchronization)
- [AWS Synchronization](#aws-synchronization)
- [Mailing Lists](#mailing-lists)
- [Tasks](#tasks)
- [Styling](#styling)
- [Development and Contributing](#development-and-contributing)
- [Getting Started](#getting-started)
- [Logging into the Backend](#logging-into-the-backend)
- [Registering a GitHub App for repository synchronisation](#registering-a-github-app-for-repository-synchronisation)
- [Registering a G Suite service account for mailing list synchronisation](#registering-a-g-suite-service-account-for-mailing-list-synchronisation)
- [Registering an AWS environment for synchronisation](#registering-an-aws-environment-for-synchronisation)
- [Dependency Management](#dependency-management)
- [Fixtures](#fixtures)
- [Tests](#tests)
Expand All @@ -44,7 +48,7 @@ This is the code for the website of [GiPHouse](http://giphouse.nl/) powered by [
- [`build-docker` job](#build-docker-job)
- [`deploy` job](#deploy-job)
- [Secrets](#secrets)
- [Server](#server)
- [Server Configuration](#server-configuration)
- [Keeping Everything Up to Date](#keeping-everything-up-to-date)

## Features
Expand Down Expand Up @@ -122,10 +126,10 @@ The room reservation is built using [FullCalendar](https://fullcalendar.io/), a
### Course, Project and Static Information
Admin users can add information about the course lectures and the projects in the backend. There are also a small amount of static HTML webpages with information about GiPHouse.

### Projects and Repositories
### Projects, Repositories and AWS
#### GitHub Synchronization
The projects module provides synchronisation functionality with a GitHub organization using the [GitHub API v3](https://developer.github.com/v3/). For this, a repository model is included in Django. Project(team)s can have one or multiple repositories, which are then synchronised with GitHub. For this functionality, a [GitHub App](https://developer.github.com/v3/apps/) must be registered and installed in the organization. Details on this are explained later.

#### GitHub Synchronization
Projects and repositories contain a field `github_team_id` and `github_repo_id` that corresponds to the respective `id` of the object on GitHub. These fields are automatically set and should not be touched under normal circumstances. Teams and repositories on GitHub that do not match one of these id's will not be touched by the GitHub synchronization.
If the `github_team_id` or `github_repo_id` are `None`, it is assumed the objects do not exist and new objects will be created on synchronization (except for archived projects and teams).

Expand All @@ -149,6 +153,72 @@ Synchronization can only be initialized via actions on specific sets of objects

Synchronization currently does not regard the role of directors of GipHouse. This needs to be configured manually. Note that it is however not possible to add directors manually to a team on GitHub, since they will be removed after each sync.

#### AWS Synchronization
The projects module provides synchronisation functionality with [AWS Organizations](https://aws.amazon.com/organizations/) using the official [boto3 Python AWS SDK](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html).
The AWS synchronisation process only applies to the current semester and is one-directional (from GiPHouse to AWS, but not vice versa).

Each project in the current semester with a team mailing list gets its own AWS member account that is part of GiPHouse's AWS organization.
Since all AWS member accounts have isolated environments, each team is able to configure their own AWS environment as desired.
The AWS member accounts are restricted in their abilities using a pre-configured [SCP policy](https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps.html) that is applied to the course semester Organizational Unit (OU) where all team member accounts reside.
For example, the SCP policy can be set such that only (certain types of) [EC2](https://aws.amazon.com/ec2/) instances may be launched.
Such specific configuration details can be found under the [Getting Started](#registering-an-aws-environment-for-synchronisation) section.

The entire AWS synchronization process, also referred to as the pipeline, can be initiated in the Django admin interface under Projects by pressing the large `SYNCHRONIZE PROJECTS OF THE CURRENT SEMESTER TO AWS` at the top-right and roughly goes through the following stages:

1. Preliminary checks
- Pipeline preconditions
1. Locatable boto3 credentials and successful AWS API connection
2. Check allowed AWS API actions based on IAM policy of caller
3. Existing organization for AWS API caller
4. AWS API caller acts under same account ID as organization's management account ID
5. SCP policy type feature enabled for organization
- Edge case checks
1. No duplicate course semester OU names
2. Create current course semester OU (if non-existent)
3. Attach SCP policy to current course semester OU (if non-existent)
4. Synchronization
- Determine new accounts to be invited based on AWS and GiPHouse data.
5. Create new AWS member accounts in AWS organization
6. Move new AWS member accounts to course semester OU

![pipeline-flowchart](resources/pipeline-flowchart.drawio.png)

After the synchronization process has finished, a response box is returned indicating success (green), soft-fail (orange) or hard-fail (red).
Verbose details for each synchronization run is logged using the `logging` module and can be accessed in the backend. for example to inspect causes of failed runs.

An example of a possible AWS Organizations environment in the form a tree is the following:
```
base (root/OU)
│
├── Fall 2022 (OU)
│ ├── team-alice@giphouse.nl (member account)
│ └── team-bob@giphouse.nl (member account)
│
├── Spring 2023 (OU)
│ ├── team-charlie@giphouse.nl (member account)
│ └── team-david@giphouse.nl (member account)
│
└── admin@giphouse.nl (management account)
```

The "base" (either root or OU), under which all relevant resources are created and operated on as part of the synchronization process, offers flexibility by being configurable in the Django admin panel.

When an AWS member account has been created for a team mailing list as part of an AWS Organization, an e-mail is sent by AWS.
This process might take some time and is under AWS' control.
It is important to be aware that gaining initial access to the member account is only possible by formally resetting the password; there is no other way.
Also note well that each project team member will receive such mails because the team mailing list works as a one-to-many mail forwarder.

By default, all newly created member accounts under an AWS organization are placed under root.
Once the member accounts have been created under root, they are automatically moved to the current course semester OU.
Note that: (1) it is not possible to create a new member account that gets placed in a specific OU and (2) new requested member accounts can not be moved unless the account creation has been finalized to `SUCCESS` and AWS does not specify an upper bound for the time it takes for a new member account creation to finalize.

Due to point (2), the code contains the variables `ACCOUNT_REQUEST_MAX_ATTEMPTS` for the number of times to check the status of a new member account request, and `ACCOUNT_REQUEST_INTERVAL_SECONDS` for the time to wait in between attempts.
These values are currently hard-coded and can be tweaked, should they cause problems with the synchronization process.

Points (1) and (2) pose the possibility of there being a time period between having a newly created member account under root and moving it to its corresponding OU that is restricted with an attached SCP policy, possibly giving the member account excessive permissions.
To mitigate this risk, every newly created account comes with a pre-defined [tag](https://docs.aws.amazon.com/tag-editor/latest/userguide/tagging.html) and the SCP policy attached to root should deny all permissions for accounts under root with the specific tag (see [Getting Started](#registering-an-aws-environment-for-synchronisation) section for more details on SCP policy and tag configuration).
The tag then automatically gets removed after the account has been moved to its destination course semester OU.

### Mailing Lists
Admin users can create mailing lists using the Django admin interface. A mailing list can be connected to projects, users and 'extra' email addresses that are not tied to a user. Relating a mailing list to a project implicitly makes the members of that project a member of the mailing list. Removing a mailing list in the Django admin will result in the corresponding mailing list to be archived or deleted in G suite during the next synchronization, respecting the 'archive instead of delete' property of the deleted mailing list. To sync a mailing list with G Suite, one can run the management command: `./manage.py sync_mailing_list` or use the button in the model admin. This will sync all mailing lists and the automatic lists into G Suite at the specified domain.

Expand Down Expand Up @@ -214,6 +284,35 @@ To enable the synchronisation feature of mailing lists to G Suite, a project and

The credentials and admin user can then be setup in Github secrets. The email of the G Suite user used to manage to the G Suite domain has to be stored in the Github secret `DJANGO_GSUITE_ADMIN_USER`. The credentials json file has to be `base64` encoded and stored in the Github secret `DJANGO_GSUITE_ADMIN_CREDENTIALS_BASE64` (you can use the linux command `base64` for encoding the json file).

#### Registering an AWS environment for synchronisation
To enable the AWS synchronisation feature, the following points need to be configured only once in advance:

- Create AWS Organizations with all features enabled.
- Ensure Service Control Policies (SCPs) feature is enabled.
- Enable AWS CloudTrail for logging account activity (optional, recommended).
- Increase AWS Organizations quota for maximum number of member accounts to expected amount.
- Default quota is set to 10.
- Expected amount should be at least the number of unique projects in the current semester.
- Set AWS API credentials for `boto3` as environment variables.
- `AWS_ACCESS_KEY_ID`: access key for AWS account.
- `AWS_SECRET_ACCESS_KEY`: secret key for AWS account.
- **(!)** Currently not automated using GitHub secrets.
- AWS API caller has sufficient permissions for all synchronization actions.
- AWS API caller is IAM user acting on behalf of the management account of the AWS Organizations.
- Ensure you are logged in as the management account when creating the IAM user for API access.
- Pre-defined IAM policies `AWSOrganizationsFullAccess` and `IAMFullAccess` are more than sufficient.
- **(!)** Restrictive custom IAM policy adhering to the principle of least privilege is recommended.
- Create SCP policies under AWS Organizations.
- SCP policy restricting member accounts under root with a custom key-value tag.
- Manually attach policy to root.
- SCP policy for course semester OUs (e.g. to only allow EC2 resources of a specific type).
- Automatically attached to course semester OUs.
- Configure a current AWS Policy under `Projects/AWS Policies` in the Django admin panel.
- Set the base ID (root or OU) value.
- Set the SCP policy ID value for course semester OUs.
- Set the restricting custom key-value tag specified in the root SCP policy.
- Mark the `Is current policy` checkbox to make the configuration active.

### Dependency Management
The Python dependencies are managed using a tool called [Poetry](https://python-poetry.org/), which automatically creates virtual environments that ease development and makes it easy to manage the dependencies. See the [Poetry documentation](https://python-poetry.org/docs/) for more information.

Expand Down Expand Up @@ -312,6 +411,7 @@ This repository is public and the GitHub Actions CI runner logs are also public,
The current server is an Amazon Web Services Elastic Cloud Computing (AWS EC2) instance that runs Ubuntu 18.04. EC2 instances have a default `ubuntu` user, that is allowed to execute `sudo` without password. The `docker-compose.yaml` file includes all services that are necessary to run the website in a production environment. That is why Docker is the only dependency on the host.

These steps are the necessary setup for a production server.

1. Add the SSH public keys of engineers to the `authorized_keys` of the `ubuntu` user.
2. Disable SSH password login.
3. Install `docker` and `docker-compose`.
Expand Down
Loading