Installing Metagov¶

This documentation will walk through how to set up Metagov to work with a Driver. If you want to set up Metagov for local development, follow the instructions at Local Development instead.

There are three ways to work with Metagov:

Import the Django app directly into a Django-based Driver.
Communicate with a hosted version of Metagov via public (authenticated) HTTP endpoints.
Install your own version of Metagov and communicate locally via HTTP.

Instructions for #1 and #3 are written below. We are in the process of launching a hosted version of Metagov (#2). Once that’s set up, documentation on how to work with it will be linked here.

Using Metagov with a Django-based Driver¶

If you’re using PolicyKit as your Driver, head over to the PolicyKit Documentation for instructions on how to install PolicyKit on your server. The following instructions should work for all Django-based Drivers, not just PolicyKit.

Installation & Setup¶

Install the most recent version of Metagov directly or by adding it to your project’s requirements.txt:

pip install -e git+https://github.com/metagov/gateway.git#egg=metagov&subdirectory=metagov

You can then add Metagov to your project’s settings.py. Note that you need to install each plugin that you want to use individually:

INSTALLED_APPS = [
    ... # existing apps
    'metagov.plugins.slack',
    'metagov.plugins.opencollective',
    'metagov.plugins.github',
    'metagov.plugins.sourcecred',
    'metagov.plugins.loomio',
    'metagov.plugins.discord',
    'metagov.plugins.discourse',
    'metagov.plugins.example', #for testing
    'metagov.core',
]

You should also add a variable named METAGOV_SETTINGS to your settings.py. This is used to configure plugins, for example:

METAGOV_SETTINGS = {
    "SLACK": {
        "CLIENT_ID": env("SLACK_CLIENT_ID", default=None),
        "CLIENT_SECRET": env("SLACK_CLIENT_SECRET", default=None),
        "SIGNING_SECRET": env("SLACK_SIGNING_SECRET", default=None),
        "APP_ID": env("SLACK_APP_ID", default=None),
    }
}

Read more about installing and configuring plugins.

Usage¶

Use the Metagov app by importing it from core:

from metagov.core.app import MetagovApp

metagov = MetagovApp()
community = metagov.get_community("metagov_slug")

community.enable_plugin(
    name="opencollective",
    config={"collective_slug": "mycollective", api_key": "XYZ"}
)
metagov_community.perform_action(plugin_name="opencollective", action_id="process-expense")

plugin_instance = community.get_plugin("opencollective")
plugin.start_process("vote", title="A test vote on OpenCollective")

Request Handlers¶

Metagov allows your Driver to talk to third party platforms through a class called MetagovRequestHandler. The handler takes care of two main types of communication: webhook events sent by the third party platform, and authentication (usually via oauth). Most of the business logic is handled by Metagov core and by the plugins, however you will need to make sure you’re exposing these endpoints.

You can do this in two ways. The endpoints are defined in metagov.core.urls.py, which can be added to your project’s urls.py. Alternatively, you can import MetagovRequestHandler directly and create your own urls and/or custom views which call it, for example:

from metagov.core.app import MetagovApp
from metagov.core.handlers import MetagovRequestHandler

metagov = MetagovApp()
metagov_handler = MetagovRequestHandler(metagov)

def plugin_auth_callback(request, plugin_name):
    return metagov_handler.handle_oauth_callback(request, plugin_name)

Signal Handlers¶

When Metagov receives an event from a third-party platform, it emits a platform_event_created Signal. Your Driver should use signal receivers to capture these events and perform actions, for example:

from django.dispatch import receiver

@receiver(platform_event_created, sender=Github)
def github_event_receiver(sender, instance, event_type, data, initiator, **kwargs):
    logger.debug(f"Received {event_type} event from {instance}")
    # custom logic here

Metagov also emits a governance_process_updated signal when a GovernanceProcess is updated. Again, use receivers to capture the signal, for example:

@receiver(governance_process_updated, sender=GithubIssueReactVote)
def github_vote_updated_receiver(sender, instance, status, outcome, errors, **kwargs):
    ...

Note that in both of these examples, we use the sender parameter in the receiver decorator to filter out events and governance process updates not relevant to a specific platform. You can then check the sender’s subclass to determine the relevant platform - or ignore that information, if not needed.

To receive all events simply drop the sender argument from the receiver decorator’s parameters. If you want to recieve all Plugin events (but not other events) you’ll need to check that the sender is a subclass of Plugin:

@receiver(platform_event_created)
def metagov_event_receiver(sender, instance, event_type, data, initiator, **kwargs):
    if not issubclass(sender, Plugin):
        return
    # do something...

Warning

It is easy to accidentally cause signals to be duplicated. Please do not assume all signals sent with Metagov are unique, and instead make use of dispatch_uid to weed out duplicate signals.

Installing your own version of Metagov and communicating via local HTTP¶

If you’re using Metagov alongside an existing system, make sure that you’re installing Metagov on the same server. This is necessary because Metagov and your system will communicate over the local network.

We’ll assume that you don’t have Python or apache2 installed on your Ubuntu system. These installation instructions have only been tested on Ubuntu 20.04.

Clone Metagov¶

Clone the Metagov repository (or your fork):

git clone https://github.com/metagov/metagov-prototype.git
cd metagov-prototype/metagov

Install Dependencies¶

Install Python3, and create and activate a new virtual environment by following this tutorial from Digital Ocean: how to install python on ubuntu 20.0.4.

Next, install the Metagov requirements:

pip install --upgrade pip
pip install -r requirements.txt

Set up the Metagov Environment¶

Set up an .env file for storing secrets, and generate a new DJANGO_SECRET_KEY:

cp metagov/.env.example metagov/.env
DJANGO_SECRET_KEY=$(python manage.py shell -c 'from django.core.management import utils; print(utils.get_random_secret_key())')
echo "DJANGO_SECRET_KEY=$DJANGO_SECRET_KEY" >> metagov/.env

Next, open up your .env file and set the following values:

DEBUG=False
ALLOWED_HOSTS=<your host>
DATABASE_PATH=<your database path> # Recommended: /var/databases/metagov/db.sqlite3

Make sure that your database path is not inside the Metagov repository directory, because you need to grant the apache2 user (www-data) access to the database its parent folder.

Set up the Database and Static Files¶

Run python manage.py migrate to set up your database.

Run python manage.py collectstatic to create static files.

To test that everything is working correctly, enter the Django shell:

python manage.py shell_plus

Deploy with Apache web server¶

Now that you have Metagov installed on your server, you can deploy it on Apache web server. Make sure you have a domain dedicated to Metagov that is pointing to your server’s IP address.

Note

In the remaining examples, make sure to substitute the following values:

$METAGOV_REPO is the path to your metagov-prototype repository root. (/metagov-prototype)

$METAGOV_ENV is the path to your metagov virtual environment. (/environments/metagov_env)

$SERVER_NAME is your server name. (metagov.mysite.com)

Install and start apache2

sudo apt-get install apache2 libapache2-mod-wsgi-py3
sudo service apache2 start

Create a new apache2 config file:

cd /etc/apache2/sites-available
# replace SERVER_NAME (ie metagov.mysite.org.conf)
cp default-ssl.conf SERVER_NAME.conf

Edit the config file to look like this:

<IfModule mod_ssl.c>
        <VirtualHost _default_:443>
            ServerName $SERVER_NAME
            ServerAdmin webmaster@localhost
            Alias /static $METAGOV_REPO/metagov/static

            # 🚨 IMPORTANT: Restrict internal endpoints to local traffic 🚨
            <Location /api/internal>
                Require ip YOUR-IP-ADDRESS
            </Location>

            # Grant access to static files for the API docs.
            <Directory $METAGOV_REPO/metagov/static>
                    Require all granted
            </Directory>

            # Grant access to wsgi.py file. This is the Django server.
            <Directory $METAGOV_REPO/metagov/metagov>
                <Files wsgi.py>
                        Require all granted
                </Files>
            </Directory>

            WSGIDaemonProcess metagov python-home=$METAGOV_ENV python-path=$METAGOV_REPO/metagov
            WSGIProcessGroup metagov
            WSGIScriptAlias / $METAGOV_REPO/metagov/metagov/wsgi.py

            # .. REST ELIDED
        </VirtualHost>
</IfModule>

Test your config with apache2ctl configtest. You should get a “Syntax OK” as a response.

Enable your site:

# activate your config
a2ensite /etc/apache2/sites-available/$SERVER_NAME.conf

# disable the default config
sudo a2dissite 000-default-le-ssl.conf

Get an SSL certificate and set it up to auto-renew using LetsEncrypt:

sudo apt install certbot python3-certbot-apache
sudo certbot --apache

Add the certificates to your $SERVER_NAME.conf file:

SSLCertificateFile /etc/letsencrypt/live/$SERVER_NAME/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/$SERVER_NAME/privkey.pem

Reload the config:
```
systemctl reload apache2
```
Give the Apache2 user access to the database directory and the logging directory (update paths as needed):
```
sudo chown -R www-data:www-data /var/log/django
sudo chown -R www-data:www-data /var/databases/metagov
```
Load your site in the browser.

Check for errors at /var/log/apache2/error.log and /var/log/django/debug.log (or whatever logging path you have defined in settings.py). The www-data user should own the Django log directory and have write-access to the log file.

Any time you update the code, you’ll need to run systemctl reload apache2 to reload the server.

Set up Celery¶

Metagov uses Celery to run scheduled tasks for Governance Processes and Plugin listeners. Follow these instructions to run a celery daemon on your Ubuntu machine using systemd. For more information about configuration options, see the Celery Daemonization.

Create RabbitMQ virtual host¶

Install RabbitMQ and create a virtual host:

sudo apt-get install rabbitmq-server

sudo rabbitmqctl add_user 'username' 'password'
sudo rabbitmqctl add_vhost 'metagov-vhost'
sudo rabbitmqctl set_permissions -p 'metagov-vhost' 'username' '.*' '.*' '.*'

In metagov/settings.py, set the CELERY_BROKER_URL as follows, substituting values for your RabbitMQ username, password, and virtual host:

CELERY_BROKER_URL = "amqp://USERNAME:PASSWORD@localhost:5672/CUSTOMVIRTUALHOST"

Create celery user¶

If you don’t already have a celery user, create one:

sudo useradd celery -d /home/celery -b /bin/bash

Give the celery user access to necessary pid and log folders:

sudo useradd celery -d /home/celery -b /bin/bash
sudo mkdir /var/log/celery
sudo chown -R celery:celery /var/log/celery
sudo chmod -R 755 /var/log/celery

sudo mkdir /var/run/celery
sudo chown -R celery:celery /var/run/celery
sudo chmod -R 755 /var/run/celery

The celery user will also need write access to the Django log file and the database. To give celery access, create a group that contains both www-data (the apache2 user) and celery. For example, if your Django logs are in /var/log/django and your database is in /var/databases:

sudo groupadd www-and-celery
sudo usermod -a -G www-and-celery celery
sudo usermod -a -G www-and-celery www-data

# give the group read-write access to logs
sudo chgrp -R www-and-celery /var/log/django
sudo chmod -R 775 /var/log/django

# give the group read-write access to database
sudo chgrp -R www-and-celery /var/databases
sudo chmod -R 775 /var/databases

Create Celery configuration files¶

Next, you’ll need to create three Celery configuration files for Metagov:

`/etc/conf.d/celery-metagov`¶

CELERYD_NODES="mg1"

# Absolute or relative path to the 'celery' command:
CELERY_BIN="$METAGOV_ENV/bin/celery"

# App instance to use
CELERY_APP="metagov"

# How to call manage.py
CELERYD_MULTI="multi"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=4"

# - %n will be replaced with the first part of the nodename.
# - %I will be replaced with the current child process index
#   and is important when using the prefork pool to avoid race conditions.
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_LOG_LEVEL="INFO"

# you may wish to add these options for Celery Beat
CELERYBEAT_PID_FILE="/var/run/celery/metagov_beat.pid"
CELERYBEAT_LOG_FILE="/var/log/celery/metagov_beat.log"

`/etc/systemd/system/celery-metagov.service`¶

[Unit]
Description=Celery Service
After=network.target

[Service]
Type=forking
User=celery
Group=celery
EnvironmentFile=/etc/conf.d/celery-metagov
WorkingDirectory=$METAGOV_REPO/metagov
ExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} \
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} \
--pidfile=${CELERYD_PID_FILE}'
ExecReload=/bin/sh -c '${CELERY_BIN} multi restart ${CELERYD_NODES} \
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'

[Install]
WantedBy=multi-user.target

`/etc/systemd/system/celerybeat-metagov.service`¶

[Unit]
Description=Celery Beat Service
After=network.target

[Service]
Type=simple
User=celery
Group=celery
EnvironmentFile=/etc/conf.d/celery-metagov
WorkingDirectory=$METAGOV_REPO/metagov
ExecStart=/bin/sh -c '${CELERY_BIN} -A ${CELERY_APP}  \
beat --pidfile=${CELERYBEAT_PID_FILE} \
--logfile=${CELERYBEAT_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} \
--schedule=/var/run/celery/celerybeat-metagov-schedule'

[Install]
WantedBy=multi-user.target

After creating the files (and after any time you change them) run the following command:

sudo systemctl daemon-reload

Start Celery services¶

# Start RabbitMQ
sudo service rabbitmq-server start

# Start celery and celerybeat services
systemctl start celery-metagov celerybeat-metagov

# Check status of all celery services
systemctl status 'celery*'
systemctl list-units | grep celery

# Inspect celery metagov logs
less /var/log/celery/mg1.log          # logs from the worker
less /var/log/celery/metagov_beat.log # logs from celerybeat
less /var/log/django/metagov.log      # tasks should log to metagov's normal file handler

# Restart celery. You'll need to do this whenever the task code changes.
systemctl restart celery-metagov

Troubleshooting: If celery or celerybeat fail to start up as a service, try running celery directly to see if there are errors in your code:

celery -A metagov worker -l info --uid celery
celery -A metagov beat -l info --uid celery --schedule=/var/run/celery/celerybeat-metagov-schedule

Plugins¶

Some plugins require administrator setup before they can be used.

Slack¶

In order to use the Metagov Slack plugin, the Metagov server administrator needs to create a new Slack App and store its credentials on the server where Metagov is being used:

Go to https://api.slack.com/apps
Click “Create New App” and select “From an app manifest”
Paste in the manifest.yaml file. Replace $METAGOV_SERVER with the URL for your Metagov server under redirect_urls and request_url. Optional: adjust scopes, events, and bot name as desired.
Click “Manage Distribution”->”Activate Public Distribution.” This step is necessary if you want your app to be installable to multiple Slack workspaces.
In your Django app’s .env file, set the values for Slack’s App ID, Client ID, Client Secret, and Signing Secret.
In the Slack app management page, verify the URLs for the OAuth callback, the Events Subscription Request URL, and the Interactivity Request URL.

Discord¶

Go to https://discord.com/developers/applications
Click “New Application”

3. Under General, add the interactions endpoint URL [SERVER_URL]/api/hooks/discord 3. Under OAuth2, add the redirect URL [SERVER_URL]/auth/discord/callback 4. Add a new Bot and enable these options:

Public Bot

Requires OAuth2 Code Grant

Presence Intent

Server Members Intent

In your Django app’s .env file, set the values for Discord’s bot token, client ID, client secret, and public key.

Twitter¶

Create a new account for your bot
Apply for a developer account for that account
Go to the developer portal and create a new Project (NOT a standalone app). Follow the prompts.
On completion, you should see the API Key, API Secret Key, and Bearer Token.
On the Metagov server, copy metagov/plugins/twitter/.env.example to metagov/plugins/twitter/.env.
In your Django app’s .env file, fill in the Twitter values. To get the values for TWITTER_ACCESS_TOKEN and TWITTER_ACCESS_TOKEN_SECRET, you’ll need to generate a new access token and secret in the developer portal.

Github¶

In order to ues the Metagov Github plugin, the Metagov server administrator needs to create a new Github app and link it to Metagov:

Create a metagov app and get the app ID. You can follow this guide. Don’t forget to set permissions and subscribe to events.
On Github, generate and download a private key. Put the private key in the github plugin folder.
In your Django app’s .env file, fill in the GitHub values. Put the app ID in the file as well as the path to your private key.

The plugin should now work. To use the app in their community, an admin will have to install the app manually on Github. They will then provide the installation ID and organization name as configuration parameters when enabling the plugin. We are working to make this process smoother in the future.