Installing Metagov¶
This documentation will walk through how to set up Metagov to work with a Driver. If you want to set up Metagov for local development, follow the instructions at Local Development instead.
There are three ways to work with Metagov:
Import the Django app directly into a Django-based Driver.
Communicate with a hosted version of Metagov via public (authenticated) HTTP endpoints.
Install your own version of Metagov and communicate locally via HTTP.
Instructions for #1 and #3 are written below. We are in the process of launching a hosted version of Metagov (#2). Once that’s set up, documentation on how to work with it will be linked here.
Using Metagov with a Django-based Driver¶
If you’re using PolicyKit as your Driver, head over to the PolicyKit Documentation for instructions on how to install PolicyKit on your server. The following instructions should work for all Django-based Drivers, not just PolicyKit.
Installation & Setup¶
Install the most recent version of Metagov directly or by adding it to your project’s requirements.txt:
pip install -e git+https://github.com/metagov/gateway.git#egg=metagov&subdirectory=metagov
You can then add Metagov to your project’s settings.py
. Note that you need to install each plugin that you want to use individually:
INSTALLED_APPS = [
... # existing apps
'metagov.plugins.slack',
'metagov.plugins.opencollective',
'metagov.plugins.github',
'metagov.plugins.sourcecred',
'metagov.plugins.loomio',
'metagov.plugins.discord',
'metagov.plugins.discourse',
'metagov.plugins.example', #for testing
'metagov.core',
]
You should also add a variable named METAGOV_SETTINGS to your settings.py
. This is used to configure plugins, for example:
METAGOV_SETTINGS = {
"SLACK": {
"CLIENT_ID": env("SLACK_CLIENT_ID", default=None),
"CLIENT_SECRET": env("SLACK_CLIENT_SECRET", default=None),
"SIGNING_SECRET": env("SLACK_SIGNING_SECRET", default=None),
"APP_ID": env("SLACK_APP_ID", default=None),
}
}
Read more about installing and configuring plugins.
Usage¶
Use the Metagov app by importing it from core:
from metagov.core.app import MetagovApp
metagov = MetagovApp()
community = metagov.get_community("metagov_slug")
community.enable_plugin(
name="opencollective",
config={"collective_slug": "mycollective", api_key": "XYZ"}
)
metagov_community.perform_action(plugin_name="opencollective", action_id="process-expense")
plugin_instance = community.get_plugin("opencollective")
plugin.start_process("vote", title="A test vote on OpenCollective")
Request Handlers¶
Metagov allows your Driver to talk to third party platforms through a class called MetagovRequestHandler. The handler takes care of two main types of communication: webhook events sent by the third party platform, and authentication (usually via oauth). Most of the business logic is handled by Metagov core and by the plugins, however you will need to make sure you’re exposing these endpoints.
You can do this in two ways. The endpoints are defined in metagov.core.urls.py
, which can be added to your project’s urls.py
. Alternatively, you can import MetagovRequestHandler directly and create your own urls and/or custom views which call it, for example:
from metagov.core.app import MetagovApp
from metagov.core.handlers import MetagovRequestHandler
metagov = MetagovApp()
metagov_handler = MetagovRequestHandler(metagov)
def plugin_auth_callback(request, plugin_name):
return metagov_handler.handle_oauth_callback(request, plugin_name)
Signal Handlers¶
When Metagov receives an event from a third-party platform, it emits a platform_event_created Signal. Your Driver should use signal receivers to capture these events and perform actions, for example:
from django.dispatch import receiver
@receiver(platform_event_created, sender=Github)
def github_event_receiver(sender, instance, event_type, data, initiator, **kwargs):
logger.debug(f"Received {event_type} event from {instance}")
# custom logic here
Metagov also emits a governance_process_updated signal when a GovernanceProcess is updated. Again, use receivers to capture the signal, for example:
@receiver(governance_process_updated, sender=GithubIssueReactVote)
def github_vote_updated_receiver(sender, instance, status, outcome, errors, **kwargs):
...
Note that in both of these examples, we use the sender parameter in the receiver decorator to filter out events and governance process updates not relevant to a specific platform. You can then check the sender’s subclass to determine the relevant platform - or ignore that information, if not needed.
To receive all events simply drop the sender argument from the receiver decorator’s parameters. If you want to recieve all Plugin events (but not other events) you’ll need to check that the sender is a subclass of Plugin:
@receiver(platform_event_created)
def metagov_event_receiver(sender, instance, event_type, data, initiator, **kwargs):
if not issubclass(sender, Plugin):
return
# do something...
Warning
It is easy to accidentally cause signals to be duplicated. Please do not assume all signals sent with Metagov are unique, and instead make use of dispatch_uid to weed out duplicate signals.
Installing your own version of Metagov and communicating via local HTTP¶
If you’re using Metagov alongside an existing system, make sure that you’re installing Metagov on the same server. This is necessary because Metagov and your system will communicate over the local network.
We’ll assume that you don’t have Python or apache2 installed on your Ubuntu system. These installation instructions have only been tested on Ubuntu 20.04.
Clone Metagov¶
Clone the Metagov repository (or your fork):
git clone https://github.com/metagov/metagov-prototype.git
cd metagov-prototype/metagov
Install Dependencies¶
Install Python3, and create and activate a new virtual environment by following this tutorial from Digital Ocean: how to install python on ubuntu 20.0.4.
Next, install the Metagov requirements:
pip install --upgrade pip
pip install -r requirements.txt
Set up the Metagov Environment¶
Set up an .env
file for storing secrets, and generate a new DJANGO_SECRET_KEY:
cp metagov/.env.example metagov/.env
DJANGO_SECRET_KEY=$(python manage.py shell -c 'from django.core.management import utils; print(utils.get_random_secret_key())')
echo "DJANGO_SECRET_KEY=$DJANGO_SECRET_KEY" >> metagov/.env
Next, open up your .env
file and set the following values:
DEBUG=False
ALLOWED_HOSTS=<your host>
DATABASE_PATH=<your database path> # Recommended: /var/databases/metagov/db.sqlite3
Make sure that your database path is not inside the Metagov repository directory, because you need to grant the apache2 user (www-data
) access to the database its parent folder.
Set up the Database and Static Files¶
Run python manage.py migrate
to set up your database.
Run python manage.py collectstatic
to create static files.
To test that everything is working correctly, enter the Django shell:
python manage.py shell_plus
Deploy with Apache web server¶
Now that you have Metagov installed on your server, you can deploy it on Apache web server. Make sure you have a domain dedicated to Metagov that is pointing to your server’s IP address.
Note
In the remaining examples, make sure to substitute the following values:
$METAGOV_REPO
is the path to your metagov-prototype repository root. (/metagov-prototype
)
$METAGOV_ENV
is the path to your metagov virtual environment. (/environments/metagov_env
)
$SERVER_NAME
is your server name. (metagov.mysite.com
)
Install and start apache2
sudo apt-get install apache2 libapache2-mod-wsgi-py3 sudo service apache2 start
Create a new apache2 config file:
cd /etc/apache2/sites-available # replace SERVER_NAME (ie metagov.mysite.org.conf) cp default-ssl.conf SERVER_NAME.conf
Edit the config file to look like this:
<IfModule mod_ssl.c> <VirtualHost _default_:443> ServerName $SERVER_NAME ServerAdmin webmaster@localhost Alias /static $METAGOV_REPO/metagov/static # 🚨 IMPORTANT: Restrict internal endpoints to local traffic 🚨 <Location /api/internal> Require ip YOUR-IP-ADDRESS </Location> # Grant access to static files for the API docs. <Directory $METAGOV_REPO/metagov/static> Require all granted </Directory> # Grant access to wsgi.py file. This is the Django server. <Directory $METAGOV_REPO/metagov/metagov> <Files wsgi.py> Require all granted </Files> </Directory> WSGIDaemonProcess metagov python-home=$METAGOV_ENV python-path=$METAGOV_REPO/metagov WSGIProcessGroup metagov WSGIScriptAlias / $METAGOV_REPO/metagov/metagov/wsgi.py # .. REST ELIDED </VirtualHost> </IfModule>
Test your config with
apache2ctl configtest
. You should get a “Syntax OK” as a response.Enable your site:
# activate your config a2ensite /etc/apache2/sites-available/$SERVER_NAME.conf # disable the default config sudo a2dissite 000-default-le-ssl.conf
Get an SSL certificate and set it up to auto-renew using LetsEncrypt:
sudo apt install certbot python3-certbot-apache sudo certbot --apache
Add the certificates to your
$SERVER_NAME.conf
file:SSLCertificateFile /etc/letsencrypt/live/$SERVER_NAME/fullchain.pem SSLCertificateKeyFile /etc/letsencrypt/live/$SERVER_NAME/privkey.pem
Reload the config:
systemctl reload apache2
Give the Apache2 user access to the database directory and the logging directory (update paths as needed):
sudo chown -R www-data:www-data /var/log/django sudo chown -R www-data:www-data /var/databases/metagov
Load your site in the browser.
Check for errors at
/var/log/apache2/error.log
and/var/log/django/debug.log
(or whatever logging path you have defined insettings.py
). Thewww-data
user should own the Django log directory and have write-access to the log file.
Any time you update the code, you’ll need to run
systemctl reload apache2
to reload the server.
Set up Celery¶
Metagov uses Celery to run scheduled tasks for Governance Processes and Plugin listeners.
Follow these instructions to run a celery daemon on your Ubuntu machine using systemd
.
For more information about configuration options, see the Celery Daemonization.
Create RabbitMQ virtual host¶
Install RabbitMQ and create a virtual host:
sudo apt-get install rabbitmq-server
sudo rabbitmqctl add_user 'username' 'password'
sudo rabbitmqctl add_vhost 'metagov-vhost'
sudo rabbitmqctl set_permissions -p 'metagov-vhost' 'username' '.*' '.*' '.*'
In metagov/settings.py
, set the CELERY_BROKER_URL
as follows, substituting values for your RabbitMQ username, password, and virtual host:
CELERY_BROKER_URL = "amqp://USERNAME:PASSWORD@localhost:5672/CUSTOMVIRTUALHOST"
Create celery user¶
If you don’t already have a celery
user, create one:
sudo useradd celery -d /home/celery -b /bin/bash
Give the celery
user access to necessary pid and log folders:
sudo useradd celery -d /home/celery -b /bin/bash
sudo mkdir /var/log/celery
sudo chown -R celery:celery /var/log/celery
sudo chmod -R 755 /var/log/celery
sudo mkdir /var/run/celery
sudo chown -R celery:celery /var/run/celery
sudo chmod -R 755 /var/run/celery
The celery
user will also need write access to the Django log file and the database.
To give celery
access, create a group that contains both www-data
(the apache2 user) and celery
.
For example, if your Django logs are in /var/log/django
and your database is in /var/databases
:
sudo groupadd www-and-celery
sudo usermod -a -G www-and-celery celery
sudo usermod -a -G www-and-celery www-data
# give the group read-write access to logs
sudo chgrp -R www-and-celery /var/log/django
sudo chmod -R 775 /var/log/django
# give the group read-write access to database
sudo chgrp -R www-and-celery /var/databases
sudo chmod -R 775 /var/databases
Create Celery configuration files¶
Next, you’ll need to create three Celery configuration files for Metagov:
/etc/conf.d/celery-metagov
¶
CELERYD_NODES="mg1"
# Absolute or relative path to the 'celery' command:
CELERY_BIN="$METAGOV_ENV/bin/celery"
# App instance to use
CELERY_APP="metagov"
# How to call manage.py
CELERYD_MULTI="multi"
# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=4"
# - %n will be replaced with the first part of the nodename.
# - %I will be replaced with the current child process index
# and is important when using the prefork pool to avoid race conditions.
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_LOG_LEVEL="INFO"
# you may wish to add these options for Celery Beat
CELERYBEAT_PID_FILE="/var/run/celery/metagov_beat.pid"
CELERYBEAT_LOG_FILE="/var/log/celery/metagov_beat.log"
/etc/systemd/system/celery-metagov.service
¶
[Unit]
Description=Celery Service
After=network.target
[Service]
Type=forking
User=celery
Group=celery
EnvironmentFile=/etc/conf.d/celery-metagov
WorkingDirectory=$METAGOV_REPO/metagov
ExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} \
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} \
--pidfile=${CELERYD_PID_FILE}'
ExecReload=/bin/sh -c '${CELERY_BIN} multi restart ${CELERYD_NODES} \
-A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \
--logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'
[Install]
WantedBy=multi-user.target
/etc/systemd/system/celerybeat-metagov.service
¶
[Unit]
Description=Celery Beat Service
After=network.target
[Service]
Type=simple
User=celery
Group=celery
EnvironmentFile=/etc/conf.d/celery-metagov
WorkingDirectory=$METAGOV_REPO/metagov
ExecStart=/bin/sh -c '${CELERY_BIN} -A ${CELERY_APP} \
beat --pidfile=${CELERYBEAT_PID_FILE} \
--logfile=${CELERYBEAT_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} \
--schedule=/var/run/celery/celerybeat-metagov-schedule'
[Install]
WantedBy=multi-user.target
After creating the files (and after any time you change them) run the following command:
sudo systemctl daemon-reload
Start Celery services¶
# Start RabbitMQ
sudo service rabbitmq-server start
# Start celery and celerybeat services
systemctl start celery-metagov celerybeat-metagov
# Check status of all celery services
systemctl status 'celery*'
systemctl list-units | grep celery
# Inspect celery metagov logs
less /var/log/celery/mg1.log # logs from the worker
less /var/log/celery/metagov_beat.log # logs from celerybeat
less /var/log/django/metagov.log # tasks should log to metagov's normal file handler
# Restart celery. You'll need to do this whenever the task code changes.
systemctl restart celery-metagov
Troubleshooting: If celery or celerybeat fail to start up as a service, try running celery directly to see if there are errors in your code:
celery -A metagov worker -l info --uid celery
celery -A metagov beat -l info --uid celery --schedule=/var/run/celery/celerybeat-metagov-schedule
Plugins¶
Some plugins require administrator setup before they can be used.
Slack¶
In order to use the Metagov Slack plugin, the Metagov server administrator needs to create a new Slack App and store its credentials on the server where Metagov is being used:
Click “Create New App” and select “From an app manifest”
Paste in the manifest.yaml file. Replace
$METAGOV_SERVER
with the URL for your Metagov server underredirect_urls
andrequest_url
. Optional: adjust scopes, events, and bot name as desired.Click “Manage Distribution”->”Activate Public Distribution.” This step is necessary if you want your app to be installable to multiple Slack workspaces.
In your Django app’s
.env
file, set the values for Slack’s App ID, Client ID, Client Secret, and Signing Secret.In the Slack app management page, verify the URLs for the OAuth callback, the Events Subscription Request URL, and the Interactivity Request URL.
Discord¶
Click “New Application”
3. Under General, add the interactions endpoint URL [SERVER_URL]/api/hooks/discord
3. Under OAuth2, add the redirect URL [SERVER_URL]/auth/discord/callback
4. Add a new Bot and enable these options:
Public Bot
Requires OAuth2 Code Grant
Presence Intent
Server Members Intent
In your Django app’s
.env
file, set the values for Discord’s bot token, client ID, client secret, and public key.
Twitter¶
Create a new account for your bot
Apply for a developer account for that account
Go to the developer portal and create a new Project (NOT a standalone app). Follow the prompts.
On completion, you should see the API Key, API Secret Key, and Bearer Token.
On the Metagov server, copy
metagov/plugins/twitter/.env.example
tometagov/plugins/twitter/.env
.In your Django app’s
.env
file, fill in the Twitter values. To get the values forTWITTER_ACCESS_TOKEN
andTWITTER_ACCESS_TOKEN_SECRET
, you’ll need to generate a new access token and secret in the developer portal.
Github¶
In order to ues the Metagov Github plugin, the Metagov server administrator needs to create a new Github app and link it to Metagov:
Create a metagov app and get the app ID. You can follow this guide. Don’t forget to set permissions and subscribe to events.
On Github, generate and download a private key. Put the private key in the github plugin folder.
In your Django app’s
.env
file, fill in the GitHub values. Put the app ID in the file as well as the path to your private key.
The plugin should now work. To use the app in their community, an admin will have to install the app manually on Github. They will then provide the installation ID and organization name as configuration parameters when enabling the plugin. We are working to make this process smoother in the future.