08_Deploy_tutorial

This page is a moving target
- Things may get out of date, due to the incompleteness of everything
- Other contributors nee to start working on their parts of the system
Assumes an archlinux machine/vm
Assumes root user everywhere
Change usernames and passwords from examples here

Overview

VM Infrastructure

the meili executable only cares about its config.toml, in it contains values for:
- which data dir to write to
- api key
- exposed host address port
- performance tuning params
the mysql part only cares about the configs stored in /etc/my.cnf.d/my.cnf, lots of things to tweak here
nginx only cares about whats in /etc/nginx/, other than basic http/https/server configs:
- ayase-quarts unix socket it has to proxy pass to
- meili's debug page it has to proxy passs to if its meili is in dev mode
- the media storage directory
ayase quart has more moving parts:
- the python deps:
  - uses virtualenv-wrapper to switch between sets of deps easily
  - workon your_env; pip install ~/aq/requirements.txt
- hypercorn:
  - connection to nginx via uds
  - manages the app workers
  - configs in ~/hyperconf.toml
- the code: git pull main to pull in the source
- the boards we want to expose: meta.py
- the connection to meili and mysql: configs.py

Prelim setup

# find good mirrors
reflector --sort rate --latest 10 --protocol https --country US --save /etc/pacman.d/mirrorlist

# update package db + keyring first
pacman -Sy && yes Y | pacman -S archlinux-keyring

# minimum packages
# extra favorites: neovim zstd stow ncdu htop eza bat fd xh ripgrep tmux rsync
yes Y | pacman -S git chrony cronie curl ufw

# start base services
systemctl enable --now cronie chronyd ufw

# upgrade full system & remove packages
yes Y | pacman -Syu && yes Y | pacman -Scc


# firewall up
ufw allow ssh
ufw allow http
ufw allow https

# create meili dir
mkdir /srv/meili

# create media dir
mkdir /srv/http/media

# download a thumbnail dump
curl -L -o /srv/http/media/4ch-be-thumbs_20151031.tar.gz https://archive.org/download/4ch.be/4ch-be-thumbs_20151031.tar.gz

# unpack thumbnail dump
tar -xzf /srv/http/media/4ch-be-thumbs_20151031.tar.gz

# rename
mv /srv/http/media/boards /srv/http/media/neo

# download aq
git clone --depth 1 https://github.com/sky-cake/ayase-quart.git ~/aq

# generate secret file
tr -dc A-Za-z0-9 < /dev/urandom | head -c 64 > ~/aq/src/secret.txt

# reboot, check everything still works
reboot -h now

Native setup (no docker/containers)

Files edited with nano have their example content later down

# packages for native
yes y | pacman -S python python-virtualenvwrapper mariadb nginx nginx-mod-brotli

# set virtualenv dir
export WORKON_HOME=~/.virtualenvs

# export virtualenv commands to sesssion
source /usr/bin/virtualenvwrapper.sh

# create .bashrc if not exists
[[ -f ~/.bashrc ]] && touch ~/.bashrc

# virtualenv commands in bashrc
echo "export WORKON_HOME=~/.virtualenvs" >> ~/.bashrc
echo "source /usr/bin/virtualenvwrapper.sh" >> ~/.bashrc

# must source .bashrc for each new ssh connection until full logout

## BEGIN MARIADB

# backup default config if exists and copy config over (edit as needed)
[[ -f /etc/my.cnf.d/my.cnf ]] && mv /etc/my.cnf.d/my.cnf /etc/my.cnf.d/my.cnf.bak
nano /etc/my.cnf.d/my.cnf

# must happen before starting systemd service, but after my.cnf planted for innodb support
mariadb-install-db --user=mysql --basedir=/usr --datadir=/var/lib/mysql

# start mariadb systemd service
systemctl enable --now mariadb

# create db and aq user
mariadb -e "CREATE DATABASE IF NOT EXISTS hayden CHARACTER SET = 'utf8mb4' COLLATE = 'utf8mb4_general_ci';"
mariadb -e "CREATE USER 'hayden_user'@'localhost' IDENTIFIED BY 'hayden_password';"
mariadb -e "GRANT ALL PRIVILEGES ON hayden.* TO 'hayden_user'@'localhost'; FLUSH PRIVILEGES;"

# get an sql dump
curl -LO https://archive.org/download/4ch.be/4ch-be-db_20151031.sql.gz

# load dump
gunzip -c ~/4ch-be-db_20151031.sql.gz | sed -e 's/ENGINE=TokuDB/ENGINE=InnoDB/g' | sed -e 's/CHARSET=utf8 /CHARSET=utf8mb4 /g' | sed -e 's/CHARSET=utf8mb4.*;/CHARSET=utf8mb4;/g' | mariadb hayden

## END MARIADB

## BEGIN MEILI

# direct meilisearch download
curl -L https://github.com/meilisearch/meilisearch/releases/download/v1.9.0/meilisearch-linux-amd64 -o /srv/meili/meilisearch
chmod +x /srv/meili/meilisearch

# Setup meili config
nano /srv/meili/config.toml

# start meili and put in bg
/srv/meili/meilisearch --config-file-path=/srv/meili/config.toml &

## END MEILI

## BEGIN AYASE-QUART

# create virtualenv
mkvirtualenv aq1

# activate it
workon aq1

# install deps
pip install -r aq/requirements.txt

# fix AQ files

# get into src dir
cd aq/src

# edit hypercorn config
nano ~/hyperconf.toml

# start aq into bg
hypercorn --config ~/hyperconf.toml main:app &

# create index in gui

# populate meili
python -m search load g ck mu

# back to home
cd
## END AYASE-QUART

## BEGIN NGINX
systemctl enable --now nginx

# Insert ayase-quart config and modify nginx
nano /etc/nginx/aq.conf
nano /etc/nginx/nginx.conf

# check configs are valid
nginx -t

# reload with good configs
nginx -s reload
## END NGINX

AQ config

~/aq/src/configs.py
copy the template over
- cp ~/aq/src/rename_to_configs.py ~/aq/src/configs.py
Edit the relevant parts

class CONSTS(NamedTuple):
	TESTING = False
	autoreload = False

	# site details
	site_name = "Ayase Quart"
	site_url = "https://aq.domain.tld" # change to actual site address

	# mysql settings
	db_type = DbType.mysql
	db_host = '127.0.0.1'
	db_port = 3306
    db_database = 'hayden'
    db_user = 'hayden_user'
    db_password = 'hayden_password'

	# media link generation base paths
	image_uri = "/static/neo/{board_shortname}/image"
    thumb_uri = "/static/neo/{board_shortname}/thumb"

	# search engine settings
	index_search_provider = IndexSearchType.meili
	index_search_host = 'http://127.0.0.1:7700' # set in /srv/meili/config.toml
    index_search_auth_key = '84661694560523325849' # set in /srv/meili/config.toml
    index_search_config = dict(
        headers={
            'content-type': 'application/json',
            'Authorization': f'Bearer {index_search_auth_key}',
        }
    )

Meili Config

/srv/meili/config.toml

no_analytics = true
env = "development" # Value must be either `production` or `development`.
master_key = "84661694560523325849" # change this, its the api key for the debug page too

http_addr = '127.0.0.1:7700' # 0.0.0.0 to expose over network
http_payload_size_limit = "100 MB"

log_level = "INFO" # `OFF`, `ERROR`, `WARN`, `INFO`, `DEBUG`, `TRACE`

max_indexing_threads = 4 # set to all number of real cores (not SMT/HT)
max_indexing_memory = "2 GiB" # Set to full ram - (mysql innodb + 256MB + 80MB/hypercorn worker)

db_path = "/srv/meili/data" # main storage
dump_dir = "/srv/meili/dumps/"
snapshot_dir = "/srv/meili/snapshots/"

ignore_missing_dump = false
ignore_dump_if_db_exists = false
schedule_snapshot = false # false or interval in seconds
ignore_missing_snapshot = false
ignore_snapshot_if_db_exists = false

Nginx config

/etc/nginx/aq.conf
In addtion to everything else in /etc/nginx/nginx.conf

Replace domain.tld with your own

Generate certs with

 yes y | pacman -S certbot
 certbot certonly --manual --agree-tos --register-unsafely-without-email --preferred-challenges dns -d *.domain.tld -d domain.tld
 # follow onscreen instructions

Copy the certs over to nginx dir or change config to point to /etc/letsencrypt/live/domain.tld

Import it in nginx.conf with include /etc/nginx/aq.conf; in the http block

# Cert & key
ssl_certificate /etc/nginx/certs/domain.tld/fullchain.pem;
ssl_certificate_key /etc/nginx/certs/domain.tld/privkey.pem;

# Ca-cert for ocsp
ssl_trusted_certificate /etc/nginx/certs/domain.tld/chain.pem;

upstream aqbackend {
	# don't put in /tmp, not multi process usable
	server unix:/var/run/aq.sock;
	keepalive 400;
}

# main server
server {
	listen 443 ssl http2;
	server_name domain.tld;

	location / {
		proxy_pass http://aqbackend/;
		proxy_http_version 1.1;

		proxy_set_header Host $http_host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Proto $scheme;

		proxy_buffering off;
		proxy_request_buffering off;
		proxy_redirect off;
		client_max_body_size 1M;
		proxy_read_timeout 300;
		proxy_connect_timeout 300;

	}

	location /static {
		alias /root/aq/src/static;
		# 30 days
		add_header 'Cache-Control' 'max-age=2592000';
	}

	location /static/neo {
		alias /srv/http/media/neo;

		add_header 'Cache-Control' 'max-age=2592000';
		etag off;
	}
}

# Meili's JS won't load from a sub-path, needs a subdomain of its down
server {
	listen 443 ssl http2;
	server_name meili.domain.tld;

	location / {
		proxy_pass http://127.0.0.1:7700;
		proxy_http_version 1.1;

		proxy_set_header Host $http_host;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
		proxy_set_header X-Forwarded-Proto $scheme;

		proxy_buffering off;
		proxy_request_buffering off;
	}
}

# https redirect
server {
	listen 80;
	listen [::]:80;
	server_name domain.tld;
	default_type "";
	return 301 https://$host$request_uri;
}

# https catchall
server {
	listen 443 ssl http2;
	server_name .domain.tld;
	return 444;
}

Mysql config

/etc/my.cnf.d/my.cnf

[client]
port = 3306
socket = /run/mysqld/mysqld.sock
default-character-set = utf8mb4

[mysqld_safe]
socket = /run/mysqld/mysqld.sock
nice = 0

[mysqld]
pid-file = /run/mysqld/mysqld.pid
# bind-address = 127.0.0.1 # default is localhost
port = 3306
socket = /run/mysqld/mysqld.sock
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
lc_messages_dir = /usr/share/mariadb
lc_messages = en_US

# UTC timezone
default-time-zone=+00:00

old-mode='NO_DUP_KEY_WARNINGS_WITH_IGNORE'

skip-external-locking


key_buffer_size = 128M
max_connections = 100
connect_timeout = 5

# affects insert query sizes of mysqldumps
max_allowed_packet = 512M

# must be higher than pool connecting lifetime
wait_timeout = 3600
net_read_timeout = 3600
net_write_timeout = 3600

thread_cache_size = 128
thread_stack = 192K
sort_buffer_size = 4M
bulk_insert_buffer_size = 32M
tmp_table_size = 32M
max_heap_table_size = 32M

#performance_schema = on
character_set_server = utf8mb4
collation_server = utf8mb4_general_ci
transaction_isolation = READ-COMMITTED
binlog_format = MIXED

large_pages = ON

query_cache_type = OFF


skip-log-bin = 1
max_binlog_size = 100M

# Innodb
default_storage_engine = InnoDB
innodb_page_size = 4K

innodb_buffer_pool_size = 256M
innodb_log_buffer_size = 8M
innodb_file_per_table = 1
innodb_open_files = 1000
innodb_io_capacity = 3000
innodb_flush_method = O_DIRECT
innodb_stats_on_metadata = OFF

innodb_flush_log_at_trx_commit = 2


[mysqldump]
quick
quote-names
max_allowed_packet = 256M

Hypercorn config

~/hyperconf.toml

bind = 'unix:/var/run/aq.sock'
# bind = 'http://0.0.0.0:9001'
backlog = 500
umask = 79 # must be decimal int, not octal, 0o117 == 79
user = 33 # http user
group = 33 # http group
workers = 4 # num cores * 2
loglevel = 'WARNING'

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

08_Deploy_tutorial

Overview

Prelim setup

Native setup (no docker/containers)

AQ config

Meili Config

Nginx config

Mysql config

Hypercorn config

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally