diff --git a/Chapter11/Temporal_GraphML.ipynb b/Chapter11/Temporal_GraphML.ipynb new file mode 100644 index 0000000..2e5d13b --- /dev/null +++ b/Chapter11/Temporal_GraphML.ipynb @@ -0,0 +1,966 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "su-ySan88ru6" + }, + "source": [ + "# Temporal GraphML" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "IJ0eLdE5lh3d" + }, + "source": [ + "In this notebook, we will introduce representative examples of the machine learning approaches for dealing with temporal graphs. We will offer a general understanding of their implementation using publicly available frameworks." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "UfEu4ta08v0e" + }, + "source": [ + "## Temporal Matrix Factorization\n", + "\n", + "Temporal Matrix Factorization model (TMF) by Yu et al. (2017) is a method used for temporal link prediction, particularly in dynamic network scenarios. This technique leverages matrix factorization with temporal dynamics to model the evolution of links in a dynamic network over time.\n", + "\n", + "We adopt the implementation provided in the publicly available library [OpenTLP](https://github.com/KuroginQin/OpenTLP). It integrates an encoder-decoder architecture, where the encoder learns model parameters through matrix factorization, and the decoder generates predictions based on these parameters." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "AnL-blMhkR5a", + "outputId": "697a0d81-c5ca-43b9-dbc1-4496cc4d5ddb" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Cloning into 'OpenTLP'...\n", + "remote: Enumerating objects: 147, done.\u001b[K\n", + "remote: Counting objects: 100% (32/32), done.\u001b[K\n", + "remote: Compressing objects: 100% (5/5), done.\u001b[K\n", + "remote: Total 147 (delta 27), reused 27 (delta 27), pack-reused 115 (from 1)\u001b[K\n", + "Receiving objects: 100% (147/147), 13.13 MiB | 19.66 MiB/s, done.\n", + "Resolving deltas: 100% (68/68), done.\n" + ] + } + ], + "source": [ + "# Donwload the OpenTLP repository\n", + "!git clone https://github.com/KuroginQin/OpenTLP.git" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "sDiVZdas-d5_" + }, + "source": [ + "OpenTLP contains a set of useful temporal graph data. Let's unzip it as follows:" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "j-UOMplglz0c", + "outputId": "d353dc92-d81b-439e-cdce-23ad530aa989" + }, + "outputs": [], + "source": [ + "# Unzip the sample graph\n", + "import zipfile\n", + "zipfile.ZipFile('OpenTLP/Python/data/data.zip').extractall('data')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "QRriEYix-m2C" + }, + "source": [ + "For the TMF example, the Enron dataset is used as a case study.\n", + "The dataset consists of temporal snapshots of a graph with 184 nodes over 26 time points. Historical edge sequences (`edge_seq`) are loaded, representing graph snapshots at different timestamps.\n", + "\n", + "Let's now implementing the TMF example. This code is adapted from the [OpenTLP examples](https://github.com/KuroginQin/OpenTLP/blob/main/Python/TMF_demo1.py).\n", + "\n", + "### 1. Model Setup\n", + "- TMF is implemented with the following parameters:\n", + " - **Latent dimensionality of node embeddings** (`hid_dim = 64`).\n", + " - **Regularization coefficients** for model optimization (`alpha`, `beta`, and `theta`).\n", + " - **Learning rate** for gradient-based optimization.\n", + " - A **sliding window of historical snapshots** (`win_size = 5`) is used to predict the adjacency structure at the next time step.\n", + "\n", + "### 2. Training the TMF Model\n", + "- For each time step after the historical window (`win_size` to `num_snaps`):\n", + " - The model uses the last `win_size` adjacency matrices to learn a low-dimensional representation of the graph.\n", + " - The learned representation is used to predict the adjacency matrix for the current time step.\n", + "- The adjacency matrices are refined to ensure symmetry and zero diagonal elements.\n", + "\n", + "### 3. Evaluation\n", + "- The **Area Under the Curve (AUC)** score is computed to evaluate the quality of predictions against the ground truth adjacency matrix at each time step.\n", + "- The average AUC and standard deviation across all time steps are reported as metrics of model performance.\n", + "\n", + "## Results\n", + "The model iteratively predicts the graph structure for each snapshot and computes the corresponding AUC. This provides insight into the TMF's ability to generalize and learn temporal patterns from historical graph data." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "import sys\n", + "sys.path.append('OpenTLP/Python/')" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "GlafCysq_Nuq", + "outputId": "e3e307c5-fef0-438e-e0c9-7f25e2f8fd13" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Snapshot 5: AUC = 0.752615\n", + "Snapshot 6: AUC = 0.797639\n", + "Snapshot 7: AUC = 0.765659\n", + "Snapshot 8: AUC = 0.834835\n", + "Snapshot 9: AUC = 0.860552\n", + "Snapshot 10: AUC = 0.855638\n", + "Snapshot 11: AUC = 0.880935\n", + "Snapshot 12: AUC = 0.836431\n", + "Snapshot 13: AUC = 0.874893\n", + "Snapshot 14: AUC = 0.864066\n", + "Snapshot 15: AUC = 0.879439\n", + "Snapshot 16: AUC = 0.772748\n", + "Snapshot 17: AUC = 0.800971\n", + "Snapshot 18: AUC = 0.805514\n", + "Snapshot 19: AUC = 0.760164\n", + "Snapshot 20: AUC = 0.806832\n", + "Snapshot 21: AUC = 0.805647\n", + "Snapshot 22: AUC = 0.857036\n", + "Snapshot 23: AUC = 0.923815\n", + "Snapshot 24: AUC = 0.859344\n", + "Snapshot 25: AUC = 0.856243\n", + "Mean AUC: 0.831001\n", + "Standard Deviation of AUC: 0.046278\n" + ] + } + ], + "source": [ + "# Import necessary libraries and modules\n", + "import numpy as np\n", + "import torch\n", + "from TMF.TMF import TMF # Custom TMF implementation\n", + "from utils import get_adj_un, get_AUC # Utility functions for adjacency and evaluation\n", + "\n", + "# Check if GPU is available, otherwise use CPU\n", + "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n", + "\n", + "# ====================\n", + "# Dataset and model configuration\n", + "data_name = 'Enron' # Name of the dataset\n", + "num_nodes = 184 # Total number of nodes in the graph\n", + "num_snaps = 26 # Total number of snapshots (time points)\n", + "hid_dim = 64 # Dimensionality of the latent space\n", + "theta = 0.1 # Regularization parameter for model training\n", + "alpha = 0.01 # TMF-specific hyperparameter\n", + "beta = 0.01 # TMF-specific hyperparameter\n", + "\n", + "# Load edge sequences from dataset\n", + "edge_seq = np.load(f'data/{data_name}_edge_seq.npy', allow_pickle=True)\n", + "\n", + "# ====================\n", + "# Training hyperparameters\n", + "learn_rate = 1e-3 # Learning rate for the optimizer\n", + "win_size = 5 # Size of the historical window for snapshots\n", + "num_epochs = 200 # Number of training epochs\n", + "\n", + "# ====================\n", + "# Initialize a list to store AUC scores for each snapshot\n", + "AUC_list = []\n", + "\n", + "# Iterate through snapshots, starting after the initial window size\n", + "for tau in range(win_size, num_snaps):\n", + " # Ground truth edges for the current snapshot\n", + " edges = edge_seq[tau]\n", + " gnd = get_adj_un(edges, num_nodes) # Generate ground truth adjacency matrix\n", + "\n", + " # Collect adjacency matrices for historical snapshots within the window\n", + " adj_list = []\n", + " for t in range(tau - win_size, tau):\n", + " edges = edge_seq[t]\n", + " adj = get_adj_un(edges, num_nodes)\n", + " adj_tnr = torch.FloatTensor(adj).to(device)\n", + " adj_list.append(adj_tnr)\n", + "\n", + " # Initialize and train the TMF model\n", + " TMF_model = TMF(num_nodes,\n", + " hid_dim,\n", + " win_size,\n", + " num_epochs,\n", + " alpha,\n", + " beta,\n", + " theta,\n", + " learn_rate,\n", + " device)\n", + "\n", + " adj_est = TMF_model.TMF_fun(adj_list) # Predict adjacency matrix for the current snapshot\n", + "\n", + " # Convert predicted adjacency matrix to NumPy array if necessary\n", + " adj_est = adj_est.cpu().data.numpy() if torch.cuda.is_available() else adj_est.data.numpy()\n", + "\n", + " # Refine the predicted adjacency matrix\n", + " adj_est = (adj_est + adj_est.T) / 2 # Ensure symmetry\n", + " np.fill_diagonal(adj_est, 0) # Set diagonal elements to 0 (no self-loops)\n", + "\n", + " # Evaluate prediction quality using AUC metric\n", + " AUC = get_AUC(adj_est, gnd, num_nodes)\n", + " AUC_list.append(AUC)\n", + " print(f'Snapshot {tau}: AUC = {AUC:.6f}')\n", + "\n", + "# ====================\n", + "# Compute mean and standard deviation of AUC scores\n", + "AUC_mean = np.mean(AUC_list)\n", + "AUC_std = np.std(AUC_list, ddof=1)\n", + "\n", + "# Display overall results\n", + "print(f'Mean AUC: {AUC_mean:.6f}')\n", + "print(f'Standard Deviation of AUC: {AUC_std:.6f}')" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "uKW8CUdMrYZX" + }, + "outputs": [], + "source": [] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "p37_4aR1vzrw" + }, + "source": [ + "## Temporal Random Walk\n", + "We pesent here a Temporal Random Walk-based method called CTDNE, by Nguyen et al. (2018), for learning time-preserving embedding.\n", + "\n", + "This code is adapted from [StellarGraph](https://colab.research.google.com/github/stellargraph/stellargraph/blob/master/demos/link-prediction/ctdne-link-prediction.ipynb#scrollTo=I2Vw-NfmeMU5)." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4kC5hO9NJKKb" + }, + "source": [ + "In this example we will be using again the Enron dataset. You can find a specific class handling the Enron dataset in StellarGraph." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "LSHjNRIAJlym", + "outputId": "043f13c8-99ab-4040-8ff9-f0e1384efc6a" + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2025-06-23 12:11:15.352421: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n", + "2025-06-23 12:11:15.352784: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n", + "2025-06-23 12:11:16.775450: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory\n", + "2025-06-23 12:11:16.775493: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)\n", + "2025-06-23 12:11:16.775520: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (08c0bdca2fee): /proc/driver/nvidia/version does not exist\n", + "2025-06-23 12:11:16.775759: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA\n", + "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "A dataset of edges that represent emails sent from one employee to another.There are 50572 edges, and each of them contains timestamp information. Edges refer to 151 unique node IDs in total.Ryan A. Rossi and Nesreen K. Ahmed “The Network Data Repository with Interactive Graph Analytics and Visualization” (2015)\n" + ] + } + ], + "source": [ + "from stellargraph.datasets import IAEnronEmployees\n", + "\n", + "dataset = IAEnronEmployees()\n", + "print(dataset.description)\n", + "full_graph, edges = dataset.load()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "HMntKbp3Kngp" + }, + "source": [ + "In this example, we show how random walks can be obtained from time graphs, and how they can be used to generate network embeddings for a link prediction task.\n", + "\n", + "Since we will be address a link prediction task, let's prepare the graph for the task: Let's split the edges into two parts:\n", + "\n", + "* the oldest edges are used to create the graph structure\n", + "* the recent edges are what we are interested in predicting - we randomly split this part further into training and test sets." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "id": "_ZtuSGtfLGNJ" + }, + "outputs": [], + "source": [ + "import pandas as pd\n", + "from stellargraph import StellarGraph\n", + "\n", + "# Finally, let's create an instance of the StellarGraph class\n", + "graph = StellarGraph(\n", + " nodes=pd.DataFrame(index=full_graph.nodes()),\n", + " edges=edges,\n", + " edge_weight_column=\"time\",\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "urrcUoAOL38I" + }, + "source": [ + "It's now time for running the Temporal Random Walk algorithm" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "uaPnd-BZL-RR", + "outputId": "89bb875f-e990-4f90-b430-a602bd6444c2" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Number of temporal random walks: 1730\n" + ] + } + ], + "source": [ + "from stellargraph.data import TemporalRandomWalk\n", + "from gensim.models import Word2Vec\n", + "\n", + "num_walks_per_node = 10\n", + "walk_length = 80\n", + "context_window_size = 10\n", + "\n", + "num_cw = len(graph.nodes()) * num_walks_per_node * (walk_length - context_window_size + 1)\n", + "\n", + "temporal_rw = TemporalRandomWalk(graph)\n", + "temporal_walks = temporal_rw.run(\n", + " num_cw=num_cw,\n", + " cw_size=context_window_size,\n", + " max_walk_length=walk_length,\n", + " walk_bias=\"exponential\",\n", + ")\n", + "\n", + "print(\"Number of temporal random walks: {}\".format(len(temporal_walks)))" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "id": "IyRwcekhNAt9" + }, + "outputs": [], + "source": [ + "embedding_size = 128\n", + "temporal_model = Word2Vec(\n", + " temporal_walks,\n", + " vector_size=embedding_size, # \"size\" in older gensim versions\n", + " window=context_window_size,\n", + " min_count=0,\n", + " sg=1,\n", + " workers=2,\n", + " epochs=1, # \"iter\" in older gensim versions\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7ubW4bxyMsKV" + }, + "source": [ + "Let's visualize the embeddings:" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 657 + }, + "id": "VLeUlo9wMpm7", + "outputId": "0ab52d41-7306-480f-81a3-b41aa85218f0" + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlkAAAJdCAYAAAABaWJRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzde3xT9f0/8FfuaZombbEBKlDkZhGmSDdQFFFEGKJON9GBDtCpKDpENt2c2wQnc+qmftUJ6KYCuwg4N7/fn1e8zQuKioLcBdQgNym9JG3Tprl8fn90CU2TtEl7Ts45Oa/n48FDm+Zy0pxz8j7vz/vz/hiEEAJEREREJCmj0htARERElI8YZBERERHJgEEWERERkQwYZBERERHJgEEWERERkQwYZBERERHJgEEWERERkQwYZBERERHJgEEWERERkQwYZJGmzZkzBwMHDlR6M1Juh8FgwKJFi3K+LUq9bjY++ugjjBs3DoWFhTAYDNi0aZPSm6Q5atn3Y9566y0YDAa89dZbSm9K1s4++2yMHDkyJ6+V6fG5aNEiGAyGhNsGDhyIOXPmyLNhJAsGWXnGYDBk9C92IqyursbNN9+MyspKFBQUwOPxYMyYMfj5z3+OxsbG+PPOmTMHBoMBJ598MlKtxGQwGHDTTTfFf/7qq686ff3f//73sv8t8tmLL76o+kAqnVAohOnTp6O2thYPPvggVq1ahYqKipT33b59OxYtWoSvvvoqtxtJRCQBs9IbQNJatWpVws8rV67EunXrkm4fPnw4amtr8e1vfxt+vx9XX301KisrUVNTg88++wxLly7FDTfcAKfTmfC4LVu24LnnnsMPfvCDjLZnxowZOP/885NuP/XUU7N8Z6k98cQTiEajkjyX1Jqbm2E2y3OIvfjii/jTn/6UMtCS83WlsHfvXni9XjzxxBO45pprOr3v9u3bsXjxYpx99tmqytoQKWHXrl0wGpkb0RL1nompW6688sqEnz/44AOsW7cu6XYAuP/++7Fv3z689957GDduXMLv/H4/rFZrwm0FBQXo378/7rrrLnz/+99PSmWnMnr06JSvLRWLxSLbc/eU3W7X1etm6siRIwCA4uJihbdEXQKBABwOh9KbQSpms9mU3gTKEkNiHdu7dy9MJhNOO+20pN+5XK6kL2uj0Yhf/epX+Oyzz/Cvf/1Ltu36wx/+AIPBAK/Xm/S722+/HVarFXV1dQBS16U888wzqKqqQlFREVwuF771rW/hf/7nf+K/T1XrAABPP/00DAZDwtDU888/j2nTpqG8vBw2mw2DBw/Gb3/7W0QikS7fR8fai4aGBixYsAADBw6EzWaDx+PBeeedh08++SR+n3feeQfTp0/HgAEDYLPZ0L9/f9xyyy1obm6O32fOnDn405/+FH+N2L90rwsAn376KaZOnQqXywWn04lzzz0XH3zwQcr3/95772HhwoUoKytDYWEhLrnkElRXV3f5fgHgjTfewPjx41FYWIji4mJ873vfw44dOxK2fcKECQCA6dOnw2Aw4Oyzz075XE8//TSmT58OADjnnHOShroB4KWXXoq/XlFREaZNm4Zt27YlPM+cOXPgdDqxb98+XHDBBXA6nTj++OPjf8MtW7Zg4sSJKCwsREVFBf7+97+n/Lu8/fbbmDt3Lnr16gWXy4VZs2bF98P2HnvsMYwYMQI2mw3l5eW48cYbUV9fn3CfWA3Qxo0bcdZZZ8HhcOCXv/wlgJ7tc6kMHDgQF1xwAd59912MGTMGdrsdgwYNwsqVK5Pu+8UXX2D69OkoLS2Fw+HAaaedhhdeeCHpfvv378fFF1+MwsJCeDwe3HLLLQgGgylff8OGDfjud78Lt9sNh8OBCRMm4L333sto24PBIO68804MGTIkfjzcdtttSa8VK1dYu3YtTjrpJBQUFOD000/Hli1bAADLly/HkCFDYLfbcfbZZ6cdft64cSPGjRuHgoICnHDCCVi2bFm3tykYDOKWW25BWVkZioqKcNFFF2H//v0pX/fdd9/Fd77zHdjtdgwePBjLly9Peb+ONVnZHLPRaBSLFi1CeXk5HA4HzjnnHGzfvj3pOUOhEBYvXoyhQ4fCbrejV69eOPPMM7Fu3bqU20SdYyZLxyoqKhCJRLBq1SrMnj07o8fMnDkTv/3tb3HXXXfhkksu6TKbFQgEcPTo0aTbi4uL0w5pXXbZZbjtttuwZs0a3HrrrQm/W7NmDSZPnoySkpKUj123bh1mzJiBc889F/feey8AYMeOHXjvvfdw8803Z/IWEzz99NNwOp1YuHAhnE4n3njjDfzmN7+B3+/H/fffn9VzXX/99Xj22Wdx00034aSTTkJNTQ3effdd7NixA6NHjwYArF27FoFAADfccAN69eqFDz/8EI888gj279+PtWvXAgDmzp2LgwcPphwGTmXbtm0YP348XC4XbrvtNlgsFixfvhxnn302/vOf/2Ds2LEJ9//JT36CkpIS3Hnnnfjqq6/w0EMP4aabbsLq1as7fZ3XXnsNU6dOxaBBg7Bo0SI0NzfjkUcewRlnnIFPPvkEAwcOxNy5c3H88cfjd7/7HebPn4/vfOc76N27d8rnO+usszB//nw8/PDD+OUvf4nhw4cDQPy/sf12ypQpuPfeexEIBLB06VKceeaZ+PTTTxOC70gkgqlTp+Kss87Cfffdh7/97W+46aabUFhYiDvuuANXXHEFvv/972PZsmWYNWsWTj/9dJxwwgkJ23PTTTehuLgYixYtwq5du7B06VJ4vd54wTfQFsAvXrwYkyZNwg033BC/30cffYT33nsvIfNaU1ODqVOn4oc//CGuvPLK+N9Byn0uZs+ePbj00kvx4x//GLNnz8aTTz6JOXPmoKqqCiNGjAAAfPPNNxg3bhwCgQDmz5+PXr16YcWKFbjooovw7LPP4pJLLgHQNhx97rnnYt++fZg/fz7Ky8uxatUqvPHGG0mv+8Ybb2Dq1KmoqqrCnXfeCaPRiKeeegoTJ07EO++8gzFjxqTd5mg0iosuugjvvvsurrvuOgwfPhxbtmzBgw8+iM8//xz//ve/E+7/zjvv4H//939x4403AgDuueceXHDBBbjtttvw2GOPYd68eairq8N9992Hq6++Oml76+rqcP755+Oyyy7DjBkzsGbNGtxwww2wWq24+uqrs96ma665Bn/9618xc+ZMjBs3Dm+88QamTZuW9D63bNmCyZMno6ysDIsWLUI4HMadd96Z9rhIJZNj9vbbb8d9992HCy+8EFOmTMHmzZsxZcoUtLS0JDzXokWLcM899+Caa67BmDFj4Pf78fHHH+OTTz7Beeedl/E20X8Jyms33nijSPcxHz58WJSVlQkAorKyUlx//fXi73//u6ivr0+67+zZs0VhYaEQQogVK1YIAOK5556L/x6AuPHGG+M/f/nllwJA2n/vv/9+p9t9+umni6qqqoTbPvzwQwFArFy5MmG7Kioq4j/ffPPNwuVyiXA4nPa577zzzpR/k6eeekoAEF9++WX8tkAgkHS/uXPnCofDIVpaWtJuhxBtf5M777wz/rPb7U74G6WS6vXuueceYTAYhNfrjd/W2efa8XUvvvhiYbVaxd69e+O3HTx4UBQVFYmzzjorflvs/U+aNElEo9H47bfccoswmUwp94v2Ro0aJTwej6ipqYnftnnzZmE0GsWsWbPit7355psCgFi7dm2nzyeEEGvXrhUAxJtvvplwe0NDgyguLhbXXnttwu2HDx8Wbrc74fbZs2cLAOJ3v/td/La6ujpRUFAgDAaDeOaZZ+K379y5M+nvF/u7VFVVidbW1vjt9913nwAgnn/+eSGEEEeOHBFWq1VMnjxZRCKR+P0effRRAUA8+eST8dsmTJggAIhly5Ylveee7HOpVFRUCADi7bffjt925MgRYbPZxE9/+tP4bQsWLBAAxDvvvBO/raGhQZxwwgli4MCB8ff00EMPCQBizZo18fs1NTWJIUOGJHxW0WhUDB06VEyZMiVhfwoEAuKEE04Q5513XqfbvWrVKmE0GhO2Rwghli1bJgCI9957L34bAGGz2RKO3eXLlwsAok+fPsLv98dvv/3225OO89jn8cc//jF+WzAYjO/Tsc89023atGmTACDmzZuXcL+ZM2emPD7tdnvC8b19+3ZhMpmSjvGKigoxe/bs+M+ZHrOHDx8WZrNZXHzxxQnPt2jRIgEg4TlPOeUUMW3aNEHS4HChjvXu3RubN2/G9ddfj7q6OixbtgwzZ86Ex+PBb3/725SzCAHgiiuuwNChQ3HXXXelvU/Mddddh3Xr1iX9O+mkkzp93OWXX46NGzdi79698dtWr14Nm82G733ve2kfV1xcjKamJslS2wUFBfH/b2howNGjRzF+/HgEAgHs3Lkzq+cqLi7Ghg0bcPDgwYxer6mpCUePHsW4ceMghMCnn36a9fZHIhG8+uqruPjiizFo0KD47X379sXMmTPx7rvvwu/3JzzmuuuuS8hQjh8/HpFIJOXwbcyhQ4ewadMmzJkzB6WlpfHbTz75ZJx33nl48cUXs972zqxbtw719fWYMWMGjh49Gv9nMpkwduxYvPnmm0mPaV9kX1xcjBNPPBGFhYW47LLL4refeOKJKC4uxhdffJH0+Ouuuy4hE3XDDTfAbDbH39trr72G1tZWLFiwIKE4+dprr4XL5UoadrPZbLjqqquSXkfKfS7mpJNOwvjx4+M/l5WV4cQTT0x4ny+++CLGjBmDM888M36b0+nEddddh6+++grbt2+P369v37649NJL4/dzOBy47rrrEl5z06ZN2L17N2bOnImampr4Z9TU1IRzzz0Xb7/9dqeTVtauXYvhw4ejsrIy4TOeOHEiACR9xueee25C9jKWof3BD36AoqKipNs7fsZmsxlz586N/2y1WjF37lwcOXIEGzduzGqbYvvE/PnzE15jwYIFCT9HIhG88soruPjiizFgwID47cOHD8eUKVPS/m066uqYff311xEOhzFv3ryEx/3kJz9Jeq7i4mJs27YNu3fvzvj1KT0GWTrXt29fLF26FIcOHcKuXbvw8MMPo6ysDL/5zW/wl7/8JeVjTCYTfvWrX2HTpk1JKfuOhg4dikmTJiX9c7lcnT5u+vTpMBqN8XS3EAJr166N1xWlM2/ePAwbNgxTp05Fv379cPXVV+Pll1/u4q+Q3rZt23DJJZfA7XbD5XKhrKwsXsjv8/myeq777rsPW7duRf/+/TFmzBgsWrQo6US/b9++eKDidDpRVlYWr2HK9vWAthYdgUAAJ554YtLvhg8fjmg0iq+//jrh9vYnewDxodlU9UcxsZN5uteJfblKJfYFMHHiRJSVlSX8e/XVV+PF9TF2ux1lZWUJt7ndbvTr1y9pyNvtdqd8r0OHDk342el0om/fvvH6nnR/A6vVikGDBiUFqccff3zS5BJA2n0upuNnCrR9ru3fp9frTfv5xX4f+++QIUOS/m4dHxv7jGbPnp30Gf35z39GMBjs9P3s3r0b27ZtS3rssGHDACDpM+74Ht1uNwCgf//+KW/v+BmXl5ejsLAw4bbYa8U+40y3yev1wmg0YvDgwZ3+jaqrq9Hc3Jy0b6W6b2e6OmZjn92QIUMS7ldaWppUenHXXXehvr4ew4YNw7e+9S3ceuut+OyzzzLeFkrEmiwC0FY4OmzYMAwbNgzTpk3D0KFD8be//S3tFPsrrrgiXpt18cUXS7495eXlGD9+PNasWYNf/vKX+OCDD7Bv3754nVU6Ho8HmzZtwiuvvIKXXnoJL730Ep566inMmjULK1asiL/XVDoWFtfX12PChAlwuVy46667MHjwYNjtdnzyySf4+c9/nnXriMsuuwzjx4/Hv/71L7z66qu4//77ce+99+K5557D1KlTEYlEcN5556G2thY///nPUVlZicLCQhw4cABz5szJWasKk8mU8vauspa5FPtbrFq1Cn369En6fcd6v3TvScn32j5jFSP1PhejxPuMbev999+PUaNGpbxPxxYxHR//rW99Cw888EDK33cMnnLxGWe7Tbki5Xs866yzsHfvXjz//PN49dVX8ec//xkPPvggli1b1mXLFUrGIIuSDBo0CCUlJTh06FDa+8SyWXPmzMHzzz8vy3ZcfvnlmDdvHnbt2oXVq1fD4XDgwgsv7PJxVqsVF154IS688EJEo1HMmzcPy5cvx69//WsMGTIkfuVWX1+f0EagY6bhrbfeQk1NDZ577jmcddZZ8du//PLLbr+nvn37Yt68eZg3bx6OHDmC0aNHY8mSJZg6dSq2bNmCzz//HCtWrMCsWbPij0k19JlJ+wygbVjI4XBg165dSb/buXMnjEajJF8MsWai6V7nuOOOS8oSZCLd+4xlCDweDyZNmpT183bH7t27cc4558R/bmxsxKFDh+J94Nr/DdoPzba2tuLLL7/MaDvl2OcyVVFRkfbzi/0+9t+tW7dCCJHw+XR8bOwzcrlc3fqMBg8ejM2bN+Pcc8/NeH/viYMHD6KpqSlhP/38888BID4Mmek2VVRUIBqNYu/evQkZqY5/o7KyMhQUFKQcmkv1WXRX7LPbs2dPwoSOmpqalFnb0tJSXHXVVbjqqqvQ2NiIs846C4sWLWKQ1Q0cLtSxDRs2pBzC+fDDD1FTU9NluvrKK6/EkCFDsHjxYlm27wc/+AFMJhP+8Y9/YO3atbjgggu6/KKuqalJ+NloNOLkk08GgPgU69jJ/+23347fr6mpKZ7pioldHba/GmxtbcVjjz2W9XuJRCJJQyMejwfl5eXx7Ur1ekKIhPYTMbG/Q8fWAB2ZTCZMnjwZzz//fMK09W+++QZ///vfceaZZ3Y5dJuJvn37YtSoUVixYkXCNm3duhWvvvpqyoa0mUj3PqdMmQKXy4Xf/e53CIVCSY/LtOVENh5//PGE11q6dCnC4TCmTp0KAJg0aRKsVisefvjhhM/wL3/5C3w+X8qZZR1Juc9l6/zzz8eHH36I999/P35bU1MTHn/8cQwcODBeR3n++efj4MGDePbZZ+P3CwQCePzxxxOer6qqCoMHD8Yf/vCHhNUjYrr6jC677DIcOHAATzzxRNLvmpubJR1+BoBwOJzQOqG1tRXLly9HWVkZqqqqstqm2D7x8MMPJ9znoYceSvjZZDJhypQp+Pe//419+/bFb9+xYwdeeeUVad4Y2urVzGYzli5dmnD7o48+mnTfjudQp9OJIUOGpG3RQZ1jJkvHVq1ahb/97W+45JJLUFVVBavVih07duDJJ5+E3W6P9+1Jx2Qy4Y477khZvBvzySef4K9//WvS7YMHD8bpp5/e6fN7PB6cc845eOCBB9DQ0IDLL7+8y/d0zTXXoLa2FhMnTkS/fv3g9XrxyCOPYNSoUfHaksmTJ2PAgAH48Y9/jFtvvRUmkwlPPvkkysrKEk5048aNQ0lJCWbPno358+fDYDBg1apV3UrBNzQ0oF+/frj00ktxyimnwOl04rXXXsNHH32EP/7xjwCAyspKDB48GD/72c9w4MABuFwu/POf/0x5pRk76c+fPx9TpkyByWTCD3/4w5Svfffdd2PdunU488wzMW/ePJjNZixfvhzBYBD33Xdf1u8lnfvvvx9Tp07F6aefjh//+MfxFg5ut7vbSwCNGjUKJpMJ9957L3w+H2w2GyZOnAiPx4OlS5fiRz/6EUaPHo0f/vCH8c/vhRdewBlnnJHyC6QnWltbce655+Kyyy7Drl278Nhjj+HMM8/ERRddBKAtK3H77bdj8eLF+O53v4uLLroofr/vfOc7GTXllXKfy9YvfvEL/OMf/8DUqVMxf/58lJaWYsWKFfjyyy/xz3/+M17Mf+211+LRRx/FrFmzsHHjRvTt2xerVq1KaqRqNBrx5z//GVOnTsWIESNw1VVX4fjjj8eBAwfw5ptvwuVy4f/+7//Sbs+PfvQjrFmzBtdffz3efPNNnHHGGYhEIti5cyfWrFmDV155Bd/+9rcle//l5eW499578dVXX2HYsGFYvXo1Nm3ahMcffzw+4SHTbRo1ahRmzJiBxx57DD6fD+PGjcPrr7+OPXv2JL3u4sWL8fLLL2P8+PGYN28ewuEwHnnkEYwYMUKyWqjevXvj5ptvxh//+EdcdNFF+O53v4vNmzfjpZdewnHHHZeQlTvppJNw9tlno6qqCqWlpfj444/jrWeoG3I+n5FyqrOp/p999pm49dZbxejRo0Vpaakwm82ib9++Yvr06eKTTz5JuG/7Fg7thUIhMXjw4KxbOLSfMtyZJ554QgAQRUVForm5Oen3HaexP/vss2Ly5MnC4/EIq9UqBgwYIObOnSsOHTqU8LiNGzeKsWPHxu/zwAMPpGzh8N5774nTTjtNFBQUiPLycnHbbbeJV155JamtQFctHILBoLj11lvFKaecIoqKikRhYaE45ZRTxGOPPZbwmO3bt4tJkyYJp9MpjjvuOHHttdeKzZs3CwDiqaeeit8vHA6Ln/zkJ6KsrEwYDIaEzxgdpogLIcQnn3wipkyZIpxOp3A4HOKcc84R69evT7hP7P1/9NFHCbfHWi50bKOQymuvvSbOOOMMUVBQIFwul7jwwgvF9u3bUz5fJi0chGjbBwYNGhSf0t5+O958800xZcoU4Xa7hd1uF4MHDxZz5swRH3/8cfw+6fbdCRMmiBEjRiTdXlFRkTCFPfZ3+c9//iOuu+46UVJSIpxOp7jiiisS2lXEPProo6KyslJYLBbRu3dvccMNN4i6urqMXluInu1zqXR8P+23YcKECQm37d27V1x66aWiuLhY2O12MWbMGPH//t//S3qs1+sVF110kXA4HOK4444TN998s3j55ZdT7ieffvqp+P73vy969eolbDabqKioEJdddpl4/fXXu9z21tZWce+994oRI0YIm80mSkpKRFVVlVi8eLHw+Xzx+3U8/whx7Bx0//33J9yeav+LfR4ff/yxOP3004XdbhcVFRXi0Ucf7fY2NTc3i/nz54tevXqJwsJCceGFF4qvv/465fH5n//8R1RVVQmr1SoGDRokli1blrLVTLoWDpkcs+FwWPz6178Wffr0EQUFBWLixIlix44dolevXuL666+P3+/uu+8WY8aMEcXFxaKgoEBUVlaKJUuWJLQvocwZhFBRNSsRkco8/fTTuOqqq/DRRx9JmjkhUlp9fT1KSkpw991344477lB6c/ISa7KIiIjyXPuluWJiNWLplrainmNNFhERUZ5bvXo1nn76aZx//vlwOp1499138Y9//AOTJ0/GGWecofTm5S0GWURERHnu5JNPhtlsxn333Qe/3x8vhr/77ruV3rS8xposIiIiIhmwJouIiIhIBgyyiIiIiGSg+ZqsaDSKgwcPoqioKCdLLxAREZF+CSHQ0NCA8vLyeJPedDQfZB08eFCxRTmJiIhIn77++mv069ev0/toPsgqKioC0PZmpViDjYiIiCgdv9+P/v37x+OPzmg+yIoNEbpcLgZZRERElBOZlCix8J2IiIhIBgyyiIiIiGTAIIuIiIhIBgyyiIiIiGTAIIuIiIhIBgyyiIiIiGTAIIuIiIhIBgyyiIiIiGTAIIuIiIhIBgyyiIiIiGTAIIuIiIhIBgyyiIiIiGTAIIuIiIhIBgyyiIiIiGTAIIuIiIhIBgyyiIiIiGTAIIuIiIhIBmalN4CIiLITjQp8fqQBvkAIbocFwzxFMBoNij0PEaXGIIuISEM2emuxYr0Xe440ojUcgdVswhCPE7PHVaCqojTnz0NE6XG4kIhIIzZ6a7HkhR3YesAHl92MfiUOuOxmbDvow5IXdmCjtzanz0NEnWOQRUSkAdGowIr1XtQHQhjYy4FCmxkmowGFNjMqSh3wNYewcr0X0ajIyfMQUdcYZBERacDOb/zYesAHu9mIpmAEQhwLggwGA8qcNuw+0ojPjzR0+jyfH2nAniON8BTZYDAk1l9l8zxE1DXWZBERqdxGby0eePVz7K8LwGw0wmgECq1tw3zFDgsAwG4x4WhjEL5AqNPn8gVCaA1HYLfYUv4+0+choq4xk0VEpGKx+ilvbVuAZTEZYDYa0RAMY/eRBtT/NxhqCbUVr7v/G3Sl43ZYYDWb0BKKpPx9ps9DRF1jkEUkg2hUYOdhPzZ8UYOdh/2sb6FuaV8/NdTjRFGBGaGogMloQIHZiHBUYH99ANFoFNWNQQz1ODHMU9Tpcw7zFGGIx4nqxmDCkCMACCEyfp5Mt5/HAekZhwuJJNZxarzFZERvlx0Th3sw5oRS9iKijLWvnzIaDOhX4sDubxrQHIrAajLCajKioTmM3dVN8BTZMGtcRZf7ltFowOxxFfHsWJnTBrulLbNV3RiEu8CS0fN0hS0iiACD6HgpozF+vx9utxs+nw8ul0vpzckbPW1SqNcmh7GhnfpACJ4iG4LhKPbVBtDQEoLRYEAftx0n9yvmFw1lZMMXNbjjX1vQr8QB03+Pn/rmEPbXBdAUjCASjSISBb4zsAQLJw/rcZ+soR4nZkmwb3Y8DjoGcXdMG879nzQrm7iDmSxK0tMrUL1ewXacGu9rDmNvdSPCUQGH1YxgOIKGlhC2HWjrRcQvGupK+/qpQlvb6bq4wAJ3gRtNwTAaWkJoCUdxx7ThOKncndVzV1WU4tT+JZJfDHU8DmIzGAttZjisJnhrA1i53otT+5fo4sKL9I01WZSgp00K9dzksP3QDgDsrwsgHBUoMBthNhpgM5sQDAv0clrZi4gykq5+ygCg0GpCMBzFt8rdqOzTvSy+0WhAZR8Xxg7qhco+LkmCHraIIDqGQRbF9bRJod6bHB6bGm9CUzCCptYwrCZj/IvGZDQgKgTCUcEvGspIrH7KXWCBtzaApmAYkahAUzAMb21AsvopKbU/DlKxW0xoDUfYIoJ0gUGWDLQ6o6anV6B6v4JtP7QTikYRFYCp3Z8hEhUwGgywmIz8oqGMVVWU4o5pwzGi3A1/Sxj76wLwt4QxstytyiFntoggOoY1WRLLdT2SlAXmPW1SqPcmh7GhnW0HfehVaIXRAEQEYDYAAkBrOIICqxmhSBTNrfyioczJVT8lh/bHgcNqSrjgirWIGFnulqRFBJHaMciSUPKMGhtaQqB6qEcAACAASURBVJF4PZLUV51SB3Spimzb6+oKtKeP17r2U+OPNgZhM5sQaI1AmAwIhqOIRAVEaxg7DvkRiQJlRVY0tORnwKlHcs+ojdVPqV2uWkQQaQGHCyWS63okOQrMe9qkMJdNDtUqNrQz8vhiFNnNiAqBptYIwpEojAbAbDQCMMBqbvuCuefFnXk9GUAvNnprsWD1JixcvRl3/GsLFq7ejAWrN+n2s9XaECeRXJjJkkg29Ug9vRqVa4p0T69AeQXbpv3QzoYvavCnN/eiPtAKk8kIAQFXgRn9ih1wF5g5nT0P5DqDrRVaGuIkkguDLInksh5JzoAudgUaG4Y82hiE1WzCyHJ3Rk0Ke/r4fNF+aOeZD79GH5cNZpMRFqMRhbZjdSpSBt+Ue+wJ1TmtDHESyYVBlkRyWY8kd0DXkyvQaFSg0GbGzDEDUNfciuICC0oKrbq9gvUFQghFoujtOtaxu718nwyQ73KZwSYi7WGQJZFczqjJRUDXnSvQzgrx9RhgAZwMkO/0PqOWiDrHwneJ5LJpoBoLzPXc6b0zavysSDrsCUVEnWGQJaFczahRWxdovXd674zaPiuSFoNoIuqMQXQ8M2hMNqth54rU/XLSPV+q4bmhHmdGBeZSbuPOw34sXL0ZLrs55ZBYUzAMf0sYD1x+im7rUnryWVHPydnDKpbF9TWHUs6o1evsQqJ8lU3cwZosGUg5o6arhqPdKVCXuokp61K6xunsypF7FQbOqCWidJjJUrHk/js9v0KW4zmZySK1kmN/T0fuju9EpA7ZxB2syVIpOeqc5KqdYl0KqVGuawVjGeyxg3qhso+LARYRMchSq2z67yj5nACLu0md5NrfiYgyxSBLpY7VOZlS/t5uMaE1HMmqzkmO54zhWmWkNnLu7/kmGhXYediPDV/UYOdhvy5nAhPJgYXvKiVHE0u5G2OyuJvUhI1gMyP3xAAiPWMmS6XkqHPKRe0U61JILVgr2DU2ESaSF4MslZKjzom1U6Qn3N87xybCRPJjkKVictQ5sXaK9CTf9/ee1FJxYgCR/FiTpXJy1Dmxdor0JF/3957WUrGJMJH8GGRpgJQd5OV8TiK1yrf9PbnJqg0toUi8liqTLB0nBhDJj8OFpGqcWk6USKpaKk4MIJIfM1kEQJ1LgnBqOVGybGqpOsvexSYGLHlhB7y1gZSLW+t5YgCRFBhkUbeCGbmDMimGQ4jykZS1VFzcmkheDLJ0rjvBjNwZpo7DIbGr9UKbGQ6rCd7aAFau9+LU/iW8yibdkbqWKl8nBhCpAWuydKw7tR25aF7IqeVE6clRS8UmwkTyYJClYnIXfWcbzOSqeSHXnCNKj01WibSDw4UqlYui72xrO6QquO0Kp5YTdY61VETawCBLhXJV9J1tMJOr5oWx4ZBtB31wWE0JAV1sOGRkuZtTy0nXWEtFpH4cLlSZXK4nlm1tR/ugLBWpMkwcDiHKjBprqdjbjugYZrJUJldDckD2fXJymWHicAhR9yjZ84697YgSMchSmVyvJ5ZNMJPr5oUcDiGtUirQUTLIYW87omQMslRGiaLvbIKZXGeY8m3NOcp/SgU6SgY57G1HlBqDLJVRqug7m2CGGSai1JQKdJQOcnJZ5kCkJSx8VxmtFH2rseCWSEm5nLTSkdINfNnbjig1BlkqFBuSG1Huhr8ljP11AfhbwhhZ7mZdA5FKKRnoKB3k5GrmMZHWcLhQZt0tgE01JDfkOCf2HG3Ehi9qOERHpDK5nrTSntINfNnbjig1Blky6mkBbPs6qY3eWixcu5lTo3NMyenwpC1KBjpKBzm5nnlMpBUG0bELpcb4/X643W74fD64XOopqEwugE084WQz7Cflc1Hm2POHshGNCixYvQnbDvpQUepICnS8tQGMLHfjwctHyRJsxM4TvuZQyiAnF+eJVMfMUI+Tve0or2QTdzDIkkHsZLv1gC9hpg+Q/clWyueizDGwpe5QOtDpbpAjZcaW2V/Kd9nEHRwulIGU05k5NTr3lJ4OT9ql9EoF3WmvInXGlr3tiI5hkCUDKQtglSym1SsGttQTSveRyybIYZd2InkxyJKBlAWwSs8a0iMGttRTWsjmMGNLJD/2yZJBbKZPdWMQHUveYjN9hnqcGc30kfK5KDPs+UN6oHQDUyI9YJAlAym7tmulA3w+YWBLeqB0A1MiPWCQJRMpu7azA3xuMbAlPWDGlkh+rMmSkZQFsEoX0+qN0rPEiOSmdANTUie24JAWgyyZSVkAq4Vi2nzCwJbyGbu0U0dswCw9NiMlItIxdmkngA2Ys8FmpERElBFmbIntPOTDIIuISOdYiqBvbMAsHwZZRESUgMXP+aWrz5MNmOXDIIuIiOJY/JxfMvk8ubKIfNgni4iIABwrft56wAeX3Yx+JQ647Ob4WoYbvbVKbyJlIdPPkw2Y5aN4kLVo0SIYDIaEf5WVlUpvFhGRrnQsfi60mWEyGlBoM6Oi1AFfcwgr13sRjWp6QrpuZPN5sgGzfFQxXDhixAi89tpr8Z/NZlVsFhGRbqip+FkrNWFq3s5sP082YJaHKqIZs9mMPn36KL0ZRES6pZbiZ63UhKl9O7vzebKdh/QUHy4EgN27d6O8vByDBg3CFVdcgX379qW9bzAYhN/vT/hHRKRV0ajAzsN+bPiiBjsP+xUbjlPDWoZaqQnTwnZ29/OMtfMYO6gXKvu4GGD1kOKZrLFjx+Lpp5/GiSeeiEOHDmHx4sUYP348tm7diqKi5CK7e+65B4sXL1ZgS4mIpKWmbIjSaxlqpSGmVrZT6c+T2iieyZo6dSqmT5+Ok08+GVOmTMGLL76I+vp6rFmzJuX9b7/9dvh8vvi/r7/+OsdbTETUc2rLhihd/JxNDZGStLKdSn+e1EbxIKuj4uJiDBs2DHv27En5e5vNBpfLlfCPiEhL1DqTL1b8PKLcDX9LGPvrAvC3hDGy3C372nXHaohMKX9vt5jQGo4o3hBTK9sJKPt5UhvFhws7amxsxN69e/GjH/1I6U0hIpKFmmbydaRU8bNWGmJqZTtjWMyuLMWDrJ/97Ge48MILUVFRgYMHD+LOO++EyWTCjBkzlN40IiJZqGUmXzpKrGWolRoipbazJ+0iuDalchQPsvbv348ZM2agpqYGZWVlOPPMM/HBBx+grKxM6U0jIpKF1rIhuRCrIVrywg54awMoc9pgt7T9jaobg6qpIVJiO9U0QYKyYxAde+hrjN/vh9vths/nY30WEWlCNCqwYPUmbDvoQ0WpIykb4q0NYGS5Gw9ePkrxoCLXUgUUQz1O1TXEzNV2xiZI1AdC8BQlB3Ssrcq9bOIOxTNZRER6o5WsjRK0UkOUi+3USrsISo9BFhGRAriMSXpaqSGSezvVPEGCMsMgi4hIIVrJ2pAy1D5BgrrGIIuISEFaydpQ7nGChPaprhkpERERHWsXUd0YRMc5arF2EUM9TsXbWlB6DLKIiIhUiEvjaB+DLCIiIpXi0jjaxposIiIiFeMECe1ikEVERLrQk6VplH5NTpDQJgZZRESU95RYmobL4RBrsoiIKK/FlqbZesAHl92MfiUOuOxmbDvow5IXdmCjtzYvXpPUh0EWERHlrY5L0xTazDAZDSi0mVFR6oCvOYSV672IRqVbxleJ1yR1YpBFRER5K5ulabT8mqRODLKIiChvHVuaxpTy93aLCa3hiKRL0yjxmqRODLKIiChvtV+aJhU5lqZR4jVJnRhkERFR3lJiaRouh0MxDLKIiAhAW8H2zsN+bPiiBjsP+/OiMFuJpWm4HA7FGETHMFtj/H4/3G43fD4fXC42aiMi6o587+mU6v0N9TgxK8d9suR+TZJfNnEHgywiIp2L9XSqD4TgKbLBbmmrJ6puDMJdYMmbNfK03PGd1CObuIMd34mIdKxjT6dYy4FCmxkOqwne2gBWrvfi1P4lmggOOgtqlFiahsvh6BuDLCIiHcump5Pag4V8H/Ik7WHhOxGRjuVLTycuY0NqxCCLiEjH8qGnE5exIbVikEVEpGP50NOJy9iQWjHIIiLSsXzo6ZQvQ56UfxhkERHpXFVFKe6YNhwjyt3wt4Sxvy4Af0sYI8vdmmjfkA9DnpSfOLuQiIhQVVGKU/uXaLKnU2zIc9tBHxxWU8KQYWzIc2S5W9VDnpSfmMkiIiIAx3o6jR3UC5V9XJoIsID8GPKk/MQgi4iINE/rQ56UnzhcSEREeUHLQ56UnxhkERFR3uAyNqQmHC4kIiIikgGDLCIiIiIZMMgiIiIikgGDLCIiIiIZMMgiIiIikgGDLCIiIiIZMMgiIiIikgH7ZBEREWUhGhVseEoZYZBFRESUoY3eWqxY78WeI41oDUdgNZswxOPE7HEVXLqHknC4kIiIKAMbvbVY8sIObD3gg8tuRr8SB1x2M7Yd9GHJCzuw0Vur9CaqTjQqsPOwHxu+qMHOw35Eo0LpTcopZrKIiIi6EI0KrFjvRX0ghIG9HDAY2oYHC21mOKwmeGsDWLnei1P7l3Do8L+Y9WMmi4iIqEufH2nAniON8BTZ4gFWjMFgQJnTht1HGvH5kQaFtlBdmPVrw0wWEdF/saCZ0vEFQmgNR2C32FL+3m4x4WhjEL5AKMdbpj5yZ/20dJwyyCKivJXNyZhDG9QZt8MCq9mEllAEhbbkr86WUNs+43ZYFNg6dckm61fZx5XVc2vtOGWQRUR5KZuTcWxooz4QgqfIBrvFhpZQJD60cce04ao8gVPuDPMUYYjHiW0HfXBYTQnBgxAC1Y1BjCx3Y5inSMGtVAe5sn6ZHKen9i9RVZaLQRYR5Z1sgiYWNFMmjEYDZo+rwJIXdsBbG0CZ0wa7pS2zVd0YhLvAglnjKriPQJ6sXybH6UPrdqPYYcHe6ibVZLlY+E5EeaXjybjQZobJaEChzYyKUgd8zSGsXO+NTyVnQTNlqqqiFHdMG44R5W74W8LYXxeAvyWMkeVuZjvbiWX9qhuDECKxZUMs6zfU48wq69fVcWo3G/Gxtw6ffl2vqkJ7ZrKIKK9kWw/Sk6ENLRXgkjSqKkpVNySlNnJk/To7TtsCt1aEo1GUOa3x7JkastEMsohIUkoHHtkGTd0d2tBaAS5Jx2g0ZF2wrTexrF/sGDnaGITVbMLIcjdmdeMY6ew4bQpG0NgShsVohNVsSvhdTwvte4pBFhFJRg2BR7ZBU3cKmlkoT9Q1KbN+nR2nrZEIQtEoShyWlMe8ku01WJNFRJJQS/PBbOtBYkMb7gILvLUBNAXDiEQFmoJheGsDSUMb2dZ8EelZLOs3dlAvVPZxdTur3dlxWt3YCrPRgLIiO1I9u5LtNRhkEVGPqSnwyDZoArIraGahvPT0vr4dZSbdcTq6fzGqKkrQEopIVmgvFQ4XElGPydl8sDu6Uw+S6dAGO39LSw1DzGqkdG2jWqU7Tj/9uk6V7TUYZBFRj6kx8OhOPUgmBc1KdP7O1y9c1ralxsCzc6mOU6kL7aXCIIuIekytS47IMQss152/8/ULt/0Qc0UvBwKtEfhbQrCYjBhQ6sA+nTaBZeDZfWpsr8GaLCLqMTmaD6pVd2q+ukstkwnkEBtiLrCasP2QH9sO+rHzcAO2HfRj+yE/7BaT7mrb1FTbqFVSFdpLtj2KvjoR5YVUgUc4KlDdEMTOww2wmo248rT8WXIkF52/8/0L1xcIwdfcin01TWhoDsNsNKDAYoLZaEBDSxj7aprga27Ni9q2TAv7Oaki/3C4kIgk0b4m4rP99Tja2IpQJAqLyQCr2YhVH3hhNCJtAKK1uiO5hybUNplAakV2MxpawghFBApt5vjUe7PRAJPRhKbWMBpawiiya/trKpvhXjXWNlLPaHvvJSJVqaooRVQI/ObfDSiym1HmtKLEYUUwHO20pkSrdUdydv7O+y9cQ7v/CgG0DyTbDzmrN85OkOoiITbjLdP6KrXWNlL3McgiIslEowKr3t+HYDiK4X2K4hkYs8mYdg0xFvqmlu9fuA3N4Xg2qzkchdVkhMkARATQGmn7uchuRkNzWOlN7VKqi4TBZYWoC4Tiw72xY6Gz9fRyPamC5MeaLCKSTLY1JVquO5K7gWa+TyZwOyxwF1jRv6QARTYzwtEomsNRhKNRFNnN6F9SAHeBtcdBpNyfU7rJCZu+rsdGbx0KLMaM66tyOamCcoOZLCKSTLZDXFqtO8rF8GbsC1eNDRal0D5rc1LfIgRaowhFo7AYjXBYjdhX19zjrI3cn1PHi4T22arjnFYc9regurEVvV32pP073XCvWvs9UfcwyCIiyWQ7xKXFuqNcDm/m8xdu+yByX10zypw2uOwWtIQi2FfX3OMgMhefU2cXCVazCRajEY0tYTQFI3B2KODvbLhXjf2eqHsYZBGRZLKtKdFa3VFnmYt0dTY9lc9fuHIFkbn6nDq7SCi0meG0m1AXCKE1EkH7r9tM6qvknFRBucMgi4gkk+0Ql1oLfdO1k1BqeDOfv3DlCCJz9Tl1dpFgAFBWZEdDSxjVja2wmU15NdxLmWGQRUSSyiY7oca6o87qeMIRobnhTS2QOojM1TB0VxcJLaEIvj2wBMUFVuytbsqr4V7KDIMsIpJcNtkJNdUddVXHc+VpFZoa3tSrXA1DZ3KRsGDSsLwd7qWuMcgiIllkk51QQ91RJnU8b+2qxuCyQmw/5FfV8CYlyuUwdKYXCfk63EudY5BFRKqgdN1RJnU8e4404prxA3Ggvlk1w5uULNfD0Gq4SCB1YpBFRITM63iOL3aoZniT0sv1MLTSFwmkTgyyiEhz5FhMOps6nso+LmYuNIAZJlIagywi0hS5unhnW8fDzIU28HMiJXHtQiLSjHTrxMVm/2301nb7ubluHBFJjUEWEWlCLhaTjtXxjCh3w98Sxv66APwtYYwsd0u6XA4R6QOHC4lIE3LVxZt1PEQkFQZZRKQJuVxMmnU8RCQFDhcSkSa0n/2XCrutE5HaMMgiIk2Izf6rbgxCiMS6q9jsv6EeJ7utE5FqMMgiIk3g7D8i0hoGWUSkGZz9R0RawsJ3ItIUzv7THzk6/BPlAoMsItIcLc/+Y8CQHbk6/BPlAoMsIqIcYcCQnViH//pACJ4iG+wWG1pCkXiHfw4Rk9qxJouIKAfkXBIoH+Wiwz+R3BhkERHJjAFD9rLp8E+kVgyyiIhkxoAhe8c6/JtS/t5uMaE1HJGkwz+RXBhkERHJjAFD9tjhn/IBgywiIpkxYMgeO/xTPmCQRUQkMwYM2WOHf8oHDLKIiGTGgKF72OGftM4gOl5WaYzf74fb7YbP54PLpc3mhESkD6n6ZA31ODGLfbI6xQaupCbZxB1sRkpElCNcEihRpsGTljv8k76pIsj605/+hPvvvx+HDx/GKaecgkceeQRjxoxRerNIRnJcmfJql7SAAUMbdr8nPVA8yFq9ejUWLlyIZcuWYezYsXjooYcwZcoU7Nq1Cx6PR+nNIxnIcXLlCZtIO7hcDumF4oXvDzzwAK699lpcddVVOOmkk7Bs2TI4HA48+eSTSm8ayUCOpUW4XAmRdrD7PemJokFWa2srNm7ciEmTJsVvMxqNmDRpEt5///2UjwkGg/D7/Qn/SBvkOLnyhE2kLex+T3qiaJB19OhRRCIR9O7dO+H23r174/Dhwykfc88998Dtdsf/9e/fPxebShLI9OS687AfOw/7seGLGuw87O80QMrmhB2Nioyfl4jkwe73pCeK12Rl6/bbb8fChQvjP/v9fgZaGnHs5GpL+Xu7xYSv6wJY8sIO1P33vl3VVmXynEcbg/jwy1o89uZe1mwRKax99/tCW/JXELvfUz5RNJN13HHHwWQy4Ztvvkm4/ZtvvkGfPn1SPsZms8HlciX8I23oammR6sYgapta4a0JZFxblclyJeGowD827GPNFpEKsPs96YmiQZbVakVVVRVef/31+G3RaBSvv/46Tj/9dAW3jOTQ2ck1KgS8NQEYjQYM9RRmXFvV5Qm7IYhQJIqWUIQ1W0QqwO73pCeKzy5cuHAhnnjiCaxYsQI7duzADTfcgKamJlx11VVKbxpJrLOT654jjYhEBQaWOmA0Ju6WnRXDdnXCtlqMsJiM6O2ys8iWSCW4XA7pheI1WZdffjmqq6vxm9/8BocPH8aoUaPw8ssvJxXDU36InVxjPa2ONgZhNZswoNSBqBAoK7KnfFystipVMWy65xxZ7kZVRQlWvv9Vp0W26Z6XiOTD7vekB1y7kBTRsTt7VAj8bM1ncNnNKYthm4Jh+FvCeODyU9J2y07V8f3zIw1YuHpzj56XiKgnuBpFfuHahaR6HZcWiUYFhnic2HbQB4fVlDC0FyuGHVnu7rQYNtVyJbGarZ48LxFRd3E1Cn1TvCaLCJCvGJZFtkSkFK5GQQyySDXkKoZlkS0R5RpXoyCAw4WkMnIVw7LIlohyKZvVKKSsB2X9l7owyCLVSVVbpebnJSLqKNPVKKSc2cz6L/XhcCEREZHEMlmNQsrlg1j/pU4MsoiI8ggXQleHXC4fxPov9eJwIVGWWPNAasXhIvWIzWxe8sIOeGsDKHPaYLe0ZbaqG4OSzmxWqv6LusYgiygL/BIjtYoNF9UHQvAU2WC32NASisSHiziTNvc6W41iloTnDCXqvygzDLKIMsQvMVKrjsNFsWxGoc0Mh9UEb20AK9d7cWr/EmZdcywXM5vb13+lWtlC6vovyhxrsogywJoHUrNshoso92Izm8cO6oXKPi7JA91c1n9RdhhkEWWAX2KkZseGi9IvhN4ajnC4KE9xZQv1YpBFlAF+iZGa5bpdAKkPV7ZQJ9ZkEWWANQ+kZlwInQCubKFGzGQRZYA1D6RmHC6iGLnrvyg7DLKIMsAvMVI7DhcRqY9BdLws1xi/3w+32w2fzweXi03W1CbfGnem6pM11OOUtOcNUU/k2zEH5Od7Iu3KJu5gTRbJJh8bd7LmgdQu3xZCz8fzCOkHM1kqpfUrt+TGnYnLSXD4goi6wvMIqREzWRqn9Ss3dp8mop7ieYTyAQvfVSZ25bb1gA8uuxn9Shxw2c3xpVs2emuV3sQusXEnUW5EowI7D/ux4Ysa7Dzsz6sVB3geoXzATJaK5MuVGxcrJZKf1jPeXeF5hPIBM1kqki9Xbuw+TSQvqTPeasyI8TxC+YCZLBXJlys3dp8mko/UGW+1ZsR4HlEPrU/EUhKDLBVR89It2RxkscadS17YAW9tAGXO5FlBbNxJ1D3ZZLy7auWQPHvPhpZQJJ4RU3L2nprOI3oOMtQahGsFgywVUeuVW3cOslj36djjjjYGYTWbMLLczcadRD0gVcZbCzWgajiP6DnIUHMQrhUMslRETVduMT05yNi4k0h6UmW8pcyIyUnJ84iegwwtBOEdqTHjyCBLZdRw5RYjxUGWb92niZQmVcZbSzWgSpxHtBhkSEkrQXiMWjOODLJUSC0ZIK0dZER6IFXGW801oGqg9/OfloJwNWcc2cJBpWJXbmMH9UJlH5ciV0rHDjJTyt/bLSa0hiOqOMiI9CSW8R5R7oa/JYz9dQH4W8IYWe7O+AsllhGrbgyi4+pqsYzYUI9Tt7P39H7+00oLjY4Zx0KbGSajAYU2MypKHfA1h7ByvVextiTMZFFa+XKlq8ZxeqKe6mnGW401oGqSL+e/7lLrRKyO1J5xZJBFaWnlIOuMWsfpiaTQ01olKWpA8/UiRmvnP6k/B60E4Wof1mSQRWlp5SBLR83j9ERq0ZOMWD5fxGjp/CfX56CmiVjpqD3jaBAdB+M1xu/3w+12w+fzweXKv+JDNUh1AA/1OFVzkKUSjQosWL0JWw/4EmYGAW1Xod7aAEaWu/Hg5aNUcZIk0prki5jEACRfLmLUfv7Lxeeg5mxl7Fy/7aAPFaW5OddnE3cwk0VdUstsx2yofZyeSMv01N5Azee/XH0Oam7Fo/aMI4MsyoiaD7JU1D5OT6RleruIUev5T2+fQzpqHtZkkEV5Se3j9ERaxosYdeDncIxaM44MsigvaW1mEJGW8CJGHbT0OeSirkuNGUcGWZSX1D5OT6RlvIhRB618Dvk8C7Ur7PhOeUuKrthElCx2EeMusMBbG0BTMIxIVKApGIa3NsCLmBzRwucQm/249YAPLrsZ/UoccNnN8VY6G721im1bLrCFA8lO6em/Sr8+Ub5Se3sDvVDr55CvrXTYwiGH+AXeOTWkidU4Tk+UD9RabKw3av0cOPuRQVaPqCGAUDN2XCfSlu5cNPIiRh3U+Dlw9iODrG5jANE5PTUrJMoHvGgkqWlp9qNcWPjeDR0DiEKbGSajAYU2MypKHfA1h7ByvRfRqKbL3XokmzQxESlL78XJJI/Y7MfqxiA6ln/HZj8O9TgVn/0oJwZZ3cAAomvH0sSmlL+3W0xoDUfyOk1MpAXtLxorejkgAPhbQhAABuTgojEaFdh52I8NX9Rg52G/ri9O840WZj/KjcOF3cBx5q4xTUykDbGLxgKrCdsP+dEUjCAqBIwGAwptJhwnY3Eyhyjzn5qXvMkFBlndwACia1ppkkekd75ACL7mVvibQ4hEAavZCJPRiEhUoKEljEAwDFeBRfKLRta16odaZz/mAocLu4HjzF1jmphIG4rsZjS0hBGKCBRYTTAbDTAAMBsNKLCYEPpvsFVkl+6anHWt+hOb/Th2UC9U9nHp5tzPIKsbGEBkhh3XiTTA0O6/HXtTt/9ZwtMZ61pJLzhc2E16H2fOlJ7TxERa0NAcjmezmsNRWE1GmAxARACtkbafi+xmNDSHJXtN1rVqG5twZ45BVg8wgMiMGpvkEVEbt8MCd4EVxQUWHG1sRVNrGK0CMBrahhKPK7RCwCBpjSnrWrWLkxWywyCrhxhAEJGWtZ+kclLfIgRaowhFo7AYjXBYjdhX1yz5JBVOjNEmTlbIHmuyiIh0rH2N6b66ZhgMgMtugcEA7KtrlqXGlHWt2sPJCt3DIIuISOeUmKTCFQQNLwAAIABJREFUiTHawskK3cPhQiIiUqTGlHWt2sHJCt3DIIuIiAAoU2PKulZt4GSF7uFwIRHlJa6JRyQdNuHuHmayusB+IETaw2nmRNKKTVZY8sIOeGsDKHPaYLe0ZbaqG4OcrJCGQXQMSTXG7/fD7XbD5/PB5eLipUR6lzzNPPGLgEXVRN2X6ntxqMepqybc2cQdDLLSkPNEzewYkTyiUYEFqzdh6wEfBvZyJPVf+qqmCRWlhZg7YTCKC3nsEXWH3r/Dsok7OFyYQsd+ILETdaHNDIfVBG9tACvXe3Fq/5Ksdyxmx4jk09k0c19zGL7mMNZ/UYM91Y0oslt47FGP6DXY4GSFzDHISiGbfiDZ7Gj50i1XrycWUr9008zrAyHsPtKAUETAYAB6Oa2wm02aO/ZIPXjBTJlgkJWCHP1A5MyO5RJPLKRmqaaZCyGwvy6AcFTAajIgIgywmU2aO/ZIPfLlgpnkxxYOKbQ/UafSnX4g+dAtN3Zi2XrAB5fdjH4lDrjs5viJZaO3VulNJJ1LNc28KRhBU2sYVpMRoahAoc0UD8C0cuxRbnXW/oPLy1A2mMlKQY7FS7XeLTdfMnGU31JNMw+GIwhHBSLRCCwmI/qVONB+D1X7sUe51VW2Xq5yEspPzGSlIMfipXJkx3IpHzJxpA8d18SraWqFEIDDasLQ3kUoLkg8xtR+7FHuZJKtP3bBbEr5HHaLCa3hCIN2AsBMVlqxE3XsiuZoYxBWswkjy93d6gciR3Ysl7SeiSN9ab8mXl1TK5b/5wt8XRuA2554ytPCsUe5kWm2fu7Zg7i8DGWMQVYnpFy8VKpuuUrN7OO6VaQ17aeZW81GdqqmTmWarYeApi+YKbcYZHVByn4gPc2OZTKzT64gTOuZONI3qTPTlH8yzdY3tIS5vAxljEFWjnU3O5bJlGEAsrVX4LpVpHVSZqZJekr338smW1/Zx8WgnTLCZXU0oKulQry1AZS7C9AYDMPXLO96bVy3ioikpob+e7Hz7LaDPlSUpj7Pjix348HLR8WDP6UDQ1IGl9XJM13VChxXaMXm/fVwFVgwzOOUtb0CswFEJCW1NPbsTraey8tQVxhkaUBXtQIR0ZbK7ldSkJO+LTyxEGmHmrMtauu/x9o9khqDLA3oqlYgEAwDMKDQmvrjZHsFIn1SwzBcZ9TY2JPZepISm5FqQKqlQmKEEPAHw7BbjGlPAmyvQKQ/WlgGS62NPWPZ+rGDeqGyj4sBFnUbgywN6KoDfZnTipP7uXE0TRBW3RjEUI+T7RWIdEIr6+tpfSWMnuhsfUTKHxwu1IiuagUAsL1CJ9Rcl0IkNTUNw3V27Om1/57ah3FJOgyyNCRVrcCQ45zYc7QRvkAIV55WgTd3HsHe6iYWbLbDExrpjVqWwerq2NNj/z21zKak3GCQpTHtZ/Zt9NZi4drNCSewwWWFuGb8QBxf7GDGBjyhkT6pYRmsTI89Pc3oU9tsSpIfgyyNSncC237IjwP1zbhj2nDdt1ngCY30SulhuGyPPb3M6FPTMC7lBgvfNUgrRa1Ky+aERpRPuposI/cwXHeOPT3M6FPrbEqSD4MsDWLwkBme0EjPYsNwI8rd8LeEsb8uAH9LGCPL3bIPk/PYS03Psyn1isOFGqSWola1U0NdCpGSlBqG47GXmtLDuJR7zGRpEK+GMtNVE1f2DyM9UGIYjsdeakoP41LuMcjSIJ7AMsMTGpEyeOylp+QwLuWeQXT8ltYYv98Pt9sNn88Hl0s/szFiswt9zaGUvWV4sB6TqlfPUI8z76aHE6kNj7302CBZu7KJOxhkaRhPYJnjCY1IGTz2KN8wyNIRnsCIiIhyJ5u4g7MLNa59B3giIr3hhSapGYMsIiLSJK5LSmrH2YVERKQ5sck/Ww/44LKb0a/EAZfdHF8bcaO3VulNJGKQRURE2sKlxUgrFA2yBg4cCIPBkPDv97//vZKbREREKselxUgrFK/Juuuuu3DttdfGfy4q0ncDTTVhQSkRqRGXFiOtUDzIKioqQp8+fZTeDOqABaVEpFZSrI3Ii0jKBUX7ZA0cOBAtLS0IhUIYMGAAZs6ciVtuuQVmc/rYLxgMIhgMxn/2+/3o37+/bvtkySFWUFofCMFTxG7yRPkgn4KKaFRgwepN2HbQh4pSR9JCy97aAEaWu/Hg5aNSvkdeRFJPaKZP1vz58zF69GiUlpZi/fr1uP3223Ho0CE88MADaR9zzz33YPHixTncSn3pWFAaO3kV2sxwWE3w1gawcr0Xp/Yv0ewJmkhv8i2oiK2NuOSFHfDWBlIuLZZubcTki0gbWkKR+KxEXkSSlCTPZP3iF7/Avffe2+l9duzYgcrKyqTbn3zyScydOxeNjY2w2VKPtedzJksNV5o7D/uxcPVmuOzmlGn4pmAY/pYwHrj8FDZBpQTp9l817Nd6ls+Z6WyXFotlwLYe8CVcRAKZZcCIAIUzWT/96U8xZ86cTu8zaNCglLePHTsW4XAYX331FU488cSU97HZbGkDMC1Ty5UmC0qpO9Ltv2MHlWLDF7WK79d6le+Z6aqKUpzavyTjID6bWYm8iCQpSB5klZWVoaysrFuP3bRpE4xGIzwej8RbpW5qSl9LUVBK+pJu/93orcVrO75BicOC/iUODssoQA9BRTZLi/EiknJNsZqs999/Hxs2bMA555yDoqIivP/++7jllltw5ZVXoqSkRKnNyjm1XWkO8xRhiMeJbQd9cFhNSen06sYgRpa7MczDVhuUfv91WE0IRwRCkShCkSgcNjMMyJ8MilYwqEjEi0jKNcWakdpsNjzzzDOYMGECRowYgSVLluCWW27B448/rtQmKUJtTfViBaXuAgu8tQE0BcOIRAWagmF4awOdFpSS/qTbf5uCETS1hmE3mxBojaIpGI7/Tq/NIqNRgZ2H/djwRQ12HvbnpBt5+6AiFb0FFbGLyOrGIDqWI8cuIod6nLyIJMkolskaPXo0PvjgA6VeXjXUeKVZVVGKO6YNj9fYHG0Mwmo2YWS5O21BKelTuv03FI0iKgC72YCWcFs2qz29ZVCUqrlkZjpRT2YlEnWH4s1I9U6t6etsC0pJn9LtvxajEUYDEIoIGA0GWEyJSXM9ZVCUrLlkUJGMF5GUSwyyFKbmK81sCkpJn9Ltv4U2EwqtZtQGWlHisCQEYErv17mkhppLBhXJeBFJucIgS2G80iQt62z/NZvaMlgWkxGBYFiX+7VaZvcxqEjGi0jKBQZZKsArTdKydPvvtytKMaZdnyw97tdqqrlkUEGUewyyVIJXmqRlne2/M74zQLf7tVprLokoNxhkqQivNEnL0u2/et6v1VZzySWOiHKLQRYRkUzUVHOplqW78oVSASsDZW2RfIHoXMtmoUYiIiVku5CxHK+fr4tEK0GpgJWBsjpkE3cwyCIiygElMx8LVm/C1gO+hDYSQNuQpbc2gJHlbjx4+ShmRDKgVMDKQFk9sok7FFtWh4hIT2K1aWMH9UJlH1fOAhq1Ld2lZR37nhXazDAZDSi0mVFR6oCvOYSV672SL5mk1OtSzzHIIiJFKbGmn54cayNhSvl7u8WE1nBEN0sc9YRSASsDZe1i4TsRKYY1JvJjGwnpKNX3TE391ig7zGQR6YiaskaxGpOtB3xw2c3oV+KAy26Or+m30Vur2Lblk1gbierGIDqW4MbaSAz1OPN+iSMptA9YU+luwNrVcSnX65L8mMkiUlAui6HVlDVSw5p+eqGmNhJaJ0ffs0yOS7X1W6PMMZNFpJCN3losWL0JC1dvxh3/2oKFqzdjwepNsmRw1JY1Yo1JbrOKsaWPRpS74W8JY39dAP6WMEaWuzkrLQuxgNVdYIG3NoCmYBiRqEBTMAxvbSDrgDXT41Lq16XcYSaLSAHJ07FtaAlF4idXKb/41Jg1itWY2Cw2NAbDCEWisJiMKLSZYUD+15gokVXk0l3SkGqt2WyPy1SvazEZMaDEgXOGe1BoMyMaFfw8VYZBFlGO5TroySZrlKvlb9wOC8JRga0H6hEMC0SFgNFgQKHNhH4lDliMhrytMcllgN2Rnpc4kpIUAWt3jsv2r/vhF7V4Y+cRfNMQxMr1X+GZD7/mpBEV4nAhUY7leqhMjVP4G1pC8DWH4G8Jw2QACiwmmI0GNLSE8flhP76uC+RlMTb7HeWPnvY96+5xaTQa0BQM49+bDmBfbUAVw/+UHoMsohzL5OQaDEewZb9Pknodtc1MikYFVr2/DwUWEwosJoSiApGogMkAWE1GtISjCLRGcOVp+Vdjwlo0iunucclAXVs4XEiUY131LapuDKK6IYilb+2F0YAe1+uobWZSLNAYUOpAKCKwvy6AptYwWgVgNAAuuwVFdjOKCvLv9MR+RxTT3eNSjcP/lB4zWUQ51lnforpAK/YcaURECJQ5rZIMA6htZlL7TF6xw4IR5S6M6OtGZZ8ijOjrxshyF8xGQ14GGmrLKpJyuntcqnH4n9JjkEWUY+lOro3BMD7/phEAUNm7CE67RbJhADVN4e8YaBgMBjjtZpQ4rHDazQiGo3kbaLAxKLXXneOSgbq25F8+nkgDUk3HjqJtuGxIWSGKHdaE+0sxDKCWKfxqG77MJTYGpY6yPS71fPxoEYMsIoV0PLnuqw1g2Vt7UFZkT3l/Kep11DCFX++BhlR9lih/ZHNc6v340RoGWUQKan9ydTsssFnMuljIV++BhlqyiqRNej9+tIRBFpFK6G0YQO+BhhqyiqRdej9+tIJBFpFK6HEYgIEGUffx+FE/BllEKtLdYYBoVPCKlrgfEKkMgywilcl2GECJxYZJfbgfEKmPQXRs1qIxfr8fbrcbPp8PLhfTpqQvyYsNJw4v5roHFimD+wFR7mQTd7AZKZFGcQ0zArgfEKkZgywijeJiwwRwPyBSMwZZRBrFNcwI4H5ApGYsfCfSqPZrmOV781JKr+N+IAA0BcMIRaKwmNquo7kfECmDQRaRRumteSml1n4/aI1EcaC+GU3BCKJCwIC2IcNTB3A/IFIChwuJNCrWvNRdYIG3NoCmYBiRqEBTMAxvbSAvm5dSsth+YDIasO2gH75ACCYDYDUZERECoUgU3/iD+PTrOqU3lUh3GGQRaViseemIcjf8LWHsrwvA3xLGyHI3p+3ryKn9S+Ap+v/t3XtwlGf5//HP7ibZHHcJNAtG0hQKipVaI8ixM0ZFqF9wxOnQ34zaAtNBW1NbCipga6naQG21OjKdAm3l4E+HehhFpNXBSlumtFKhOIJQ2i9twKQhoYHdhJSc9vn+wWxCyIFNyP3czybv18zONLub5OIByif3fT3Xnak0v09pAZ9a4o5a43GFs9I1sTCktrjDHYaABWwXAikuVc4wYxq5Ocdq6lV3rlkTC0OSfGqJx5Xu9ysneGEbOT3gb7/DkGNYAPcQsoBBwOtnmDGN3KzEHYZZeUEFugmumekBnW5o4g5DwGVsFwIwKjGN/FBlVKHMNI3Oz1YoM02Hq6Iq33lE+yvqbJeY8i6+w7A73GkK2EHIAmAM08jdkbjDsLahSZeelJa403R8JJc7DAGXEbIAGMM0cndwpyngTYQsAMYM9DTyeNzR0eqY/nH8PR2tjrECdhHuNAW8h8Z3AMYM5FR6mucvL1XuNO0r7kxFqiJkATBmoKbSJ5rnzza2KJIXVGZ6UOdb2tqb51mp6eD1O037inCNVMZ2IQBjBqJXaLA1z7PlmTzuTEWqYyULgFGJXqHEasTphiZlpAU0sTCs25JYjehL87zXV3BYlUnepeE68XufE0xTdkZAFXWN2rq3QiVF+WwdwrMIWQCMu5JeoY7m+WC3r6fKoE22PPtmMIVrDF2ELACu6G+v0EA2z9vCqkzfDZZwjaGNniwAnjYYBm0yL6zvmGKPwYCQBcDTBsOgzYGeFzYUDIZwDRCyAHheqg/aZFWm7wZDuAboyQKQElJ50OZAzQsbaq70zlTANkIWgJSRqoM2E6sy5TuPqKKuUQW5QWWmX1jZqm1oYlWmF6kcrgGfc+lmd4qJxWIKh8OKRqMKhVLvf74Aho7u5mSNj+SyKgOkkL7kDlayAMAlrMoAQwshCwBclKpbngD6jrsLAQAADCBkAQAAGEDIAgAAMICQBQAAYAAhCwAAwABCFgAAgAGELAAAAAMIWQAAAAYQsgAAAAwgZAEAABhAyAIAADCAkAUAAGAAIQsAAMAAQhYAAIABhCwAAAADCFkAAAAGELIAAAAMIGQBAAAYQMgCAAAwgJAFAABgACELAADAAEIWAACAAYQsAAAAAwhZAAAABhCyAAAADCBkAQAAGEDIAgAAMICQBQAAYAAhCwAAwABCFgAAgAGELAAAAAMIWQAAAAYQsgAAAAwgZAEAABhAyAIAADCAkAUAAGCAsZBVXl6uGTNmKDs7W8OGDev2PSdOnNDcuXOVnZ2tSCSib3/722ptbTVVEgAAgGvSTH3h5uZmLViwQNOnT9fTTz/d5fW2tjbNnTtXo0aN0t69e/Xuu+/qtttuU3p6utasWWOqLAAAAFf4HMdxTH6DzZs3a+nSpTp79myn55977jnNmzdPVVVVGjlypCRp/fr1WrFihWpra5WRkZHU14/FYgqHw4pGowqFQgNePwAAQEJfcoe1nqxXXnlF119/fXvAkqQ5c+YoFovp8OHDPX5eU1OTYrFYpwcAAIDXWAtZ1dXVnQKWpPaPq6ure/y8tWvXKhwOtz+KioqM1gkAANAffQpZK1eulM/n6/Vx9OhRU7VKklatWqVoNNr+OHnypNHvBwAA0B99anxfvny5Fi1a1Ot7xo4dm9TXGjVqlPbt29fpuVOnTrW/1pNgMKhgMJjU9wAAALClTyGroKBABQUFA/KNp0+frvLyctXU1CgSiUiSdu3apVAopOuuu25AvgcAAIAtxkY4nDhxQnV1dTpx4oTa2tp08OBBSdK4ceOUm5ur2bNn67rrrtOtt96qRx55RNXV1br//vtVVlbGShUAAEh5xkY4LFq0SFu2bOny/O7du1VaWipJqqio0J133qkXXnhBOTk5WrhwoR5++GGlpSWf/RjhAAAA3NKX3GF8TpZphCwAAOCWlJiTBQAAMJgRsgAAAAwgZAEAABhAyAIAADCAkAUAAGAAIQsAAMAAQhYAAIABhCwAAAADCFkAAAAGELIAAAAMIGQBAAAYQMgCAAAwgJAFAABgACELAADAAEIWAACAAYQsAAAAAwhZAAAABqTZLgAAcHnxuKNjNfWKNrYonJ2uD0Xy5Pf7bJcFoBeELADwuP0Vddqyt0Jv1TSoubVNGWkBjYvkauGMYk0qHm67PAA9YLsQADxsf0Wdynce0aHKqEKZaRqdn61QZpoOV0VVvvOI9lfU2S4RQA8IWQDgUfG4oy17K3S2sUXXjMhWTjBNAb9POcE0FQ/PVvT9Fm3dW6F43LFdKoBuELIAwKOO1dTrrZoGRfKC8vk691/5fD4V5Ab1Zk2DjtXUW6oQQG8IWQDgUdHGFjW3tikzPdDt65npATW3tina2OJyZQCSQcgCAI8KZ6crIy2g8y1t3b5+vuVCE3w4O93lygAkg5AFAB71oUiexkVyVdvQJMfp3HflOI5qG5o0PpKrD0XyLFUIoDeELADwKL/fp4UzihXOSldFXaPONbWqLe7oXFOrKuoaFc5K120zipmXBXgUIQsAPGxS8XDdN/cj+mhhWLHzrfrvmUbFzrdqYmFY9839CHOyAA9jGCkAeNyk4uEqKcpn4juQYghZAJAC/H6fJowK2S4DQB8QsgAAQwrnQMIthCwAwJDBOZBwE43vAOBh8bijo9Ux/eP4ezpaHeMInSvAOZBwGytZAOBRrLoMnEvPgUwcU5QTTFN2RkAVdY3aurdCJUX5bB1iwLCSBQAexKrLwOIcSNhAyAIAj7l01SUnmKaA36ecYJqKh2cr+n6Ltu6tML51OJi2KjkHEjawXQgAHtOXVRdTYx0G21blxedA5gS7/tPHOZAwgZUsAPAY26sug3GrknMgYQMhCwA85uJVl+6YXHXxylZlT7X1d/uScyBhA9uFAOAxiVWXw1VRZWcEOm0ZJlZdJhaGjay6eGGrsjsDsX2ZOAcy8XVONzQpIy2giYVh3Zai26DwNkIWAHhMYtWlfOcRVdQ1qiA3qMz0CytbtQ1NRlddOrYqg92+npke0OmGJlcbxBPbl2cbWxTJCyozPajzLW3t25d9OSibcyDhJkIWAHiQrVUXrzWIm5hvxTmQcAshCwA84tIz9UqK8lXy/9xddbG5Vdkdr25fAskgZAGAB3hlZILNrcrueHH7EkgWdxcCgGVeG5mQ2Kr8aGFYsfOt+u+ZRsXOt2piYbhP/U8DweadlsCVYiULACzy6pl6XmkQ99r2JdAXrGQBgEVePlMv0SA+dewITRgVsnIHHvOtkMoIWQBgke3p7qnAS9uXQF+wXQgALrr0DsK8rDRPjUzwKq9sXwJ9QcgCAJd0dwfhtQU5Gp6ToXej79NzdBnMt0KqIWQBgAt6mlr+n3djCvh9Cvh9nhiZAGDgELIAwLBk7iAsHJapYVkZ+t/ac5ypBwwShCwAMCyZOwjfa2jRqs9/RH6/j54jYJAgZAGAYclOLa8/36qpY0e4XB0AUxjhAACGMbUcGJoIWQBgWGJqeW1DkxzH6fRa4g7C8ZFc7iAEBhlCFgAYxtRyYGgiZAGAC1J5ank87uhodUz/OP6ejlbHFI87l/8kADS+A4BbUnFqeXcDVMdFcrWQ0RLAZRGyAMBFqTS1vKcBqoeroirfecTzK3CAbWwXAgC6uHSAak4wTQG/TznBNBUPz1b0/RZt3VvB1iHQC0IWAKCLZAaovlnToGM19ZYqBLyP7UIAQBfJDlCNNrZc0feJx52U6lED+oKQBQDo4uIBqjnBrv9UDMQAVZrqMdixXQgA6ML0ANVEU/2hyqhCmWkanZ+tUGZae1P9/oq6gfhlAFYRsgAAXZgcoEpTPYYKQhYAoFumBqjSVI+hgp4sAECPTAxQdaupHrCNkAUA6NVAD1B1o6ke8AK2CwHAYwb7WYGmm+oBr2AlCwA8xItjDQZ6llWiqb585xFV1DWqIDeozPQLK1u1DU1X1FQPeInPufTHiBQTi8UUDocVjUYVCqXGeWAA0J2uZwV2Dh42zgo0Gfq6+9rjI7m6jTlZ8LC+5A5WsgDAAy4da5C46y4nmKbsjIAq6hq1dW+FSoryXVvhMX1AtImmesBLCFkA4AF9GWswkE3oPXEr9A10Uz3gJTS+A4AHdIw1CHT7emZ6QM2tba6NNWCWFXDlWMkCAA/w2liDK5llxaHPwAWELADwgMRYg8NVUWVnBDqtHiXGGkwsDLs21qC/oc+Ld0cCtrBdCAAeYPKswP7ozywrDn0GOiNkAYBHmDorUOr7gNO+hj4OfQa6YrsQADzExFiD/m7hJUJf4nNPNzQpIy2giYXhLrOsvHZ3JOAFhCwA8JiBHGtwpbOukg19HPoMdEXIAoBBaqBmXSUT+rx2dyTgBfRkAYAlpg+CdnPWFYc+A12xkgUAFrgx6sDNLTwOfQa6YiULAFzm1qiDi7fwujPQW3gm744EUhErWQDgIjcPgrYx4JRDn4EOrGQBgIvc7JOyNeA00Sg/dewITRgVImBhyCJkAYCL3D4Imi08wB62CwHARTZGHbCFB9hByAIAF5nok4rHncsGqIEccAogOYQsAHDRQI86cGMUBID+oScLAFzWU5/URwvD+uq0YrW2OUkNJ3VrFASA/mElCwAsuLRPqvJso3YfrdVTe95OakXKzVEQAPqHlSwAsCTRJ5UW8On/v3pCh6tiSa9IuTkKAkD/GAtZ5eXlmjFjhrKzszVs2LBu3+Pz+bo8tm3bZqokAPCcS1ekcoJpCvh9ygmmqXh4tqLvt2jr3oouW4duj4LoqXaTZy8Cqc7YdmFzc7MWLFig6dOn6+mnn+7xfZs2bdJNN93U/nFPgQwABqO+rEhdfHegjVEQF6PhHrg8YyHr+9//viRp8+bNvb5v2LBhGjVqlKkyAMDT+nuIs40jcxISDfdnG1sUyQsqMz2o8y1t7dubDDkFLrDek1VWVqarrrpKU6ZM0S9+8Qs5Tu/LzU1NTYrFYp0eAJCq+nuIs60jc/q7vQkMRVZD1g9+8AP95je/0a5du3TzzTfrG9/4htatW9fr56xdu1bhcLj9UVRU5FK1ADDwEitStQ1NXX7ITKxIjY/kdrsiZePIHBrugeT1abtw5cqV+tGPftTre44cOaIJEyYk9fW+973vtf93SUmJzp07p0cffVR33313j5+zatUqLVu2rP3jWCxG0AKQsq50OKnbR+b0d3sTGIr6FLKWL1+uRYsW9fqesWPH9ruYqVOn6oc//KGampoUDHb/FzgYDPb4GgCkosSKVKKR/HRDkzLSAppYGNZtSTSSu3VkTjzuqK6xWa1xR3XnmnVVXlCXRjnTDfdAKulTyCooKFBBQYGpWnTw4EHl5+cTogAMOV4/xDlxN+Gbp+p1uqFJVWfPa1jsfRUNz9GwrAuBynTDPZBqjN1deOLECdXV1enEiRNqa2vTwYMHJUnjxo1Tbm6uduzYoVOnTmnatGnKzMzUrl27tGbNGn3rW98yVRIAeJpXD3Huejdhnt6sqdeZxhada4rpQyPzFEzz9+vsRWAwMxayHnjgAW3ZsqX945KSEknS7t27VVpaqvT0dD3++OO699575TiOxo0bp8cee0xLliwxVRIAoI+6O74nJyh9eGRI/z3TqDONzXqzpkFXD89OensTGCp8zuVmJnhcLBZTOBxWNBpVKOS9nwABIJUdrY5p2TP/UigzrcvQU8dxLjS5v9+qlf8zQXOuG8UKFga9vuQO63OyAADe1dvxPT6fT8NzgkoP+DQ8O4OABVyCkAUA6FF/h6UCIGQBAHpxJcNSgaGOkAUA6JGt43uAwYCQBQDolY3je4DBwNgIBwCuNdejAAAJL0lEQVTA4OH1YamAFxGyAABJ8eqwVMCr2C4EAAAwgJAFAABgACELAADAAEIWAACAAYQsAAAAAwhZAAAABhCyAAAADCBkAQAAGEDIAgAAMICQBQAAYAAhCwAAwABCFgAAgAGELAAAAAMIWQAAAAYQsgAAAAwgZAEAABiQZruAK+U4jiQpFotZrgQAAAx2ibyRyB+9SfmQVV9fL0kqKiqyXAkAABgq6uvrFQ6He32Pz0kminlYPB5XVVWV8vLy5PP5bJfTb7FYTEVFRTp58qRCoZDtcqziWnTgWnTgWnTgWlzAdejAtehg+lo4jqP6+noVFhbK7++96yrlV7L8fr9Gjx5tu4wBEwqFhvxfkASuRQeuRQeuRQeuxQVchw5ciw4mr8XlVrASaHwHAAAwgJAFAABgQODBBx980HYRuCAQCKi0tFRpaSm/i3vFuBYduBYduBYduBYXcB06cC06eOVapHzjOwAAgBexXQgAAGAAIQsAAMAAQhYAAIABhCwAAAADCFkedezYMX3xi1/UVVddpVAopBtvvFG7d++2XZYVO3fu1NSpU5WVlaX8/HzNnz/fdklWNTU16eMf/7h8Pp8OHjxouxzXvfPOO7r99ts1ZswYZWVl6dprr9Xq1avV3NxsuzRXPP7447rmmmuUmZmpqVOnat++fbZLct3atWv1yU9+Unl5eYpEIpo/f77eeOMN22V5wsMPPyyfz6elS5faLsWKyspKffWrX9WIESOUlZWl66+/Xv/85z+t1UPI8qh58+aptbVVf//737V//37dcMMNmjdvnqqrq22X5qrf//73uvXWW7V48WL961//0ssvv6wvf/nLtsuy6jvf+Y4KCwttl2HN0aNHFY/HtWHDBh0+fFg//elPtX79en33u9+1XZpxzzzzjJYtW6bVq1frwIEDuuGGGzRnzhzV1NTYLs1VL774osrKyvTqq69q165damlp0ezZs3Xu3DnbpVn12muvacOGDfrYxz5muxQrzpw5o5kzZyo9PV3PPfec/vOf/+gnP/mJ8vPz7RXlwHNqa2sdSc5LL73U/lwsFnMkObt27bJYmbtaWlqcD37wg85TTz1luxTPePbZZ50JEyY4hw8fdiQ5r7/+uu2SPOGRRx5xxowZY7sM46ZMmeKUlZW1f9zW1uYUFhY6a9eutViVfTU1NY4k58UXX7RdijX19fXO+PHjnV27djmf+tSnnHvuucd2Sa5bsWKFc+ONN9ouoxNWsjxoxIgR+vCHP6ytW7fq3Llzam1t1YYNGxSJRDRp0iTb5bnmwIEDqqyslN/vV0lJiT7wgQ/o85//vA4dOmS7NCtOnTqlJUuW6Je//KWys7Ntl+Mp0WhUw4cPt12GUc3Nzdq/f79mzZrV/pzf79esWbP0yiuvWKzMvmg0KkmD/s9Ab8rKyjR37txOfz6Gmj/96U+aPHmyFixYoEgkopKSEj355JNWayJkeZDP59Pf/vY3vf7668rLy1NmZqYee+wx/eUvf7G77Omy48ePS5IefPBB3X///frzn/+s/Px8lZaWqq6uznJ17nIcR4sWLdIdd9yhyZMn2y7HU9566y2tW7dOX//6122XYtTp06fV1tamkSNHdnp+5MiRQ66N4GLxeFxLly7VzJkzNXHiRNvlWLFt2zYdOHBAa9eutV2KVcePH9cTTzyh8ePH669//avuvPNO3X333dqyZYu1mghZLlq5cqV8Pl+vj6NHj8pxHJWVlSkSiWjPnj3at2+f5s+fry984Qt69913bf8yrliy1yEej0uS7rvvPt18882aNGmSNm3aJJ/Pp9/+9reWfxUDI9lrsW7dOtXX12vVqlW2SzYm2WtxscrKSt10001asGCBlixZYqly2FRWVqZDhw5p27Zttkux4uTJk7rnnnv0q1/9SpmZmbbLsSoej+sTn/iE1qxZo5KSEn3ta1/TkiVLtH79ems1cayOi2pra/Xee+/1+p6xY8dqz549mj17ts6cOaNQKNT+2vjx43X77bdr5cqVpks1Ktnr8PLLL+szn/mM9uzZoxtvvLH9talTp2rWrFkqLy83XapxyV6LW265RTt27JDP52t/vq2tTYFAQF/5yles/qQ2UJK9FhkZGZKkqqoqlZaWatq0adq8ebP8/sH9M2Nzc7Oys7P1u9/9rtMdtgsXLtTZs2e1fft2i9XZcdddd2n79u166aWXNGbMGNvlWPHHP/5RX/rSlxQIBNqfa2trk8/nk9/vV1NTU6fXBrPi4mJ97nOf01NPPdX+3BNPPKGHHnpIlZWVVmriFEkXFRQUqKCg4LLva2xslKQu/2j4/f721Z1Ulux1mDRpkoLBoN544432kNXS0qJ33nlHxcXFpst0RbLX4uc//7keeuih9o+rqqo0Z84cPfPMM5o6darJEl2T7LWQLqxgffrTn25f3RzsAUuSMjIyNGnSJD3//PPtISsej+v555/XXXfdZbk6dzmOo29+85v6wx/+oBdeeGHIBixJ+uxnP6t///vfnZ5bvHixJkyYoBUrVgyZgCVJM2fO7DLK49ixY1b/vSBkedD06dOVn5+vhQsX6oEHHlBWVpaefPJJvf3225o7d67t8lwTCoV0xx13aPXq1SoqKlJxcbEeffRRSdKCBQssV+euq6++utPHubm5kqRrr71Wo0ePtlGSNZWVlSotLVVxcbF+/OMfq7a2tv21UaNGWazMvGXLlmnhwoWaPHmypkyZop/97Gc6d+6cFi9ebLs0V5WVlenXv/61tm/frry8vPaetHA4rKysLMvVuSsvL69LL1pOTo5GjBgx5HrU7r33Xs2YMUNr1qzRLbfcon379mnjxo3auHGjvaKs3tuIHr322mvO7NmzneHDhzt5eXnOtGnTnGeffdZ2Wa5rbm52li9f7kQiEScvL8+ZNWuWc+jQIdtlWff2228P2REOmzZtciR1+xgK1q1b51x99dVORkaGM2XKFOfVV1+1XZLrevr937Rpk+3SPGGojnBwHMfZsWOHM3HiRCcYDDoTJkxwNm7caLUeerIAAAAMGPyNDAAAABYQsgAAAAwgZAEAABhAyAIAADCAkAUAAGAAIQsAAMAAQhYAAIABhCwAAAADCFkAAAAGELIAAAAMIGQBAAAYQMgCAAAw4P8AjsUfKQW3mNIAAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import numpy as np\n", + "np.random.seed(5)\n", + "\n", + "import matplotlib.pyplot as plt\n", + "from sklearn.manifold import TSNE\n", + "%matplotlib inline\n", + "\n", + "def plot_tsne(title, x, y=None):\n", + " tsne = TSNE(n_components=2)\n", + " x_t = tsne.fit_transform(x)\n", + "\n", + " plt.figure(figsize=(7, 7))\n", + " plt.title(title)\n", + " alpha = 0.7 if y is None else 0.5\n", + "\n", + " scatter = plt.scatter(x_t[:, 0], x_t[:, 1], c=y, cmap=\"jet\", alpha=alpha)\n", + " if y is not None:\n", + " plt.legend(*scatter.legend_elements(), loc=\"lower left\", title=\"Classes\")\n", + "\n", + "temporal_node_embeddings = temporal_model.wv.vectors\n", + "plot_tsne(\"TSNE visualisation of temporal node embeddings\", temporal_node_embeddings);" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "dXukSMRqCcct" + }, + "source": [ + "You may want to use these embeddings for downstream tasks such as link prediction! To this aim, you can split the dataset in order to create \"future\" link examples. Check [the stellargraph repo](https://colab.research.google.com/github/stellargraph/stellargraph/blob/master/demos/link-prediction/ctdne-link-prediction.ipynb#scrollTo=I2Vw-NfmeMU5) for a full example" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_SBxlsmfUkMx" + }, + "source": [ + "## Temporal Graph Neural Network\n", + "In this example, we will explore the implementation of Temporal Graph Networks (TGN) using PyTorch Geometric (PyG). TGNs are designed to handle dynamic graphs where interactions between nodes occur at different timestamps. We'll use the Wikipedia dataset from JODIE, where nodes represent users and articles, and edges represent user-article interactions." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "VhcWnVDsVAOn" + }, + "source": [ + "First, let's set up our environment and load the data.\n", + "We use the JODIE Wikipedia dataset, which contains temporal interactions between users and articles.\n", + "\n", + "The `TemporalDataLoader` is specially designed for temporal graphs. The `neg_sampling_ratio=1.0` means for each positive edge, we sample one negative edge for training." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "id": "WylvEgEsUPEK" + }, + "outputs": [], + "source": [ + "# Setup and Data Loading\n", + "import os.path as osp\n", + "import torch\n", + "from sklearn.metrics import average_precision_score, roc_auc_score\n", + "from torch.nn import Linear\n", + "from torch_geometric.datasets import JODIEDataset\n", + "from torch_geometric.loader import TemporalDataLoader\n", + "from torch_geometric.nn.models.tgn import LastNeighborLoader\n", + "\n", + "# Device configuration\n", + "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n", + "\n", + "# Load Wikipedia dataset from JODIE\n", + "path = osp.join('data', 'JODIE')\n", + "dataset = JODIEDataset(path, name='wikipedia')\n", + "data = dataset[0]\n", + "data = data.to(device) # Move data to GPU if available\n", + "\n", + "# Split dataset into train, validation, and test sets\n", + "train_data, val_data, test_data = data.train_val_test_split(\n", + " val_ratio=0.15, test_ratio=0.15)\n", + "\n", + "# Create data loaders with negative sampling\n", + "train_loader = TemporalDataLoader(\n", + " train_data,\n", + " batch_size=200,\n", + " neg_sampling_ratio=1.0,\n", + ")\n", + "\n", + "val_loader = TemporalDataLoader(\n", + " val_data,\n", + " batch_size=200,\n", + " neg_sampling_ratio=1.0,\n", + ")\n", + "\n", + "test_loader = TemporalDataLoader(\n", + " test_data,\n", + " batch_size=200,\n", + " neg_sampling_ratio=1.0,\n", + ")\n", + "\n", + "neighbor_loader = LastNeighborLoader(data.num_nodes, size=10, device=device)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "x3JPyEy3VPOn" + }, + "source": [ + "Let's now proceed implementing the key components of TGN.\n", + "* The memory module is a key component of TGN that maintains node states over time" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "id": "sX2UdSRFVb8Y" + }, + "outputs": [], + "source": [ + "from torch_geometric.nn import TGNMemory, TransformerConv\n", + "from torch_geometric.nn.models.tgn import (\n", + " IdentityMessage,\n", + " LastAggregator,\n", + " LastNeighborLoader,\n", + ")\n", + "\n", + "memory_dim = 100\n", + "time_dim = 100\n", + "embedding_dim = 100\n", + "\n", + "memory = TGNMemory(\n", + " data.num_nodes, # Number of nodes in the graph\n", + " data.msg.size(-1), # Message dimension\n", + " memory_dim, # Memory dimension\n", + " time_dim, # Time encoding dimension\n", + " message_module=IdentityMessage(data.msg.size(-1), memory_dim, time_dim),\n", + " aggregator_module=LastAggregator(),\n", + ").to(device)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "b-paMXo2VqOq" + }, + "source": [ + "* Together with the `TGNMemory`, we will also create a GNN for obtaining the embeddings. In this example, we will define a `GraphAttentionEmbedding` class, which uses the `TransformerConv` module (a message passing module implemented in PyTorch)." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": { + "id": "x_MgWFMZVusK" + }, + "outputs": [], + "source": [ + "class GraphAttentionEmbedding(torch.nn.Module):\n", + " def __init__(self, in_channels, out_channels, msg_dim, time_enc):\n", + " super().__init__()\n", + " self.time_enc = time_enc\n", + " edge_dim = msg_dim + time_enc.out_channels\n", + " self.conv = TransformerConv(in_channels, out_channels // 2, heads=2,\n", + " dropout=0.1, edge_dim=edge_dim)\n", + "\n", + " def forward(self, x, last_update, edge_index, t, msg):\n", + " # Compute relative temporal encoding\n", + " rel_t = last_update[edge_index[0]] - t\n", + " rel_t_enc = self.time_enc(rel_t.to(x.dtype))\n", + " # Concatenate temporal and message features\n", + " edge_attr = torch.cat([rel_t_enc, msg], dim=-1)\n", + " return self.conv(x, edge_index, edge_attr)\n", + "\n", + "# Create the GNN\n", + "gnn = GraphAttentionEmbedding(\n", + " in_channels=memory_dim,\n", + " out_channels=embedding_dim,\n", + " msg_dim=data.msg.size(-1),\n", + " time_enc=memory.time_enc,\n", + ").to(device)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "jbRFwBECV0My" + }, + "source": [ + "The GraphAttentionEmbedding uses a transformer-based graph convolution that:\n", + "1. Encodes temporal information using relative timestamps\n", + "2. Combines temporal encodings with edge messages\n", + "3. Applies multi-head attention to compute node embeddings" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "K0AapIP9V7FC" + }, + "source": [ + "* Finally, let's use a simple MLP that predicts link probabilities between node pairs:" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": { + "id": "NH_0oS0OV_Sx" + }, + "outputs": [], + "source": [ + "class LinkPredictor(torch.nn.Module):\n", + " def __init__(self, in_channels):\n", + " super().__init__()\n", + " self.lin_src = Linear(in_channels, in_channels)\n", + " self.lin_dst = Linear(in_channels, in_channels)\n", + " self.lin_final = Linear(in_channels, 1)\n", + "\n", + " def forward(self, z_src, z_dst):\n", + " h = self.lin_src(z_src) + self.lin_dst(z_dst)\n", + " h = h.relu()\n", + " return self.lin_final(h)\n", + "\n", + "# Create the LinkPredictor Object\n", + "link_pred = LinkPredictor(in_channels=embedding_dim).to(device)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "poGFIqhKWD02" + }, + "source": [ + "### Training\n", + "A few important points about the training:\n", + "\n", + "1. Memory and neighbor states are reset at the start of each epoch\n", + "For each batch, we first compute temporal neighborhoods using neighbor_loader\n", + "2. Node embeddings are computed using the current memory state and graph attention\n", + "3. The model predicts both positive and negative links\n", + "After prediction, we update the memory with the true interactions\n" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": { + "id": "IkpNhogDbN-K" + }, + "outputs": [], + "source": [ + "# Let's define the optimizer and the Loss function\n", + "optimizer = torch.optim.Adam(set(memory.parameters()) | set(gnn.parameters()) | set(link_pred.parameters()), lr=0.0001)\n", + "criterion = torch.nn.BCEWithLogitsLoss()\n", + "\n", + "# Helper vector to map global node indices to local ones.\n", + "assoc = torch.empty(data.num_nodes, dtype=torch.long, device=device)" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": { + "id": "6ddZznkJWEgP" + }, + "outputs": [], + "source": [ + "def train():\n", + " # Reset memory and neighbor loader states\n", + " memory.reset_state()\n", + " neighbor_loader.reset_state()\n", + "\n", + " total_loss = 0\n", + " for batch in train_loader:\n", + " optimizer.zero_grad()\n", + " batch = batch.to(device)\n", + "\n", + " # Get temporal neighborhood\n", + " n_id, edge_index, e_id = neighbor_loader(batch.n_id)\n", + " assoc[n_id] = torch.arange(n_id.size(0), device=device)\n", + "\n", + " # Compute node embeddings\n", + " z, last_update = memory(n_id)\n", + " z = gnn(z, last_update, edge_index, data.t[e_id].to(device),\n", + " data.msg[e_id].to(device))\n", + "\n", + " # Predict positive and negative links\n", + " pos_out = link_pred(z[assoc[batch.src]], z[assoc[batch.dst]])\n", + " neg_out = link_pred(z[assoc[batch.src]], z[assoc[batch.neg_dst]])\n", + "\n", + " # Compute binary cross entropy loss\n", + " loss = criterion(pos_out, torch.ones_like(pos_out))\n", + " loss += criterion(neg_out, torch.zeros_like(neg_out))\n", + "\n", + " # Update memory and graph structure\n", + " memory.update_state(batch.src, batch.dst, batch.t, batch.msg)\n", + " neighbor_loader.insert(batch.src, batch.dst)\n", + "\n", + " loss.backward()\n", + " optimizer.step()\n", + " memory.detach()\n", + " total_loss += float(loss) * batch.num_events\n", + "\n", + " return total_loss / train_data.num_events" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "0mAVcbdiVxjT" + }, + "source": [ + "Let's also implement a testing function for model evaluation" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "id": "Fg0UOoVEWdkJ" + }, + "outputs": [], + "source": [ + "@torch.no_grad()\n", + "def test(loader):\n", + " # Set all modules to evaluation mode\n", + " memory.eval()\n", + " gnn.eval()\n", + " link_pred.eval()\n", + "\n", + " # Set random seed for reproducible negative sampling\n", + " torch.manual_seed(12345)\n", + "\n", + " aps, aucs = [], []\n", + " for batch in loader:\n", + " batch = batch.to(device)\n", + "\n", + " # Get temporal neighborhood for current batch\n", + " n_id, edge_index, e_id = neighbor_loader(batch.n_id)\n", + " # Create mapping from global to local node indices\n", + " assoc[n_id] = torch.arange(n_id.size(0), device=device)\n", + "\n", + " # Get node embeddings from memory\n", + " z, last_update = memory(n_id)\n", + " # Update embeddings using graph attention\n", + " z = gnn(z, last_update, edge_index, data.t[e_id].to(device),\n", + " data.msg[e_id].to(device))\n", + "\n", + " # Predict on positive and negative edges\n", + " pos_out = link_pred(z[assoc[batch.src]], z[assoc[batch.dst]])\n", + " neg_out = link_pred(z[assoc[batch.src]], z[assoc[batch.neg_dst]])\n", + "\n", + " # Combine predictions and convert to probabilities\n", + " y_pred = torch.cat([pos_out, neg_out], dim=0).sigmoid().cpu()\n", + " # Create ground truth labels (1 for positive edges, 0 for negative)\n", + " y_true = torch.cat(\n", + " [torch.ones(pos_out.size(0)),\n", + " torch.zeros(neg_out.size(0))], dim=0)\n", + "\n", + " # Calculate metrics\n", + " aps.append(average_precision_score(y_true, y_pred))\n", + " aucs.append(roc_auc_score(y_true, y_pred))\n", + "\n", + " # Update memory and graph with ground truth interactions\n", + " memory.update_state(batch.src, batch.dst, batch.t, batch.msg)\n", + " neighbor_loader.insert(batch.src, batch.dst)\n", + "\n", + " # Return average metrics across all batches\n", + " return float(torch.tensor(aps).mean()), float(torch.tensor(aucs).mean())" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "sl-cqTNYWtYH" + }, + "source": [ + "We evaluate the model using two metrics:\n", + "\n", + "* Average Precision Score (AP): Measures the precision-recall trade-off\n", + "* Area Under ROC Curve (AUC): Measures the model's ability to distinguish between classes" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "TCvdFDqkXtpN" + }, + "source": [ + "Finally, let's train and test the model" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "9I8tl_PeXpWh", + "outputId": "591f36a7-0f65-43c8-b833-9fd2943c8782" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch: 01, Loss: 1.1259\n", + "Val AP: 0.8647, Val AUC: 0.8768\n", + "Test AP: 0.8350, Test AUC: 0.8548\n", + "Epoch: 02, Loss: 0.9952\n", + "Val AP: 0.8293, Val AUC: 0.8436\n", + "Test AP: 0.8149, Test AUC: 0.8225\n", + "Epoch: 03, Loss: 0.9123\n", + "Val AP: 0.8679, Val AUC: 0.8697\n", + "Test AP: 0.8468, Test AUC: 0.8471\n", + "Epoch: 04, Loss: 0.8480\n", + "Val AP: 0.8882, Val AUC: 0.8845\n", + "Test AP: 0.8682, Test AUC: 0.8646\n", + "Epoch: 05, Loss: 0.7954\n", + "Val AP: 0.9040, Val AUC: 0.8983\n", + "Test AP: 0.8839, Test AUC: 0.8776\n", + "Epoch: 06, Loss: 0.7564\n", + "Val AP: 0.9130, Val AUC: 0.9057\n", + "Test AP: 0.8935, Test AUC: 0.8834\n", + "Epoch: 07, Loss: 0.7330\n", + "Val AP: 0.9173, Val AUC: 0.9098\n", + "Test AP: 0.9002, Test AUC: 0.8911\n" + ] + } + ], + "source": [ + "# Training and evaluation loop\n", + "for epoch in range(1, 51):\n", + " loss = train()\n", + " print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}')\n", + "\n", + " # Evaluate on validation and test sets\n", + " val_ap, val_auc = test(val_loader)\n", + " test_ap, test_auc = test(test_loader)\n", + "\n", + " print(f'Val AP: {val_ap:.4f}, Val AUC: {val_auc:.4f}')\n", + " print(f'Test AP: {test_ap:.4f}, Test AUC: {test_auc:.4f}')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "xyT5TyO7XwAl" + }, + "source": [ + "The model achieves around 93.50% performance on the Wikipedia dataset. Notice that the performance differs slightly from the original TGN paper as noted in the PyTorch Geometric repository.\n", + "\n", + "Here, a slightly different evaluation setup is used. Predictions within the same batch are made in parallel, meaning that interactions occurring later in the batch do not have access to any information about earlier interactions in the same batch. By contrast, the original TGN paper's code allows access to earlier interactions in the batch when sampling node neighborhoods for later interactions. While both methods are valid, we, in collaboration with the authors of the paper, chose to present this version as it is more realistic and provides a better testing ground for future methodologies." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ZcdNbcCEQwnQ" + }, + "source": [ + "### Final notes\n", + "* The interested reader can take a look at [Pytorch Geometric Temporal](https://pytorch-geometric-temporal.readthedocs.io/en/latest/modules/root.html) a temporal graph neural network extension library for PyTorch Geometric.\n", + "* A DGL implementation of TGN can be found [here](https://github.com/ytchx1999/TGN-DGL)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "0_1XwgXXRFYM" + }, + "outputs": [], + "source": [] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "chap4", + "language": "python", + "name": "chap4" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.20" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/Chapter11/requirements.txt b/Chapter11/requirements.txt new file mode 100644 index 0000000..104a080 --- /dev/null +++ b/Chapter11/requirements.txt @@ -0,0 +1,132 @@ +absl-py==2.1.0 ; python_version >= "3.8" and python_version < "3.9" +aiohappyeyeballs==2.4.3 ; python_version >= "3.8" and python_version < "3.9" +aiohttp==3.10.10 ; python_version >= "3.8" and python_version < "3.9" +aiosignal==1.3.1 ; python_version >= "3.8" and python_version < "3.9" +annotated-types==0.7.0 ; python_version >= "3.8" and python_version < "3.9" +appnope==0.1.4 ; python_version >= "3.8" and python_version < "3.9" and (platform_system == "Darwin" or sys_platform == "darwin") +asttokens==2.4.1 ; python_version >= "3.8" and python_version < "3.9" +astunparse==1.6.3 ; python_version >= "3.8" and python_version < "3.9" +async-timeout==4.0.3 ; python_version >= "3.8" and python_version < "3.9" +attrs==24.2.0 ; python_version >= "3.8" and python_version < "3.9" +backcall==0.2.0 ; python_version >= "3.8" and python_version < "3.9" +cachetools==5.5.0 ; python_version >= "3.8" and python_version < "3.9" +certifi==2024.8.30 ; python_version >= "3.8" and python_version < "3.9" +cffi==1.17.1 ; python_version >= "3.8" and python_version < "3.9" and implementation_name == "pypy" +chardet==5.2.0 ; python_version >= "3.8" and python_version < "3.9" +charset-normalizer==3.4.0 ; python_version >= "3.8" and python_version < "3.9" +colorama==0.4.6 ; python_version >= "3.8" and python_version < "3.9" and (sys_platform == "win32" or platform_system == "Windows") +comm==0.2.2 ; python_version >= "3.8" and python_version < "3.9" +cycler==0.12.1 ; python_version >= "3.8" and python_version < "3.9" +debugpy==1.8.7 ; python_version >= "3.8" and python_version < "3.9" +decorator==5.1.1 ; python_version >= "3.8" and python_version < "3.9" +dgl @ https://data.dgl.ai/wheels/torch-2.1/dgl-2.4.0-cp38-cp38-manylinux1_x86_64.whl ; python_version >= "3.8" and python_version < "3.9" +executing==2.1.0 ; python_version >= "3.8" and python_version < "3.9" +filelock==3.16.1 ; python_version >= "3.8" and python_version < "3.9" +flatbuffers==2.0.7 ; python_version >= "3.8" and python_version < "3.9" +frozenlist==1.4.1 ; python_version >= "3.8" and python_version < "3.9" +fsspec==2024.9.0 ; python_version >= "3.8" and python_version < "3.9" +gast==0.4.0 ; python_version >= "3.8" and python_version < "3.9" +gensim==4.3.3 ; python_version >= "3.8" and python_version < "3.9" +google-auth-oauthlib==1.0.0 ; python_version >= "3.8" and python_version < "3.9" +google-auth==2.35.0 ; python_version >= "3.8" and python_version < "3.9" +google-pasta==0.2.0 ; python_version >= "3.8" and python_version < "3.9" +grpcio==1.66.2 ; python_version >= "3.8" and python_version < "3.9" +h5py==3.11.0 ; python_version >= "3.8" and python_version < "3.9" +idna==3.10 ; python_version >= "3.8" and python_version < "3.9" +importlib-metadata==8.5.0 ; python_version >= "3.8" and python_version < "3.9" +ipykernel==6.29.5 ; python_version >= "3.8" and python_version < "3.9" +ipython==8.12.3 ; python_version >= "3.8" and python_version < "3.9" +jedi==0.19.1 ; python_version >= "3.8" and python_version < "3.9" +jinja2==3.1.4 ; python_version >= "3.8" and python_version < "3.9" +joblib==1.4.2 ; python_version >= "3.8" and python_version < "3.9" +jupyter-client==8.6.3 ; python_version >= "3.8" and python_version < "3.9" +jupyter-core==5.7.2 ; python_version >= "3.8" and python_version < "3.9" +keras-preprocessing==1.1.2 ; python_version >= "3.8" and python_version < "3.9" +keras==2.7.0 ; python_version >= "3.8" and python_version < "3.9" +kiwisolver==1.4.7 ; python_version >= "3.8" and python_version < "3.9" +libclang==18.1.1 ; python_version >= "3.8" and python_version < "3.9" +lightning-utilities==0.11.7 ; python_version >= "3.8" and python_version < "3.9" +markdown==3.7 ; python_version >= "3.8" and python_version < "3.9" +markupsafe==2.1.5 ; python_version >= "3.8" and python_version < "3.9" +matplotlib-inline==0.1.7 ; python_version >= "3.8" and python_version < "3.9" +matplotlib==3.2.2 ; python_version >= "3.8" and python_version < "3.9" +mpmath==1.3.0 ; python_version >= "3.8" and python_version < "3.9" +multidict==6.1.0 ; python_version >= "3.8" and python_version < "3.9" +nest-asyncio==1.6.0 ; python_version >= "3.8" and python_version < "3.9" +networkx==2.5 ; python_version >= "3.8" and python_version < "3.9" +neural-structured-learning==1.3.1 ; python_version >= "3.8" and python_version < "3.9" +numpy==1.21.6 ; python_version >= "3.8" and python_version < "3.9" +nvidia-cublas-cu12==12.1.3.1 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.8" and python_version < "3.9" +nvidia-cuda-cupti-cu12==12.1.105 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.8" and python_version < "3.9" +nvidia-cuda-nvrtc-cu12==12.1.105 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.8" and python_version < "3.9" +nvidia-cuda-runtime-cu12==12.1.105 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.8" and python_version < "3.9" +nvidia-cudnn-cu12==8.9.2.26 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.8" and python_version < "3.9" +nvidia-cufft-cu12==11.0.2.54 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.8" and python_version < "3.9" +nvidia-curand-cu12==10.3.2.106 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.8" and python_version < "3.9" +nvidia-cusolver-cu12==11.4.5.107 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.8" and python_version < "3.9" +nvidia-cusparse-cu12==12.1.0.106 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.8" and python_version < "3.9" +nvidia-nccl-cu12==2.18.1 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.8" and python_version < "3.9" +nvidia-nvjitlink-cu12==12.6.77 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.8" and python_version < "3.9" +nvidia-nvtx-cu12==12.1.105 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.8" and python_version < "3.9" +oauthlib==3.2.2 ; python_version >= "3.8" and python_version < "3.9" +opt-einsum==3.4.0 ; python_version >= "3.8" and python_version < "3.9" +packaging==24.1 ; python_version >= "3.8" and python_version < "3.9" +pandas==2.0.3 ; python_version >= "3.8" and python_version < "3.9" +parso==0.8.4 ; python_version >= "3.8" and python_version < "3.9" +pexpect==4.9.0 ; python_version >= "3.8" and python_version < "3.9" and sys_platform != "win32" +pickleshare==0.7.5 ; python_version >= "3.8" and python_version < "3.9" +pillow==10.4.0 ; python_version >= "3.8" and python_version < "3.9" +platformdirs==4.3.6 ; python_version >= "3.8" and python_version < "3.9" +prompt-toolkit==3.0.48 ; python_version >= "3.8" and python_version < "3.9" +propcache==0.2.0 ; python_version >= "3.8" and python_version < "3.9" +protobuf==3.20.3 ; python_version >= "3.8" and python_version < "3.9" +psutil==6.0.0 ; python_version >= "3.8" and python_version < "3.9" +ptyprocess==0.7.0 ; python_version >= "3.8" and python_version < "3.9" and sys_platform != "win32" +pure-eval==0.2.3 ; python_version >= "3.8" and python_version < "3.9" +pyasn1-modules==0.4.1 ; python_version >= "3.8" and python_version < "3.9" +pyasn1==0.6.1 ; python_version >= "3.8" and python_version < "3.9" +pycparser==2.22 ; python_version >= "3.8" and python_version < "3.9" and implementation_name == "pypy" +pydantic-core==2.23.4 ; python_version >= "3.8" and python_version < "3.9" +pydantic==2.9.2 ; python_version >= "3.8" and python_version < "3.9" +pygments==2.18.0 ; python_version >= "3.8" and python_version < "3.9" +pyparsing==3.1.4 ; python_version >= "3.8" and python_version < "3.9" +python-dateutil==2.9.0.post0 ; python_version >= "3.8" and python_version < "3.9" +pytz==2024.2 ; python_version >= "3.8" and python_version < "3.9" +pywin32==307 ; sys_platform == "win32" and platform_python_implementation != "PyPy" and python_version >= "3.8" and python_version < "3.9" +pyyaml==6.0.2 ; python_version >= "3.8" and python_version < "3.9" +pyzmq==26.2.0 ; python_version >= "3.8" and python_version < "3.9" +requests-oauthlib==2.0.0 ; python_version >= "3.8" and python_version < "3.9" +requests==2.32.3 ; python_version >= "3.8" and python_version < "3.9" +rsa==4.9 ; python_version >= "3.8" and python_version < "3.9" +scikit-learn==1.3.2 ; python_version >= "3.8" and python_version < "3.9" +scipy==1.10.1 ; python_version >= "3.8" and python_version < "3.9" +setuptools==75.1.0 ; python_version >= "3.8" and python_version < "3.9" +six==1.16.0 ; python_version >= "3.8" and python_version < "3.9" +smart-open==7.0.5 ; python_version >= "3.8" and python_version < "3.9" +stack-data==0.6.3 ; python_version >= "3.8" and python_version < "3.9" +stellargraph==1.2.1 ; python_version >= "3.8" and python_version < "3.9" +sympy==1.13.3 ; python_version >= "3.8" and python_version < "3.9" +tensorboard-data-server==0.7.2 ; python_version >= "3.8" and python_version < "3.9" +tensorboard==2.14.0 ; python_version >= "3.8" and python_version < "3.9" +tensorflow-estimator==2.7.0 ; python_version >= "3.8" and python_version < "3.9" +tensorflow-io-gcs-filesystem==0.21.0 ; python_version >= "3.8" and python_version < "3.9" +tensorflow==2.7.2 ; python_version >= "3.8" and python_version < "3.9" +termcolor==2.4.0 ; python_version >= "3.8" and python_version < "3.9" +threadpoolctl==3.5.0 ; python_version >= "3.8" and python_version < "3.9" +torch-geometric==2.6.1 ; python_version >= "3.8" and python_version < "3.9" +torch==2.1.2 ; python_version >= "3.8" and python_version < "3.9" +torchmetrics==1.4.3 ; python_version >= "3.8" and python_version < "3.9" +torchvision==0.16.2 ; python_version >= "3.8" and python_version < "3.9" +tornado==6.4.1 ; python_version >= "3.8" and python_version < "3.9" +tqdm==4.66.5 ; python_version >= "3.8" and python_version < "3.9" +traitlets==5.14.3 ; python_version >= "3.8" and python_version < "3.9" +triton==2.1.0 ; platform_system == "Linux" and platform_machine == "x86_64" and python_version >= "3.8" and python_version < "3.9" +typing-extensions==4.12.2 ; python_version >= "3.8" and python_version < "3.9" +tzdata==2024.2 ; python_version >= "3.8" and python_version < "3.9" +urllib3==2.2.3 ; python_version >= "3.8" and python_version < "3.9" +wcwidth==0.2.13 ; python_version >= "3.8" and python_version < "3.9" +werkzeug==3.0.4 ; python_version >= "3.8" and python_version < "3.9" +wheel==0.44.0 ; python_version >= "3.8" and python_version < "3.9" +wrapt==1.16.0 ; python_version >= "3.8" and python_version < "3.9" +yarl==1.14.0 ; python_version >= "3.8" and python_version < "3.9" +zipp==3.20.2 ; python_version >= "3.8" and python_version < "3.9" diff --git a/docker/Dockerfile b/docker/Dockerfile index 3530d5e..d060efb 100644 --- a/docker/Dockerfile +++ b/docker/Dockerfile @@ -85,3 +85,9 @@ RUN ls -d -1 */ | grep -v -e Chapter10 | xargs rm -rf RUN conda create -n chap10 python=3.10 RUN conda run -n chap10 pip install -r Chapter10/requirements.txt RUN conda run -n chap10 python -m ipykernel install --name chap10 --user + +FROM base as chap11 +RUN ls -d -1 */ | grep -v -e Chapter11 | xargs rm -rf +RUN conda create -n chap11 python=3.8 +RUN conda run -n chap11 pip install -r Chapter11/requirements.txt +RUN conda run -n chap11 python -m ipykernel install --name chap11 --user