Chapter 13/Neural Machine Translation (LSTM).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neural Machine Translation (NMT)\n",
    "\n",
    "In the universe of natural language processing (NLP), text classification is a relatively simple task. At the opposite end of the difficulty spectrum lies neural machine translation (NMT), which uses deep learning to translate text from one language to another – for example, turning “Hello world” into “Salut le monde.” NMT has proven superior to the [rules-based machine translation](https://en.wikipedia.org/wiki/Rule-based_machine_translation) (RBMT) and [statistical machine translation](https://en.wikipedia.org/wiki/Statistical_machine_translation) (SMT) systems that predated the explosion of deep learning and today is the basis for virtually all state-of-the-art text-translation services. Let's use Keras and TensorFlow to build an NMT model that translates English to French."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data file **en-fr.txt** is a subset of a [larger file](http://www.manythings.org/anki/fra-eng.zip) containing more than 190,000 phrases and their corresponding translations compiled as part of the [Tatoeba project](https://tatoeba.org/). The file is tab-delimited. Each line contains an English phrase, the equivalent French phrase, and an attribution identifying where the translation came from. We don’t need the attributions, so load the dataset into a Pandas DataFrame, remove the attribution column, and shuffle and reindex the rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>en</th>\n",
       "      <th>fr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>You're very clever.</td>\n",
       "      <td>Vous êtes fort ingénieuse.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Are there kids?</td>\n",
       "      <td>Y a-t-il des enfants ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Come in.</td>\n",
       "      <td>Entrez !</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Where's Boston?</td>\n",
       "      <td>Où est Boston ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>You see what I mean?</td>\n",
       "      <td>Vous voyez ce que je veux dire ?</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     en                                fr\n",
       "0   You're very clever.        Vous êtes fort ingénieuse.\n",
       "1       Are there kids?            Y a-t-il des enfants ?\n",
       "2              Come in.                          Entrez !\n",
       "3       Where's Boston?                   Où est Boston ?\n",
       "4  You see what I mean?  Vous voyez ce que je veux dire ?"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('Data/en-fr.txt', names=['en', 'fr', 'attr'], usecols=['en', 'fr'], sep='\\t')\n",
    "df = df.sample(frac=1, random_state=42)\n",
    "df = df.reset_index(drop=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clean the text by removing punctuation symbols and numbers, converting characters to lowercase, and replacing Unicode characters with their ASCII equivalents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>en</th>\n",
       "      <th>fr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>youre very clever</td>\n",
       "      <td>vous etes fort ingenieuse</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>are there kids</td>\n",
       "      <td>y atil des enfants</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>come in</td>\n",
       "      <td>entrez</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>wheres boston</td>\n",
       "      <td>ou est boston</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>you see what i mean</td>\n",
       "      <td>vous voyez ce que je veux dire</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    en                               fr\n",
       "0    youre very clever        vous etes fort ingenieuse\n",
       "1       are there kids              y atil des enfants \n",
       "2              come in                           entrez\n",
       "3        wheres boston                    ou est boston\n",
       "4  you see what i mean  vous voyez ce que je veux dire "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "from unicodedata import normalize\n",
    "\n",
    "def clean_text(text):\n",
    "    text = normalize('NFD', text.lower())\n",
    "    text = re.sub('[^A-Za-z ]+', '', text)\n",
    "    return text\n",
    "\n",
    "df['en'] = df['en'].apply(lambda row: clean_text(row))\n",
    "df['fr'] = df['fr'].apply(lambda row: clean_text(row))\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to scan the phrases and determine the maximum length of the English phrases and then of the French phrases. These lengths will determine the lengths of the sequences input to and output from the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Max phrase length (English): 7\n",
      "Max phrase length (French): 14\n"
     ]
    }
   ],
   "source": [
    "en = df['en']\n",
    "fr = df['fr']\n",
    "\n",
    "en_max_len = max(len(line.split()) for line in en)\n",
    "fr_max_len = max(len(line.split()) for line in fr)\n",
    "\n",
    "print(f'Max phrase length (English): {en_max_len}')\n",
    "print(f'Max phrase length (French): {fr_max_len}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now fit one `Tokenizer` to the English phrases and another `Tokenizer` to their French equivalents, and generate padded sequences for all the phrases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "\n",
    "en_tokenizer = Tokenizer()\n",
    "en_tokenizer.fit_on_texts(en)\n",
    "en_sequences = en_tokenizer.texts_to_sequences(en)\n",
    "en_x = pad_sequences(en_sequences, maxlen=en_max_len, padding='post')\n",
    "\n",
    "fr_tokenizer = Tokenizer()\n",
    "fr_tokenizer.fit_on_texts(fr)\n",
    "fr_sequences = fr_tokenizer.texts_to_sequences(fr)\n",
    "fr_y = pad_sequences(fr_sequences, maxlen=fr_max_len, padding='post')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the vocabulary sizes from the `Tokenizer` instances:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary size (English): 6033\n",
      "Vocabulary size (French): 12195\n"
     ]
    }
   ],
   "source": [
    "en_vocab_size = len(en_tokenizer.word_index) + 1\n",
    "fr_vocab_size = len(fr_tokenizer.word_index) + 1\n",
    "\n",
    "print(f'Vocabulary size (English): {en_vocab_size}')\n",
    "print(f'Vocabulary size (French): {fr_vocab_size}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build and train a model\n",
    "\n",
    "Now it's time to define a model. We'll start with an `Embedding` layer followed by an `LSTM` layer that encodes English phrases input to the model. A second `LSTM` layer decodes those phrases into their French equivalents. In between is a `RepeatVector` layer whose purpose is to reshape the output from the first `LSTM` layer for input to the second. The final layer is a softmax classification layer that outputs probabilities for each word in the French vocabulary. The `TimeDistributed` wrapper ensures that the model outputs a set of probabilities for each token (word) in the output rather than just one set for the entire sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "____________________________________________________________________________________________________\n",
      " Layer (type)                                Output Shape                            Param #        \n",
      "====================================================================================================\n",
      " embedding (Embedding)                       (None, 7, 256)                          1544448        \n",
      "                                                                                                    \n",
      " lstm (LSTM)                                 (None, 256)                             525312         \n",
      "                                                                                                    \n",
      " repeat_vector (RepeatVector)                (None, 14, 256)                         0              \n",
      "                                                                                                    \n",
      " lstm_1 (LSTM)                               (None, 14, 256)                         525312         \n",
      "                                                                                                    \n",
      " dropout (Dropout)                           (None, 14, 256)                         0              \n",
      "                                                                                                    \n",
      " time_distributed (TimeDistributed)          (None, 14, 12195)                       3134115        \n",
      "                                                                                                    \n",
      "====================================================================================================\n",
      "Total params: 5,729,187\n",
      "Trainable params: 5,729,187\n",
      "Non-trainable params: 0\n",
      "____________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Embedding, LSTM, RepeatVector, TimeDistributed, Dropout\n",
    "\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(en_vocab_size, 256, input_length=en_max_len, mask_zero=True))\n",
    "model.add(LSTM(256))\n",
    "model.add(RepeatVector(fr_max_len))\n",
    "model.add(LSTM(256, return_sequences=True))\n",
    "model.add(Dropout(0.4))\n",
    "model.add(TimeDistributed(Dense(fr_vocab_size, activation='softmax')))\n",
    "model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "model.summary(line_length=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the model, and use an `EarlyStopping` callback to end training if the validation accuracy fails to improve for three consecutive epochs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "800/800 [==============================] - 134s 164ms/step - loss: 2.1994 - accuracy: 0.7258 - val_loss: 1.8796 - val_accuracy: 0.7378\n",
      "Epoch 2/50\n",
      "800/800 [==============================] - 137s 171ms/step - loss: 1.8238 - accuracy: 0.7436 - val_loss: 1.7514 - val_accuracy: 0.7520\n",
      "Epoch 3/50\n",
      "800/800 [==============================] - 135s 169ms/step - loss: 1.6835 - accuracy: 0.7583 - val_loss: 1.6041 - val_accuracy: 0.7728\n",
      "Epoch 4/50\n",
      "800/800 [==============================] - 135s 168ms/step - loss: 1.5247 - accuracy: 0.7764 - val_loss: 1.4834 - val_accuracy: 0.7850\n",
      "Epoch 5/50\n",
      "800/800 [==============================] - 134s 168ms/step - loss: 1.3972 - accuracy: 0.7872 - val_loss: 1.3885 - val_accuracy: 0.7938\n",
      "Epoch 6/50\n",
      "800/800 [==============================] - 137s 171ms/step - loss: 1.2890 - accuracy: 0.7951 - val_loss: 1.3138 - val_accuracy: 0.8002\n",
      "Epoch 7/50\n",
      "800/800 [==============================] - 136s 170ms/step - loss: 1.1905 - accuracy: 0.8017 - val_loss: 1.2486 - val_accuracy: 0.8053\n",
      "Epoch 8/50\n",
      "800/800 [==============================] - 139s 173ms/step - loss: 1.1001 - accuracy: 0.8076 - val_loss: 1.1928 - val_accuracy: 0.8080\n",
      "Epoch 9/50\n",
      "800/800 [==============================] - 145s 181ms/step - loss: 1.0201 - accuracy: 0.8128 - val_loss: 1.1492 - val_accuracy: 0.8117\n",
      "Epoch 10/50\n",
      "800/800 [==============================] - 146s 182ms/step - loss: 0.9446 - accuracy: 0.8177 - val_loss: 1.1087 - val_accuracy: 0.8159\n",
      "Epoch 11/50\n",
      "800/800 [==============================] - 153s 191ms/step - loss: 0.8779 - accuracy: 0.8237 - val_loss: 1.0760 - val_accuracy: 0.8198\n",
      "Epoch 12/50\n",
      "800/800 [==============================] - 185s 231ms/step - loss: 0.8181 - accuracy: 0.8289 - val_loss: 1.0516 - val_accuracy: 0.8218\n",
      "Epoch 13/50\n",
      "800/800 [==============================] - 189s 236ms/step - loss: 0.7636 - accuracy: 0.8340 - val_loss: 1.0257 - val_accuracy: 0.8247\n",
      "Epoch 14/50\n",
      "800/800 [==============================] - 182s 227ms/step - loss: 0.7133 - accuracy: 0.8399 - val_loss: 1.0078 - val_accuracy: 0.8272\n",
      "Epoch 15/50\n",
      "800/800 [==============================] - 176s 220ms/step - loss: 0.6725 - accuracy: 0.8451 - val_loss: 1.0002 - val_accuracy: 0.8292\n",
      "Epoch 16/50\n",
      "800/800 [==============================] - 177s 221ms/step - loss: 0.6337 - accuracy: 0.8500 - val_loss: 0.9895 - val_accuracy: 0.8307\n",
      "Epoch 17/50\n",
      "800/800 [==============================] - 180s 225ms/step - loss: 0.6023 - accuracy: 0.8547 - val_loss: 0.9769 - val_accuracy: 0.8330\n",
      "Epoch 18/50\n",
      "800/800 [==============================] - 179s 224ms/step - loss: 0.5711 - accuracy: 0.8597 - val_loss: 0.9712 - val_accuracy: 0.8329\n",
      "Epoch 19/50\n",
      "800/800 [==============================] - 150s 188ms/step - loss: 0.5453 - accuracy: 0.8639 - val_loss: 0.9658 - val_accuracy: 0.8345\n",
      "Epoch 20/50\n",
      "800/800 [==============================] - 180s 225ms/step - loss: 0.5234 - accuracy: 0.8670 - val_loss: 0.9629 - val_accuracy: 0.8354\n",
      "Epoch 21/50\n",
      "800/800 [==============================] - 173s 217ms/step - loss: 0.5013 - accuracy: 0.8710 - val_loss: 0.9572 - val_accuracy: 0.8364\n",
      "Epoch 22/50\n",
      "800/800 [==============================] - 191s 239ms/step - loss: 0.4836 - accuracy: 0.8738 - val_loss: 0.9584 - val_accuracy: 0.8364\n",
      "Epoch 23/50\n",
      "800/800 [==============================] - 188s 236ms/step - loss: 0.4656 - accuracy: 0.8770 - val_loss: 0.9624 - val_accuracy: 0.8362\n",
      "Epoch 24/50\n",
      "800/800 [==============================] - 187s 233ms/step - loss: 0.4500 - accuracy: 0.8794 - val_loss: 0.9597 - val_accuracy: 0.8377\n",
      "Epoch 25/50\n",
      "800/800 [==============================] - 191s 239ms/step - loss: 0.4371 - accuracy: 0.8815 - val_loss: 0.9643 - val_accuracy: 0.8377\n",
      "Epoch 26/50\n",
      "800/800 [==============================] - 187s 234ms/step - loss: 0.4227 - accuracy: 0.8846 - val_loss: 0.9618 - val_accuracy: 0.8373\n",
      "Epoch 27/50\n",
      "800/800 [==============================] - 156s 196ms/step - loss: 0.4105 - accuracy: 0.8868 - val_loss: 0.9624 - val_accuracy: 0.8378\n",
      "Epoch 28/50\n",
      "800/800 [==============================] - 179s 224ms/step - loss: 0.3990 - accuracy: 0.8889 - val_loss: 0.9663 - val_accuracy: 0.8389\n",
      "Epoch 29/50\n",
      "800/800 [==============================] - 187s 233ms/step - loss: 0.3893 - accuracy: 0.8906 - val_loss: 0.9646 - val_accuracy: 0.8383\n",
      "Epoch 30/50\n",
      "800/800 [==============================] - 155s 193ms/step - loss: 0.3793 - accuracy: 0.8926 - val_loss: 0.9710 - val_accuracy: 0.8391\n",
      "Epoch 31/50\n",
      "800/800 [==============================] - 195s 244ms/step - loss: 0.3706 - accuracy: 0.8942 - val_loss: 0.9778 - val_accuracy: 0.8399\n",
      "Epoch 32/50\n",
      "800/800 [==============================] - 200s 250ms/step - loss: 0.3629 - accuracy: 0.8957 - val_loss: 0.9732 - val_accuracy: 0.8387\n",
      "Epoch 33/50\n",
      "800/800 [==============================] - 201s 251ms/step - loss: 0.3545 - accuracy: 0.8970 - val_loss: 0.9732 - val_accuracy: 0.8395\n",
      "Epoch 34/50\n",
      "800/800 [==============================] - 188s 235ms/step - loss: 0.3473 - accuracy: 0.8987 - val_loss: 0.9863 - val_accuracy: 0.8397\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.callbacks import EarlyStopping\n",
    "\n",
    "callback = EarlyStopping(monitor='val_accuracy', patience=3, restore_best_weights=True)\n",
    "hist = model.fit(en_x, fr_y, epochs=50, batch_size=50, validation_split=0.2, callbacks=[callback])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the per-epoch training and validation accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEXCAYAAACDChKsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAABYrElEQVR4nO3dd0BV5f/A8fe9XC4bGQJu3Bsc4d7mCkTRNFFTyfJb/TLLlpRp5shypGZlWWZqwzRNw9yaWmKucuHeKHvIhrvO7w/yGgGC5GXo5/WXzxnP+ZyH6/3c85xznkelKIqCEEIIcY/UZR2AEEKIikkSiBBCiBKRBCKEEKJEJIEIIYQoEUkgQgghSkQSiBBCiBKRBCKKbebMmQwcOJCBAwfSvHlz+vbtay5nZ2cXu55x48Zx8eLFu26zaNEiNmzY8B8jvn9OnjxJz5498y1/8803mTp1ar7l27ZtY8CAAYXWt379ep599lkAJk+eTHh4eLGP+W9r167l22+/BeD7779n6dKlRe5zL95//32aN29OTEzMfa1XVHyasg5AVBxvv/22+d89e/Zk3rx5+Pj43HM9X3zxRZHbvPTSS/dcb1kYMWIEISEhvPXWW9ja2pqXr1mzhpEjRxarjlmzZv2nGI4ePUqDBg0AGD58+H+q699ycnLYsGEDffv25ZtvvuG11167r/WLik0SiLgvFi9ezLFjx4iLi6NRo0aEhoYydepUEhMTiY+Pp3r16ixcuBB3d3d69uzJokWLyMzMZMGCBdSsWZMLFy6g0+mYOnUq7du3JzQ0lAYNGvD000/j4+PD//73P/bv309cXByjR48mJCQEo9HInDlz2L17N05OTvj6+nLp0iVWrVqVJ7bMzEymTZvG1atXSUlJwcHBgXnz5lG3bl1GjRpFy5Yt+fPPP4mOjuaRRx7hgw8+QK1W891337FixQocHR1p2LBhgeft4+NDnTp12Lp1K0FBQQDcuHGDU6dOsXjxYn788Ud++OEH9Ho9KSkpjBs3jhEjRuSpY9SoUYwcOZJ+/foVesyEhIQC2/PPP/9k9+7d7N+/H1tbW5KSkkhOTmbq1KlcuHCB6dOnc+vWLVQqFWPHjiUoKIiDBw8W2u7/9ssvv1CrVi1CQkJ4+umneeGFF7CzswPgypUrTJ06laSkJNRqNc8//zz+/v6FLr/9d7/9o+N22dXVlZEjR1KvXj1u3rzJqlWrWL9+PTt37iQnJ4esrCwmTZpE7969MRgMzJ07lz179mBlZUWrVq145513CAwMZMqUKXTu3BnI/bHToEEDxowZU7IPtCgW6cIS983Nmzf56aefmDdvHr/88gstW7bkhx9+YNeuXdja2rJx48Z8+5w4cYKxY8eyYcMGhgwZwscff5xvG51Oh6urK6tXr+ajjz5i/vz55OTksHbtWiIiIti0aROrV68mMjKywLj27duHs7Mza9asYdu2bTRv3tzc5QNw/fp1Vq1axc8//8wff/zBoUOHOHPmDB9//DHffPMN69atw9rautDzHjlyJOvWrTOX165dy8CBA1EUhbVr17J06VI2bNjAggULmDt3bqH13O2YhbVn79696dmzJyEhIXmueAwGA88//zyjRo0iLCyML774gg8//JC//vqr2O0OuV1iAwYMwMfHBw8PD3766SfzuldeeYV+/frxyy+/sHTpUj788EPS09MLXX43MTEx/N///R/btm1Dr9cTHh7ON998Q1hYGBMnTuSjjz4C4LvvviMiIoKNGzeyadMmMjIy2Lx5M8OHD2ft2rUApKens2vXLgYNGnTXY4r/ThKIuG9atmyJRpN7UTtmzBhat27N8uXLmTZtGhcuXCAzMzPfPtWqVaNJkyYANG3alJSUlALrfvTRRwFo1qwZOp2OzMxM9u7dy8CBA7GxsUGr1TJs2LAC9+3Xrx+DBg1i1apVzJw5k0OHDuWJpUePHqjVahwdHfH29iYlJYUDBw7QqVMnPDw8AAqtG8Df358LFy5w/fp1DAYD69evZ8SIETg4OPDZZ5+xd+9eFi5cyGeffVZgG9x2t2MWtz1vu3r1Kjk5OfTp0wcALy8v+vTpw2+//QYUr90jIiI4c+YMAQEBAAQFBbFy5UoUReHWrVucPXuWoUOHAlC1alV27tyJwWAocLmjo2OhsQJoNBpatmwJQPXq1fnggw8ICwtj3rx5rF69moyMDADCw8MZOHAgtra2qNVqFi5cSFBQEIMHDyY8PJykpCR+/vlnunfvjrOz812PKf47SSDivrG3tzf/e+7cuebuiWHDhtGpUycKGnbtn/cNVCpVgdsA2NjYmLcBUBTFnKxuU6sL/jh/9913TJ48GVtbWwIDA+nfv3+e4xQUw79jsbKyKvS8bWxsGDx4MOvWrWPPnj00bNiQ2rVrExMTQ1BQEDdv3uSRRx7h5ZdfLrSOgs7/n8csbnveZjKZ8i1TFAWDwVDoOf/bd999h0aj4fHHH6dnz56sWrWKq1evsm/fPnPb3/57AFy+fNkc87+X337I4p/H0el05n9rtVpznREREQQHB5Oenk6nTp145plnzNv9+2+ekJBAXFwczs7O9OvXj59//pl169bd93tBomCSQIRF/P7774wZM4agoCDc3d0JDw/HaDTe12N069aNn3/+GZ1Oh8FgyNO98u9YBg0axNChQ6lTpw67d+8uMpaOHTuyf/9+85NHhdV92/Dhw9m8eTPr1683dyWdOnUKNzc3/u///o8uXbrw66+/AhR67Lsd827taWVlZU4Mt9WpUwdra2u2b98OQGxsLNu2baNjx453PY/bUlNT+eWXX/jss8/YvXs3u3fvZt++fQwYMICvv/4aR0dHmjVrZn5SLjo6muHDh5OdnV3g8rS0NNzc3Dh16hQAx44dIz4+vsBjHz58mObNm/PUU0/Rtm1bdu3aZT7XDh06sGnTJnQ6HSaTiWnTpvHLL78AuV2Jt6+QfH19i3We4r+Rm+jCIl544QXmzJnDp59+ipWVFa1bt+b69ev39RiDBw/mypUrBAUFYW9vT40aNcw3eP9p7NixTJ06lfXr12NlZUWzZs04f/78Xetu1KgRr7/+OmPGjMHBwaHIL6SaNWtSp04dzp8/T/fu3QHo1KkTP/74I/369cPOzg5fX1/c3Ny4du3aPR/zbu3ZtWtXZsyYkacua2trPv30U2bOnMnixYsxGo288MILtG/fnoMHD971XCA3edWrVy/fjfXnn3+egIAAzp8/z/z583n33XdZtWoVKpWKWbNm4eHhUejy1157jWnTpvHDDz/QrFkzmjVrVuCx+/fvz/bt2/H398fa2poOHTqQkpJCeno6wcHB3Lx5k8GDB6MoCm3btmXUqFEANG7cmEqVKhEcHFzk+Yn7QyXDuYuK6vfffycxMZGBAwcCue+p2NjY8Prrr5dxZKIsXL9+nVGjRrF169YCf0iI+0+6sESF1aBBAzZs2MCAAQMICAggOTmZ5557rqzDEmVg0aJFDB8+nEmTJknyKEVyBSKEEKJE5ApECCFEiUgCEUIIUSKSQIQQQpSIJBAhhBAl8lC9B5KcnIHJlPeZAXd3RxIT7z5OT3lVUWOXuEuXxF26HqS41WoVrq4Ohe7zUCUQk0nJl0BuL6+oKmrsEnfpkrhL18MSt3RhCSGEKBFJIEIIIUrEogkkLCwMf39/evfunWf+hdv27t1LYGAggYGBvPrqq+Yhm1NTU/nf//7HY489xsiRI82Drul0Ol5//XUee+wxBg0axKVLlywZvhBCiLuwWAKJjY1lwYIFfPfdd2zcuJEffvghzzzYqamphIaGsmDBAsLCwmjcuDELFiwAYOHChfj5+bFlyxaGDh1qnvJz1apV2NnZsWXLFt566y1CQ0MtFb4QQogiWCyBhIeH0759e1xcXLC3t6dv375s3brVvP7q1atUq1aN+vXrA7mT+uzcuROAPXv2EBgYCOSOzLlv3z70ej179uxhwIABALRp04bk5GSioqIsdQpCCCHuwmJPYcXFxZlnVgPw9PTkxIkT5vLtCXfOnj1L48aN2bJlCwkJCfn21Wg0ODo6kpSUlK9ODw8PYmJiqFatWrFicncveFY0Dw+nez6/8qKixi5xly6Ju3Q9LHFbLIEUNEbjP2cpc3Z25oMPPmDKlCmYTCaeeOKJu847Xdhsc4UtL0hiYnq+x9Q8PJyIj08rdh3lSUWNXeIuXRJ36SrruFMydFyJTuVqdCpXotO4GpNKu6ZejOjV8K77FRS3Wq0q9Ic3WDCBeHl5ceTIEXM5Li4OT09Pc9loNFKlShXWrl0L5E5jWbNmTSD3aiUhIYEqVapgMBhIT0/HxcUFT09P4uPj8fb2BiA+Pj5PnUII8TDJzNZzLSaNKzFp5qSRmJoDgEoF1So70KJeZdo18bLI8S2WQDp27MjixYtJSkrCzs6O7du355k1TaVSMXbsWNauXYunpydfffUV/v7+QO5UpRs2bOC5555j8+bN+Pn5YW1tTbdu3di4cSN+fn4cOXIEGxubYndfCSFERZWaqSM6IYOoxEyiEjKISsggOjGDW+l35pX3dLGjXvVK9PJzpk5VZ2p5OWKrtey74ha9Apk4cSKjR49Gr9czZMgQfH19GTduHBMmTMDHx4fp06fzzDPPoNPp6NChA08//TQAL730EqGhoQQEBODk5MS8efMAGDVqFFOnTiUgIACtVsucOXMsFb4QQpS6HJ2RG/HpXI9NIzIuPTdZJGaSnqU3b2OjtaKauwPNartRtbIDtbwcqV3FGUe7wm8BWMpDNaGU3AMpHyTu0iVxl67ixp2SoeN6bJo5WVyPTSc2KZPb31D2NhqqezhQrbIDVd0dqFbZnmruDrg62eS5n2zJuMvsHogQQojcB4qSUnO4FpvG1ZjchHEtJo2UjDvdT5Ur2VLT05H2Tb2o6eVILU8n3JwtkyjuJ0kgQghxnyiKQmxSJn+ejePa34niakyauQvq9o3tZnXcqOXlhLeXIzU9HbG3Lf3up/tBEogQQpSQoihEJWZy/noyZ6/f4nzkLfOVhZVaRbXKDrRsUJnaVZzw9nKihqcjNtZWZRz1/SMJRAghismkKETFZ3Au8hbnridzLvIWaZm5VxeuTjY08XalVRMvKjtqqeHhgLXmwUkWBZEEIoQQhcjI1nM1Oo3Lf79jceFGirk7yt3ZhuZ13GlUy4XGtVzwcLFDpVJV2Jv/JSEJRAghAL3ByPXYdC5Hp3Ll77e4Y5Myzeu93OxpUc+dRrVcaVzLhcoudmUYbfkgCUQI8VDK0Rk5f+MWZ64lc/ZaMpFx6Rj/fsy/kqOWulWd6dS8CnWqOVOnilOFvdFtSZJAhBAPBb3BxOWoFM5cS+bMtWQuR6ViNClYqVXUq+ZMv3a1qF3FmbrVnHF1sinrcCsESSBCiAeSSVGIjE0n4moSZ64mceFGCjqDCZUKaldxpm/bWjTxdqV+jUoP1JNRpUkSiBDigZGSoSPiSiKnriRx+koSqX8/IVXDw4GuLavRxNuVRjVdsbeVr777QVpRCFFhGYwmLt5I4dSVJE5dSeR6bDoATvbWNKvjRvM6bjSr404lB20ZR/pgkgQihKhQTIrCuWvJ/HYymr8uJJCjM+bex6heice71aV5HXdqejmiLufDgDwIJIEIISqExJRs9p+K5vcT0SSkZGNno6FdEy9a1HOnsbcrdjbydVbapMWFEOWW3mDirwvx/HYimtNXklCAJt6uDO5al9YNPdDKze8yJQlECFGuGIwmrkansf73K/x6JJKMbANuzjYEdqpNJ5+qeMgLfOWGJBAhRJkymRSuxaZx9nru+xkXIlPI0RvRWKlp3bAyXXxzn55Sq+WeRnkjCUQIUapMisLN+AzO/v1C37nIW2TlGACo6m5Px+ZVaOLtSudHapKdkVPG0Yq7kQQihLC4rBwDEVeSOH4pgZOXEs3vZ3i62NGmsQeNa7nS2NsVF8c7b4A72WslgZRzkkCEEBYRl5zJ8YuJHL+UwLnrtzCaFOxtNPjUc6d5HTca13LFvZJtWYcp/gOLJpCwsDCWLFmCXq8nJCSEkSNH5lkfERHB1KlT0ev1VK1alblz5+Ls7MzgwYMxGo0AZGdnExkZyb59+9DpdAQEBFCrVi0AKleuzLJlyyx5CkKIYjIpChcib5mTRnRi7ki2Vd3t6d2mJi3quVO/RiWs1OoyjlTcLxZLILGxsSxYsID169ej1WoJDg6mXbt21K9f37zNrFmzmDBhAt26deP9999n2bJlTJw4kfXr15u3eeONNxg0aBCVK1dm27ZtBAYGMn36dEuFLYS4R6kZOn4/Gc2+Y1HE3crCSq2icS0XureqTot67ni62pd1iMJCLJZAwsPDad++PS4uLgD07duXrVu3Mn78ePM2JpOJjIwMALKysqhUqVKeOg4cOMDZs2eZPXs2ACdPnuT8+fMMHjwYR0dHJk+eTKNGjSx1CkKIQpgUhbPXktl7LIo/z8djNCk0rOnCwM51aNmgsrzU95Cw2F85Li4ODw8Pc9nT05MTJ07k2SY0NJSnnnqK9957Dzs7O9asWZNn/UcffcTEiROxssp9WcjGxoagoCCCg4PZu3cvL7zwAps3b0arlXFuhCgNqRk69p+KZu+xKOKSs3Cw1fDoIzXo2qIa1So7lHV4opRZLIEoipJvmeofY9NkZ2czefJkVqxYga+vL8uXL2fSpEksXboUgAsXLpCcnEyPHj3M+7z44ovmf3fr1o358+dz+fJlGjduXKyY3N0dC1zu4eFUrP3Lo4oau8Rduv5L3AajieMX4tl1OJIDJ6MwGBWa1XXnycea0Mm3mkXfBn8Y27ss3WvcFksgXl5eHDlyxFyOi4vD09PTXD5//jw2Njb4+voCMGzYMBYtWmRev3PnTvz9/fPUuWrVKvr374+rqyuQm6Q0muKfQmJiOiZT3sRWkecvrqixS9ylqyRxK4rCpZup/HE6hsNn40jL1ONgq6FHqxp0bVmN6n9fbaTcyiyippJ7mNq7PCgobrVaVegPb7BgAunYsSOLFy8mKSkJOzs7tm/fzowZM8zrvb29iYmJ4fLly9StW5ddu3bh4+NjXn/s2DHGjBmTp87Dhw+TnZ3NuHHjOHToECaTibp161rqFIR46NyMT+eP07EcPB1LQko21ho1LetXpn0zL5rXccdaI09QiTssegUyceJERo8ejV6vZ8iQIfj6+jJu3DgmTJiAj48Ps2fP5uWXX0ZRFNzd3XnvvffM+0dGRuLl5ZWnzsmTJxMaGsrGjRuxsbFh/vz5qOWRQCH+k5T0HMJPxXAgIpYb8emoVNCsthsDO9ehdUMPuSEuCqVSCrpZ8YCSLqzyQeIuXYXFnZSazZaD19l7LAqD0US9as60a+pFmyZe5WICpgetvcu7ctWFJYQonxJTstn8xzV+OxGFokCH5lXwb+9NFTd5X0PcG0kgQjwk4m9l8cuBa+w/GQ1AF9+q+Lf3prIMjy5KSBKIEA+4qIR0Vv1yhvBTMajV0K1lNfzbe+PmLONQif9GEogQD6Db84bvPR7FkbNxWFmp6flIdR5r542rk03RFQhRDJJAhHiAJKVms/9kNL/9PW+4vY2GAV3r0c2nCpUcJXGI+0sSiBAVnMFo4vjFBH47Ec3Jy4koCjSu5cKgrnV5pKEH1au5VMingkT5JwlEiAoqOjGDfcejCD8VQ1qmHhdHLQEdvOnsU1VGwBWlQhKIEBVMVEIGG3+/wuGzcVipVbSoX5kuvlVpXtdN5toQpUoSiBAVRGxyJj//foU/Tsei1VgR0MGbXn41y8VLf+LhJAlEiHIuISWLsP1X2X8yBo2Vir5tatGvfS2c7SVxiLIlCUSIcio5LYdN4VfZdzwKlQp6tq5OQAdveZpKlBuSQIQoZ9Kz9GwKv8ruP2+iKApdfKvSv2NtefFPlDuSQIQoJxRF4Y/TsazedYH0LD2dmlclsFNtPGSoEVFOSQIRohyITcpk1fZznL6aTN1qzrw6rCW1vCrmrHbi4SEJRIgypDeY2HrwGmHh17DWqHiyT0O6t6yOWq0qemchypgkECHKyLnryazcdo7oxEzaNPZkeK8GuMgNclGBSAIRopSlZ+lZ8+tFfj8RTeVKtkx8ogU+dd3LOiwh7pkkECFKiUlR2H8ymrW/XiIrx4B/e28CO9XGxtqqrEMTokQkgQhRCi5FpfDdjvNciU6jfvVKjO7biBqehU8VKkRFIAlECAtKSc/hx72X2H8yhkqOWsYFNqV9Uy9UKrlJLio+iyaQsLAwlixZgl6vJyQkhJEjR+ZZHxERwdSpU9Hr9VStWpW5c+fi7OzM4cOHGT9+PFWqVAGgadOmzJ49m9TUVF577TUiIyNxc3Nj4cKFeHh4WPIUhCgRg9HEziM3+Hn/FfQGE/7tvQno4I2djfxmEw8Oi32aY2NjWbBgAevXr0er1RIcHEy7du2oX7++eZtZs2YxYcIEunXrxvvvv8+yZcuYOHEiJ0+eZOzYsTz77LN56ly4cCF+fn4sXbqUDRs2MGvWLBYuXGipUxCiRE5dTuS7nReIScqkRT13gh9tgJebDK8uHjwWG/s5PDyc9u3b4+Ligr29PX379mXr1q15tjGZTGRkZACQlZWFrW3uUA0nT55k//79BAUF8dxzzxEdHQ3Anj17CAwMBKB///7s27cPvV5vqVMQ4p7E3criox9P8OGa4yiKwstDfXlpaAtJHuKBZbErkLi4uDzdS56enpw4cSLPNqGhoTz11FO899572NnZsWbNGgCcnJwICAigV69efP/990ycOJHVq1fnqVOj0eDo6EhSUhJeXl7FisndveCblh4eFfeN34oa+4MUt05vZN3uC6zdfQGNlYqQgKYM6FoXa035ebrqQWrviuBhidtiCURRlHzL/nnjMDs7m8mTJ7NixQp8fX1Zvnw5kyZNYunSpUyfPt283fDhw5k/fz5paQVPyam+hwl0EhPTMZnyxuXh4VRhp/usqLE/SHGfuJTAtzvOE38rm7ZNPBnWswGuTjbcSs4soyjze5DauyJ4kOJWq1WF/vAGC3ZheXl5kZCQYC7HxcXh6elpLp8/fx4bGxt8fX0BGDZsGIcOHcJkMrFkyRKMRmOe+jQaDZ6enuY6DQYD6enpuLi4WOoUhChUQkoWi9edYOHaE2is1LwW3JLnBjbH1UneJBcPD4slkI4dO3LgwAGSkpLIyspi+/btdO3a1bze29ubmJgYLl++DMCuXbvw8fFBrVazY8cOtm3bBsCGDRto0aIFdnZ2dOvWjQ0bNgCwefNm/Pz8sLa2ttQpCJGPwWjilwNXefuLg0RcTeLxbnV5d2xbmtZ2K+vQhCh1KqWgvqb7JCwsjM8//xy9Xs+QIUMYN24c48aNY8KECfj4+LB3717mz5+Poii4u7szY8YMatasyYULF5gyZQppaWm4ubkxZ84cqlatyq1btwgNDSUyMhInJyfmzZtHjRo1ih2PdGGVDxU17qjkbD5ee4yYpExaN/Rg+KMNcK9U/ufoqKjtLXGXrpJ0YVk0gZQ3kkDKh4oWd0qGjtW7LnDwdCyeLnaM6N0Q33oVZ+yqitbet0ncpaskCUTeahKiEIqisP9kDD/svkCO3sjwPo3o7lulXD1dJURZkgQiRAHibmWxcutZTl9NpkGNSoQ81hjfxlUq5C9LISxFEogQ/2A0mdhx+AYbfruMWq1iVJ+GdGtVHbWMXSVEPpJAhPjb9dg0lm85y7WYNFrWr8yTfRri5lz+b5ILUVYkgYiHnk5v5Of9V9l68DqOdhqeD2qOXyMPGTFXiCJIAhEPtctRqXwRFkFschadfavyRI/6ONrJu0Xiv1NyMjBlpWDlUg0A/eVDmG7FYNN6QG750iGUnHS0TXvmbq8o9/SjxRh3GaxtsXLNrV93Zg9ql6poqjYCIOfoBqw866Kp6Xs/TysPi71IKER5ZlIUtvxxjdnfHMVgNPFacEvG+jeR5PEQMWUko7/4B4o+BwBjwjVyjqzHlJ1mXm+MuYBizB2wVVEUFEOOeX9jwjV0Z/aYy7rTv5K56QNzOefwOjI3zrqz/c0z6C/sN5cNlw+hP7vXXM7e8TFZ2xbdWX/9BIbrd8YPzNzyITkH15jLWdsWoj+57c7xDq3FcPnwnXiOb8EYdfYeWuTeyRWIeOikpOfw5abTRFxN5pFGHoQ81hgHW0kcpUExGcBoRGVtg6KYMMVfQeXghtrBFcVkzC07upvLxqSbqB1cUNk4mMfXK+xXumIyoeSko7K2RaXRouRkYIg6i5VXPdT2LhhjL5Id/i223Z7Gyq0GxrhLZO/+DPvB07CqXBtTUiS6P8OwbtwNAMO1v8j5fSUOTy5EZe+CPmIXOeHf4PjU56isbTBEnkB3eB3WDTqi0mhBrQa1FYopdxgm64adsarWxByfTedR2Kru/Ga36z0+T0KyqtIQlDtDOOUc/AGVoxuaWrlXEGonD1T2Lub1tr1eQO3gai47PDEblebOUDqOT31m8W5YuQIRD5UTlxKZ+tUhLtxIYUy/RvxfUHNJHhaiKAqGyJMY46/klg060pc/j+72r2ajgcwNM+78KjfoyNw4E8OlgwCYstLJ/HEy+gsHcvfPvEX6F2PR/f2r3ZSRTPoPoRiu/pVbTr5BxqoJGCJzf7WbUuPI3rEY09/Hx9oOlY0D/P0Fr6nWBPshM1C7Vs9d3bAzjuOWoXLIHZZG490Ku8deQWWbO0Kt2r0m2rZDgdxEpm3SA4cnF4JV7udH27gb9v6voVLnvidk5VkX67ptzO2hUuX/uv3nF77Wty/aFv7msp3/a9j1Gm8u23Yehdanj7msqdoItfOd8QXVds6orO/UVxr38OQKRDwU9AYT6/ZeYvvhSGp4OPDsiOZUr+xQ1mGVOiU7HSU7DbVLVQB0EbtQslKx8RsE5Ha7KLpMbDuNyi0fXINiMmLbYXhu+dCPoFZj4zc4t3x0IyprLVrfx3LLf/6MytoGrU9fVCoV2Xu+xKpmc+y6j0Ol0aJtPSD3lzaAWoNdv1dQu+TOPIpGi91jr5rLKq0tto/+H1aVa+WWNVq0rQOxcvc2n4+Vey2wyZ1vRe3ojk3HkVi51cwtu1bDfvC75i9ZK7fq2Pu/Zt5XZeOAlU3ez8A/v+TVDq55fuFrqjYy318AUNk6Ysmv6H8eu7ySBCIeeDFJmXy+MYJrsWn0bF2dJ3rUR2v9YL5NbkpPxJQah+bvrhPd6V+JS76MqtPTAGSHf4Mx9hKOw+fmbp94DVNaonl/xaADg+5OWZ9j/sUOYMq8heofUyiYEq6av8ABjDdP30kI5P6K/ucXoU2rQPO/VWq1uXsmt2yFpqaPuay2tsG6Xts7620czIkLcr9g7Xr9X5712ua975Q1NlhVvpNsxP0nCUQ8sBRFIfxUDN9sP4/GSsWLg31o1dCj6B3LMcWgw5R0A7VHHVQqFbqze9Gf2oH949NRqdToI3ahO7kNx6e/QKVSo+gyMaQlofn7CR9t00dR6rc312fbdWye+m9faZjLnUflKdt1fyZvue9Lecr2gaF5ylbuNUt8rqL8kwQiHkjJaTl8s/0cf11IoFFNF8YFNq2QLwWa0hIwXP0T60ZdUGnt0J/dS074tziMXIDKwRWV1h51pSq5Vw3Wtlg37oamdmvz/jYtA/DwCDYPwWJVpUFZnYp4AEkCEQ8Uk6Kw56+b/LjnEkaTwtDu9ejbthZqddm+FKgoJlQqNcb4K2TtWoJtxxFoarXEmHyTrO2Lse08Gk31phgTI8ne/Rk2nUejqdoI060ocg58h7qyd24ffK2WqBzcUGlzu42s67bJc6NWXckLKhVvimch/qsiE0hSUhJubjJZjij/bsans2LrOS7eTKFpbVdG922Ep6t90TtakJKTQcaGGWib9ULbvBdq12pYedQ1P9mj0mixcq+FSmuXW7ayRu1SFZV17tWSVdXGODy5EPXfj2+qnT1QO1fsbjjx4CgygfTv358OHTowfPhw/Pz8SiMmIe6J3mBkU/g1Nv9xDTsbDU8HNKFj8yqlOhTJ7beIFUUhe8diVM5e2LYflvukj1d91I6584eoNDbYPfqceT+1k0eeG8FqlyrY9b7z6KZKo819x0CIcqjIBLJ7925++eUX5syZQ1ZWFsHBwQwcOBBHx8InGRGitJy7nsyKreeIScqkQzMvhj3aAGd7y37hmrJSUTKSsKpcG4Cs3Z+jZKViH/A6KpUKlb2r+QoD8t94FuJBUWQCsbW15fHHH+fxxx/n4MGDvPXWW8ybN4+goCDGjx+Pu3vFmZlNPDgysvWs/fUi+45HU7mSLa8Ma0HzOpb5LBrjLmOMPge9ngAg58D3GKPO4PjkQgCsvOqj6LPM2//7ySUhHlTFuom+b98+1q5dy9GjRwkMDGTw4MHs3buX559/njVr1hRdgRD30Z/n41m17RxpmXr6tavFwE51sNHev/c6jEk3MFz8A63fIFRqKwyRJ9EdC8PUPQgArU9flMZdzd1W2maP3rdjC1GRFJlAunfvjqurKyNGjGDu3LnY2ube3GvUqBE//PDDXfcNCwtjyZIl6PV6QkJCGDlyZJ71ERERTJ06Fb1eT9WqVZk7dy7Ozs5cunSJKVOmkJGRga2tLdOmTaNJkyZERUUREBBArVq5b6ZWrlyZZcuWlfTcRQWTmqHj2x3nOXw2jpqejrw8tAXeVZyK3rEIptR4dKd3o/Xpg9rBFVNyFLoTW9A06ICVa3W0zXuh9e2HWqMFcrDyqP2fjynEg6DIBPLhhx/SqFEjHBwc0Ol0JCYmmrutdu3aVeh+sbGxLFiwgPXr16PVagkODqZdu3bUr1/fvM2sWbOYMGEC3bp14/3332fZsmVMnDiRt99+m//973/06NGDAwcOMGnSJH7++WdOnjxJYGAg06dPvw+nLioKRVE4eDqW73ZeIFtnYFCXOjzW3huNVcmGclMMORguH0ZduQ5WbtVR9FnoT21HU60JagdXNN4tcQz51DxOkcrm4RvyRIjiKPJ/YExMDIMG5Y6Tc/PmTQICAti9e3eRFYeHh9O+fXtcXFywt7enb9++bN26Nc82JpOJjIwMALKyssxXN0OHDqVr165A7pVOdHQ0ACdPnuT8+fMMHjyY0aNHc+7cuXs4VVERJafl8NGPJ1gadhpPVzveeaotgZ3q3HPyMKXEYEy68XfBRPa+5RguHwJA7VYDx5BPzcNq5D75ZFNYVUKIvxX5v/Czzz5j5cqVANSpU4effvqJxYsXF1lxXFwcHh53nlf39PQkNjY2zzahoaFMnjyZzp07Ex4eTnBwMACDBw/Gyiq3T/ujjz6iV69eANjY2BAUFMT69et5+umneeGFF9DpdIgHj6Io7Dsexdtf/sGZa8kM61mft5585J4GQFRyMsx1Zf4yF93hdQCotHY4DJmF9pGg3LJKLQlDiBIosgvLZDJRpcqdwdGqVq2KyWQqsuLbY/f/0z+fy8/Ozmby5MmsWLECX19fli9fzqRJk1i6dKl5/zlz5nD8+HFzAnvxxRfN+3fr1o358+dz+fJlGjduXGQ8AO7uBT967OHx3/vRy0pFjf1uccckZvDx2mMcv5CAT73KjH+iBdUq39tj4/GbPiXnxllqPLsIlUpFVtBLaCp5YO3y93FL2G4PYnuXZxJ36brXuItMIG5ubqxevZohQ4agUqn46aefqFy5cpEVe3l5ceTIEXM5Li4OT887Y9efP38eGxsbfH1zuw2GDRvGokW5s3EZDAYmTZpEbGwsK1euxMkp96RWrVpF//79cXXNHd1TURQ0muKPxpKYmI7JlDexeXg4mccJqmgqaux3i/uPiBhWbDuHChjVtxHdWlZDrShFnqcx7jK6E1uw/XvYcINXc9QOVYmPS8mdn8G+FuiB/9BeD2J7l2cSd+kqKG61WlXoD28oRhfW9OnTWbNmDb6+vvj6+rJmzRreeeedIoPp2LEjBw4cICkpiaysLLZv326+rwHg7e1NTEwMly9fBnJvyPv45A7l/MEHH5Cens5XX31lTh4Ahw8f5scffwTg0KFDmEwm6tatW2QsovzL0Rn5avMZloadpqanI9OfbkuPVtVRFzb7nKJgjL2IKTMlt6zPxhhzAVNKbjeppnYrtM17mSf3EULcfyqloL6mAqSkpGBlZXVPb6CHhYXx+eefo9frGTJkCOPGjWPcuHFMmDABHx8f9u7dy/z581EUBXd3d2bMmIGDgwOdO3emRo0a2NnZmevauHEjsbGxhIaGEh8fj42NDbNmzSp29xXIFUh58e+4I+PS+WzjKWISMwnoWJuBnWtjpc7720ZRFJSUGFCpUFeqginzFhnfvIy2zRBsWvVHUUygkGeuCkvHXVFI3KXrQYq7qCuQIhNIUlISP//8MxkZGSiKgslk4tq1a8yfP//+RF2KJIGUD7fjVhSFPcei+H7nBRxsNYwLbErT2ncG7jQmR4E+GyvPuigmE+krX8C6bhvzHBb6y4fQ1PAxD0RYWnFXNBJ36XqQ4i4qgRR5A+Hll1/G1taWixcv0rFjR8LDw3nkkUf+e7TioZaZrWf5lrMcPRdP8zpuPNO/Kc4OWhRdljkhZO/5ApWVNfYD3kKlVmP36PO5c1/8zbpu28KqF0KUgiKv96Oioli6dCldu3blySef5Pvvv+f69eulEZt4QJ29lsQ7Xx3m2IUEhvaox8tPtMDZQUvOX5vIWPNm7rSqgG2nUdj+YyBCTU1f8/zWQoiyV+QVyO0nrmrXrs358+cZMGAABoPB4oGJB4/RZGLboUh+2ncZVycbQp9sTW1NIipdJtg4YFW1IeizzXNwW3nKAxJClGdFJhB3d3e+/PJLWrZsyeLFi3F0dCQ9Pb00YhMPkIs3U1i17RyRcel0alGN4T3qYatLJmP1dLRtHsemVSCaKg3RVGlY1qEKIYqpyAQyffp0fvnlF/z8/GjevDkfffQRr732WmnEJh4AaZk61u65xO8nonF1suG1bja0rJ6IzrYx2Hpi2+sFNDWal3WYQogSKDKBfPDBB8yZMweA119/nddff93iQYmKz/T3UCTr9lwiW2ekX7taDOhUG2X/clIPXcGmRntUaqs883kLISqWIhPI2bNnzfMeCFEcV2NSWbXtPFeiU2lXXeEJp8NUeqQ5aq0GpcNwPKpWJiE5u6zDFEL8R0UmEA8PDwICAmjRogUODncGsnv77bctGpioeDKy9azfd5k9f96kkkPuex1tvbVk/bwZU2osamcPVLaOqDTWgCQQISq6IhNIq1ataNWqVWnEIiooRVE4fDaO73acJy1Lz6s1/6RWZVscm+UOXeMQPNeib4gLIcpGkQlk/PjxpRGHqKCS03JYte0cNy9fxt2rOhOfaEmVmGwwGcxdn5I8hHgwFZlAAgMDC1weFhZ234MRFcft+TrW/HqRpqorvO2yB9u+b2NdxQmq+Jd1eEKIUlBkApkyZYr533q9np07d+YZll08fOKS0tm36RfOxpnwrtaIIb0HoL3pgca1almHJoQoRUUmkLZt84431LFjR4KDg3n++ectFpQon0wmhR1HItm47yKhjvvwq1OfWoODc4dc9wgo6/CEEKWs+LMx/S05OZm4uDhLxCLKsbgjO0n8azdrEh/Fp15lHDuF4lqlujzeLcRD7J7vgURFRTFs2DCLBSTKByUnA/2lQ1jVa8fWP+O49sc12tqqee6x2vj51pXEIYS4t3sgKpUKNzc36tWrZ9GgRNlQsnPHOFPZOmJMjiLn9xVsO5LA5ih3HmnUloZ9R+Fsry3jKIUQ5UWRz1fWqlWLzZs307ZtW9zd3Zk/fz4JCQmlEZsoRUpOBunfvowuYicAp9MrsTg7iJ1xlQl5rDH/F9RckocQIo8iE0hoaKh53vHq1avTtm1b3nzzTYsHJixPf/EAOUd+AkBl44BN++FQszWrd11g4Y+nyHaoxtSQtnRtUU26rIQQ+RSZQJKTkxk9ejQANjY2hISEEB8fb/HAhOUZYy9huBmBYsyd3yWpagfe/yWW7YcjebR1DaaMeYRqlR2KqEUI8bAq8h6I0WgkNjYWLy8vABISEihiGnVRTilGA7oTW9HUboWVa3Vs2j0Bag2oVOw/Gc0328+jsVLx4mAfWjX0KOtwhRDlXJEJJCQkhKCgILp06YJKpSI8PJw33nijWJWHhYWxZMkS9Ho9ISEhjBw5Ms/6iIgIpk6dil6vp2rVqsydOxdnZ2dSU1N57bXXiIyMxM3NjYULF+Lh4YFOp2Py5MmcOnUKW1tb5s2bJzf074Giy0R3YguYDFg9Uh2VRktWjoFV28/xR0QsjWq6MC6wKW7OtmUdqhCiAiiyC2vIkCEsX76cpk2b0rx5c7766qtChzf5p9jYWBYsWMB3333Hxo0b+eGHH7h48WKebWbNmsWECRP4+eefqVOnDsuWLQNg4cKF+Pn5sWXLFoYOHcqsWbMAWLVqFXZ2dmzZsoW33nqL0NDQkpzzQ0XJyUAXsRNFUVDbOeMwZCY2jwQBEJ2YwfQVRzh4OpagLnV4fXgrSR5CiGIrMoHExsayevVqQkJC6NSpEwsWLCjWPZDw8HDat2+Pi4sL9vb29O3bl61bt+bZxmQykZGRAUBWVha2trlfXnv27DEnqf79+7Nv3z70ej179uxhwIABALRp04bk5GSioqLu7YwfMvoL4eSEf4sp+QYAagdXAE5cSmTmyqNkZut5Y3grBnSqg1otN8qFEMVXZAKZNGlSvqew3nrrrSIrjouLw8PjTj+6p6cnsbGxebYJDQ1l8uTJdO7cmfDwcIKDg/Ptq9FocHR0JCkpKV+dHh4exMTEFOM0Hy6KyYApLTfJWzftif3g6Vi51cxdpyhsOXiNRWuP41HJlilj/GhUy7UswxVCVFBF3gMp6CmsDRs2FFlxQTfa//koaHZ2NpMnT2bFihX4+vqyfPlyJk2axNKlSwusT13IkOCFLS+Iu7tjgcs9PJyKXUd5U1DscRsWknPjHDWeXYja2ga8XADI0Rv5eO0x9hy9QacW1Xh5WCtsbe55NJv7oqK2ucRduiTu0nWvcVvsKSwvLy+OHDliLsfFxeUZxff8+fPY2Njg6+sLwLBhw1i0aBGQe7WSkJBAlSpVMBgMpKen4+LigqenJ/Hx8Xh7ewMQHx9/TyMDJyamYzLljd3Dw4n4+LRi11GeFBa7qUEPNJ5NSLylA3RA7rwdH68/wZXoNAZ1qUP/jrVJS82iLM68ora5xF26JO7SVVDcarWq0B/eUIwurNtPYb3xxhu88cYbDBo0iGeeeabIYDp27MiBAwdISkoiKyuL7du307VrV/N6b29vYmJiuHz5MgC7du3Cx8cHgG7dupmvcjZv3oyfnx/W1tZ069aNjRs3AnDkyBFsbGyoVq1akbE8DPRn96E7uQ0AK8+6WDfoaF53KSqF6SsOE5WYyfjBPgR2qiMvBgoh/rMir0CGDBlC8+bN+eOPP7CysqJWrVqsXLmyyCexvLy8mDhxIqNHj0av1zNkyBB8fX0ZN24cEyZMwMfHh9mzZ/Pyyy+jKAru7u689957ALz00kuEhoYSEBCAk5MT8+bNA2DUqFFMnTqVgIAAtFotc+bMuQ9NUPEpioLhxikUXSbWzXujUt35XbD/ZDQrtp7FxdGGV0e1pIZH4b8mhBDiXqiUYvRHpaSksHr1ar777jsyMzMZNWoUEyZMKI347qsHrQvL1c5AYlImajtnFIMO1Fao1FYAmBSFtb9eZNuhSJp4u/J8UHMc7azLOOJcFbXNJe7SJXGXrpJ0Yd31CuTy5ct8/fXXhIWFUb16dbKzs9m9ezdOThXzBtGDRDEZiFr5NoqjJ/aPvYJKc2egQ5OisHLrWfYdj6ZH6+oMf7QBGiuZl1wIcX8VmkDGjRtHREQE/v7+rFy5Eh8fH3r27CnJo5xQqTW49xxNGnn/HiaTwvLNZ9h/Kob+Hb0Z1EXm7hBCWEahCeTMmTM0bdqUBg0aULt2bQD5IioHjLEXUQw6NNWb4tC4HZn/uOQ0mkws23SGP07HEtS5DgM61ynDSIUQD7pC+zX27NnD448/zqZNm+jcuTMTJkwgJyenNGMT/6IoCjmH1pIT/i2KyZRnncFo4vOfT/PH6Vge71ZXkocQwuIKTSAajYbHHnuMVatWsW7dOjw9PcnOzqZPnz58//33pRmj+JtKpcKuzwTs+k1E9Y8XKA1GE59tjODI2Tie6FGfgA61yy5IIcRDo1h3VuvXr8/bb7/Nb7/9xtNPP82aNWssHZf4B1NWKjlHN6CYTKhsHFA7VTav0xtMfLL+JH+ej2dErwb0a1erDCMVQjxM7unRHDs7O4YNG8ZPP/1kqXhEAQyXD6M7thlTSnSe5Tl6I4vXn+D4pURG9W1EL7+aZRShEOJhVDYDIYl7om32KJqavqid7wwkmaM3MmPZH0RcTiLkscZ0bSFv5AshSpe8HFBOKSYj2eHfYkrNHVX3n8lDpzeyaO1xTl5MYGxAE0keQogyIQmknFLS4tFfCMdwMyLfutW7LnD2+i0mjniETj5VyyA6IYSQLqxyS12pCg5PzEZt55xn+cHTsew5FoV/e2+6t65RIYdMEEI8GOQKpJzRn/sN3Zk9APmSR2xSJl9vPUv9GpUY1FXe8xBClC1JIOWM4dpfGC4dzDfnit5gZMmGU2jUKp4b0Ayre5hISwghLEG6sMoZ297jQZeVb9iY1bsvcj0unZeG+OLmbFtG0QkhxB3yM7acMESfQ8nJQKVSo7JxyLPu8Nk4fv3zJv3a1qJF/cqF1CCEEKVLEkg5oOhzyN6+mOzfVuRbF5ecyfLNZ6hXzZnB3eqWQXRCCFEw6cIqB1TWNtj5v5rvykNvMLFkQwRWahXPDWwuc3oIIcoV+UYqY4ouCwArjzqonT3zrFvz60WuxaYxNqAJ7pXkvocQonyRBFKGTBnJZKx+A/253/KtO3I2jl1Hb9CnTU1aNfAoYG8hhChbkkDKkEqjRePdCiuvBnmWx93KYvmWs9Sp6syQ7vXKKDohhLg7i94DCQsLY8mSJej1ekJCQhg5cqR53ZkzZwgNDTWXk5KSqFSpEitWrGDs2LHm5WlpaSQnJ/PXX39x+PBhxo8fT5UqVQBo2rQps2fPtuQpWJTKxgHbbmPzLDMYTXy+8RQq4PmBzeS+hxCi3LJYAomNjWXBggWsX78erVZLcHAw7dq1o379+gA0adKEjRs3ApCVlcXQoUOZNm0a7u7u5uUmk4kxY8YwceJEAE6ePMnYsWN59tlnLRV2qTBlppAT/g027YNRO7rnWbfjSCRXotMYP9iHyi52ZRShEEIUzWI/b8PDw2nfvj0uLi7Y29vTt29ftm7dWuC2n3/+OW3atMHPzy/P8nXr1mFnZ0dgYCCQm0D2799PUFAQzz33HNHR0QVVV+6ZEq9jjDqLYsg7RbDBaGLH4UiaeLvSuqHc9xBClG8WuwKJi4vDw+POl6CnpycnTpzIt11qaipr1qwhLCwsz3Kj0ciSJUtYsmSJeZmTkxMBAQH06tWL77//nokTJ7J69epix+Tu7ljgcg8Pp2LXcV94dMTk8whqa5s8i/ccjeRWuo6XglsXO6ZSj/0+kbhLl8Rduh6WuC2WQP49lhOQb3gOyL1P0qtXL9zd83bl/Pbbb9SpU4dGjRqZl02fPt387+HDhzN//nzS0tJwcireSScmpmMy5Y3Lw8Op1Ea0NWWlYoq/iqaW799LdOZ1iqLw464LVHW3p6a7XbFiKs3Y7yeJu3RJ3KXrQYpbrVYV+sMbLNiF5eXlRUJCgrkcFxeHp6dnvu127tyJv79/kctNJhNLlizBaDTm2U6jqTjvQur+3EjW9o8wZSTnW3fu+i2uxabRp01N1AUkWiGEKG8slkA6duzIgQMHSEpKIisri+3bt9O1a9c82yiKQkREBK1atcq3/7Fjx/LcE1Gr1ezYsYNt27YBsGHDBlq0aIGdXcW50WzTbhj2A95C7eCab932w5E42lnToVmVMohMCCHunUWvQCZOnMjo0aMJCgqif//++Pr6Mm7cOE6ePAnkPrprbW2NjY1Nvv0jIyPNj+ve9sEHH7By5UoCAgJYt24dM2fOtFT4FqHSaLHyzD+eVUxSJscuJtCzdXW01lZlEJkQQtw7lVLQzYoHVFneA8ne9zVWtXyxrt0637pV287x24lo5v5fRyo5aItd54PU11oRSNylS+IuXeXqHoi4Q9FlYow5j5IWn29depae/Sej6dDM656ShxBClLWKcwe6AlNp7bEfOgsUY751v/51E53BRJ82NcsgMiGEKDlJIBammIyACpVaDaq8za03mNh99AbN67hR3aPwy0QhhCiPpAvLwgxXjpLx/WuYUuPyrTt0JpaUDB1929Yqg8iEEOK/kQRiYSoHF6yqNUblmHcqWkVR2HboOtU9HGhaO/9jvUIIUd5JF5aFaao0RFOlYb7lp68lcyM+g6f8Gxf4hr4QQpR3cgViQcb4Kyg5GQWu234oEmcHLe2byouDQoiKSRKIhSiKiawdH5O1+/N8624mZHDyciKPtq6OtUb+BEKIikm6sCxGhV3vF0Ex5Vuz43Ak1ho13VtVL4O4hBDi/pAEYiEqlQorj9r5lqdm6Ag/FUNnnyo42cuLg0KIikv6TyxAyU4n5+AaTOlJ+db9+tdNDEYTveXFQSFEBScJxAKMMRfQndiKost7A11vMLL7zxu0qOdOVXeHMopOCCHuD+nCsgBN7VY4jFqE2jbvRFcHImJJy9TTR14cFEI8AOQK5D67Pbjxv5OH3mBiU/hVvL2caFzLpQwiE0KI+0sSyH2WE/4tWXu+zLd89583SEjJZkiPevLioBDigSBdWPeZSmsH6ryTQmVk69kUfpXmdd1oVtutjCITQoj7SxLIfWbT5vF8y34Jv0ZmtoGh3euXQURCCGEZ0oV1HxU04m7CrSx2Ho2ko08VanrKkO1CiAeHJJD7xHQrmozVb6A/91ue5et/u4xKpWJQl/xzoQshREVm0S6ssLAwlixZgl6vJyQkhJEjR5rXnTlzhtDQUHM5KSmJSpUqsWnTJjZs2MC8efNwd3cHoHv37kycOJGoqChef/11EhMTqVOnDvPmzcPBoXy8T6Gyc8amw3CsavqYl12NSeWPiFgCOnjj5mxbhtEJIcT9Z7EEEhsby4IFC1i/fj1arZbg4GDatWtH/fq59wGaNGnCxo0bAcjKymLo0KFMmzYNgJMnTxIaGkr//v3z1Pnuu+8yYsQIAgIC+OSTT/j00095/fXXLXUK90Rl44DWp6+5rCgKa3ZfxNHOmsfaeZdhZELkpSgKycnx6HTZgFLW4RQqLk6NyZR/LLnyrmLGrSIz0xE7O9d7ekrUYgkkPDyc9u3b4+LiAkDfvn3ZunUr48ePz7ft559/Tps2bfDz8wNyE8i1a9dYunQpDRs2ZMqUKdjb23P48GE++eQTAAYPHsyTTz5ZLhKI7uR21K7V0dRoZl528nIiZ6/fYkSvBtjbyrMKovxIT09BpVLh5VUDlar89mJrNGoMhor2RVwx41YUE6mpSRiNKTg5uRR7P4t9euLi4vDw8DCXPT09iY2Nzbddamoqa9asyZNYPDw8ePHFF9m4cSNVq1Zl+vTpJCcn4+joiEajMW9TUH2lTTEa0J/ejeHyQfMyk0lh7a+X8HS1kxF3RbmTlZWOk5NLuU4eonSpVGoqVXIlKyv9nvaz2E/j229k/1NBl0ZhYWH06tXLfL8DMF9lADzzzDP06tWLN954o1j13Y27e8FPQXl4OBW4vLiU5xaiGHSobXPvx2w/eI2bCRmEjm5D1SqV/lPdRfmvsZcVibt0/TPuuDgFGxtthXihVVNB58upiHErigpQ7ukzbrEE4uXlxZEjR8zluLg4PD098223c+dOnn32WXM5LS2NdevWERISAuQmIo1Gg5ubG+np6RiNRqysrIiPjy+wvrtJTEzHZMqb2Dw8nIiPT7unem4zpSehsndBpVbfDp4cnZFVm09Tr5ozDao6lrju4vgvsZclibt0/Ttuk8mE0ahQnu9/QMXsCoKKHbfJZMrzWVGrVYX+8AYLdmF17NiRAwcOkJSURFZWFtu3b6dr1655tlEUhYiICFq1amVeZm9vz5dffsnx48cB+Oabb+jduzfW1tb4+fmxefNmADZs2JCvvtKkmExkbV1A1vaP8izffvg6t9J1PNGzfoX4hSdEWZo//wNCQkbw5JND6d69PSEhIwgJGcEvv/xc7DpCQkbcdf3vv+/lyy8/+6+higJY9Apk4sSJjB49Gr1ez5AhQ/D19WXcuHFMmDABHx8fkpKSsLa2xsbGxryflZUVCxcuZNq0aWRnZ1O7dm3mzJkDwDvvvENoaChLliyhatWqfPjhh5YKv2gqFdrWA1Bp7kwKlZqhY/PB67Ru6EGDGi5lF5sQFcSrr04CIDo6ihdffJavv/7unusoap/OnbvRuXO3EsUn7s6ijwcFBgYSGBiYZ9kXX3xh/re7uzv79+/Pt5+fnx8//fRTvuXVq1dn1apV9z/QElCpVFjXbZNn2cb9V9DrTTzeTV4aFOK/CgoKoGnT5ly4cI5PP/2SNWu+5+jRw6SmpuLi4sKsWXNwd69M585+/P77EZYt+5yEhHgiI68TGxtD//4DGTPmaTZvDuOvv44yefI0hgwJpG9ffw4dOkBWVjZvv/0ujRs34fLli8ya9S5Go5EWLVryxx/h/PDDhjzxXL58kQUL5pKVlUVychLBwU8ydGgwqakpzJ49g+vXr2JtreXll1+hZUs/tm/fysqVywAVTZo0ZdKkt1mxYhkATz+d220/ZEggixd/zl9/HWXLlk2kpNyiU6eu9O7dt1jHevHFiURH3+TIkcNMmzYLgK++WopWq+XJJ0Ms/jeS50tLQHd8M9g4oG1851dNdGIGe/+KolurajJZlKhQ9p+M5vcT0Rapu7NvVTr5VC3x/u3bd2T69NncuBHJ9etX+eyzr1Cr1cyYMZXt27cyfPiTeba/ePECn376JenpaTzxRBCDBz+Rr85KlSrxxRcr+fHH1axa9RWzZs1l5sxpjBv3HB06dOaHH77FaDTm2y8sbCNjxjyNn19bbt68QUjICIYODeaLLz6jRo2azJ49j0uXLjJ37ixmzPiAxYs/ZNmyVXh6ejFjxhTCw3+/67nGx8fxzTdr0Wg0LFo0v1jHmjNnFgsWfMLnn39KZmYmdnZ2bN++lY8//rzEbX4vKt6jAmVMUUwYbkRgjD6XZ/n6vZextlYzoFOdMopMiAdP06bNAahRoybjx08kLGwDixcvICLiJFlZmfm2b93aD2tra1xd3XB2diYjI/9jqe3adQSgbt36pKamkpqaQkxMNB06dAYgIGBggbGMH/8yOp2OVauWs3Tpp+bjHzt2lL59/QGoV68+X365glOnTuDj0wJPTy8ApkyZQdeu3e96rg0bNja/plDcY33++XLs7e3p0KETe/fu5sSJY1SvXoPKlT0KPc79JFcg90ilUmPn/xoYdeZl0YkZHD0fT2DH2lRy0N5lbyHKn04+/+0qwZJu3x89e/YM06ZNJjh4BD16PIqVlbrAVwW02jv//1QqVZHbKIqCWm1V4Hb/NnVqKE5OznTq1IVHH+3Drl3bAcxf+rddvXol37Lk5OQCYzIYDPnO9V6Ode3aVWrWrEVAwABWrFhGtWrV8ffPO4KHJckVyD0wpcSi6LJQqVSoNHf+2NsORWKtUfPoIzXKMDohHlzHjh2lVatHCAoaQu3adTl06OB9Gy7E0dGRGjVqcOBA7v3YHTu2FvgE5eHDh3jmmefo0qU7x479CfD3PZPW5i/4a9eu8vLL42ncuCmnT58iMTEBgMWLP+T33/dSqZILV65cBsizvqTHevXVF1GpVLRo0Yq4uDj+/PMIXbp0vy/tUhxyBVJMiqKQtftzUEzYD3rH/AFLydARfiqGzj5VcJarDyEs4tFH+/DWW68zZkwwVlYa6tWrT3R01H2rf/Lkd5k9ezpffPEp9eo1yHM1cNvYseN4/vlncHJypGZNb6pWrUZ0dBRPP/0sH3wwkzFjhmNlZcW0aTPw8PDkpZde5ZVXXsRkMtK8uS/+/oGkp6exd+9unnxyKI0aNaZBg0YFxlPcY02ZMt38XdStWw9SUlLyXGFZmkopzrXbA+K/vkhojLuMkpOB5h8j7v607zKbwq8yc1y7Ur95/qC82FZRPChxx8Rco0qV8j/AZ2m+kLd8+RcEBg6icuXK7N27m+3btzBr1twS1VXaLxIqioJer2fixBeYMOFVGjVqXKJ6NBo1N25cyfPZKOpFQrkCuQdWnnkfz83RG/n1r5u0qF9ZnrwSogLz8qrCxIn/h0ajwcnJmdDQKWUdUrElJiby5JNDGTAgqMTJo6QkgRRDzrFNoMtG22ZwngHowk9Gk56lp1+7WmUYnRDiv/L3D8TfP7DoDcuhypUrs3Xrr2VybEkgxaCkJaDkZOZJHiaTwrbDkdSp6kyDGpYdMFEIIcojSSDFYNslBMWU98Wivy4kEJecxfNB9WTMKyHEQ0ke4y0mldoqT3nboetUrmRL64aVyygiIYQoW5JASuDizRQu3kyhd5uaWKmlCYUQDyf59iuBbYeuY2+joYtv+Xx7VwghSoMkkHsUl5zJn+fi6dG6OrZauYUkxH/xf//3DDt2bM2zLCsrC3//R7l1K7nAfWbNmsbmzWEkJMTz2msTCtymc2e/ux43Kuoms2dPB+Ds2dO8//6MEkQvJIHco+2HI1GrVTJsiRD3gb9/IDt2bMuzbO/e3bRu/QguLq533bdyZQ/mzfvortsUJiYmmps3bwDQuHHTCvXeR3kiP6HvQXqWnt9PRNOhWRVcHPMPdSCEuDc9e/bmk08WkZqagrNz7uPw27Zt5oknRvDnn0dZsuRjcnKySUtL4/nnJ9CzZy/zvrcnofrxxzCio6OYPn0KWVlZNGvW3LxNfHwcs2fPID09jcTEBHr16svzz7/IokXziIq6yfz5H9Cjx6N89dVSPv54KdevX2POnFmkpaVia2vHyy+/RpMmzZg1axoODo6cO3eG+Pg4nnpqHAEBA/Kcy+1jZWSkkZBw51g5OTl8+OEHnDhxDI1GQ0jIMzz6aB8OHz7Ixx8vRFFMVKlSlXfemcnevb+a5y4BGD/+f4wd+z8Aliz5CKPRRN269Xj22RcKPK+CjlWpkgtffvkZn332FQBbtmwiIuIkr7325n/++8kVyD349a+b6Awm+rStWdahCHFfZYbNRn/uNwAUkyG3fCE8t2zIyS1fOphb1mXmlq8cAcCUnUZm2GwM1/7KLWfeyi1HnijyuPb29nTp0o3du3cCkJAQz/Xr12jXrgNr164mNHQKX331LaGhU/j66y8KrWfBgjn4+wfy9dff4ePTwrx8x45t9O7dl6VLv2bFitX89NOP3Lp1i5deeo1GjZqYZ0S8bcaMKQwdGsyKFat58cVXePvtSeh0uSNvx8XF8umnX/LBBwv45JNF+WK4faxly1bmOda6dT+QlZXFt9/+yMKFn7J8+ZfodDqmT5/C229PY+XKH6hXrwFbtmy6a1tFRl7no48+4+233y30vAo6lq9vSxITE81XXFu2bOKxx+7PS5OSQIpJbzCy6+gNmtd1o4ZH4WPDCCHuTUDAAHbuzO3G2r59C337+qNWq5k2bSaXL1/k66+/ZPXqb8jKyiq0jr/+Osqjj/YGoE+fx8zDno8YMQovryp8990qFi2ah8GgJzu74HoyMzO5ceMG3br1BKB5cx+cnZ25fv0aAG3btkOlUlG3bj1SU1Py7X/7WN9+uzLPsY4d+5M+ffqhVqtxd6/MN9+s4fLli3h4eJgHU3z22RcYMiT4ru1Us6Y3jo6Odz2vgo6l1Wp57LEAtm3bTExMDElJSXmu0v4L6cIqpgMRsaRm6OjXVoYtEQ8e+8A73RkqtSZvWWOTt6y1z1NW2zrlLdu75CkXpUWLViQmJhAbG8O2bVt4773cQQyfe+5pWrV6hFatHuGRR9rw7rtv36UWlXmgVJVKhfrvx+sXL15AVNRNevfuR9eu3Tly5FChc38oiinfOkXBPDuhVmtjrr8gt4/Vr99jdOrUzXwsK6u8X7M3bkTmW5aenk5mZka++UKMxoLnCynsvAo6lpdXFfz9A3n11RfRarX06+dfYPwlIVcgxWBSFLYduk4tT0eaeN/9xp4Q4t499lh/VqxYhrOzM9Wr1yA1NYXr16/z9NO508weOvTHXef/8PNry7Ztm4Hcm/C3u52OHDnIiBGj6NmzF3FxscTHx2EymbCy0uSbttbBwZHq1Wuwd+9uAE6dOklSUiJ169Yr1jncPtajj/bOc6yWLVuxe/dOFEUhOTmJ8eP/R/Xq1bl165Z5bpBvv13Bhg3rqFTJhWvXrqAoClFRN7l48eJdj/Xv8yroWHq9jipVquLh4cmGDevo1y+gWOdTHBa9AgkLC2PJkiXo9XpCQkIYOXKked2ZM2cIDQ01l5OSkqhUqRKbNm3i6NGjvPfeexgMBlxcXHjvvfeoXr06hw8fZvz48VSpUgWApk2bMnv2bEueAgAnLyUSnZjJuMCmMmyJEBbQr18AQ4cO4M03pwLg7FyJAQOCGDXqCRwcHGjWzJfs7OxCu7FeeeUNZsyYys8/r6dx46bY2+eOjv3kkyHMmDEVR0cn3NzcaNy4KVFRN2nYsBHp6WnMmDElzxS2U6fOYO7c91i27HOsrbXMmjUHa2vrYp3D7WM5OTnh6nrnWIMGDWXhwrmEhAwHYOLE17G3d2DKlOnMnPkOBoOeatVqMGXKdDQaDb/8spHhwx/H29sbX9+Wdz3Wv8+rsGMB9OrVhz17dt/X6W4tNh9IbGwsw4cPZ/369Wi1WoKDg/nwww+pX79+vm2zsrIYOnQo06ZNw8/Pj549e/Lpp5/SuHFjfvzxR3bt2sWSJUv46quv0Ov1PPvssyWKqaTzgcz57k9ik7P44LkOaKzKz0XbgzI/RUXxoMQt84FYVnmM22AwMGPGVHr27GW+x/NvJZkPxGLfhuHh4bRv3x4XFxfs7e3p27cvW7duLXDbzz//nDZt2uDn54dOp+Oll16icePcce0bNWpEdHQ0ACdPnmT//v0EBQXx3HPPmZdbkqIoXIlOw7+9d7lKHkIIURyKohAU9Bhqtfq+T3drsS6suLg4PDzuXCp5enpy4kT+x/pSU1NZs2YNYWFhQO6E9wMH5l5SmkwmPv74Y3r1yn3228nJiYCAAHr16sX333/PxIkTWb16dbFjKiyTeng43XW/ldP6Ym9bvMvY0lZU7OWVxF26/hl3XJwajaZi/BiqKHH+W3mLe+vWXcXaTq1W39Nn3GIJpKCesYLuH4SFhdGrVy/c3d3zLNfpdISGhmIwGMxdVtOnTzevHz58OPPnzyctLQ0np+Kd8H+Z0jYjLbtYxyhND0qXSkXxoMRtMpnKXRdLQcpjV1BxVOS4TSZTns9KmXVheXl5kZCQYC7HxcXh6emZb7udO3fi75/3sbKMjAyeeeYZDAYDS5YswdraGpPJxJIlS/I9OXH7eW8hRPFZ6NanqMBK8pmwWALp2LEjBw4cICkpiaysLLZv307Xrl3zbKMoChEREbRq1SrP8tdffx1vb28WLVqEVqvNDVStZseOHWzblvvC0YYNG2jRogV2dnaWOgUhHkgajZaMjFRJIsJMURTS0lLQaLT3tJ/Ffr57eXkxceJERo8ejV6vZ8iQIfj6+jJu3DgmTJiAj48PSUlJWFtb53lB5vTp0+zatYv69esTFBQE5N4/+eKLL/jggw+YMmUKn3zyCW5ubsyZM8dS4QvxwHJ19SA5OZ709FtlHcpdqdXqu777UV5V1LgdHOxxdb23R3wt9hhvefRf7oGURxU1dom7dEncpetBirvM7oEIIYR4sEkCEUIIUSIP1SNManXBw5AUtrwiqKixS9ylS+IuXQ9K3EWdx0N1D0QIIcT9I11YQgghSkQSiBBCiBKRBCKEEKJEJIEIIYQoEUkgQgghSkQSiBBCiBKRBCKEEKJEJIEIIYQoEUkgQgghSuShTiBhYWH4+/vTu3dvvv3227IOp9hGjx5NQEAAAwcOZODAgRw/frysQ7qr9PR0+vfvz40bNwAIDw8nMDCQPn36sGDBgjKOrnD/jvvNN9+kT58+5nbfsWNHGUeY38cff0xAQAABAQHm6Q4qQnsXFHdFaO9Fixbh7+9PQEAAy5cvBypGe0PBsd9zmysPqZiYGKVHjx5KcnKykpGRoQQGBioXLlwo67CKZDKZlE6dOil6vb6sQymWY8eOKf3791eaNWumREZGKllZWUq3bt2U69evK3q9Xhk7dqyyZ8+esg4zn3/HrSiK0r9/fyU2NraMIyvc/v37lWHDhik5OTmKTqdTRo8erYSFhZX79i4o7u3bt5f79j548KASHBys6PV6JSsrS+nRo4dy5syZct/eilJw7JcuXbrnNn9or0DCw8Np3749Li4u2Nvb07dvX7Zu3VrWYRXp8uXLqFQqxo0bx4ABA/jmm2/KOqS7WrNmDe+88455OuMTJ07g7e1NzZo10Wg0BAYGlst2/3fcmZmZREVFMWXKFAIDA/noo4/K3aRBHh4ehIaGotVqsba2pl69ely9erXct3dBcUdFRZX79m7bti0rV65Eo9GQmJiI0WgkNTW13Lc3FBy7jY3NPbf5Q5tA4uLi8PC4M/uWp6cnsbGxZRhR8aSmptKhQwc++eQTvv76a1avXs3+/fvLOqxCzZo1Cz8/P3O5orT7v+NOTEykffv2vPfee6xZs4YjR47w448/lmGE+TVo0ICWLVsCcPXqVTZv3oxKpSr37V1Q3F26dCn37Q1gbW3NRx99REBAAB06dKgwn2/IH7vRaLznNn9oE4hSwCDEKlX5H4K5VatWzJkzB3t7e9zc3BgyZAh79+4t67CKraK2e82aNfnkk09wd3fHzs6OUaNGldt2v3DhAmPHjmXSpEnUqlUr3/ry2t7/jLtu3boVpr0nTJjAgQMHiI6O5urVq/nWl9f2hryxHzhw4J7b/KFNIF5eXiQkJJjLcXFx5u6K8uzIkSMcOHDAXFYUBY2m4kzrUlHb/dy5c2zbts1cLq/tfvToUUJCQnj11VcZNGhQhWnvf8ddEdr70qVLnDlzBgA7Ozv69OnDwYMHK0R7FxT75s2b77nNH9oE0rFjRw4cOEBSUhJZWVls376drl27lnVYRUpLS2POnDnk5OSQnp7OTz/9RO/evcs6rGJr0aIFV65c4dq1axiNRjZt2lQh2l1RFN577z1SUlLQ6/X88MMP5a7do6OjeeGFF5g3bx4BAQFAxWjvguKuCO1948YN3n77bXQ6HTqdjl27dhEcHFzu2xsKjr1Nmzb33OblK6WXIi8vLyZOnMjo0aPR6/UMGTIEX1/fsg6rSD169OD48eMEBQVhMpkYMWIErVq1Kuuwis3Gxob333+fF198kZycHLp160a/fv3KOqwiNW7cmP/9738MHz4cg8FAnz596N+/f1mHlceyZcvIycnh/fffNy8LDg4u9+1dWNzlvb27detm/r9oZWVFnz59CAgIwM3NrVy3NxQc+/jx43F1db2nNpcZCYUQQpTIQ9uFJYQQ4r+RBCKEEKJEJIEIIYQoEUkgQgghSkQSiBBCiBJ5aB/jFeJ+atSoEQ0bNkStzvub7JNPPqFGjRr3/VgHDhzAzc3tvtYrxL2SBCLEfbJixQr5UhcPFUkgQljYwYMHmTNnDl5eXkRGRmJra8v7779PvXr1SEtL49133+Xs2bOoVCq6dOnCK6+8gkaj4fjx48ycOZOsrCysra1544036NChAwCLFy/m+PHj3Lp1i6effpqRI0eW8VmKh5EkECHukzFjxuTpwqpRowaffPIJAKdPn+bNN9/Ez8+P77//ntdff53169czc+ZMXFxcCAsLQ6/X8/zzz/PVV1/x1FNP8cILLzBz5ky6d+/OqVOnePPNN9m4cSOQO7jjO++8w+nTpxk2bBhPPPEE1tbWZXLe4uElCUSI++RuXViNGzc2Dw//+OOPM336dJKTk9m3bx/ff/89KpUKrVZLcHAwK1asoFOnTqjVarp37w5A8+bNCQsLM9d3e4iJJk2aoNPpSE9Px9XV1bInKMS/yFNYQpQCKyurPGVFUbCysso3YY/JZMJgMGBlZZVvGPDz589jMBgAzKOk3t5GRiQSZUESiBCl4OzZs5w9exaAH374gdatW+Ps7Eznzp359ttvURQFnU7HmjVr6NixI3Xr1kWlUpknC4uIiGDMmDHlblY+8XCTLiwh7pN/3wMBeOWVV7C1taVy5cosXLiQmzdv4ubmxpw5cwB4++23mTlzJoGBgej1erp06cJzzz2HVqtl8eLFvPfee8yZMwdra2sWL16MVqsti1MTokAyGq8QFnbw4EFmzJjBpk2byjoUIe4r6cISQghRInIFIoQQokTkCkQIIUSJSAIRQghRIpJAhBBClIgkECGEECUiCUQIIUSJSAIRQghRIv8Po2qjpH6xX5sAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "sns.set()\n",
    "\n",
    "acc = hist.history['accuracy']\n",
    "val = hist.history['val_accuracy']\n",
    "epochs = range(1, len(acc) + 1)\n",
    "\n",
    "plt.plot(epochs, acc, '-', label='Training accuracy')\n",
    "plt.plot(epochs, val, ':', label='Validation accuracy')\n",
    "plt.title('Training and Validation Accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(loc='lower right')\n",
    "plt.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use the model to translate text\n",
    "\n",
    "Can the model really translate English to French? Use a subset of the phrases reserved to validate the model to make predictions and see for yourself. Note the use of the French `Tokenizer`’s `sequences_to_texts` method to convert sequences representing French phrases predicted by the model into words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "its fall now => cest maintenant maintenant\n",
      "im losing => je suis en train\n",
      "it was quite funny => cetait fut amusant amusant\n",
      "thats not unusual => ce nest pas inhabituel\n",
      "i think ill do that => je pense que je le\n",
      "tom looks different => tom a lair different\n",
      "its worth a try => ca vaut le coup\n",
      "fortune smiled on him => la la lui a souri\n",
      "lets hit the road => taillons la\n",
      "i love winning => jadore gagner\n"
     ]
    }
   ],
   "source": [
    "def translate_text(text, model, en_tokenizer, fr_tokenizer, en_max_len):\n",
    "    sequence = en_tokenizer.texts_to_sequences([text])\n",
    "    padded_sequence = pad_sequences(sequence, maxlen=en_max_len, padding='post')\n",
    "    prediction = model.predict(padded_sequence, verbose=0)[0]\n",
    "    indexes = [np.argmax(idx) for idx in prediction]\n",
    "    return fr_tokenizer.sequences_to_texts([indexes])[0]\n",
    "\n",
    "texts = en[40000:40010].values\n",
    "\n",
    "for text in texts:\n",
    "    translated = translate_text(text, model, en_tokenizer, fr_tokenizer, en_max_len)\n",
    "    print(f'{text} => {translated}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, see how the model translates \"Hello world\" into French:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'salut le monde'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "translate_text('Hello world', model, en_tokenizer, fr_tokenizer, en_max_len)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model isn't perfect, but it's not bad, either. The vocabularies that you trained with are relatively small, but if you expanded the vocabularies and deepened the model – for example, if you included two `LSTM` layers for encoding and two for decoding – you'd have the makings of a decent NMT model."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}