{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Lesson6-AWSML-Production-ML-App.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [
        "c_Id55m6Jsbu",
        "bq4VmHjPpMOR"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/noahgift/aws-ml-guide/blob/master/Lesson6_AWSML_Production_ML_App.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "j8TnIYSn1rvg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Lesson 6:  Create a Production Machine Learning Application\n",
        "\n",
        "[Watch Lesson 6:  Create a Production Machine Learning Application Video](https://learning.oreilly.com/videos/aws-certified-machine/9780135556597/9780135556597-ACML_01_06_00)"
      ]
    },
    {
      "metadata": {
        "id": "c_Id55m6Jsbu",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Pragmatic AI Labs\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "e5p96AqpSDZa",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "![alt text](https://paiml.com/images/logo_with_slogan_white_background.png)\n",
        "\n",
        "This notebook was produced by [Pragmatic AI Labs](https://paiml.com/).  You can continue learning about these topics by:\n",
        "\n",
        "*   Buying a copy of [Pragmatic AI: An Introduction to Cloud-Based Machine Learning](http://www.informit.com/store/pragmatic-ai-an-introduction-to-cloud-based-machine-9780134863863) from Informit.\n",
        "*   Buying a copy of  [Pragmatic AI: An Introduction to Cloud-Based Machine Learning](https://www.amazon.com/Pragmatic-AI-Introduction-Cloud-Based-Learning/dp/0134863860) from Amazon\n",
        "*   Reading an online copy of [Pragmatic AI:Pragmatic AI: An Introduction to Cloud-Based Machine Learning](https://www.safaribooksonline.com/library/view/pragmatic-ai-an/9780134863924/)\n",
        "*  Watching video [Essential Machine Learning and AI with Python and Jupyter Notebook-Video-SafariOnline](https://www.safaribooksonline.com/videos/essential-machine-learning/9780135261118) on Safari Books Online.\n",
        "* Watching video [AWS Certified Machine Learning-Speciality](https://learning.oreilly.com/videos/aws-certified-machine/9780135556597)\n",
        "* Purchasing video [Essential Machine Learning and AI with Python and Jupyter Notebook- Purchase Video](http://www.informit.com/store/essential-machine-learning-and-ai-with-python-and-jupyter-9780135261095)\n",
        "*   Viewing more content at [noahgift.com](https://noahgift.com/)\n"
      ]
    },
    {
      "metadata": {
        "id": "bq4VmHjPpMOR",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Load AWS API Keys"
      ]
    },
    {
      "metadata": {
        "id": "aWrzIk7WpRoh",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Put keys in local or remote GDrive:  \n",
        "\n",
        "`cp ~/.aws/credentials /Users/myname/Google\\ Drive/awsml/`"
      ]
    },
    {
      "metadata": {
        "id": "hPWO_zyRopXN",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Mount GDrive\n"
      ]
    },
    {
      "metadata": {
        "id": "XI73HZNLobp4",
        "colab_type": "code",
        "outputId": "66f597a9-6c94-4ffd-b323-21869a4eb988",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive', force_remount=True)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/gdrive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "UNyzZwgmoxwm",
        "colab_type": "code",
        "outputId": "80a98691-3926-4666-82c7-f27f606fbfaf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "import os;os.listdir(\"/content/gdrive/My Drive/awsml\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['kaggle.json', 'credentials', 'config']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "metadata": {
        "id": "fYu0ekUlqPk6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Install Boto"
      ]
    },
    {
      "metadata": {
        "id": "dJDDrUkWrYRY",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!pip -q install boto3\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "FpJhrpSQsK5E",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Create API Config"
      ]
    },
    {
      "metadata": {
        "id": "QxRwGOZtsN0-",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!mkdir -p ~/.aws &&\\\n",
        "  cp /content/gdrive/My\\ Drive/awsml/credentials ~/.aws/credentials "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Kj977UW3rph_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Test Comprehend API Call"
      ]
    },
    {
      "metadata": {
        "id": "P-A8Cia-raT0",
        "colab_type": "code",
        "outputId": "4290a9a8-667f-4b06-edf3-aa4c2ae121f2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 238
        }
      },
      "cell_type": "code",
      "source": [
        "import boto3\n",
        "comprehend = boto3.client(service_name='comprehend', region_name=\"us-east-1\")\n",
        "text = \"There is smoke in San Francisco\"\n",
        "comprehend.detect_sentiment(Text=text, LanguageCode='en')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'ResponseMetadata': {'HTTPHeaders': {'connection': 'keep-alive',\n",
              "   'content-length': '160',\n",
              "   'content-type': 'application/x-amz-json-1.1',\n",
              "   'date': 'Thu, 22 Nov 2018 00:21:54 GMT',\n",
              "   'x-amzn-requestid': '9d69a0a9-edec-11e8-8560-532dc7aa62ea'},\n",
              "  'HTTPStatusCode': 200,\n",
              "  'RequestId': '9d69a0a9-edec-11e8-8560-532dc7aa62ea',\n",
              "  'RetryAttempts': 0},\n",
              " 'Sentiment': 'NEUTRAL',\n",
              " 'SentimentScore': {'Mixed': 0.008628507144749165,\n",
              "  'Negative': 0.1037612184882164,\n",
              "  'Neutral': 0.8582549691200256,\n",
              "  'Positive': 0.0293553676456213}}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 50
        }
      ]
    },
    {
      "metadata": {
        "id": "WcZCtRRN6J2l",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 6.1 Create Machine Learning Data Pipeline"
      ]
    },
    {
      "metadata": {
        "id": "j8aR-lyQ6RJC",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Data Pipeline Patterns"
      ]
    },
    {
      "metadata": {
        "id": "BPunH1cUi6LG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "![data_engineers](https://user-images.githubusercontent.com/58792/49955060-c062bf00-feb6-11e8-8204-79d629ab2862.png)"
      ]
    },
    {
      "metadata": {
        "id": "UOJ8_Si6gV9S",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### [Anti-Pattern] Accidental Data Pipeline\n"
      ]
    },
    {
      "metadata": {
        "id": "jC81G-qkhLyK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "* Production SQL Server is center"
      ]
    },
    {
      "metadata": {
        "id": "6CAXI7TA05HC",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### [Pattern] Batch Data Lake"
      ]
    },
    {
      "metadata": {
        "id": "bj4O5iMu1AaV",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "* Batch ML Pipeline to and from Data Lake"
      ]
    },
    {
      "metadata": {
        "id": "ESc_0Wr51kOs",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "#### [Pattern]  Batch Infinite Resources on Production"
      ]
    },
    {
      "metadata": {
        "id": "tkTG6JTY1spd",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "* Environment uses "
      ]
    },
    {
      "metadata": {
        "id": "YW22mRwZ6RSl",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 6.2 EDA with Sagemaker"
      ]
    },
    {
      "metadata": {
        "id": "6mp2NtUJ6RVg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### [Demo] County Census Clustering"
      ]
    },
    {
      "metadata": {
        "id": "1nyJrOrKNJmH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "* [Original AWS Article on Census Clustering](https://aws.amazon.com/blogs/machine-learning/analyze-us-census-data-for-population-segmentation-using-amazon-sagemaker/)"
      ]
    },
    {
      "metadata": {
        "id": "fXpNEXuP6RYH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 6.3 Create Machine Model using AWS Sagemaker"
      ]
    },
    {
      "metadata": {
        "id": "srx9xZir6Rag",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### [Demo] Sagemaker Pipeline"
      ]
    },
    {
      "metadata": {
        "id": "D6zp_J2K-iVn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## 6.4 Deploy Machine Learning Model using AWS Sagemaker"
      ]
    },
    {
      "metadata": {
        "id": "kvp2uP_C-qAi",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### [Demo] Sagemaker Pipeline"
      ]
    },
    {
      "metadata": {
        "id": "9QiFs5qdew49",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        ""
      ]
    }
  ]
}