![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
]
},
{
"metadata": {
"id": "iwV-bBLUtTQu",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# Lesson 3 Exploratory Data Analysis\n",
"\n",
"[Watch Lesson 3: Exploratory Data Analysis on AWS Video](https://learning.oreilly.com/videos/aws-certified-machine/9780135556597/9780135556597-ACML_01_03_00)"
]
},
{
"metadata": {
"id": "c_Id55m6Jsbu",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"## Pragmatic AI Labs\n",
"\n"
]
},
{
"metadata": {
"id": "e5p96AqpSDZa",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"\n",
"\n",
"This notebook was produced by [Pragmatic AI Labs](https://paiml.com/). You can continue learning about these topics by:\n",
"\n",
"* Buying a copy of [Pragmatic AI: An Introduction to Cloud-Based Machine Learning](http://www.informit.com/store/pragmatic-ai-an-introduction-to-cloud-based-machine-9780134863863) from Informit.\n",
"* Buying a copy of [Pragmatic AI: An Introduction to Cloud-Based Machine Learning](https://www.amazon.com/Pragmatic-AI-Introduction-Cloud-Based-Learning/dp/0134863860) from Amazon\n",
"* Reading an online copy of [Pragmatic AI:Pragmatic AI: An Introduction to Cloud-Based Machine Learning](https://www.safaribooksonline.com/library/view/pragmatic-ai-an/9780134863924/)\n",
"* Watching video [Essential Machine Learning and AI with Python and Jupyter Notebook-Video-SafariOnline](https://www.safaribooksonline.com/videos/essential-machine-learning/9780135261118) on Safari Books Online.\n",
"* Watching video [AWS Certified Machine Learning-Speciality](https://learning.oreilly.com/videos/aws-certified-machine/9780135556597)\n",
"* Purchasing video [Essential Machine Learning and AI with Python and Jupyter Notebook- Purchase Video](http://www.informit.com/store/essential-machine-learning-and-ai-with-python-and-jupyter-9780135261095)\n",
"* Viewing more content at [noahgift.com](https://noahgift.com/)\n"
]
},
{
"metadata": {
"id": "bq4VmHjPpMOR",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"## Load AWS API Keys"
]
},
{
"metadata": {
"id": "aWrzIk7WpRoh",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"Put keys in local or remote GDrive: \n",
"\n",
"`cp ~/.aws/credentials /Users/myname/Google\\ Drive/awsml/`"
]
},
{
"metadata": {
"id": "hPWO_zyRopXN",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Mount GDrive\n"
]
},
{
"metadata": {
"id": "XI73HZNLobp4",
"colab_type": "code",
"outputId": "66f597a9-6c94-4ffd-b323-21869a4eb988",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"cell_type": "code",
"source": [
"from google.colab import drive\n",
"drive.mount('/content/gdrive', force_remount=True)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Mounted at /content/gdrive\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "UNyzZwgmoxwm",
"colab_type": "code",
"outputId": "80a98691-3926-4666-82c7-f27f606fbfaf",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"cell_type": "code",
"source": [
"import os;os.listdir(\"/content/gdrive/My Drive/awsml\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['kaggle.json', 'credentials', 'config']"
]
},
"metadata": {
"tags": []
},
"execution_count": 27
}
]
},
{
"metadata": {
"id": "fYu0ekUlqPk6",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Install Boto"
]
},
{
"metadata": {
"id": "dJDDrUkWrYRY",
"colab_type": "code",
"colab": {}
},
"cell_type": "code",
"source": [
"!pip -q install boto3\n"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "FpJhrpSQsK5E",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Create API Config"
]
},
{
"metadata": {
"id": "QxRwGOZtsN0-",
"colab_type": "code",
"colab": {}
},
"cell_type": "code",
"source": [
"!mkdir -p ~/.aws &&\\\n",
" cp /content/gdrive/My\\ Drive/awsml/credentials ~/.aws/credentials "
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "Kj977UW3rph_",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Test Comprehend API Call"
]
},
{
"metadata": {
"id": "P-A8Cia-raT0",
"colab_type": "code",
"outputId": "4290a9a8-667f-4b06-edf3-aa4c2ae121f2",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 238
}
},
"cell_type": "code",
"source": [
"import boto3\n",
"comprehend = boto3.client(service_name='comprehend', region_name=\"us-east-1\")\n",
"text = \"There is smoke in San Francisco\"\n",
"comprehend.detect_sentiment(Text=text, LanguageCode='en')"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"{'ResponseMetadata': {'HTTPHeaders': {'connection': 'keep-alive',\n",
" 'content-length': '160',\n",
" 'content-type': 'application/x-amz-json-1.1',\n",
" 'date': 'Thu, 22 Nov 2018 00:21:54 GMT',\n",
" 'x-amzn-requestid': '9d69a0a9-edec-11e8-8560-532dc7aa62ea'},\n",
" 'HTTPStatusCode': 200,\n",
" 'RequestId': '9d69a0a9-edec-11e8-8560-532dc7aa62ea',\n",
" 'RetryAttempts': 0},\n",
" 'Sentiment': 'NEUTRAL',\n",
" 'SentimentScore': {'Mixed': 0.008628507144749165,\n",
" 'Negative': 0.1037612184882164,\n",
" 'Neutral': 0.8582549691200256,\n",
" 'Positive': 0.0293553676456213}}"
]
},
"metadata": {
"tags": []
},
"execution_count": 50
}
]
},
{
"metadata": {
"id": "hYtm6PMXsrQd",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"## 3.1 Data Visualization"
]
},
{
"metadata": {
"id": "q5nioy1LuCTQ",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### AWS Quicksite"
]
},
{
"metadata": {
"id": "9TIbWnzcYJ9w",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"\n",
"\n",
"* Business Intelligence Service\n",
"* Creates Automated Visualizations\n",
"\n",
"\n",
"\n",
""
]
},
{
"metadata": {
"id": "g7hVS3d-nh0T",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"#### [Demo] Quicksite"
]
},
{
"metadata": {
"id": "9NbCNan3ZLTF",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Part of EDA Cycle\n",
"\n",
"* used to detect outliers\n",
"* See data Distribution"
]
},
{
"metadata": {
"id": "Hk3QVRT5oFHT",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"[Demo] Plotly"
]
},
{
"metadata": {
"id": "WirfPfS8zQ1G",
"colab_type": "code",
"colab": {}
},
"cell_type": "code",
"source": [
"df = pd.read_csv(\"https://raw.githubusercontent.com/noahgift/real_estate_ml/master/data/Zip_Zhvi_SingleFamilyResidence.csv\")\n"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "PQan_MCrP7eF",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"**Clean Up DataFrame**\n",
"Rename RegionName to ZipCode and Change Zip Code to String\n",
"\n"
]
},
{
"metadata": {
"id": "93nEMwVWP5ic",
"colab_type": "code",
"outputId": "de69504c-1d73-4e12-9ec2-5c7a6ae9ea75",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 270
}
},
"cell_type": "code",
"source": [
"df.rename(columns={\"RegionName\":\"ZipCode\"}, inplace=True)\n",
"df[\"ZipCode\"]=df[\"ZipCode\"].map(lambda x: \"{:.0f}\".format(x))\n",
"df[\"RegionID\"]=df[\"RegionID\"].map(lambda x: \"{:.0f}\".format(x))\n",
"df.head()"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"| \n", " | RegionID | \n", "ZipCode | \n", "City | \n", "State | \n", "Metro | \n", "CountyName | \n", "SizeRank | \n", "1996-04 | \n", "1996-05 | \n", "1996-06 | \n", "... | \n", "2016-12 | \n", "2017-01 | \n", "2017-02 | \n", "2017-03 | \n", "2017-04 | \n", "2017-05 | \n", "2017-06 | \n", "2017-07 | \n", "2017-08 | \n", "2017-09 | \n", "
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "84654 | \n", "60657 | \n", "Chicago | \n", "IL | \n", "Chicago | \n", "Cook | \n", "1.0 | \n", "420800.0 | \n", "423500.0 | \n", "426200.0 | \n", "... | \n", "1097900.0 | \n", "1098300.0 | \n", "1094700.0 | \n", "1088500.0 | \n", "1081200.0 | \n", "1073900.0 | \n", "1064300.0 | \n", "1054300.0 | \n", "1048500.0 | \n", "1044400.0 | \n", "
| 1 | \n", "84616 | \n", "60614 | \n", "Chicago | \n", "IL | \n", "Chicago | \n", "Cook | \n", "2.0 | \n", "542400.0 | \n", "546700.0 | \n", "551700.0 | \n", "... | \n", "1522800.0 | \n", "1525900.0 | \n", "1525000.0 | \n", "1526100.0 | \n", "1528700.0 | \n", "1526700.0 | \n", "1518900.0 | \n", "1515800.0 | \n", "1519900.0 | \n", "1525300.0 | \n", "
| 2 | \n", "93144 | \n", "79936 | \n", "El Paso | \n", "TX | \n", "El Paso | \n", "El Paso | \n", "3.0 | \n", "70900.0 | \n", "71200.0 | \n", "71100.0 | \n", "... | \n", "114200.0 | \n", "114300.0 | \n", "114200.0 | \n", "114000.0 | \n", "113800.0 | \n", "114000.0 | \n", "114000.0 | \n", "113800.0 | \n", "113500.0 | \n", "113300.0 | \n", "
| 3 | \n", "84640 | \n", "60640 | \n", "Chicago | \n", "IL | \n", "Chicago | \n", "Cook | \n", "4.0 | \n", "298200.0 | \n", "297400.0 | \n", "295300.0 | \n", "... | \n", "739400.0 | \n", "743100.0 | \n", "741500.0 | \n", "736300.0 | \n", "729500.0 | \n", "727700.0 | \n", "726000.0 | \n", "718800.0 | \n", "713400.0 | \n", "710900.0 | \n", "
| 4 | \n", "61807 | \n", "10467 | \n", "New York | \n", "NY | \n", "New York | \n", "Bronx | \n", "5.0 | \n", "NaN | \n", "NaN | \n", "NaN | \n", "... | \n", "391600.0 | \n", "388900.0 | \n", "388800.0 | \n", "391100.0 | \n", "394400.0 | \n", "396900.0 | \n", "398600.0 | \n", "400500.0 | \n", "402600.0 | \n", "403700.0 | \n", "
5 rows × 265 columns
\n", "