{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "Lesson7-AWSML-Case-Studies.ipynb",
"version": "0.3.2",
"provenance": [],
"collapsed_sections": [
"bq4VmHjPpMOR",
"WcZCtRRN6J2l",
"YW22mRwZ6RSl",
"D6zp_J2K-iVn",
"o3j8pTFwixkQ",
"MUp-WrPOmJVu"
],
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\")
"
]
},
{
"metadata": {
"id": "j8TnIYSn1rvg",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"# Lesson 7: Case Studies\n",
"\n",
"[Watch Lesson 7: Case Studies Video](https://learning.oreilly.com/videos/aws-certified-machine/9780135556597/9780135556597-ACML_01_07_00)"
]
},
{
"metadata": {
"id": "c_Id55m6Jsbu",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"## Pragmatic AI Labs\n",
"\n"
]
},
{
"metadata": {
"id": "e5p96AqpSDZa",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"\n",
"\n",
"This notebook was produced by [Pragmatic AI Labs](https://paiml.com/). You can continue learning about these topics by:\n",
"\n",
"* Buying a copy of [Pragmatic AI: An Introduction to Cloud-Based Machine Learning](http://www.informit.com/store/pragmatic-ai-an-introduction-to-cloud-based-machine-9780134863863) from Informit.\n",
"* Buying a copy of [Pragmatic AI: An Introduction to Cloud-Based Machine Learning](https://www.amazon.com/Pragmatic-AI-Introduction-Cloud-Based-Learning/dp/0134863860) from Amazon\n",
"* Reading an online copy of [Pragmatic AI:Pragmatic AI: An Introduction to Cloud-Based Machine Learning](https://www.safaribooksonline.com/library/view/pragmatic-ai-an/9780134863924/)\n",
"* Watching video [Essential Machine Learning and AI with Python and Jupyter Notebook-Video-SafariOnline](https://www.safaribooksonline.com/videos/essential-machine-learning/9780135261118) on Safari Books Online.\n",
"* Watching video [AWS Certified Machine Learning-Speciality](https://learning.oreilly.com/videos/aws-certified-machine/9780135556597)\n",
"* Purchasing video [Essential Machine Learning and AI with Python and Jupyter Notebook- Purchase Video](http://www.informit.com/store/essential-machine-learning-and-ai-with-python-and-jupyter-9780135261095)\n",
"* Viewing more content at [noahgift.com](https://noahgift.com/)\n"
]
},
{
"metadata": {
"id": "bq4VmHjPpMOR",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"## Load AWS API Keys"
]
},
{
"metadata": {
"id": "aWrzIk7WpRoh",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"Put keys in local or remote GDrive: \n",
"\n",
"`cp ~/.aws/credentials /Users/myname/Google\\ Drive/awsml/`"
]
},
{
"metadata": {
"id": "hPWO_zyRopXN",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Mount GDrive\n"
]
},
{
"metadata": {
"id": "XI73HZNLobp4",
"colab_type": "code",
"outputId": "fd9be4a2-0f47-4580-9793-49a6f4e44e64",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"cell_type": "code",
"source": [
"from google.colab import drive\n",
"drive.mount('/content/gdrive', force_remount=True)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Mounted at /content/gdrive\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "UNyzZwgmoxwm",
"colab_type": "code",
"outputId": "80a98691-3926-4666-82c7-f27f606fbfaf",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 34
}
},
"cell_type": "code",
"source": [
"import os;os.listdir(\"/content/gdrive/My Drive/awsml\")"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"['kaggle.json', 'credentials', 'config']"
]
},
"metadata": {
"tags": []
},
"execution_count": 27
}
]
},
{
"metadata": {
"id": "fYu0ekUlqPk6",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Install Boto"
]
},
{
"metadata": {
"id": "dJDDrUkWrYRY",
"colab_type": "code",
"colab": {}
},
"cell_type": "code",
"source": [
"!pip -q install boto3\n"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "FpJhrpSQsK5E",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Create API Config"
]
},
{
"metadata": {
"id": "QxRwGOZtsN0-",
"colab_type": "code",
"colab": {}
},
"cell_type": "code",
"source": [
"!mkdir -p ~/.aws &&\\\n",
" cp /content/gdrive/My\\ Drive/awsml/credentials ~/.aws/credentials "
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "Kj977UW3rph_",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Test Comprehend API Call"
]
},
{
"metadata": {
"id": "P-A8Cia-raT0",
"colab_type": "code",
"outputId": "8dc13e19-1f30-4631-cb67-2a4e05a61aec",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 238
}
},
"cell_type": "code",
"source": [
"import boto3\n",
"comprehend = boto3.client(service_name='comprehend', region_name=\"us-east-1\")\n",
"text = \"There is smoke in San Francisco\"\n",
"comprehend.detect_sentiment(Text=text, LanguageCode='en')"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"{'ResponseMetadata': {'HTTPHeaders': {'connection': 'keep-alive',\n",
" 'content-length': '160',\n",
" 'content-type': 'application/x-amz-json-1.1',\n",
" 'date': 'Fri, 14 Dec 2018 16:11:55 GMT',\n",
" 'x-amzn-requestid': 'fa00db36-ffba-11e8-882b-8bc33ca9084d'},\n",
" 'HTTPStatusCode': 200,\n",
" 'RequestId': 'fa00db36-ffba-11e8-882b-8bc33ca9084d',\n",
" 'RetryAttempts': 0},\n",
" 'Sentiment': 'NEUTRAL',\n",
" 'SentimentScore': {'Mixed': 0.008628507144749165,\n",
" 'Negative': 0.1037612184882164,\n",
" 'Neutral': 0.8582549691200256,\n",
" 'Positive': 0.0293553676456213}}"
]
},
"metadata": {
"tags": []
},
"execution_count": 4
}
]
},
{
"metadata": {
"id": "WcZCtRRN6J2l",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"## 7.1 Sagemaker Features"
]
},
{
"metadata": {
"id": "ZAwxrTUcfeko",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Search"
]
},
{
"metadata": {
"id": "xJN-eXYDtpVE",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"#### [Demo] Search"
]
},
{
"metadata": {
"id": "60kLF_hRfgZd",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"[Manage Machine Learning Experiments with Search](https://docs.aws.amazon.com/sagemaker/latest/dg/search.html)\n",
"\n",
"\n",
"\n",
"* Finding training jobs\n",
"* Rank training jobs\n",
"* Tracing lineage of a model\n",
"\n"
]
},
{
"metadata": {
"id": "OVuZQBAg-gHf",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Ground Truth"
]
},
{
"metadata": {
"id": "MYIx7nJT0z2z",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"\n",
"\n",
"\n",
"\n",
"* Setup and Manage labeling jobs\n",
"* Uses active learning and human labeling\n",
"* First 500 objects labeled per month are free\n",
"\n"
]
},
{
"metadata": {
"id": "xfvNv4E237k9",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"#### [Demo] Labeling Job"
]
},
{
"metadata": {
"id": "0pU-_czP4Blj",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
""
]
},
{
"metadata": {
"id": "nl1K5TVY00Pg",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Notebook"
]
},
{
"metadata": {
"id": "guMyo-9P014a",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
""
]
},
{
"metadata": {
"id": "o6nEwQ1tGiOl",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"#### [Demo] Sagemaker Notebooks\n",
"\n",
"* Create and run Jupyter Notebooks\n",
" - Using Jupyter\n",
" - Using JupyterLab\n",
" - Using the terminal\n",
" \n",
"* Lifecycle configurations\n",
"\n",
"* Git Repositories\n",
" - public repositories can be cloned on Notebook launch\n",
"\n"
]
},
{
"metadata": {
"id": "iKGPvalK02Yt",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Training"
]
},
{
"metadata": {
"id": "WwNsFi6602eK",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
""
]
},
{
"metadata": {
"id": "UGQQyDJtUN6D",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"#### [Demo] Sagemaker Training\n",
"\n",
"* Algorithms\n",
" - Create algorithm\n",
" - Subscribe [AWS Marketplace](https://aws.amazon.com/marketplace/search/results?page=1&filters=fulfillment_options%2Cresource_type&fulfillment_options=SAGEMAKER&resource_type=ALGORITHM)\n",
"\n",
" \n",
"* Training Jobs\n",
"\n",
"* HyperParameter Tuning Jobs\n"
]
},
{
"metadata": {
"id": "F9YxzTDT04lq",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Inference"
]
},
{
"metadata": {
"id": "9OALWADP05aM",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
""
]
},
{
"metadata": {
"id": "RQqvKpzIp1Sl",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"#### [Demo] Sagemaker Inference\n",
"\n",
"* Compilation jobs\n",
"\n",
"* Model packages\n",
"\n",
"* Models\n",
"\n",
"* Endpoint configurations\n",
"\n",
"* Endpoints\n",
"\n",
"* Batch transform jobs\n"
]
},
{
"metadata": {
"id": "1Q52AI5CW3eh",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Built in Sagemaker Algorithms"
]
},
{
"metadata": {
"id": "NzsQUYX7W5Xa",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"Table of [algorithms provided by Amazon Sagemaker](https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-algo-docker-registry-paths.html)\n",
"\n",
"\n"
]
},
{
"metadata": {
"id": "E4mpkoTycugo",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"\n",
"\n",
"\n"
]
},
{
"metadata": {
"id": "YW22mRwZ6RSl",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"## 7.2 DeepLense Features"
]
},
{
"metadata": {
"id": "AIPltTzbsRY6",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
""
]
},
{
"metadata": {
"id": "QOB1dQJHtmOi",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
""
]
},
{
"metadata": {
"id": "KNYr5yn4txIw",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
""
]
},
{
"metadata": {
"id": "zpGAF0WS-mdS",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"#### [Demo] DeepLense"
]
},
{
"metadata": {
"id": "AGmBAdl8Z1EQ",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
""
]
},
{
"metadata": {
"id": "fXpNEXuP6RYH",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"## 7.3 Kinesis Features"
]
},
{
"metadata": {
"id": "LV-LMaYT-wuO",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"[Kinesis FAQ](https://aws.amazon.com/kinesis/data-streams/faqs/)\n",
"\n",
"* Processes Data in Real-Time\n",
"* Can process hundreds of TBs an hour\n",
"* Example inputs are: \n",
" - logs\n",
" - financial transactions\n",
" * Streaming Data"
]
},
{
"metadata": {
"id": "Tnye93h6h0jn",
"colab_type": "code",
"colab": {}
},
"cell_type": "code",
"source": [
"!pip install -q sensible"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "gCR4VAgV75DC",
"colab_type": "code",
"colab": {}
},
"cell_type": "code",
"source": [
"import boto3\n"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "dQVPZr-c799V",
"colab_type": "code",
"colab": {}
},
"cell_type": "code",
"source": [
"import asyncio\n",
"import time\n",
"import datetime\n",
"import uuid\n",
"import boto3\n",
"import json\n",
"from sensible.loginit import logger\n",
"\n",
"LOG = logger(__name__)\n",
"\n",
"def firehose_client(region_name=\"us-east-1\"):\n",
" \"\"\"Kinesis Firehose client\"\"\"\n",
"\n",
" firehose_conn = boto3.client(\"firehose\", region_name=region_name)\n",
" extra_msg = {\"region_name\": region_name, \"aws_service\": \"firehose\"}\n",
" LOG.info(\"firehose connection initiated\", extra=extra_msg)\n",
" return firehose_conn\n",
"\n",
"async def put_record(data,\n",
" client,\n",
" delivery_stream_name=\"aws-ml-cert\"):\n",
" \"\"\"\n",
" See this:\n",
" http://boto3.readthedocs.io/en/latest/reference/services/\n",
" firehose.html#Firehose.Client.put_record\n",
" \"\"\"\n",
" extra_msg = {\"aws_service\": \"firehose\"}\n",
" LOG.info(f\"Pushing record to firehose: {data}\", extra=extra_msg)\n",
" response = client.put_record(\n",
" DeliveryStreamName=delivery_stream_name,\n",
" Record={\n",
" 'Data': data\n",
" }\n",
" )\n",
" return response\n",
"\n",
"\n",
"def gen_uuid_events():\n",
" \"\"\"Creates a time stamped UUID based event\"\"\"\n",
"\n",
" current_time = 'test-{date:%Y-%m-%d %H:%M:%S}'.format(date=datetime.datetime.now())\n",
" event_id = str(uuid.uuid4())\n",
" event = {event_id:current_time}\n",
" return json.dumps(event)\n",
"\n",
"def send_async_firehose_events(count=100):\n",
" \"\"\"Async sends events to firehose\"\"\"\n",
"\n",
" start = time.time() \n",
" client = firehose_client()\n",
" extra_msg = {\"aws_service\": \"firehose\"}\n",
" loop = asyncio.get_event_loop()\n",
" tasks = []\n",
" LOG.info(f\"sending aysnc events TOTAL {count}\",extra=extra_msg)\n",
" num = 0\n",
" for _ in range(count):\n",
" tasks.append(asyncio.ensure_future(put_record(gen_uuid_events(), client)))\n",
" LOG.info(f\"sending aysnc events: COUNT {num}/{count}\")\n",
" num +=1\n",
" loop.run_until_complete(asyncio.wait(tasks))\n",
" loop.close()\n",
" end = time.time() \n",
" LOG.info(\"Total time: {}\".format(end - start))\n",
"\n"
],
"execution_count": 0,
"outputs": []
},
{
"metadata": {
"id": "nhWXJvwc8hNp",
"colab_type": "code",
"outputId": "6f5cd3d1-bb57-4381-812e-bd9dfad79071",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 428
}
},
"cell_type": "code",
"source": [
"send_async_firehose_events(10)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"2018-12-14 18:49:58,211 - __main__ - INFO - firehose connection initiated\n",
"2018-12-14 18:49:58,213 - __main__ - INFO - sending aysnc events TOTAL 10\n",
"2018-12-14 18:49:58,214 - __main__ - INFO - sending aysnc events: COUNT 0/10\n",
"2018-12-14 18:49:58,216 - __main__ - INFO - sending aysnc events: COUNT 1/10\n",
"2018-12-14 18:49:58,220 - __main__ - INFO - sending aysnc events: COUNT 2/10\n",
"2018-12-14 18:49:58,221 - __main__ - INFO - sending aysnc events: COUNT 3/10\n",
"2018-12-14 18:49:58,225 - __main__ - INFO - sending aysnc events: COUNT 4/10\n",
"2018-12-14 18:49:58,228 - __main__ - INFO - sending aysnc events: COUNT 5/10\n",
"2018-12-14 18:49:58,231 - __main__ - INFO - sending aysnc events: COUNT 6/10\n",
"2018-12-14 18:49:58,233 - __main__ - INFO - sending aysnc events: COUNT 7/10\n",
"2018-12-14 18:49:58,236 - __main__ - INFO - sending aysnc events: COUNT 8/10\n",
"2018-12-14 18:49:58,237 - __main__ - INFO - sending aysnc events: COUNT 9/10\n",
"2018-12-14 18:49:58,242 - __main__ - INFO - Pushing record to firehose: {\"23bacac0-6eff-410f-b655-a7faa122b68f\": \"test-2018-12-14 18:49:58\"}\n",
"2018-12-14 18:49:58,364 - __main__ - INFO - Pushing record to firehose: {\"03b15c89-e0a2-46b8-b8d3-cdd71e2ab140\": \"test-2018-12-14 18:49:58\"}\n",
"2018-12-14 18:49:58,404 - __main__ - INFO - Pushing record to firehose: {\"5ff296d3-f43a-4e69-b61b-624603a354b4\": \"test-2018-12-14 18:49:58\"}\n",
"2018-12-14 18:49:58,448 - __main__ - INFO - Pushing record to firehose: {\"7ca78179-7ff7-49d1-bd6c-08ca5569737c\": \"test-2018-12-14 18:49:58\"}\n",
"2018-12-14 18:49:58,486 - __main__ - INFO - Pushing record to firehose: {\"554e587b-2d59-4e4f-95a3-75e4fb129c60\": \"test-2018-12-14 18:49:58\"}\n",
"2018-12-14 18:49:58,511 - __main__ - INFO - Pushing record to firehose: {\"30532d9e-6f3a-4366-b6ab-3bf9291c4e0b\": \"test-2018-12-14 18:49:58\"}\n",
"2018-12-14 18:49:58,548 - __main__ - INFO - Pushing record to firehose: {\"99cdff7d-32d1-424b-886e-9e93bb5e5a03\": \"test-2018-12-14 18:49:58\"}\n",
"2018-12-14 18:49:58,575 - __main__ - INFO - Pushing record to firehose: {\"8ea0c709-a8a5-4bde-9454-c6dd5691d58a\": \"test-2018-12-14 18:49:58\"}\n",
"2018-12-14 18:49:58,610 - __main__ - INFO - Pushing record to firehose: {\"99ce9671-7971-415c-9c64-ac2c82f9f478\": \"test-2018-12-14 18:49:58\"}\n",
"2018-12-14 18:49:58,643 - __main__ - INFO - Pushing record to firehose: {\"dc515833-8474-4377-9514-0f6d6fcfa403\": \"test-2018-12-14 18:49:58\"}\n",
"2018-12-14 18:49:58,677 - __main__ - INFO - Total time: 0.5094583034515381\n"
],
"name": "stderr"
}
]
},
{
"metadata": {
"id": "D6zp_J2K-iVn",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"## 7.4 AWS Flavored Python"
]
},
{
"metadata": {
"id": "PG-xBi-u-27q",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Boto3"
]
},
{
"metadata": {
"id": "o9NCTCQe5yC_",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"* Main Interface for working with AWS\n",
"\n",
"* Any Service in AWS can be communicated with Boto\n",
"* If Amazon is a country this is the language\n",
"\n"
]
},
{
"metadata": {
"id": "JjEX_5Xbe4BM",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"#### Communicate with S3"
]
},
{
"metadata": {
"id": "T8SwLHP7iZ4z",
"colab_type": "code",
"outputId": "5c79f31c-9f3c-4a8b-f0bf-8bce2233a332",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 5511
}
},
"cell_type": "code",
"source": [
"import boto3\n",
"resource = boto3.resource(\"s3\")\n",
"resource.meta.client.download_file('testntest', 'nba_2017_endorsement_full_stats.csv',\n",
"'/tmp/nba_2017_endorsement_full_stats.csv')"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"{\"message\": \"Loading JSON file: /usr/local/lib/python3.6/dist-packages/boto3/data/s3/2006-03-01/resources-1.json\"}\n",
"{\"message\": \"Loading JSON file: /usr/local/lib/python3.6/dist-packages/boto3/data/s3/2006-03-01/resources-1.json\"}\n",
"{\"message\": \"Event choose-service-name: calling handler \"}\n",
"{\"message\": \"Event choose-service-name: calling handler \"}\n",
"{\"message\": \"Loading JSON file: /usr/local/lib/python3.6/dist-packages/botocore/data/s3/2006-03-01/service-2.json\"}\n",
"{\"message\": \"Loading JSON file: /usr/local/lib/python3.6/dist-packages/botocore/data/s3/2006-03-01/service-2.json\"}\n",
"{\"message\": \"Event creating-client-class.s3: calling handler \"}\n",
"{\"message\": \"Event creating-client-class.s3: calling handler \"}\n",
"{\"message\": \"Event creating-client-class.s3: calling handler ._handler at 0x7f8fb54f9730>\"}\n",
"{\"message\": \"Event creating-client-class.s3: calling handler ._handler at 0x7f8fb54f9730>\"}\n",
"{\"message\": \"Event creating-client-class.s3: calling handler \"}\n",
"{\"message\": \"Event creating-client-class.s3: calling handler \"}\n",
"{\"message\": \"The s3 config key is not a dictionary type, ignoring its value of: None\"}\n",
"{\"message\": \"The s3 config key is not a dictionary type, ignoring its value of: None\"}\n",
"{\"message\": \"Setting s3 timeout as (60, 60)\"}\n",
"{\"message\": \"Setting s3 timeout as (60, 60)\"}\n",
"{\"message\": \"Registering retry handlers for service: s3\"}\n",
"{\"message\": \"Registering retry handlers for service: s3\"}\n",
"{\"message\": \"Defaulting to S3 virtual host style addressing with path style addressing fallback.\"}\n",
"{\"message\": \"Defaulting to S3 virtual host style addressing with path style addressing fallback.\"}\n",
"{\"message\": \"Loading s3:s3\"}\n",
"{\"message\": \"Loading s3:s3\"}\n",
"{\"message\": \"Acquiring 0\"}\n",
"{\"message\": \"Acquiring 0\"}\n",
"{\"message\": \"DownloadSubmissionTask(transfer_id=0, {'transfer_future': }) about to wait for the following futures []\"}\n",
"{\"message\": \"DownloadSubmissionTask(transfer_id=0, {'transfer_future': }) about to wait for the following futures []\"}\n",
"{\"message\": \"DownloadSubmissionTask(transfer_id=0, {'transfer_future': }) done waiting for dependent futures\"}\n",
"{\"message\": \"DownloadSubmissionTask(transfer_id=0, {'transfer_future': }) done waiting for dependent futures\"}\n",
"{\"message\": \"Executing task DownloadSubmissionTask(transfer_id=0, {'transfer_future': }) with kwargs {'client': , 'config': , 'osutil': , 'request_executor': , 'transfer_future': , 'io_executor': }\"}\n",
"{\"message\": \"Executing task DownloadSubmissionTask(transfer_id=0, {'transfer_future': }) with kwargs {'client': , 'config': , 'osutil': , 'request_executor': , 'transfer_future': , 'io_executor': }\"}\n",
"{\"message\": \"Event before-parameter-build.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event before-parameter-build.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event before-parameter-build.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event before-parameter-build.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event before-parameter-build.s3.HeadObject: calling handler >\"}\n",
"{\"message\": \"Event before-parameter-build.s3.HeadObject: calling handler >\"}\n",
"{\"message\": \"Event before-parameter-build.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event before-parameter-build.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event before-call.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event before-call.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event before-call.s3.HeadObject: calling handler >\"}\n",
"{\"message\": \"Event before-call.s3.HeadObject: calling handler >\"}\n",
"{\"message\": \"Making request for OperationModel(name=HeadObject) with params: {'url_path': '/testntest/nba_2017_endorsement_full_stats.csv', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'Boto3/1.9.62 Python/3.6.7 Linux/4.14.65+ Botocore/1.12.62 Resource'}, 'body': b'', 'url': 'https://s3.amazonaws.com/testntest/nba_2017_endorsement_full_stats.csv', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'testntest'}}}\"}\n",
"{\"message\": \"Making request for OperationModel(name=HeadObject) with params: {'url_path': '/testntest/nba_2017_endorsement_full_stats.csv', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'Boto3/1.9.62 Python/3.6.7 Linux/4.14.65+ Botocore/1.12.62 Resource'}, 'body': b'', 'url': 'https://s3.amazonaws.com/testntest/nba_2017_endorsement_full_stats.csv', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'testntest'}}}\"}\n",
"{\"message\": \"Event request-created.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event request-created.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event request-created.s3.HeadObject: calling handler >\"}\n",
"{\"message\": \"Event request-created.s3.HeadObject: calling handler >\"}\n",
"{\"message\": \"Event choose-signer.s3.HeadObject: calling handler >\"}\n",
"{\"message\": \"Event choose-signer.s3.HeadObject: calling handler >\"}\n",
"{\"message\": \"Event choose-signer.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event choose-signer.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event before-sign.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event before-sign.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Checking for DNS compatible bucket for: https://s3.amazonaws.com/testntest/nba_2017_endorsement_full_stats.csv\"}\n",
"{\"message\": \"Checking for DNS compatible bucket for: https://s3.amazonaws.com/testntest/nba_2017_endorsement_full_stats.csv\"}\n",
"{\"message\": \"URI updated to: https://testntest.s3.amazonaws.com/nba_2017_endorsement_full_stats.csv\"}\n",
"{\"message\": \"URI updated to: https://testntest.s3.amazonaws.com/nba_2017_endorsement_full_stats.csv\"}\n",
"{\"message\": \"Calculating signature using v4 auth.\"}\n",
"{\"message\": \"Calculating signature using v4 auth.\"}\n",
"{\"message\": \"CanonicalRequest:\\nHEAD\\n/nba_2017_endorsement_full_stats.csv\\n\\nhost:testntest.s3.amazonaws.com\\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\nx-amz-date:20181214T190639Z\\n\\nhost;x-amz-content-sha256;x-amz-date\\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\"}\n",
"{\"message\": \"CanonicalRequest:\\nHEAD\\n/nba_2017_endorsement_full_stats.csv\\n\\nhost:testntest.s3.amazonaws.com\\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\nx-amz-date:20181214T190639Z\\n\\nhost;x-amz-content-sha256;x-amz-date\\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\"}\n",
"{\"message\": \"StringToSign:\\nAWS4-HMAC-SHA256\\n20181214T190639Z\\n20181214/us-east-1/s3/aws4_request\\ndc464718e66f61dbbd6306f072011e817a11700464a81c78e77bac008daaa11b\"}\n",
"{\"message\": \"StringToSign:\\nAWS4-HMAC-SHA256\\n20181214T190639Z\\n20181214/us-east-1/s3/aws4_request\\ndc464718e66f61dbbd6306f072011e817a11700464a81c78e77bac008daaa11b\"}\n",
"{\"message\": \"Signature:\\ndccf5cd1ba1f8ffd257e709de85d54d05de887bf6c7a0539fb74f037e3c727de\"}\n",
"{\"message\": \"Signature:\\ndccf5cd1ba1f8ffd257e709de85d54d05de887bf6c7a0539fb74f037e3c727de\"}\n",
"{\"message\": \"Event request-created.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event request-created.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Sending http request: \"}\n",
"{\"message\": \"Sending http request: \"}\n",
"{\"message\": \"Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)\"}\n",
"{\"message\": \"Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)\"}\n",
"{\"message\": \"Starting new HTTPS connection (1): testntest.s3.amazonaws.com\"}\n",
"{\"message\": \"Starting new HTTPS connection (1): testntest.s3.amazonaws.com\"}\n",
"{\"message\": \"https://testntest.s3.amazonaws.com:443 \\\"HEAD /nba_2017_endorsement_full_stats.csv HTTP/1.1\\\" 400 0\"}\n",
"{\"message\": \"https://testntest.s3.amazonaws.com:443 \\\"HEAD /nba_2017_endorsement_full_stats.csv HTTP/1.1\\\" 400 0\"}\n",
"{\"message\": \"Response headers: {'x-amz-request-id': '22146DB0C1CFB102', 'x-amz-id-2': 'L5unIiNB7+FTulP8F6l1f4Blmuo6WKNSZ5oKnNlJI4QMN/lL6amNBiEKIyFg7MgUmogD/X4FA2o=', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Fri, 14 Dec 2018 19:06:39 GMT', 'Connection': 'close', 'Server': 'AmazonS3'}\"}\n",
"{\"message\": \"Response headers: {'x-amz-request-id': '22146DB0C1CFB102', 'x-amz-id-2': 'L5unIiNB7+FTulP8F6l1f4Blmuo6WKNSZ5oKnNlJI4QMN/lL6amNBiEKIyFg7MgUmogD/X4FA2o=', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Fri, 14 Dec 2018 19:06:39 GMT', 'Connection': 'close', 'Server': 'AmazonS3'}\"}\n",
"{\"message\": \"Response body:\\nb''\"}\n",
"{\"message\": \"Response body:\\nb''\"}\n",
"{\"message\": \"Event needs-retry.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event needs-retry.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"No retry needed.\"}\n",
"{\"message\": \"No retry needed.\"}\n",
"{\"message\": \"Event needs-retry.s3.HeadObject: calling handler >\"}\n",
"{\"message\": \"Event needs-retry.s3.HeadObject: calling handler >\"}\n",
"{\"message\": \"Event before-parameter-build.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Event before-parameter-build.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Event before-parameter-build.s3.HeadBucket: calling handler >\"}\n",
"{\"message\": \"Event before-parameter-build.s3.HeadBucket: calling handler >\"}\n",
"{\"message\": \"Event before-parameter-build.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Event before-parameter-build.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Event before-call.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Event before-call.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Event before-call.s3.HeadBucket: calling handler >\"}\n",
"{\"message\": \"Event before-call.s3.HeadBucket: calling handler >\"}\n",
"{\"message\": \"Making request for OperationModel(name=HeadBucket) with params: {'url_path': '/testntest', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'Boto3/1.9.62 Python/3.6.7 Linux/4.14.65+ Botocore/1.12.62 Resource'}, 'body': b'', 'url': 'https://s3.amazonaws.com/testntest', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'testntest'}}}\"}\n",
"{\"message\": \"Making request for OperationModel(name=HeadBucket) with params: {'url_path': '/testntest', 'query_string': {}, 'method': 'HEAD', 'headers': {'User-Agent': 'Boto3/1.9.62 Python/3.6.7 Linux/4.14.65+ Botocore/1.12.62 Resource'}, 'body': b'', 'url': 'https://s3.amazonaws.com/testntest', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'testntest'}}}\"}\n",
"{\"message\": \"Event request-created.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Event request-created.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Event request-created.s3.HeadBucket: calling handler >\"}\n",
"{\"message\": \"Event request-created.s3.HeadBucket: calling handler >\"}\n",
"{\"message\": \"Event choose-signer.s3.HeadBucket: calling handler >\"}\n",
"{\"message\": \"Event choose-signer.s3.HeadBucket: calling handler >\"}\n",
"{\"message\": \"Event choose-signer.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Event choose-signer.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Event before-sign.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Event before-sign.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Checking for DNS compatible bucket for: https://s3.amazonaws.com/testntest\"}\n",
"{\"message\": \"Checking for DNS compatible bucket for: https://s3.amazonaws.com/testntest\"}\n",
"{\"message\": \"URI updated to: https://testntest.s3.amazonaws.com/\"}\n",
"{\"message\": \"URI updated to: https://testntest.s3.amazonaws.com/\"}\n",
"{\"message\": \"Calculating signature using v4 auth.\"}\n",
"{\"message\": \"Calculating signature using v4 auth.\"}\n",
"{\"message\": \"CanonicalRequest:\\nHEAD\\n/\\n\\nhost:testntest.s3.amazonaws.com\\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\nx-amz-date:20181214T190639Z\\n\\nhost;x-amz-content-sha256;x-amz-date\\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\"}\n",
"{\"message\": \"CanonicalRequest:\\nHEAD\\n/\\n\\nhost:testntest.s3.amazonaws.com\\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\nx-amz-date:20181214T190639Z\\n\\nhost;x-amz-content-sha256;x-amz-date\\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\"}\n",
"{\"message\": \"StringToSign:\\nAWS4-HMAC-SHA256\\n20181214T190639Z\\n20181214/us-east-1/s3/aws4_request\\n799fcd8c0621e283d6b947f1e3d95eef142741deedd165dd48ee8811d5f9d852\"}\n",
"{\"message\": \"StringToSign:\\nAWS4-HMAC-SHA256\\n20181214T190639Z\\n20181214/us-east-1/s3/aws4_request\\n799fcd8c0621e283d6b947f1e3d95eef142741deedd165dd48ee8811d5f9d852\"}\n",
"{\"message\": \"Signature:\\n662dcd262b4b95999730c002394be231cf02ad3b1d8cc90da9399bfd585bc1e7\"}\n",
"{\"message\": \"Signature:\\n662dcd262b4b95999730c002394be231cf02ad3b1d8cc90da9399bfd585bc1e7\"}\n",
"{\"message\": \"Event request-created.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Event request-created.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Sending http request: \"}\n",
"{\"message\": \"Sending http request: \"}\n",
"{\"message\": \"Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)\"}\n",
"{\"message\": \"Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)\"}\n",
"{\"message\": \"Starting new HTTPS connection (2): testntest.s3.amazonaws.com\"}\n",
"{\"message\": \"Starting new HTTPS connection (2): testntest.s3.amazonaws.com\"}\n",
"{\"message\": \"https://testntest.s3.amazonaws.com:443 \\\"HEAD / HTTP/1.1\\\" 400 0\"}\n",
"{\"message\": \"https://testntest.s3.amazonaws.com:443 \\\"HEAD / HTTP/1.1\\\" 400 0\"}\n",
"{\"message\": \"Response headers: {'x-amz-bucket-region': 'us-east-2', 'x-amz-request-id': '269DCC355D0E8FC5', 'x-amz-id-2': 'zDCOaU9LZsqhOzJNjRUBAl/pNuU9j05NC1GAxcfrjdhft16LSuDDk2Ay3Hu8j6kk2kmUx55SZSo=', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Fri, 14 Dec 2018 19:06:39 GMT', 'Connection': 'close', 'Server': 'AmazonS3'}\"}\n",
"{\"message\": \"Response headers: {'x-amz-bucket-region': 'us-east-2', 'x-amz-request-id': '269DCC355D0E8FC5', 'x-amz-id-2': 'zDCOaU9LZsqhOzJNjRUBAl/pNuU9j05NC1GAxcfrjdhft16LSuDDk2Ay3Hu8j6kk2kmUx55SZSo=', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Fri, 14 Dec 2018 19:06:39 GMT', 'Connection': 'close', 'Server': 'AmazonS3'}\"}\n",
"{\"message\": \"Response body:\\nb''\"}\n",
"{\"message\": \"Response body:\\nb''\"}\n",
"{\"message\": \"Event needs-retry.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Event needs-retry.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"No retry needed.\"}\n",
"{\"message\": \"No retry needed.\"}\n",
"{\"message\": \"Event needs-retry.s3.HeadBucket: calling handler >\"}\n",
"{\"message\": \"Event needs-retry.s3.HeadBucket: calling handler >\"}\n",
"{\"message\": \"S3 client configured for region us-east-1 but the bucket testntest is in region us-east-2; Please configure the proper region to avoid multiple unnecessary redirects and signing attempts.\"}\n",
"{\"message\": \"S3 client configured for region us-east-1 but the bucket testntest is in region us-east-2; Please configure the proper region to avoid multiple unnecessary redirects and signing attempts.\"}\n",
"{\"message\": \"Updating URI from https://s3.amazonaws.com/testntest to https://s3.us-east-2.amazonaws.com/testntest\"}\n",
"{\"message\": \"Updating URI from https://s3.amazonaws.com/testntest to https://s3.us-east-2.amazonaws.com/testntest\"}\n",
"{\"message\": \"Response received to retry, sleeping for 0 seconds\"}\n",
"{\"message\": \"Response received to retry, sleeping for 0 seconds\"}\n",
"{\"message\": \"Event request-created.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Event request-created.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Event request-created.s3.HeadBucket: calling handler >\"}\n",
"{\"message\": \"Event request-created.s3.HeadBucket: calling handler >\"}\n",
"{\"message\": \"Event choose-signer.s3.HeadBucket: calling handler >\"}\n",
"{\"message\": \"Event choose-signer.s3.HeadBucket: calling handler >\"}\n",
"{\"message\": \"Event choose-signer.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Event choose-signer.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Event before-sign.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Event before-sign.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Checking for DNS compatible bucket for: https://s3.us-east-2.amazonaws.com/testntest\"}\n",
"{\"message\": \"Checking for DNS compatible bucket for: https://s3.us-east-2.amazonaws.com/testntest\"}\n",
"{\"message\": \"URI updated to: https://testntest.s3.us-east-2.amazonaws.com/\"}\n",
"{\"message\": \"URI updated to: https://testntest.s3.us-east-2.amazonaws.com/\"}\n",
"{\"message\": \"Calculating signature using v4 auth.\"}\n",
"{\"message\": \"Calculating signature using v4 auth.\"}\n",
"{\"message\": \"CanonicalRequest:\\nHEAD\\n/\\n\\nhost:testntest.s3.us-east-2.amazonaws.com\\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\nx-amz-date:20181214T190639Z\\n\\nhost;x-amz-content-sha256;x-amz-date\\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\"}\n",
"{\"message\": \"CanonicalRequest:\\nHEAD\\n/\\n\\nhost:testntest.s3.us-east-2.amazonaws.com\\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\nx-amz-date:20181214T190639Z\\n\\nhost;x-amz-content-sha256;x-amz-date\\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\"}\n",
"{\"message\": \"StringToSign:\\nAWS4-HMAC-SHA256\\n20181214T190639Z\\n20181214/us-east-2/s3/aws4_request\\n871d114aad45e44f48590659e690e4bf80e931590de143cb6d43315e11738099\"}\n",
"{\"message\": \"StringToSign:\\nAWS4-HMAC-SHA256\\n20181214T190639Z\\n20181214/us-east-2/s3/aws4_request\\n871d114aad45e44f48590659e690e4bf80e931590de143cb6d43315e11738099\"}\n",
"{\"message\": \"Signature:\\ncf9f8ade13bca861c824070d31f9e931bce82e0339dffac638ead2c7a6d3a932\"}\n",
"{\"message\": \"Signature:\\ncf9f8ade13bca861c824070d31f9e931bce82e0339dffac638ead2c7a6d3a932\"}\n",
"{\"message\": \"Event request-created.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Event request-created.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Sending http request: \"}\n",
"{\"message\": \"Sending http request: \"}\n",
"{\"message\": \"Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)\"}\n",
"{\"message\": \"Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)\"}\n",
"{\"message\": \"Starting new HTTPS connection (1): testntest.s3.us-east-2.amazonaws.com\"}\n",
"{\"message\": \"Starting new HTTPS connection (1): testntest.s3.us-east-2.amazonaws.com\"}\n",
"{\"message\": \"https://testntest.s3.us-east-2.amazonaws.com:443 \\\"HEAD / HTTP/1.1\\\" 200 0\"}\n",
"{\"message\": \"https://testntest.s3.us-east-2.amazonaws.com:443 \\\"HEAD / HTTP/1.1\\\" 200 0\"}\n",
"{\"message\": \"Response headers: {'x-amz-id-2': 'TrkXI9rge0KJy+wVICwUfQAA5KlU5u8B98N8FegMqGYacUpn+4NnGpAkkpUgGdr9pbyruUjvcUQ=', 'x-amz-request-id': 'D5556B95E431CCA8', 'Date': 'Fri, 14 Dec 2018 19:06:40 GMT', 'x-amz-bucket-region': 'us-east-2', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Server': 'AmazonS3'}\"}\n",
"{\"message\": \"Response headers: {'x-amz-id-2': 'TrkXI9rge0KJy+wVICwUfQAA5KlU5u8B98N8FegMqGYacUpn+4NnGpAkkpUgGdr9pbyruUjvcUQ=', 'x-amz-request-id': 'D5556B95E431CCA8', 'Date': 'Fri, 14 Dec 2018 19:06:40 GMT', 'x-amz-bucket-region': 'us-east-2', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Server': 'AmazonS3'}\"}\n",
"{\"message\": \"Response body:\\nb''\"}\n",
"{\"message\": \"Response body:\\nb''\"}\n",
"{\"message\": \"Event needs-retry.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"Event needs-retry.s3.HeadBucket: calling handler \"}\n",
"{\"message\": \"No retry needed.\"}\n",
"{\"message\": \"No retry needed.\"}\n",
"{\"message\": \"Event needs-retry.s3.HeadBucket: calling handler >\"}\n",
"{\"message\": \"Event needs-retry.s3.HeadBucket: calling handler >\"}\n",
"{\"message\": \"S3 request was previously redirected, not redirecting.\"}\n",
"{\"message\": \"S3 request was previously redirected, not redirecting.\"}\n",
"{\"message\": \"S3 client configured for region us-east-1 but the bucket testntest is in region us-east-2; Please configure the proper region to avoid multiple unnecessary redirects and signing attempts.\"}\n",
"{\"message\": \"S3 client configured for region us-east-1 but the bucket testntest is in region us-east-2; Please configure the proper region to avoid multiple unnecessary redirects and signing attempts.\"}\n",
"{\"message\": \"Updating URI from https://s3.amazonaws.com/testntest/nba_2017_endorsement_full_stats.csv to https://s3.us-east-2.amazonaws.com/testntest/nba_2017_endorsement_full_stats.csv\"}\n",
"{\"message\": \"Updating URI from https://s3.amazonaws.com/testntest/nba_2017_endorsement_full_stats.csv to https://s3.us-east-2.amazonaws.com/testntest/nba_2017_endorsement_full_stats.csv\"}\n",
"{\"message\": \"Response received to retry, sleeping for 0 seconds\"}\n",
"{\"message\": \"Response received to retry, sleeping for 0 seconds\"}\n",
"{\"message\": \"Event request-created.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event request-created.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event request-created.s3.HeadObject: calling handler >\"}\n",
"{\"message\": \"Event request-created.s3.HeadObject: calling handler >\"}\n",
"{\"message\": \"Event choose-signer.s3.HeadObject: calling handler >\"}\n",
"{\"message\": \"Event choose-signer.s3.HeadObject: calling handler >\"}\n",
"{\"message\": \"Event choose-signer.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event choose-signer.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event before-sign.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event before-sign.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Checking for DNS compatible bucket for: https://s3.us-east-2.amazonaws.com/testntest/nba_2017_endorsement_full_stats.csv\"}\n",
"{\"message\": \"Checking for DNS compatible bucket for: https://s3.us-east-2.amazonaws.com/testntest/nba_2017_endorsement_full_stats.csv\"}\n",
"{\"message\": \"URI updated to: https://testntest.s3.us-east-2.amazonaws.com/nba_2017_endorsement_full_stats.csv\"}\n",
"{\"message\": \"URI updated to: https://testntest.s3.us-east-2.amazonaws.com/nba_2017_endorsement_full_stats.csv\"}\n",
"{\"message\": \"Calculating signature using v4 auth.\"}\n",
"{\"message\": \"Calculating signature using v4 auth.\"}\n",
"{\"message\": \"CanonicalRequest:\\nHEAD\\n/nba_2017_endorsement_full_stats.csv\\n\\nhost:testntest.s3.us-east-2.amazonaws.com\\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\nx-amz-date:20181214T190639Z\\n\\nhost;x-amz-content-sha256;x-amz-date\\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\"}\n",
"{\"message\": \"CanonicalRequest:\\nHEAD\\n/nba_2017_endorsement_full_stats.csv\\n\\nhost:testntest.s3.us-east-2.amazonaws.com\\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\nx-amz-date:20181214T190639Z\\n\\nhost;x-amz-content-sha256;x-amz-date\\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\"}\n",
"{\"message\": \"StringToSign:\\nAWS4-HMAC-SHA256\\n20181214T190639Z\\n20181214/us-east-2/s3/aws4_request\\neb0c92849dfd44b981e88b39e8c9a0902c387dba4f88c7df0487a9cb9c53c456\"}\n",
"{\"message\": \"StringToSign:\\nAWS4-HMAC-SHA256\\n20181214T190639Z\\n20181214/us-east-2/s3/aws4_request\\neb0c92849dfd44b981e88b39e8c9a0902c387dba4f88c7df0487a9cb9c53c456\"}\n",
"{\"message\": \"Signature:\\n69c32ba74cc83cb2e6ebca4e2d7f48fa012a70988c578d8ea3112117cc81f56d\"}\n",
"{\"message\": \"Signature:\\n69c32ba74cc83cb2e6ebca4e2d7f48fa012a70988c578d8ea3112117cc81f56d\"}\n",
"{\"message\": \"Event request-created.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event request-created.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Sending http request: \"}\n",
"{\"message\": \"Sending http request: \"}\n",
"{\"message\": \"Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)\"}\n",
"{\"message\": \"Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)\"}\n",
"{\"message\": \"Starting new HTTPS connection (2): testntest.s3.us-east-2.amazonaws.com\"}\n",
"{\"message\": \"Starting new HTTPS connection (2): testntest.s3.us-east-2.amazonaws.com\"}\n",
"{\"message\": \"https://testntest.s3.us-east-2.amazonaws.com:443 \\\"HEAD /nba_2017_endorsement_full_stats.csv HTTP/1.1\\\" 200 0\"}\n",
"{\"message\": \"https://testntest.s3.us-east-2.amazonaws.com:443 \\\"HEAD /nba_2017_endorsement_full_stats.csv HTTP/1.1\\\" 200 0\"}\n",
"{\"message\": \"Response headers: {'x-amz-id-2': 'O2LxC3y1e7b9U6TdAD21AtaVybEiTaqedFKWg0/Lfn7V2PYA0sD0PjGfIosDZFv9DHEXg9KGL/0=', 'x-amz-request-id': 'AFB111AF0C49CA92', 'Date': 'Fri, 14 Dec 2018 19:06:41 GMT', 'Last-Modified': 'Wed, 29 Nov 2017 18:48:35 GMT', 'ETag': '\\\"60ea033c62d2b044f596333cd464baa3\\\"', 'Accept-Ranges': 'bytes', 'Content-Type': 'text/csv', 'Content-Length': '1447', 'Server': 'AmazonS3'}\"}\n",
"{\"message\": \"Response headers: {'x-amz-id-2': 'O2LxC3y1e7b9U6TdAD21AtaVybEiTaqedFKWg0/Lfn7V2PYA0sD0PjGfIosDZFv9DHEXg9KGL/0=', 'x-amz-request-id': 'AFB111AF0C49CA92', 'Date': 'Fri, 14 Dec 2018 19:06:41 GMT', 'Last-Modified': 'Wed, 29 Nov 2017 18:48:35 GMT', 'ETag': '\\\"60ea033c62d2b044f596333cd464baa3\\\"', 'Accept-Ranges': 'bytes', 'Content-Type': 'text/csv', 'Content-Length': '1447', 'Server': 'AmazonS3'}\"}\n",
"{\"message\": \"Response body:\\nb''\"}\n",
"{\"message\": \"Response body:\\nb''\"}\n",
"{\"message\": \"Event needs-retry.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"Event needs-retry.s3.HeadObject: calling handler \"}\n",
"{\"message\": \"No retry needed.\"}\n",
"{\"message\": \"No retry needed.\"}\n",
"{\"message\": \"Event needs-retry.s3.HeadObject: calling handler >\"}\n",
"{\"message\": \"Event needs-retry.s3.HeadObject: calling handler >\"}\n",
"{\"message\": \"S3 request was previously redirected, not redirecting.\"}\n",
"{\"message\": \"S3 request was previously redirected, not redirecting.\"}\n",
"{\"message\": \"Submitting task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'testntest', 'key': 'nba_2017_endorsement_full_stats.csv', 'extra_args': {}}) to executor for transfer request: 0.\"}\n",
"{\"message\": \"Submitting task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'testntest', 'key': 'nba_2017_endorsement_full_stats.csv', 'extra_args': {}}) to executor for transfer request: 0.\"}\n",
"{\"message\": \"Acquiring 0\"}\n",
"{\"message\": \"Acquiring 0\"}\n",
"{\"message\": \"ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'testntest', 'key': 'nba_2017_endorsement_full_stats.csv', 'extra_args': {}}) about to wait for the following futures []\"}\n",
"{\"message\": \"Releasing acquire 0/None\"}\n",
"{\"message\": \"ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'testntest', 'key': 'nba_2017_endorsement_full_stats.csv', 'extra_args': {}}) about to wait for the following futures []\"}\n",
"{\"message\": \"Releasing acquire 0/None\"}\n",
"{\"message\": \"ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'testntest', 'key': 'nba_2017_endorsement_full_stats.csv', 'extra_args': {}}) done waiting for dependent futures\"}\n",
"{\"message\": \"ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'testntest', 'key': 'nba_2017_endorsement_full_stats.csv', 'extra_args': {}}) done waiting for dependent futures\"}\n",
"{\"message\": \"Executing task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'testntest', 'key': 'nba_2017_endorsement_full_stats.csv', 'extra_args': {}}) with kwargs {'client': , 'bucket': 'testntest', 'key': 'nba_2017_endorsement_full_stats.csv', 'fileobj': , 'extra_args': {}, 'callbacks': [], 'max_attempts': 5, 'download_output_manager': , 'io_chunksize': 262144, 'bandwidth_limiter': None}\"}\n",
"{\"message\": \"Executing task ImmediatelyWriteIOGetObjectTask(transfer_id=0, {'bucket': 'testntest', 'key': 'nba_2017_endorsement_full_stats.csv', 'extra_args': {}}) with kwargs {'client': , 'bucket': 'testntest', 'key': 'nba_2017_endorsement_full_stats.csv', 'fileobj': , 'extra_args': {}, 'callbacks': [], 'max_attempts': 5, 'download_output_manager': , 'io_chunksize': 262144, 'bandwidth_limiter': None}\"}\n",
"{\"message\": \"Event before-parameter-build.s3.GetObject: calling handler \"}\n",
"{\"message\": \"Event before-parameter-build.s3.GetObject: calling handler \"}\n",
"{\"message\": \"Event before-parameter-build.s3.GetObject: calling handler \"}\n",
"{\"message\": \"Event before-parameter-build.s3.GetObject: calling handler \"}\n",
"{\"message\": \"Event before-parameter-build.s3.GetObject: calling handler >\"}\n",
"{\"message\": \"Event before-parameter-build.s3.GetObject: calling handler >\"}\n",
"{\"message\": \"Event before-parameter-build.s3.GetObject: calling handler \"}\n",
"{\"message\": \"Event before-parameter-build.s3.GetObject: calling handler \"}\n",
"{\"message\": \"Event before-call.s3.GetObject: calling handler \"}\n",
"{\"message\": \"Event before-call.s3.GetObject: calling handler \"}\n",
"{\"message\": \"Event before-call.s3.GetObject: calling handler >\"}\n",
"{\"message\": \"Event before-call.s3.GetObject: calling handler >\"}\n",
"{\"message\": \"Updating URI from https://s3.amazonaws.com/testntest/nba_2017_endorsement_full_stats.csv to https://s3.us-east-2.amazonaws.com/testntest/nba_2017_endorsement_full_stats.csv\"}\n",
"{\"message\": \"Updating URI from https://s3.amazonaws.com/testntest/nba_2017_endorsement_full_stats.csv to https://s3.us-east-2.amazonaws.com/testntest/nba_2017_endorsement_full_stats.csv\"}\n",
"{\"message\": \"Making request for OperationModel(name=GetObject) with params: {'url_path': '/testntest/nba_2017_endorsement_full_stats.csv', 'query_string': {}, 'method': 'GET', 'headers': {'User-Agent': 'Boto3/1.9.62 Python/3.6.7 Linux/4.14.65+ Botocore/1.12.62 Resource'}, 'body': b'', 'url': 'https://s3.us-east-2.amazonaws.com/testntest/nba_2017_endorsement_full_stats.csv', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'region': 'us-east-2', 'bucket': 'testntest', 'endpoint': 'https://s3.us-east-2.amazonaws.com'}}}\"}\n",
"{\"message\": \"Making request for OperationModel(name=GetObject) with params: {'url_path': '/testntest/nba_2017_endorsement_full_stats.csv', 'query_string': {}, 'method': 'GET', 'headers': {'User-Agent': 'Boto3/1.9.62 Python/3.6.7 Linux/4.14.65+ Botocore/1.12.62 Resource'}, 'body': b'', 'url': 'https://s3.us-east-2.amazonaws.com/testntest/nba_2017_endorsement_full_stats.csv', 'context': {'client_region': 'us-east-1', 'client_config': , 'has_streaming_input': False, 'auth_type': None, 'signing': {'region': 'us-east-2', 'bucket': 'testntest', 'endpoint': 'https://s3.us-east-2.amazonaws.com'}}}\"}\n",
"{\"message\": \"Event request-created.s3.GetObject: calling handler \"}\n",
"{\"message\": \"Event request-created.s3.GetObject: calling handler \"}\n",
"{\"message\": \"Event request-created.s3.GetObject: calling handler >\"}\n",
"{\"message\": \"Event request-created.s3.GetObject: calling handler >\"}\n",
"{\"message\": \"Event choose-signer.s3.GetObject: calling handler >\"}\n",
"{\"message\": \"Event choose-signer.s3.GetObject: calling handler >\"}\n",
"{\"message\": \"Event choose-signer.s3.GetObject: calling handler \"}\n",
"{\"message\": \"Event choose-signer.s3.GetObject: calling handler \"}\n",
"{\"message\": \"Event before-sign.s3.GetObject: calling handler \"}\n",
"{\"message\": \"Event before-sign.s3.GetObject: calling handler \"}\n",
"{\"message\": \"Checking for DNS compatible bucket for: https://s3.us-east-2.amazonaws.com/testntest/nba_2017_endorsement_full_stats.csv\"}\n",
"{\"message\": \"Checking for DNS compatible bucket for: https://s3.us-east-2.amazonaws.com/testntest/nba_2017_endorsement_full_stats.csv\"}\n",
"{\"message\": \"URI updated to: https://testntest.s3.us-east-2.amazonaws.com/nba_2017_endorsement_full_stats.csv\"}\n",
"{\"message\": \"URI updated to: https://testntest.s3.us-east-2.amazonaws.com/nba_2017_endorsement_full_stats.csv\"}\n",
"{\"message\": \"Calculating signature using v4 auth.\"}\n",
"{\"message\": \"Calculating signature using v4 auth.\"}\n",
"{\"message\": \"CanonicalRequest:\\nGET\\n/nba_2017_endorsement_full_stats.csv\\n\\nhost:testntest.s3.us-east-2.amazonaws.com\\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\nx-amz-date:20181214T190640Z\\n\\nhost;x-amz-content-sha256;x-amz-date\\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\"}\n",
"{\"message\": \"CanonicalRequest:\\nGET\\n/nba_2017_endorsement_full_stats.csv\\n\\nhost:testntest.s3.us-east-2.amazonaws.com\\nx-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\\nx-amz-date:20181214T190640Z\\n\\nhost;x-amz-content-sha256;x-amz-date\\ne3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\"}\n",
"{\"message\": \"StringToSign:\\nAWS4-HMAC-SHA256\\n20181214T190640Z\\n20181214/us-east-2/s3/aws4_request\\n2e243a8c95b06e393be0f7c32cee241b83423b3231c92471266e365180a3de6b\"}\n",
"{\"message\": \"StringToSign:\\nAWS4-HMAC-SHA256\\n20181214T190640Z\\n20181214/us-east-2/s3/aws4_request\\n2e243a8c95b06e393be0f7c32cee241b83423b3231c92471266e365180a3de6b\"}\n",
"{\"message\": \"Signature:\\n7c748447a0a1a146d02b3995e18ef7f34c3aa47150d51253f043c6161992ca7e\"}\n",
"{\"message\": \"Signature:\\n7c748447a0a1a146d02b3995e18ef7f34c3aa47150d51253f043c6161992ca7e\"}\n",
"{\"message\": \"Event request-created.s3.GetObject: calling handler \"}\n",
"{\"message\": \"Event request-created.s3.GetObject: calling handler \"}\n",
"{\"message\": \"Sending http request: \"}\n",
"{\"message\": \"Sending http request: \"}\n",
"{\"message\": \"Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)\"}\n",
"{\"message\": \"Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)\"}\n",
"{\"message\": \"https://testntest.s3.us-east-2.amazonaws.com:443 \\\"GET /nba_2017_endorsement_full_stats.csv HTTP/1.1\\\" 200 1447\"}\n",
"{\"message\": \"https://testntest.s3.us-east-2.amazonaws.com:443 \\\"GET /nba_2017_endorsement_full_stats.csv HTTP/1.1\\\" 200 1447\"}\n",
"{\"message\": \"Response headers: {'x-amz-id-2': 'OzI+sE1ZbetaaRTj+pWA6nPuNi0e1yrAUi1vKcvSShsXqP+LxiQTI5c3pV5y1pT6B4RR06dRBrg=', 'x-amz-request-id': '9763AE9849C42E10', 'Date': 'Fri, 14 Dec 2018 19:06:41 GMT', 'Last-Modified': 'Wed, 29 Nov 2017 18:48:35 GMT', 'ETag': '\\\"60ea033c62d2b044f596333cd464baa3\\\"', 'Accept-Ranges': 'bytes', 'Content-Type': 'text/csv', 'Content-Length': '1447', 'Server': 'AmazonS3'}\"}\n",
"{\"message\": \"Response headers: {'x-amz-id-2': 'OzI+sE1ZbetaaRTj+pWA6nPuNi0e1yrAUi1vKcvSShsXqP+LxiQTI5c3pV5y1pT6B4RR06dRBrg=', 'x-amz-request-id': '9763AE9849C42E10', 'Date': 'Fri, 14 Dec 2018 19:06:41 GMT', 'Last-Modified': 'Wed, 29 Nov 2017 18:48:35 GMT', 'ETag': '\\\"60ea033c62d2b044f596333cd464baa3\\\"', 'Accept-Ranges': 'bytes', 'Content-Type': 'text/csv', 'Content-Length': '1447', 'Server': 'AmazonS3'}\"}\n",
"{\"message\": \"Response body:\\n\"}\n",
"{\"message\": \"Response body:\\n\"}\n",
"{\"message\": \"Event needs-retry.s3.GetObject: calling handler \"}\n",
"{\"message\": \"Event needs-retry.s3.GetObject: calling handler \"}\n",
"{\"message\": \"No retry needed.\"}\n",
"{\"message\": \"No retry needed.\"}\n",
"{\"message\": \"Event needs-retry.s3.GetObject: calling handler >\"}\n",
"{\"message\": \"Event needs-retry.s3.GetObject: calling handler >\"}\n",
"{\"message\": \"IOWriteTask(transfer_id=0, {'offset': 0}) about to wait for the following futures []\"}\n",
"{\"message\": \"IOWriteTask(transfer_id=0, {'offset': 0}) about to wait for the following futures []\"}\n",
"{\"message\": \"IOWriteTask(transfer_id=0, {'offset': 0}) done waiting for dependent futures\"}\n",
"{\"message\": \"IOWriteTask(transfer_id=0, {'offset': 0}) done waiting for dependent futures\"}\n",
"{\"message\": \"Executing task IOWriteTask(transfer_id=0, {'offset': 0}) with kwargs {'fileobj': , 'offset': 0}\"}\n",
"{\"message\": \"Executing task IOWriteTask(transfer_id=0, {'offset': 0}) with kwargs {'fileobj': , 'offset': 0}\"}\n",
"{\"message\": \"IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/nba_2017_endorsement_full_stats.csv'}) about to wait for the following futures []\"}\n",
"{\"message\": \"IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/nba_2017_endorsement_full_stats.csv'}) about to wait for the following futures []\"}\n",
"{\"message\": \"IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/nba_2017_endorsement_full_stats.csv'}) done waiting for dependent futures\"}\n",
"{\"message\": \"IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/nba_2017_endorsement_full_stats.csv'}) done waiting for dependent futures\"}\n",
"{\"message\": \"Executing task IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/nba_2017_endorsement_full_stats.csv'}) with kwargs {'fileobj': , 'final_filename': '/tmp/nba_2017_endorsement_full_stats.csv', 'osutil': }\"}\n",
"{\"message\": \"Executing task IORenameFileTask(transfer_id=0, {'final_filename': '/tmp/nba_2017_endorsement_full_stats.csv'}) with kwargs {'fileobj': , 'final_filename': '/tmp/nba_2017_endorsement_full_stats.csv', 'osutil': }\"}\n",
"{\"message\": \"Releasing acquire 0/None\"}\n",
"{\"message\": \"Releasing acquire 0/None\"}\n"
],
"name": "stderr"
}
]
},
{
"metadata": {
"id": "3lK8io8Ufp6h",
"colab_type": "code",
"outputId": "5fdbcb47-ab69-487d-a1f9-5eb81398b4a5",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 85
}
},
"cell_type": "code",
"source": [
"!ls -l /tmp"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"total 4\n",
"srw------- 1 root root 0 Dec 14 16:11 drivefs_ipc.0\n",
"srw------- 1 root root 0 Dec 14 16:11 drivefs_ipc.0_shell\n",
"-rw-r--r-- 1 root root 1447 Dec 14 19:06 nba_2017_endorsement_full_stats.csv\n"
],
"name": "stdout"
}
]
},
{
"metadata": {
"id": "zsjgykvfftym",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"### Pandas"
]
},
{
"metadata": {
"id": "rfPwZDxhfx55",
"colab_type": "text"
},
"cell_type": "markdown",
"source": [
"Main Datascience library for AWS and Python\n",
"\n",
"\n",
"\n",
"* It is assumed you know about it\n",
"* Many study videos will show examples using it\n",
"\n"
]
},
{
"metadata": {
"id": "eR1y8H8ofwHP",
"colab_type": "code",
"outputId": "8f90bcc0-1732-4d12-c6f1-76f8c548ee20",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 182
}
},
"cell_type": "code",
"source": [
"import pandas as pd\n",
"\n",
"df = pd.read_csv(\"/tmp/nba_2017_endorsement_full_stats.csv\")\n",
"df.head(2)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" PLAYER | \n",
" TEAM | \n",
" SALARY_MILLIONS | \n",
" ENDORSEMENT_MILLIONS | \n",
" PCT_ATTENDANCE_STADIUM | \n",
" ATTENDANCE_TOTAL_BY_10K | \n",
" FRANCHISE_VALUE_100_MILLION | \n",
" ELO_100X | \n",
" CONF | \n",
" POSITION | \n",
" AGE | \n",
" MP | \n",
" GP | \n",
" MPG | \n",
" WINS_RPM | \n",
" PLAYER_TEAM_WINS | \n",
" WIKIPEDIA_PAGEVIEWS_10K | \n",
" TWITTER_FAVORITE_COUNT_1K | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" LeBron James | \n",
" Cleveland Cavaliers | \n",
" 30.96 | \n",
" 55.0 | \n",
" 100.0 | \n",
" 84.0 | \n",
" 12.0 | \n",
" 15.45 | \n",
" East | \n",
" SF | \n",
" 32 | \n",
" 37.8 | \n",
" 74.0 | \n",
" 37.8 | \n",
" 20.43 | \n",
" 51.0 | \n",
" 14.70 | \n",
" 5.53 | \n",
"
\n",
" \n",
" | 1 | \n",
" Kevin Durant | \n",
" Golden State Warriors | \n",
" 26.50 | \n",
" 36.0 | \n",
" 100.0 | \n",
" 80.0 | \n",
" 26.0 | \n",
" 17.70 | \n",
" West | \n",
" SF | \n",
" 28 | \n",
" 33.4 | \n",
" 62.0 | \n",
" 33.4 | \n",
" 12.24 | \n",
" 51.0 | \n",
" 6.29 | \n",
" 1.43 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" SALARY_MILLIONS | \n",
" ENDORSEMENT_MILLIONS | \n",
" PCT_ATTENDANCE_STADIUM | \n",
" ATTENDANCE_TOTAL_BY_10K | \n",
" FRANCHISE_VALUE_100_MILLION | \n",
" ELO_100X | \n",
" AGE | \n",
" MP | \n",
" GP | \n",
" MPG | \n",
" WINS_RPM | \n",
" PLAYER_TEAM_WINS | \n",
" WIKIPEDIA_PAGEVIEWS_10K | \n",
" TWITTER_FAVORITE_COUNT_1K | \n",
"
\n",
" \n",
" \n",
" \n",
" | count | \n",
" 10.000000 | \n",
" 10.000000 | \n",
" 10.000000 | \n",
" 10.000000 | \n",
" 10.000000 | \n",
" 10.000000 | \n",
" 10.000000 | \n",
" 10.000000 | \n",
" 10.000000 | \n",
" 10.000000 | \n",
" 10.000000 | \n",
" 10.000000 | \n",
" 10.000000 | \n",
" 10.000000 | \n",
"
\n",
" \n",
" | mean | \n",
" 23.216000 | \n",
" 21.700000 | \n",
" 99.800000 | \n",
" 80.200000 | \n",
" 21.375000 | \n",
" 15.678000 | \n",
" 29.300000 | \n",
" 33.890000 | \n",
" 70.800000 | \n",
" 33.890000 | \n",
" 11.506000 | \n",
" 44.100000 | \n",
" 6.532000 | \n",
" 2.764000 | \n",
"
\n",
" \n",
" | std | \n",
" 5.294438 | \n",
" 15.362653 | \n",
" 2.394438 | \n",
" 5.202563 | \n",
" 8.507554 | \n",
" 1.361142 | \n",
" 3.164034 | \n",
" 2.303837 | \n",
" 8.390471 | \n",
" 2.303837 | \n",
" 6.868487 | \n",
" 12.591443 | \n",
" 5.204233 | \n",
" 3.646399 | \n",
"
\n",
" \n",
" | min | \n",
" 12.110000 | \n",
" 8.000000 | \n",
" 94.000000 | \n",
" 70.000000 | \n",
" 10.250000 | \n",
" 13.740000 | \n",
" 24.000000 | \n",
" 29.900000 | \n",
" 60.000000 | \n",
" 29.900000 | \n",
" 1.170000 | \n",
" 26.000000 | \n",
" 2.690000 | \n",
" 0.350000 | \n",
"
\n",
" \n",
" | 25% | \n",
" 21.707500 | \n",
" 13.000000 | \n",
" 100.000000 | \n",
" 78.500000 | \n",
" 13.125000 | \n",
" 15.250000 | \n",
" 28.000000 | \n",
" 32.725000 | \n",
" 62.500000 | \n",
" 32.725000 | \n",
" 6.015000 | \n",
" 32.500000 | \n",
" 3.402500 | \n",
" 0.865000 | \n",
"
\n",
" \n",
" | 50% | \n",
" 23.880000 | \n",
" 14.500000 | \n",
" 100.000000 | \n",
" 80.500000 | \n",
" 22.500000 | \n",
" 15.450000 | \n",
" 28.000000 | \n",
" 33.850000 | \n",
" 73.000000 | \n",
" 33.850000 | \n",
" 12.860000 | \n",
" 46.500000 | \n",
" 4.475000 | \n",
" 1.485000 | \n",
"
\n",
" \n",
" | 75% | \n",
" 26.500000 | \n",
" 31.250000 | \n",
" 100.000000 | \n",
" 83.250000 | \n",
" 26.000000 | \n",
" 16.275000 | \n",
" 31.750000 | \n",
" 34.975000 | \n",
" 77.750000 | \n",
" 34.975000 | \n",
" 16.890000 | \n",
" 51.000000 | \n",
" 5.917500 | \n",
" 2.062500 | \n",
"
\n",
" \n",
" | max | \n",
" 30.960000 | \n",
" 55.000000 | \n",
" 104.000000 | \n",
" 89.000000 | \n",
" 33.000000 | \n",
" 17.700000 | \n",
" 35.000000 | \n",
" 37.800000 | \n",
" 81.000000 | \n",
" 37.800000 | \n",
" 20.430000 | \n",
" 65.000000 | \n",
" 17.570000 | \n",
" 12.280000 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" SentimentRaw | \n",
" Sentiment | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" I am very Angry | \n",
" NEGATIVE | \n",
"
\n",
" \n",
" | 1 | \n",
" We are very Happy | \n",
" POSITIVE | \n",
"
\n",
" \n",
" | 2 | \n",
" It is raining in Seattle | \n",
" NEUTRAL | \n",
"
\n",
" \n",
"
\n",
"