{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dbamman/anlp25/blob/main/1.words/HW1_Tokenization.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework 1: Tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bv5qAYY2vxLH"
   },
   "source": [
    "In this homework, you'll compare the tokenizations outputs from different classes of tokenizers. This homework is also an opportunity for you to check in on your Python proficiency; for all of the operations below (downloading a file, reading it in, counting objects), you should either be comfortable implementing them already or know how to find out how to do so yourself (if you find yourself struggling with them, we encourage you to take this class at a later date, with a bit more Python experience under your belt).\n",
    "\n",
    "We've added some space for you to write the code for each section, but feel free to create more code cells if you'd like."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_uzG_NzfvK60"
   },
   "source": [
    "## Part 1\n",
    "\n",
    "Tokenize the following document with each of these models. Feel free to use the documentation linked (and AI Assistance) to do so for this low-level operation (but again remember that you have to be able to explain what your code is doing).  For each of the tokenizers above, we want to see a list of tokens for this document (not numeric token IDs, but legible words) -- e.g., \\[\"London\", \".\", ...\\]\n",
    "\n",
    "* NLTK `word_tokenize` (https://www.nltk.org/book/ch03.html)\n",
    "* Spacy `tokenize` (https://spacy.io/usage/spacy-101#annotations-token)\n",
    "* Tiktoken BPE tokenization (https://github.com/openai/tiktoken) -- cl100k_base (GPT-3.5, GPT-4).\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 4,
     "status": "ok",
     "timestamp": 1725398235106,
     "user": {
      "displayName": "David Alexander Bamman",
      "userId": "09859182337762586004"
     },
     "user_tz": 420
    },
    "id": "9ywLVgL3v5sy"
   },
   "outputs": [],
   "source": [
    "document = \"London. Michaelmas term lately over, and the Lord Chancellor sitting in Lincoln’s Inn Hall. Implacable November weather. As much mud in the streets as if the waters had but newly retired from the face of the earth, and it would not be wonderful to meet a Megalosaurus, forty feet long or so, waddling like an elephantine lizard up Holborn Hill. Smoke lowering down from chimney-pots, making a soft black drizzle, with flakes of soot in it as big as full-grown snowflakes—gone into mourning, one might imagine, for the death of the sun. Dogs, undistinguishable in mire. Horses, scarcely better; splashed to their very blinkers. Foot passengers, jostling one another’s umbrellas in a general infection of ill temper, and losing their foot-hold at street-corners, where tens of thousands of other foot passengers have been slipping and sliding since the day broke (if this day ever broke), adding new deposits to the crust upon crust of mud, sticking at those points tenaciously to the pavement, and accumulating at compound interest.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vbhjqG9m05yS"
   },
   "source": [
    "## Part 2\n",
    "\n",
    "Examine the different tokenizations for the passage above -- i.e., actually read through them and see how they differ. In a paragraph or two, characterize the salient differences in tokenization between a.) NLTK and Spacy and b.) NLTK and BPE.  Reference real examples in the text. At the end of this homework, you want to be able to discuss the practical differences between tokenization methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Response**:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your response here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pwhNqD61uZi4"
   },
   "source": [
    "## Part 3\n",
    "\n",
    "Download the full text of *Pride and Prejudice* (https://raw.githubusercontent.com/dbamman/anlp25/main/data/1342_pride_and_prejudice.txt) and tokenize it using each of the methods above. How many word types (in the formal sense we discussed in class) does each tokenization method have for that complete file?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2vf1sRWp3LOb"
   },
   "source": [
    "## Part 4\n",
    "\n",
    "Which text has the greater type-token ratio, *Pride and Prejudice* (https://raw.githubusercontent.com/dbamman/anlp25/main/data/1342_pride_and_prejudice.txt) or *Emma* (https://raw.githubusercontent.com/dbamman/anlp25/main/data/158_emma.txt)?  Calculate the TTR for both texts using the NLTK tokenizer, but only use the first 1,000 tokens from each text when calculating its TTR."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code here:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pp_ttr = 0.0  # fill this in!\n",
    "emma_ttr = 0.0  # fill this in!\n",
    "answer = \"???\"  # fill this in!\n",
    "\n",
    "print(\"The TTR for 'Pride and Prejudice' is\", pp_ttr)\n",
    "print(\"The TTR for 'Emma' is\", emma_ttr)\n",
    "print(f\"{answer} has the higher TTR.\")"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "authorship_tag": "ABX9TyPXrkC/YsLxJHVPOh+p9SFa",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}