{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dbamman/anlp25/blob/main/11.nlp/HW10_SyntacticRelations.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LRYkjH_3dljL"
   },
   "source": [
    "# HW10: Exploring gender in books\n",
    "\n",
    "This notebook explores dependency parsing by identifying the actions and objects that are characteristically associated with characters as a function of their referential gender (\"he\"/\"she\")."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "m4OX0TrLdljO"
   },
   "outputs": [],
   "source": [
    "import math\n",
    "import operator\n",
    "\n",
    "from collections import Counter\n",
    "\n",
    "import spacy\n",
    "from tqdm import tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8Iv37AaPdljQ"
   },
   "outputs": [],
   "source": [
    "nlp = spacy.load('en_core_web_sm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "l5CKftUPdljQ"
   },
   "source": [
    "We'll run seven novels by Jane Austen through spaCy (this will take a few minutes)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "uBFrBn1wd2wj"
   },
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/refs/heads/main/data/fiction/emma.txt\n",
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/refs/heads/main/data/fiction/lady_susan.txt\n",
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/refs/heads/main/data/fiction/mansfield_park.txt\n",
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/refs/heads/main/data/fiction/northanger_abbey.txt\n",
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/refs/heads/main/data/fiction/persuasion.txt\n",
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/refs/heads/main/data/fiction/pride.txt\n",
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/refs/heads/main/data/fiction/sense_and_sensibility.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "nyq6Xhx6dljQ"
   },
   "outputs": [],
   "source": [
    "files = [\"emma.txt\", \"lady_susan.txt\", \"mansfield_park.txt\", \"northanger_abbey.txt\", \"persuasion.txt\", \"pride.txt\", \"sense_and_sensibility.txt\"]\n",
    "\n",
    "def read_all_files(filenames):\n",
    "    all_tokens = []\n",
    "\n",
    "    for filename in tqdm(filenames):\n",
    "        data = open(filename, encoding=\"utf-8\").read()\n",
    "        tokens = nlp(data)\n",
    "        all_tokens.extend(tokens)\n",
    "    return all_tokens\n",
    "\n",
    "all_tokens = read_all_files(files)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MOIwxtVydljR"
   },
   "outputs": [],
   "source": [
    "print(len(all_tokens))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up log odds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "FmCkFJy6dljR"
   },
   "outputs": [],
   "source": [
    "def logodds(counter1, counter2, display=25):\n",
    "    \"\"\"\n",
    "    Function that takes two Counter objects as inputs and prints out a ranked list of terms\n",
    "    more characteristic of the first counter than the second.  Here we'll use log-odds\n",
    "    with an uninformative prior (from Monroe et al 2008, \"Fightin Words\", eqn. 22) as our metric.\n",
    "\n",
    "    \"Category 1\" corresponds to the category of the counter1 object (the first argument)\n",
    "    \"Category 2\" corresponds to the category of the counter2 object (the second argument)\n",
    "    \"\"\"\n",
    "    vocab=dict(counter1)\n",
    "    vocab.update(dict(counter2))\n",
    "    count1_sum=sum(counter1.values())\n",
    "    count2_sum=sum(counter2.values())\n",
    "\n",
    "    ranks={}\n",
    "    alpha=0.01\n",
    "    alphaV=len(vocab)*alpha\n",
    "\n",
    "    for word in vocab:\n",
    "\n",
    "        log_odds_ratio=math.log( (counter1[word] + alpha) / (count1_sum+alphaV-counter1[word]-alpha) ) - math.log( (counter2[word] + alpha) / (count2_sum+alphaV-counter2[word]-alpha) )\n",
    "        variance=1./(counter1[word] + alpha) + 1./(counter2[word] + alpha)\n",
    "\n",
    "        ranks[word]=log_odds_ratio/math.sqrt(variance)\n",
    "\n",
    "    sorted_x = sorted(ranks.items(), key=operator.itemgetter(1), reverse=True)\n",
    "\n",
    "    print(\"Most category 1:\")\n",
    "    for k,v in sorted_x[:display]:\n",
    "        print(\"%.3f\\t%s\" % (v,k))\n",
    "\n",
    "    print(\"\\nMost category 2:\")\n",
    "    for k,v in reversed(sorted_x[-display:]):\n",
    "        print(\"%.3f\\t%s\" % (v,k))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dependency parsing with SpaCy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "njbslDssdljS"
   },
   "source": [
    "SpaCy uses the [ClearNLP dependency labels](https://github.com/clir/clearnlp-guidelines/blob/master/md/specifications/dependency_labels.md), which are very close to the Stanford typed dependencies.  See the [Stanford dependencies manual](http://people.ischool.berkeley.edu/~dbamman/DependencyManual.pdf) for more information about each tag.  Parse information is contained in the spacy token object; see the following for which attributes encode the token text, idx (position in sentence), part of speech, and dependency relation.  The syntactic head for a token is another token given in `token.head` (where all of those same token attributes are accessible)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "DORmDvD6dljS"
   },
   "outputs": [],
   "source": [
    "test_doc = nlp(\"He started his car.\")\n",
    "for token in test_doc:\n",
    "    print(\"\\t\".join(str(x) for x in [token.text, token.idx, token.tag_, token.dep_, token.head.text, token.head.idx, token.head.tag_]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E9AS8IOBdljT"
   },
   "source": [
    "**Q1**. Find the verbs that men are more characteristically the *subject* of than women.  Feel free to only consider subjects that are \"he\" and \"she\" pronouns.  This function should return two Counter objects (`he_counter` and `she_counter`) which counts the number of times a given verb has \"he\" (`he_counter`) and \"she\" (`she_counter`) as its syntactic subject."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "V-f82SpWdljT"
   },
   "outputs": [],
   "source": [
    "def count_subjects(tokens):\n",
    "    he_counter = Counter()\n",
    "    she_counter = Counter()\n",
    "\n",
    "    # your code here\n",
    "\n",
    "    return he_counter, she_counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "06FTwdZodljT"
   },
   "outputs": [],
   "source": [
    "he_counts, she_counts = count_subjects(all_tokens)\n",
    "logodds(he_counts, she_counts, display=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8WCoKEwQdljT"
   },
   "source": [
    "**Q2**. Find the verbs that men are more characteristically the *object* of than women.  Feel free to only consider objects that are \"him\" and \"her\" pronouns.  This function should return two Counter objects (`he_counter` and `she_counter`) which counts the number of times a given verb has \"him\" (`he_counter`) and \"her\" (`she_counter`) as its syntactic direct object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_P4VwJPGdljT"
   },
   "outputs": [],
   "source": [
    "def count_objects(tokens):\n",
    "    he_counter=Counter()\n",
    "    she_counter=Counter()\n",
    "\n",
    "    # your code here\n",
    "            \n",
    "    return he_counter, she_counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hIW8zRP2dljT",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "he_counts, she_counts = count_objects(all_tokens)\n",
    "logodds(he_counts, she_counts, display=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vvkOo1ymdljU"
   },
   "source": [
    "**Q3**. Find the objects that are *possessed* more frequently by men than women. Feel free to only consider possessors that are \"his\" and \"her\" pronouns.   This function should return two Counter objects (`he_counter` and `she_counter`) which counts the number of times a given term is possessed by \"he\" (`he_counter`) and \"she\" (`she_counter`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "y4cIrhFFdljU"
   },
   "outputs": [],
   "source": [
    "def count_possessions(tokens):\n",
    "    he_counter=Counter()\n",
    "    she_counter=Counter()\n",
    "\n",
    "    # your code here\n",
    "\n",
    "\n",
    "    return he_counter, she_counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hxhK_F1ZdljU"
   },
   "outputs": [],
   "source": [
    "he_counts, she_counts = count_possessions(all_tokens)\n",
    "logodds(he_counts, she_counts, display=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FLU0UpOzdljU"
   },
   "source": [
    "**Q4**. Find the actions that are men do *to women* more frequently than women do *to men*.  Feel free to only consider subjects and objects that are \"she\"/\"he\"/\"her\"/\"him\" pronouns.   This function should return two Counter objects (`he_counter` and `she_counter`) which counts the number of times a given verb has \"he\" as the subject and \"her\" as the object (`he_counter`) and \"she\" as the subject and \"him\" as the object (`she_counter`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "dTn1A8WgdljU"
   },
   "outputs": [],
   "source": [
    "def count_SVO_tuples(tokens):\n",
    "    he_counter=Counter()\n",
    "    she_counter=Counter()\n",
    "\n",
    "    # your code here\n",
    "\n",
    "    return he_counter, she_counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QozyJ4eDdljV"
   },
   "outputs": [],
   "source": [
    "he_counts, she_counts = count_SVO_tuples(all_tokens)\n",
    "logodds(he_counts, she_counts, display=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q5**. **In a few sentences,** reflect on the analysis you did above. What claims can you make about the data? What are some limitations?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}