{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ya6eZCJKdVGu"
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dbamman/anlp25/blob/main/3.embeddings/SemAxis_TODO.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CDfWc5fJbz_v"
   },
   "source": [
    "[SemAxis](https://arxiv.org/pdf/1806.05521.pdf) is a method for scoring terms along a user-defined axis (e.g., positive-negative, concrete-abstract, hot-cold), which can be used for a range of empirical questions (for one example, see [Kozlowski et al. 2019](https://journals.sagepub.com/doi/full/10.1177/0003122419877135)). In this activity, you'll implement SemAxis using word representations from Glove, and use it to explore corpus-specific conceptual associations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KK-AAbkAbz_3"
   },
   "outputs": [],
   "source": [
    "import re\n",
    "from gensim.models import KeyedVectors\n",
    "import numpy as np\n",
    "import numpy.linalg as LA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7ZKdlpmycLqT"
   },
   "outputs": [],
   "source": [
    "!wget https://github.com/dbamman/anlp25/raw/main/data/glove.6B.100d.100K.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AT2Mh2OEbz_5"
   },
   "source": [
    "In this activity, we'll be working with pre-trained word embeddings using the `gensim` library, which provides a number of functions for accessing representations for individual words and comparing them.  The representations we'll use come from [Glove](https://nlp.stanford.edu/projects/glove/), which are trained on web data from the [Common Crawl](https://en.wikipedia.org/wiki/Common_Crawl) corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "dnS-JjiJbz_5"
   },
   "outputs": [],
   "source": [
    "glove = KeyedVectors.load_word2vec_format(\"glove.6B.100d.100K.txt\", binary=False, no_header=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "FFYDU2tObz_6"
   },
   "outputs": [],
   "source": [
    "good_vector = glove[\"good\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "40b5FRppbz_6"
   },
   "outputs": [],
   "source": [
    "print(good_vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OoquxV40bz_6"
   },
   "source": [
    "Functions useful for this activity include the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "q0WThnF5bz_7"
   },
   "outputs": [],
   "source": [
    "# access the representation for a single word\n",
    "great_vector = glove[\"great\"]\n",
    "\n",
    "# use numpy to average multiple vector representations together\n",
    "vecs_to_average = [good_vector, great_vector]\n",
    "average = np.mean(vecs_to_average, axis=0)\n",
    "\n",
    "# calculate the cosine similariy between two vectors\n",
    "cosine_similarity = glove.cosine_similarities(good_vector, [great_vector])\n",
    "\n",
    "print(good_vector.shape, great_vector.shape, average.shape, cosine_similarity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "H0zMiyECbz_7"
   },
   "source": [
    "Implement the [SemAxis](https://arxiv.org/pdf/1806.05521.pdf) method as described in class. Given a set of word embeddings for positive terms $S^+ = \\{v_1^+, \\ldots v_n^+\\}$ and embeddings for negative terms $S^- = \\{v_1^-, \\ldots v_n^-\\}$ that define the endpoints of the axis, your output should be a single real-value score for an input word $w$ with word representation $v_w$:\n",
    "\n",
    "$$\n",
    "score(w)_{\\mathbf{V_\\textrm{axis}}} = \\textrm{cos}(v_w, \\mathbf{V}_\\textrm{axis})\n",
    "$$\n",
    "\n",
    "Where:\n",
    "$$\n",
    "\\mathbf{V}^+ = {1 \\over n} \\sum_1^n v_i^+\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\mathbf{V}^- = {1 \\over m} \\sum_1^m v_i^-\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\mathbf{V}_{\\textrm{axis}} = \\mathbf{V}^+ - \\mathbf{V}^-\n",
    "$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QLlvpEOebz_7"
   },
   "outputs": [],
   "source": [
    "def get_semaxis_score(vectors, positive_terms=None, negative_terms=None, target_word=None):\n",
    "\n",
    "    # See cell below for example arguments\n",
    "\n",
    "    # vectors: gensim KeyedVectors object (e.g., glove)\n",
    "    # positive_terms: list of terms defining one end of an axis\n",
    "    # negative_terms: list of terms defining the other end of an axis\n",
    "    # target_word: the term to score along that axis\n",
    "\n",
    "    # the output should be a single real number (the SemAxis score for that word).\n",
    "\n",
    "    # TODO\n",
    "\n",
    "    return score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "yxofH0orbz_8"
   },
   "outputs": [],
   "source": [
    "# should be 0.342\n",
    "get_semaxis_score(glove, positive_terms=[\"woman\", \"women\"], negative_terms=[\"man\", \"men\"], target_word=\"actress\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aygtLV0Nbz_8"
   },
   "source": [
    "Now let's score a set of target terms along that axis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OCEVNKZ_bz_8"
   },
   "outputs": [],
   "source": [
    "def score_list_of_targets(vectors, positive_terms=None, negative_terms=None, target_words=None):\n",
    "    scores=[]\n",
    "    for target in target_words:\n",
    "        scores.append((get_semaxis_score(vectors, positive_terms, negative_terms, target), target))\n",
    "\n",
    "    for k,v in reversed(sorted(scores)):\n",
    "        print(\"%.3f\\t%s\" % (k,v))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "eyDM6G7_bz_8"
   },
   "outputs": [],
   "source": [
    "targets = [\"doctor\", \"nurse\", \"actor\", \"actress\", \"mechanic\", \"librarian\", \"architect\", \"magician\", \"cook\", \"chef\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3_JD7BuNbz_8"
   },
   "outputs": [],
   "source": [
    "score_list_of_targets(glove, positive_terms=[\"woman\", \"women\"], negative_terms=[\"man\", \"men\"], target_words=targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Pe-B1nffbz_9"
   },
   "source": [
    "Define **your own concept axis** by selecting a set of positive and negative terms and illustrate its utility by scoring a set of 10 target terms (as we did above).  If you've implemented  `get_semaxis_score` above, you only need to add terms to the `positive_terms` and `negative_terms` lists below and execute this cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3wxx8bejbz_9"
   },
   "outputs": [],
   "source": [
    "positive_terms = []\n",
    "negative_terms = []\n",
    "targets = []\n",
    "\n",
    "score_list_of_targets(glove, positive_terms=positive_terms, negative_terms=negative_terms, target_words=targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HDikgTS6bz_9"
   },
   "source": [
    "Let's assume now that you're able to score all words in a vocabulary along several conceptual dimensions (like the one you've defined) for a given set of word embeddings trained on a dataset.  What could you do with that score? Brainstorm possible applications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LYlF7ctZbz_9"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}