{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LJ1_xtISD2WL"
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/dbamman/anlp25/blob/main/5.classification/HW5_FeatureExploration.ipynb)\n",
    "\n",
    "**N.B.** Once it's open on Colab, remember to save a copy (by e.g. clicking `Copy to Drive` above).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature engineering for text classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bZD-F6iB63lv"
   },
   "source": [
    "This notebook explores feature engineering for text classification.  Your task is to create two new feature functions (like `dictionary_feature` and `unigram_feature` below), and include them in the `build_features` function.  What features do you think will help for your particular problem? Your grade is *not* tied to whether accuracy goes up or down, so be creative!  You are free to read in any other external resources you like (dictionaries, document metadata, etc.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ajlr1qNk63lx"
   },
   "source": [
    "You are free to use any of the following datasets for this exercise, or to use your own (if you have your own labeled data with at least 500 examples from at least two classes, I would encourage you to use it!).  If you use your own data, just be sure to format it like the examples below; each directory has a `train.tsv`, `dev.tsv` and `test.tsv` file, where each file is tab-separated (label in the first column and text in the second column).\n",
    "\n",
    "* [Sentiment Analysis](https://ai.stanford.edu/~amaas/data/sentiment/) (Positive/Negative)\n",
    "* [Congressional Speech](https://www.cs.cornell.edu/home/llee/data/convote.html) (Democrat/Republican)\n",
    "* Library of Congress Subject Classication ([21 categories](https://en.wikipedia.org/wiki/Library_of_Congress_Classification))\n",
    "\n",
    "For whichever dataset you pick, download the data first using the code below.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "HNh-QjKm67VB"
   },
   "outputs": [],
   "source": [
    "# get LMRD data\n",
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/refs/heads/main/data/lmrd/train.tsv -O lmrd_train.tsv\n",
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/refs/heads/main/data/lmrd/dev.tsv -O lmrd_dev.tsv\n",
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/refs/heads/main/data/lmrd/test.tsv -O lmrd_test.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fJd1C1cu7QD9"
   },
   "outputs": [],
   "source": [
    "# get Convote data\n",
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/refs/heads/main/data/convote/train.tsv -O convote_train.tsv\n",
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/refs/heads/main/data/convote/dev.tsv -O convote_dev.tsv\n",
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/refs/heads/main/data/convote/test.tsv -O convote_test.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "y2kamLx07T6x"
   },
   "outputs": [],
   "source": [
    "# get LoC data\n",
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/refs/heads/main/data/loc/train.tsv -O loc_train.tsv\n",
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/refs/heads/main/data/loc/dev.tsv -O loc_dev.tsv\n",
    "!wget https://raw.githubusercontent.com/dbamman/anlp25/refs/heads/main/data/loc/test.tsv -O loc_test.tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wh28crHR63ly"
   },
   "outputs": [],
   "source": [
    "import operator\n",
    "import sys\n",
    "from collections import Counter\n",
    "\n",
    "import nltk\n",
    "from nltk import word_tokenize\n",
    "from sklearn import linear_model, preprocessing\n",
    "\n",
    "nltk.download(\"punkt\")\n",
    "nltk.download(\"punkt_tab\")\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from scipy import sparse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HaiV9Dy763lx"
   },
   "source": [
    "## Part 1: Loading data\n",
    "\n",
    "**Q1: Briefly describe your data (including the categories you're predicting).**  If you're using your own data, tell us about it; if you're using one of the datasets above, tell us something that shows you've looked at the data. How many examples are in each category?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8kDz__uf63lz"
   },
   "outputs": [],
   "source": [
    "def read_data(filename):\n",
    "    df = pd.read_csv(filename, names=[\"label\", \"text\"], sep=\"\\t\")\n",
    "    return df.text.to_list(), df.label.to_list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ej8Ge3bR63lz"
   },
   "outputs": [],
   "source": [
    "# Change this to the directory with the data you will be using.\n",
    "# The directory should contain train.tsv, dev.tsv and test.tsv\n",
    "data = \"loc\"\n",
    "\n",
    "x_train, y_train = read_data(\"%s_train.tsv\" % data)\n",
    "x_dev, y_dev = read_data(\"%s_dev.tsv\" % data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Features\n",
    "\n",
    "Here, you will hand-engineer some features for your classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0l-v6WCl63l0"
   },
   "outputs": [],
   "source": [
    "## HELPER FUNCTIONS ##\n",
    "\n",
    "def majority_class(y_train, y_dev):\n",
    "    label_counts = Counter(y_train)\n",
    "    majority = label_counts.most_common(1)[0][0]\n",
    "\n",
    "    correct = 0.\n",
    "    for label in y_dev:\n",
    "        if label == majority:\n",
    "            correct += 1\n",
    "\n",
    "    print(\"%s\\t%.3f\" % (majority, correct/len(y_dev)))\n",
    "    return correct / len(y_dev)\n",
    "\n",
    "def build_features(x_train, feature_functions):\n",
    "    data = []\n",
    "    for doc in x_train:\n",
    "        feats = {}\n",
    "        tokens = doc.split(\" \")\n",
    "\n",
    "        for function in feature_functions:\n",
    "            feats.update(function(tokens))\n",
    "\n",
    "        data.append(feats)\n",
    "    return data\n",
    "\n",
    "# This helper function converts a dictionary of feature names to unique numerical ids\n",
    "def create_vocab(data):\n",
    "    feature_vocab = {}\n",
    "    idx = 0\n",
    "    for doc in data:\n",
    "        for feat in doc:\n",
    "            if feat not in feature_vocab:\n",
    "                feature_vocab[feat] = idx\n",
    "                idx += 1\n",
    "\n",
    "    return feature_vocab\n",
    "\n",
    "# This helper function converts a dictionary of feature names to a sparse representation\n",
    "# that we can fit in a scikit-learn model.  This is important because almost all feature\n",
    "# values will be 0 for most documents (note: why?), and we don't want to save them all in\n",
    "# memory.\n",
    "\n",
    "def features_to_ids(data, feature_vocab):\n",
    "    new_data = sparse.lil_matrix((len(data), len(feature_vocab)))\n",
    "    for idx,doc in enumerate(data):\n",
    "        for f in doc:\n",
    "            if f in feature_vocab:\n",
    "                new_data[idx, feature_vocab[f]] = doc[f]\n",
    "    return new_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sPtsXae_63l0"
   },
   "source": [
    "We'll start with two feature classes -- one feature class noting the presence of a word in an external dictionary, and one feature class for the word identity (i.e., unigram).  We'll implement each feature class as a function that takes a single document as input (as a list of tokens) and returns a dict corresponding to the feature we're creating."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "m7wpOrJ563l1"
   },
   "outputs": [],
   "source": [
    "# Here's a sample dictionary we can create by inspecting the output of the Mann-Whitney test (in 2.compare/)\n",
    "\n",
    "# EDIT TO FIT YOUR DATASET\n",
    "dem_dictionary = set([\"republican\",\"cut\", \"opposition\"])\n",
    "repub_dictionary = set([\"growth\",\"economy\"])\n",
    "\n",
    "def political_dictionary_feature(tokens):\n",
    "    feats = {}\n",
    "    for word in tokens:\n",
    "        if word in dem_dictionary:\n",
    "            feats[\"word_in_dem_dictionary\"] = 1\n",
    "        if word in repub_dictionary:\n",
    "            feats[\"word_in_repub_dictionary\"] = 1\n",
    "    return feats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wtO-foMj63l1"
   },
   "outputs": [],
   "source": [
    "def unigram_feature(tokens):\n",
    "    feats = {}\n",
    "    for word in tokens:\n",
    "        feats[\"UNIGRAM_%s\" % word] = 1\n",
    "    return feats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "r7yZfUkU63l1"
   },
   "source": [
    "**Q2**: **Add first new feature function here.**  Describe your feature and why you think it will help."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PAw7PO_w63l1"
   },
   "outputs": [],
   "source": [
    "def new_feature_class_one(tokens):\n",
    "    feats={}\n",
    "    feats[\"_FILL_IN_FEATURES_HERE_\"]=1\n",
    "    return feats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1GLI_RtJ63l2"
   },
   "source": [
    "**Q3**: **Add second new feature function here.** Describe your feature and why you think it will help."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jNcqQIx863l2"
   },
   "outputs": [],
   "source": [
    "def new_feature_class_two(tokens):\n",
    "    feats={}\n",
    "    feats[\"_FILL_IN_FEATURES_HERE_\"]=1\n",
    "    return feats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "07dK4NPp63l2"
   },
   "source": [
    "We use the `build_features` helper function to aggregate together all of the information from different feature classes.  Each document has a feature dict (`feats`), and we'll update that dict with the new dict that each separate feature class is returning.  (Here you want to make sure that the keys each feature function is creating are unique so they don't get clobbered by other functions)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RMh8GzFm63l2"
   },
   "outputs": [],
   "source": [
    "# This function trains a model and returns the predicted and true labels for test data\n",
    "def evaluate(x_train, x_dev, y_train, y_dev, feature_functions):\n",
    "    x_train_feat = build_features(x_train, feature_functions)\n",
    "    x_dev_feat = build_features(x_dev, feature_functions)\n",
    "\n",
    "    # just create vocabulary from features in *training* data\n",
    "    feature_vocab = create_vocab(x_train_feat)\n",
    "\n",
    "    x_train_ids = features_to_ids(x_train_feat, feature_vocab)\n",
    "    x_dev_ids = features_to_ids(x_dev_feat, feature_vocab)\n",
    "\n",
    "    logreg = linear_model.LogisticRegression(C=1.0, solver='lbfgs', penalty='l2', max_iter=10000)\n",
    "    logreg.fit(x_train_ids, y_train)\n",
    "    predictions = logreg.predict(x_dev_ids)\n",
    "    return (predictions, y_dev)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zH7bPWTA63l3"
   },
   "outputs": [],
   "source": [
    "def print_weights(clf, vocab, n=10):\n",
    "    reverse_vocab = [None]*len(clf.coef_[0])\n",
    "    for k in vocab:\n",
    "        reverse_vocab[vocab[k]] = k\n",
    "\n",
    "    if len(clf.classes_) == 2:\n",
    "\n",
    "        weights=clf.coef_[0]\n",
    "        for feature, weight in sorted(zip(reverse_vocab, weights), key = operator.itemgetter(1))[:n]:\n",
    "            print(\"%.3f\\t%s\" % (weight, feature))\n",
    "\n",
    "        print()\n",
    "\n",
    "        for feature, weight in list(reversed(sorted(zip(reverse_vocab, weights), key = operator.itemgetter(1))))[:n]:\n",
    "            print(\"%.3f\\t%s\" % (weight, feature))\n",
    "\n",
    "    else:\n",
    "        for i, cat in enumerate(clf.classes_):\n",
    "\n",
    "            weights=clf.coef_[i]\n",
    "\n",
    "            for feature, weight in list(reversed(sorted(zip(reverse_vocab, weights), key = operator.itemgetter(1))))[:n]:\n",
    "                print(\"%s\\t%.3f\\t%s\" % (cat, weight, feature))\n",
    "            print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CFfW35rB63l3"
   },
   "outputs": [],
   "source": [
    "majority_class(y_train,y_dev)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Qs9jWjaT63l3"
   },
   "source": [
    "Explore the impact of different feature functions by evaluating them below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "u2ci2NKi63l3"
   },
   "outputs": [],
   "source": [
    "features = [unigram_feature]\n",
    "clf, vocab = pipeline(x_train, x_dev, y_train, y_dev, features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lXsGfRng63l3"
   },
   "source": [
    "If you want to print the coefficients for any of the models you train, you can do so like this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Pzzj7s4Q63l3"
   },
   "outputs": [],
   "source": [
    "print_weights(clf, vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LVwYUGla63l4"
   },
   "source": [
    "## Part 3: Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q4**: Implement a function that returns the parametric confidence interval bounds for a binomial estimator of the model accuracy. It should return a tuple of floats `(lower_bound, upper_bound)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def binomial_test(predictions, targets, significance_level=0.95):\n",
    "    # YOUR CODE HERE\n",
    "    upper_bound = 0.0\n",
    "    lower_bound = 0.0\n",
    "    return (lower_bound, upper_bound)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Q5**: Plot the performance for models trained with different combinations of features, including your two custom features. Some combinations you might try (but feel free to pick your own!):\n",
    "1. Just the dictionary features\n",
    "2. Just the unigram features\n",
    "3. Just your custom features\n",
    "4. Unigram features + custom features\n",
    "\n",
    "Make a bar plot with confidence intervals. Does incorporating your features result in a statistically significant change in performance?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jDKnZ8V263l3"
   },
   "outputs": [],
   "source": [
    "# example of how to train a classifier with the dictionary and unigram features\n",
    "features = [political_dictionary_feature, unigram_feature]\n",
    "predictions, targets = evaluate(x_train, x_dev, y_train, y_dev, features)\n",
    "lower_bound, upper_bound = binomial_test(predictions, targets)\n",
    "\n",
    "# YOUR CODE TO EVALUATE MULTIPLE MODELS HERE\n",
    "\n",
    "# YOUR CODE TO GENERATE PLOT HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lFfAJs8FEDc-"
   },
   "source": [
    "---\n",
    "\n",
    "## To submit\n",
    "\n",
    "Congratulations on finishing this homework!\n",
    "Please follow the instructions below to download the notebook file (`.ipynb`) and its printed version (`.pdf`) for submission on bCourses -- remember **all cells must be executed**.\n",
    "\n",
    "1.  Download a copy of the notebook file: `File > Download > Download .ipynb`.\n",
    "\n",
    "2.  Print the notebook as PDF (via your browser, or tools like [nbconvert](https://nbconvert.readthedocs.io/en/latest/))."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}