{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Logistic Regression\n",
"\n",
"In this exercise, we will implement a logistic regression for binary classification."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Model Function\n",
"\n",
"Given a multiset of examples $D = \\{(\\mathbf{x}_1, c_1), ..., (\\mathbf{x}_n, c_n)\\}$ (the dataset), our learning task is to predict a class output under a logistic model function\n",
"\n",
"$$ y(x) = \\sigma(\\mathbf{w}^T\\mathbf{x}),$$\n",
"\n",
"where $\\sigma$ is the sigmoid function\n",
"\n",
"$$\\sigma(z) = \\frac{1}{1 + e^{-z}}.$$\n",
"\n",
"The sigmoid function maps the space of real numbers to the $(0,1)$ interval, which allows us to interpret the output of the model function as probability for the event $\\mathbf{C} = 1$.\n",
"\n",
"**Exercise**: implement the sigmoid function and the model function.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def sigmoid(z):\n",
" return # TODO"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def model(w, X):\n",
" return # TODO"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Loss\n",
"\n",
"The loss function quantifies the error made by the model. Importantly, the loss has to be derivable, so we cannot simply use accuracy here. Instead, we rely on the pointwise logistic loss:\n",
"\n",
"$$\n",
"l_{\\sigma}(c, y(\\mathbf{x})) = \\begin{cases} -\\log(y(\\mathbf{x})) & \\text{ if } c = 1 \\\\ -\\log(1 - y(\\mathbf{x})) & \\text{ if } c = 0 \\end{cases}.\n",
"$$\n",
"\n",
"Please refer to the lecture slides for the derivation of this loss function.\n",
"\n",
"**Exercise**: implement the pointwise logistic loss for binary classification."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def loss(y, c):\n",
" \"\"\"\n",
" Pointwise logistic loss\n",
"\n",
" :param y: prediction\n",
" :param c: ground-truth label\n",
" :return: pointwise logistic loss\n",
" \"\"\"\n",
" return # TODO"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Convergence\n",
"\n",
"The last important part needed to assemble the training loop is the convergence function. As stated in the lecture slides, the function `convergence` analyses the norm of the loss gradient,\n",
"\n",
"$$||\\nabla L_\\sigma(\\mathbf{x}_t)|| = ||\\sum_{(\\mathbf{x},c)\\in D}(c-y_t(\\mathbf{x}))\\cdot\\mathbf{x}||,$$\n",
"\n",
"and compares it to a small positive bound $\\epsilon$. In addition, it may check if an upper bound for the number of iterations is reached. In comparison to the lecture, we simplify the implementation by calculating the pointwise loss gradient for each example individually, and then use the inner training loop to sum over all examples.\n",
"\n",
"This means that the convergence function for a single example will calculate $(c-y_t(\\mathbf{x}))\\cdot\\mathbf{x}$ only.\n",
"\n",
"\n",
"\n",
"**Exercise**: implement the convergence function for a single example.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def convergence(x, c, y):\n",
" \"\"\"\n",
" Convergence function for a single example.\n",
"\n",
" :param x: the feature vector for the example\n",
" :param c: the ground-truth class of the example\n",
" :param y: the model prediction for the example\n",
" :return: the pointwise direction of the steepest loss descent\n",
" \"\"\"\n",
" return # TODO"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Batch Gradient Descent\n",
"\n",
"In order to fit our model, we rely on the logistic Batch Gradient Descent (BGD$_{\\sigma}$) algorithm.\n",
"\n",
"**Exercise**: implement the BGD$_{\\sigma}$ algorithm from the lecture. Some code is already filled in.\n",
"\n",
"*Remarks*:\n",
" - calculate the global logistic loss at every iteration and save the sum of loss values in an array; we want to visualize the loss after training, so it should be returned together with the optimal weights.\n",
" - calculate the convergence for each example using the function defined before and sum up their individual contributions; decide after the inner training loop whether convergence is reached or not. Remember to calculate the norm of the summed convergence values!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def bgd(D, eta: float, eps: float, max_iter: int = 1000):\n",
" \"\"\"\n",
" Batch Gradient Descent\n",
"\n",
" :param D: multiset of examples (x, c), where x is a training vector and c is a class in {0, 1}.\n",
" :param eta: learning rate, a small positive constant\n",
" :param eps: epsilon value for convergence\n",
" :param max_iter: maximum number of training iterations (epochs)\n",
" :return: w (weight vector), l (loss vector)\n",
" \"\"\"\n",
" # Add bias dimension to data, inserting an extra 1 to every feature array\n",
" D = [(np.hstack([1, x]), c) for (x, c) in D]\n",
" # Initialize random w vector with random values + bias\n",
" w = (np.random.random(size=D[0][0].shape))\n",
" # Initialize an array to keep track of the loss\n",
" l = np.zeros(shape=max_iter, dtype=np.float64)\n",
"\n",
" # Training loop\n",
" t = 0\n",
" while True:\n",
" pass # TODO\n",
"\n",
" # Return final weights and loss history (capped to current time step)\n",
" return w, l[:t]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Predictions\n",
"\n",
"Using BGD, we can infer optimal weights for our model function. Yet, the output of the model function is the *probability* for each class, not the actual class label. Therefore, we need a `predict` function that takes a (batch of) training example(s), and returns the most probable class label. It is defined as follows:\n",
"\n",
"$$\n",
"\\text{predict}(\\mathbf{x}) = \\begin{cases}1 & \\text{ if } y(\\mathbf{x}) >= 0.5 \\\\ 0 & \\text{ if } y(\\mathbf{x}) < 0.5\\end{cases}\n",
"\n",
"$$\n",
"\n",
"**Exercise**: implement a `predict` function.\n",
"\n",
"*Remarks*: remeber to also insert the bias value to every feature vector here!"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def predict(X, w):\n",
" \"\"\"\n",
" Predict class labels for given data.\n",
"\n",
" :param X: data vector\n",
" :param w: weight vector\n",
" :return: predicted 0/1 class labels\n",
" \"\"\"\n",
" return # TODO"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Example Classification Task\n",
"\n",
"To test the capabilities of the model, we conduct a sample classification task. The goal is to classify different wines (described by different features such as *Alcohol*, *Malic Acid*, *Magnesium*, etc.) into categories. Below, the dataset is loaded and the associated description with more info on its content is printed."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.datasets import load_wine\n",
"data = load_wine()\n",
"print(data.DESCR)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Load the feature vectors X\n",
"X = data.data\n",
"# Load the class labels C\n",
"C = data.target"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Initially, wines are divided into three classes; to achieve a binary setting, we simply collapse label 2 and 3.\n",
"\n",
"**Exercise**: modify the class labels such that class 2 and 3 (labels 1 and 2) are both represented by label 1."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"C = # TODO"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Investigating the value ranges of features, we can observe that they are distributed extremely different: while some features are in the 0-1 range, feature 13 goes up to 1680."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pd.DataFrame(np.array([X.min(axis=0), X.max(axis=0)]).T, columns=[\"Minimum\", \"Maximum\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This warrants *feature scaling*. Intuitively, if we have incompatible ranges, features with larger ranges will dominate. Therefore, we apply *z-Standardization*, which means that every feature is rescaled to have a distribution of mean 0 and standard deviation of 1. We can use the existing implementation from `sklearn`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.preprocessing import StandardScaler\n",
"\n",
"X = StandardScaler().fit_transform(X)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, all features have compatible ranges:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pd.DataFrame(np.array([X.min(axis=0), X.max(axis=0)]).T, columns=[\"Minimum\", \"Maximum\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Exercise**: assemble $D$ and fit the model using $BGD_{\\sigma}$.\n",
"\n",
"*Remarks*:\n",
"- use 0.0001 as learning rate\n",
"- use 40 as epsilon\n",
"- use 1000 max iterations"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# TODO"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Exercise**: predict classes for every sample in $C$. Evaluate your result using either the `sklearn` evaluation metrics, or your own implementation from the last exercise. Play around with the model parameters and see if you can improve the results."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# TODO"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.6"
}
},
"nbformat": 4,
"nbformat_minor": 1
}