{ "cells": [ { "cell_type": "markdown", "id": "34e35d21", "metadata": {}, "source": [ "# Module 6 — Performance, I/O & Mini-Projects\n" ] }, { "cell_type": "markdown", "id": "e44c33ff", "metadata": {}, "source": [ "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/inigoochoa/comprehensive-numpy/blob/main/module_6_performance_projects.ipynb)\n" ] }, { "cell_type": "markdown", "id": "2c072f0f", "metadata": {}, "source": [ "## Welcome\n", "\n", "This is the last module of the course. The first four lessons round out\n", "your mental model of NumPy's performance characteristics — memory layout,\n", "vectorization patterns, and I/O — and the last two put everything you've\n", "learned in Modules 1-5 to work in two end-to-end mini-projects.\n", "\n", "**Learning goals**\n", "- Read `.strides` and distinguish C order from Fortran order\n", "- Avoid hidden costs: in-place ops with `out=`, preallocation vs growing\n", "- Save/load arrays with `.npy`/`.npz`/text formats (round-tripped in memory)\n", "- Build and sort structured arrays with named fields\n", "- Apply the whole course to an image-as-array mini-project\n", "- Apply the whole course to a stock-price-as-array mini-project\n", "\n", "**Setup** — run this first.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "02bb3d4f", "metadata": {}, "outputs": [], "source": [ "import io\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "print(np.__version__)\n" ] }, { "cell_type": "markdown", "id": "7d2bba70", "metadata": {}, "source": [ "## Lesson 1 — Memory Layout: Strides & Order\n", "\n", "`.strides` tells you how many **bytes** to step to move one position along\n", "each axis. For a C-order (row-major, the default) `(2, 3)` array of\n", "`float64`, moving one column over costs 8 bytes, but moving one row down\n", "costs `3 * 8 = 24` bytes — because rows are stored one after another in\n", "memory.\n", "\n", "Fortran order (`order=\"F\"`) stores **columns** contiguously instead — the\n", "strides flip. This is why `.T` (transpose) is instant and never copies data:\n", "it just swaps the strides and shape, leaving the underlying buffer alone.\n", "That also means a transposed array is usually **not C-contiguous** anymore.\n", "When an algorithm (or a library written in C) genuinely needs a contiguous\n", "buffer, use `np.ascontiguousarray(arr)` — it returns `arr` itself if it's\n", "already contiguous, and a real copy only if it isn't.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "9209762b", "metadata": {}, "outputs": [], "source": [ "a = np.arange(6, dtype=np.float64).reshape(2, 3)\n", "print(\"C-order strides:\", a.strides) # (24, 8): 24 bytes per row, 8 per column\n", "print(\"is C-contiguous:\", a.flags[\"C_CONTIGUOUS\"])\n", "\n", "at = a.T\n", "print(\"\\ntransposed strides:\", at.strides) # strides just swapped, no copy\n", "print(\"transpose is C-contiguous:\", at.flags[\"C_CONTIGUOUS\"])\n", "print(\"transpose shares memory with a:\", np.shares_memory(a, at))\n", "\n", "f = np.asfortranarray(a)\n", "print(\"\\nFortran-order strides:\", f.strides)\n" ] }, { "cell_type": "markdown", "id": "27842d1d", "metadata": {}, "source": [ "### 💪 Exercise 1.1: Spot the view\n", "\n", "Write `second_row_view(matrix)` that returns a **view** (not a copy) of row\n", "index `1` of a 2-D array, using basic slicing.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "e7a22cf9", "metadata": {}, "outputs": [], "source": [ "def second_row_view(matrix):\n", " # YOUR CODE HERE\n", " raise NotImplementedError\n" ] }, { "cell_type": "code", "execution_count": null, "id": "9d1b4f25", "metadata": {}, "outputs": [], "source": [ "m = np.arange(12).reshape(3, 4)\n", "result = second_row_view(m)\n", "print(\"second_row_view(m) ->\")\n", "print(result)\n", "\n", "shares_memory = np.shares_memory(result, m)\n", "print()\n", "print(\"np.shares_memory(result, m) ->\", shares_memory)\n", "\n", "expected = np.array([4, 5, 6, 7])\n", "_passed = np.array_equal(result, expected) and shares_memory\n", "\n", "if _passed:\n", " result[0] = 999\n", " _passed = bool(m[1, 0] == 999)\n", "\n", "if not _passed:\n", " print()\n", " print(\"Expected: a VIEW (shares memory with m) equal to\", expected,\n", " \"— and mutating it should mutate m too\")\n" ] }, { "cell_type": "markdown", "id": "39d2636b", "metadata": {}, "source": [ "
\n", "💡 Hint (click to expand)\n", "\n", "Plain slicing with `matrix[1]` (or `matrix[1, :]`) already returns a view — no `.copy()` needed.\n", "\n", "
\n" ] }, { "cell_type": "markdown", "id": "e9a0105f", "metadata": {}, "source": [ "
\n", "✅ Solution (click to expand)\n", "\n", "```python\n", "def second_row_view(matrix):\n", " return matrix[1]\n", "```\n", "\n", "
\n" ] }, { "cell_type": "markdown", "id": "b0d13fce", "metadata": {}, "source": [ "### 💪 Exercise 1.2: Force a contiguous copy\n", "\n", "Write `ensure_contiguous(arr)` that returns a C-contiguous version of\n", "`arr`. If `arr` is already C-contiguous, return it unchanged (same\n", "underlying buffer); otherwise return a new, C-contiguous array with the\n", "same shape and values.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "125774d1", "metadata": {}, "outputs": [], "source": [ "def ensure_contiguous(arr):\n", " # YOUR CODE HERE\n", " raise NotImplementedError\n" ] }, { "cell_type": "code", "execution_count": null, "id": "cd2dcea7", "metadata": {}, "outputs": [], "source": [ "base = np.arange(6).reshape(2, 3)\n", "same = ensure_contiguous(base)\n", "print(\"ensure_contiguous(base) ->\")\n", "print(same)\n", "\n", "already_contiguous_ok = np.shares_memory(same, base)\n", "print()\n", "print(\"np.shares_memory(result, base) (already contiguous, should be True) ->\", already_contiguous_ok)\n", "\n", "transposed = base.T\n", "fixed = ensure_contiguous(transposed)\n", "print()\n", "print(\"ensure_contiguous(base.T) ->\")\n", "print(fixed)\n", "print(\"is C-contiguous:\", fixed.flags[\"C_CONTIGUOUS\"])\n", "\n", "_passed = (\n", " already_contiguous_ok\n", " and fixed.flags[\"C_CONTIGUOUS\"]\n", " and np.array_equal(fixed, transposed)\n", ")\n", "\n", "if not _passed:\n", " print()\n", " print(\"Expected: base returned as-is (same memory), and a C-contiguous \"\n", " \"copy of base.T with the same values as:\")\n", " print(transposed)\n" ] }, { "cell_type": "markdown", "id": "3cdb47e6", "metadata": {}, "source": [ "
\n", "💡 Hint (click to expand)\n", "\n", "`np.ascontiguousarray` does exactly this: no-op copy if already contiguous, real copy otherwise.\n", "\n", "
\n" ] }, { "cell_type": "markdown", "id": "2816ebf2", "metadata": {}, "source": [ "
\n", "✅ Solution (click to expand)\n", "\n", "```python\n", "def ensure_contiguous(arr):\n", " return np.ascontiguousarray(arr)\n", "```\n", "\n", "
\n" ] }, { "cell_type": "markdown", "id": "a7226e39", "metadata": {}, "source": [ "## Lesson 2 — Vectorization Performance Patterns\n", "\n", "Three habits keep NumPy code fast:\n", "\n", "1. **Never grow an array in a loop.** Appending to a Python list and\n", " converting once at the end is fine; calling `np.append`/`np.concatenate`\n", " inside a loop reallocates the whole buffer every iteration.\n", "2. **Preallocate when you must fill incrementally** — `np.empty(shape)` or\n", " `np.zeros(shape)` once, then assign into slices.\n", "3. **Reuse buffers with `out=`** when you don't need the original array\n", " anymore — `np.add(a, b, out=a)` writes the result into `a`'s existing\n", " memory instead of allocating a new array.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "37bf25a7", "metadata": {}, "outputs": [], "source": [ "import time\n", "\n", "n = 2_000_000\n", "a = np.arange(n, dtype=np.float64)\n", "b = np.ones(n)\n", "\n", "start = time.perf_counter()\n", "c = a + b # allocates a brand-new array\n", "print(f\"new allocation: {time.perf_counter() - start:.5f}s\")\n", "\n", "start = time.perf_counter()\n", "np.add(a, b, out=a) # reuses a's memory, no new allocation\n", "print(f\"in-place (out=): {time.perf_counter() - start:.5f}s\")\n" ] }, { "cell_type": "markdown", "id": "a134ba3a", "metadata": {}, "source": [ "### 💪 Exercise 2.1: Increment in place\n", "\n", "Write `increment_in_place(arr, amount)` that adds `amount` to every\n", "element of `arr` **in place** (using the `out=` parameter so no new array\n", "is allocated) and returns `arr` itself.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "c58da294", "metadata": {}, "outputs": [], "source": [ "def increment_in_place(arr, amount):\n", " # YOUR CODE HERE\n", " raise NotImplementedError\n" ] }, { "cell_type": "code", "execution_count": null, "id": "4c3f30cd", "metadata": {}, "outputs": [], "source": [ "original = np.array([1.0, 2.0, 3.0])\n", "result = increment_in_place(original, 5.0)\n", "print(\"increment_in_place(np.array([1.0, 2.0, 3.0]), 5.0) ->\")\n", "print(result)\n", "\n", "same_object = result is original\n", "print()\n", "print(\"result is original (must be mutated in place) ->\", same_object)\n", "\n", "expected = np.array([6.0, 7.0, 8.0])\n", "_passed = same_object and np.allclose(original, expected)\n", "\n", "if not _passed:\n", " print()\n", " print(\"Expected: the SAME array object, mutated in place to\", expected)\n" ] }, { "cell_type": "markdown", "id": "48344850", "metadata": {}, "source": [ "
\n", "💡 Hint (click to expand)\n", "\n", "`np.add(arr, amount, out=arr)` writes the result back into `arr` and also returns `arr`.\n", "\n", "
\n" ] }, { "cell_type": "markdown", "id": "adfec8de", "metadata": {}, "source": [ "
\n", "✅ Solution (click to expand)\n", "\n", "```python\n", "def increment_in_place(arr, amount):\n", " return np.add(arr, amount, out=arr)\n", "```\n", "\n", "
\n" ] }, { "cell_type": "markdown", "id": "4384b9bc", "metadata": {}, "source": [ "### 💪 Exercise 2.2: Running totals, no loop\n", "\n", "Write `running_totals(values)` that returns the cumulative sum of a 1-D\n", "array (`result[i]` = sum of `values[0..i]`) without writing a Python `for`\n", "or `while` loop.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "aff51e13", "metadata": {}, "outputs": [], "source": [ "def running_totals(values):\n", " # YOUR CODE HERE\n", " raise NotImplementedError\n" ] }, { "cell_type": "code", "execution_count": null, "id": "a92beefe", "metadata": {}, "outputs": [], "source": [ "result = running_totals(np.array([1, 2, 3, 4]))\n", "print(\"running_totals(np.array([1, 2, 3, 4])) ->\")\n", "print(result)\n", "\n", "expected = np.array([1, 3, 6, 10])\n", "_passed = np.array_equal(result, expected)\n", "\n", "if not _passed:\n", " print()\n", " print(\"Expected:\")\n", " print(expected)\n" ] }, { "cell_type": "markdown", "id": "9af8ca8b", "metadata": {}, "source": [ "
\n", "💡 Hint (click to expand)\n", "\n", "There's a single built-in NumPy function that does exactly this: `np.cumsum`.\n", "\n", "
\n" ] }, { "cell_type": "markdown", "id": "af1a72e5", "metadata": {}, "source": [ "
\n", "✅ Solution (click to expand)\n", "\n", "```python\n", "def running_totals(values):\n", " return np.cumsum(values)\n", "```\n", "\n", "
\n" ] }, { "cell_type": "markdown", "id": "9b6b0595", "metadata": {}, "source": [ "## Lesson 3 — Saving & Loading Arrays\n", "\n", "- `np.save` / `np.load` write a **single** array to a binary `.npy` file.\n", "- `np.savez` / `np.load` write **several named** arrays to one `.npz`\n", " archive (loading returns an `NpzFile` you index like a dict).\n", "- `np.savetxt` / `np.loadtxt` write a plain-text, CSV-like format — human\n", " readable, but slower and loses exact dtype information.\n", "\n", "All three normally take a file path, but they're equally happy with an\n", "in-memory file-like object (`io.BytesIO` for the binary formats,\n", "`io.StringIO` for text) — handy for round-tripping without touching disk,\n", "which is exactly what the exercises below do. **Remember to `seek(0)`** on\n", "the buffer after writing and before reading, since writing leaves the\n", "buffer's position at the end.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "11ed83ae", "metadata": {}, "outputs": [], "source": [ "arr = np.array([1, 2, 3, 4])\n", "\n", "buf = io.BytesIO()\n", "np.save(buf, arr)\n", "buf.seek(0)\n", "loaded = np.load(buf)\n", "print(\"single array round-trip:\", loaded)\n", "\n", "buf2 = io.BytesIO()\n", "np.savez(buf2, first=arr, second=arr * 10)\n", "buf2.seek(0)\n", "archive = np.load(buf2)\n", "print(\"multi-array round-trip:\", archive[\"first\"], archive[\"second\"])\n" ] }, { "cell_type": "markdown", "id": "5648d5fd", "metadata": {}, "source": [ "### 💪 Exercise 3.1: Round-trip a single array\n", "\n", "Write `roundtrip_npy(arr)` that writes `arr` to an in-memory `io.BytesIO`\n", "buffer with `np.save`, reads it back with `np.load`, and returns the\n", "loaded array.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "dd3b234d", "metadata": {}, "outputs": [], "source": [ "def roundtrip_npy(arr):\n", " # YOUR CODE HERE\n", " raise NotImplementedError\n" ] }, { "cell_type": "code", "execution_count": null, "id": "21b8a2c2", "metadata": {}, "outputs": [], "source": [ "original = np.array([3.5, -1.0, 2.25])\n", "result = roundtrip_npy(original)\n", "print(\"roundtrip_npy(np.array([3.5, -1.0, 2.25])) ->\")\n", "print(result)\n", "\n", "_passed = np.allclose(result, original) and result is not original\n", "\n", "if not _passed:\n", " print()\n", " print(\"Expected: a freshly loaded array (not the same object), equal to\", original)\n" ] }, { "cell_type": "markdown", "id": "69ca6c95", "metadata": {}, "source": [ "
\n", "💡 Hint (click to expand)\n", "\n", "`buf = io.BytesIO()`, then `np.save(buf, arr)`, then `buf.seek(0)` before `np.load(buf)`.\n", "\n", "
\n" ] }, { "cell_type": "markdown", "id": "486359be", "metadata": {}, "source": [ "
\n", "✅ Solution (click to expand)\n", "\n", "```python\n", "def roundtrip_npy(arr):\n", " buf = io.BytesIO()\n", " np.save(buf, arr)\n", " buf.seek(0)\n", " return np.load(buf)\n", "```\n", "\n", "
\n" ] }, { "cell_type": "markdown", "id": "a99d9c89", "metadata": {}, "source": [ "### 💪 Exercise 3.2: Round-trip two named arrays\n", "\n", "Write `roundtrip_npz(a, b)` that saves `a` and `b` into one in-memory\n", "`.npz` archive under the keys `\"a\"` and `\"b\"` with `np.savez`, reloads it,\n", "and returns a plain `dict` `{\"a\": ..., \"b\": ...}` of the loaded arrays.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "a7a5a71f", "metadata": {}, "outputs": [], "source": [ "def roundtrip_npz(a, b):\n", " # YOUR CODE HERE\n", " raise NotImplementedError\n" ] }, { "cell_type": "code", "execution_count": null, "id": "bf4f0fa8", "metadata": {}, "outputs": [], "source": [ "a = np.array([1, 2, 3])\n", "b = np.array([[1, 2], [3, 4]])\n", "result = roundtrip_npz(a, b)\n", "print(\"roundtrip_npz(a, b) ->\")\n", "print(result)\n", "\n", "_passed = (\n", " isinstance(result, dict)\n", " and set(result.keys()) == {\"a\", \"b\"}\n", " and np.array_equal(result.get(\"a\"), a)\n", " and np.array_equal(result.get(\"b\"), b)\n", ")\n", "\n", "if not _passed:\n", " print()\n", " print(\"Expected:\")\n", " print({\"a\": a, \"b\": b})\n" ] }, { "cell_type": "markdown", "id": "75332fdb", "metadata": {}, "source": [ "
\n", "💡 Hint (click to expand)\n", "\n", "`np.savez(buf, a=a, b=b)`, `buf.seek(0)`, then `np.load(buf)` returns an `NpzFile` — index it with `[\"a\"]` and `[\"b\"]`.\n", "\n", "
\n" ] }, { "cell_type": "markdown", "id": "5ad0595a", "metadata": {}, "source": [ "
\n", "✅ Solution (click to expand)\n", "\n", "```python\n", "def roundtrip_npz(a, b):\n", " buf = io.BytesIO()\n", " np.savez(buf, a=a, b=b)\n", " buf.seek(0)\n", " archive = np.load(buf)\n", " return {\"a\": archive[\"a\"], \"b\": archive[\"b\"]}\n", "```\n", "\n", "
\n" ] }, { "cell_type": "markdown", "id": "72ced3b8", "metadata": {}, "source": [ "## Lesson 4 — Structured Arrays\n", "\n", "A **structured array** has a dtype made of named fields, so each element\n", "is a small fixed-layout record rather than a single number — useful for\n", "tabular data when you don't want to bring in `pandas`. Define the dtype\n", "with a list of `(field_name, field_dtype)` pairs, then index a field by\n", "name like a dict key. `np.sort(arr, order=\"field_name\")` sorts the whole\n", "array by one field.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "3600a053", "metadata": {}, "outputs": [], "source": [ "dtype = np.dtype([(\"name\", \"U10\"), (\"age\", \"i4\"), (\"score\", \"f8\")])\n", "people = np.array(\n", " [(\"Ana\", 34, 91.5), (\"Boris\", 29, 88.0), (\"Chiko\", 41, 95.25)],\n", " dtype=dtype,\n", ")\n", "print(people[\"name\"])\n", "print(people[\"score\"])\n", "\n", "by_age = np.sort(people, order=\"age\")\n", "print(by_age)\n" ] }, { "cell_type": "markdown", "id": "12705a89", "metadata": {}, "source": [ "### 💪 Exercise 4.1: Build a structured array\n", "\n", "Write `make_people(names, ages, scores)` that takes three equal-length\n", "sequences and returns a structured array with dtype\n", "`[(\"name\", \"U10\"), (\"age\", \"i4\"), (\"score\", \"f8\")]`, one record per\n", "input triple.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "0ec27f70", "metadata": {}, "outputs": [], "source": [ "def make_people(names, ages, scores):\n", " # YOUR CODE HERE\n", " raise NotImplementedError\n" ] }, { "cell_type": "code", "execution_count": null, "id": "cc881c21", "metadata": {}, "outputs": [], "source": [ "result = make_people([\"Ana\", \"Boris\"], [34, 29], [91.5, 88.0])\n", "print('make_people([\"Ana\", \"Boris\"], [34, 29], [91.5, 88.0]) ->')\n", "print(result)\n", "\n", "_passed = (\n", " result.dtype.names == (\"name\", \"age\", \"score\")\n", " and result[0][\"name\"] == \"Ana\"\n", " and result[0][\"age\"] == 34\n", " and np.isclose(result[1][\"score\"], 88.0)\n", ")\n", "\n", "if not _passed:\n", " print()\n", " print('Expected: a structured array with fields (\"name\", \"age\", \"score\"), '\n", " 'first record name \"Ana\"/age 34, second record score ≈ 88.0')\n" ] }, { "cell_type": "markdown", "id": "d9c8f44a", "metadata": {}, "source": [ "
\n", "💡 Hint (click to expand)\n", "\n", "Zip the three sequences into a list of tuples, then pass that list and the dtype straight to `np.array(...)`.\n", "\n", "
\n" ] }, { "cell_type": "markdown", "id": "999efc62", "metadata": {}, "source": [ "
\n", "✅ Solution (click to expand)\n", "\n", "```python\n", "def make_people(names, ages, scores):\n", " dtype = np.dtype([(\"name\", \"U10\"), (\"age\", \"i4\"), (\"score\", \"f8\")])\n", " records = list(zip(names, ages, scores))\n", " return np.array(records, dtype=dtype)\n", "```\n", "\n", "
\n" ] }, { "cell_type": "markdown", "id": "581392b5", "metadata": {}, "source": [ "### 💪 Exercise 4.2: Sort by score\n", "\n", "Write `sort_by_score(people)` that takes a structured array with a\n", "`\"score\"` field and returns it sorted in **ascending** order by that\n", "field.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "149ed01b", "metadata": {}, "outputs": [], "source": [ "def sort_by_score(people):\n", " # YOUR CODE HERE\n", " raise NotImplementedError\n" ] }, { "cell_type": "code", "execution_count": null, "id": "66d1e504", "metadata": {}, "outputs": [], "source": [ "dtype = np.dtype([(\"name\", \"U10\"), (\"age\", \"i4\"), (\"score\", \"f8\")])\n", "people = np.array(\n", " [(\"Ana\", 34, 91.5), (\"Boris\", 29, 88.0), (\"Chiko\", 41, 95.25)],\n", " dtype=dtype,\n", ")\n", "result = sort_by_score(people)\n", "print(\"sort_by_score(people) ->\")\n", "print(result)\n", "\n", "expected_names = [\"Boris\", \"Ana\", \"Chiko\"]\n", "_passed = list(result[\"name\"]) == expected_names and np.all(np.diff(result[\"score\"]) >= 0)\n", "\n", "if not _passed:\n", " print()\n", " print(\"Expected names in ascending-score order:\", expected_names)\n" ] }, { "cell_type": "markdown", "id": "247cdef7", "metadata": {}, "source": [ "
\n", "💡 Hint (click to expand)\n", "\n", "`np.sort` takes an `order=` keyword for structured arrays — `np.sort(people, order=\"score\")`.\n", "\n", "
\n" ] }, { "cell_type": "markdown", "id": "1ab04557", "metadata": {}, "source": [ "
\n", "✅ Solution (click to expand)\n", "\n", "```python\n", "def sort_by_score(people):\n", " return np.sort(people, order=\"score\")\n", "```\n", "\n", "
\n" ] }, { "cell_type": "markdown", "id": "67082f91", "metadata": {}, "source": [ "## Lesson 5 — Capstone: Image Manipulation with NumPy\n", "\n", "A grayscale image is nothing more than a 2-D array of brightness values\n", "(`0`-`255`). Every \"image processing\" operation below is really just\n", "array indexing, broadcasting, and boolean masking — the exact tools from\n", "Modules 1-3, no special image library required.\n", "\n", "Below we synthesize a small 40×40 image containing a bright square and a\n", "bright circle on a dark background, purely from array operations (no file\n", "is loaded, so this runs identically anywhere, including Colab).\n" ] }, { "cell_type": "code", "execution_count": null, "id": "b10b24f5", "metadata": {}, "outputs": [], "source": [ "rng = np.random.default_rng(seed=7)\n", "\n", "size = 40\n", "img = rng.integers(0, 40, size=(size, size)).astype(np.uint8) # dim background noise\n", "\n", "# bright square: rows/cols 5..15\n", "img[5:15, 5:15] = 220\n", "\n", "# bright circle of radius 10 centered at (28, 28)\n", "yy, xx = np.indices((size, size))\n", "circle_mask = (yy - 28) ** 2 + (xx - 28) ** 2 <= 10 ** 2\n", "img[circle_mask] = 200\n", "\n", "plt.imshow(img, cmap=\"gray\", vmin=0, vmax=255)\n", "plt.title(\"Synthetic grayscale image\")\n", "plt.colorbar()\n", "plt.show()\n" ] }, { "cell_type": "markdown", "id": "e1f56560", "metadata": {}, "source": [ "Every slice you write, like `img[5:15, 5:15]` above, is really Python\n", "building a `slice(5, 15)` object per axis and handing a tuple of them to\n", "`img.__getitem__`. Because that's just an object, you can construct it\n", "yourself, store it in a variable, and use it later — `row_slice = slice(5, 15)`\n", "then `img[row_slice]` is identical to `img[5:15]`. For multiple axes, a\n", "tuple of slices works the same way: `(slice(5, 15), slice(0, 10))` is the\n", "same index as `img[5:15, 0:10]`. The exercise below asks you to accept a\n", "`crop_slice` built this way, so the caller can crop to whatever region it\n", "wants.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "f75110cb", "metadata": {}, "outputs": [], "source": [ "# `a:b` syntax and an explicit slice object are the same thing.\n", "row_slice = slice(5, 15)\n", "print(np.array_equal(img[5:15, 5:15], img[row_slice, row_slice]))\n", "\n", "# A tuple of slices works for multi-dimensional indexing too.\n", "region = (slice(5, 15), slice(0, 10))\n", "print(np.array_equal(img[region], img[5:15, 0:10]))\n" ] }, { "cell_type": "markdown", "id": "24db3bf7", "metadata": {}, "source": [ "### 💪 Exercise 5.1: Threshold, crop, and measure brightness\n", "\n", "Write `analyze_image(img, threshold, crop_slice)` that performs three\n", "chained steps on a 2-D grayscale array `img`:\n", "\n", "1. Build a boolean `\"mask\"` the same shape as `img` where pixels are\n", " strictly greater than `threshold`.\n", "2. Build a `\"crop\"`: the sub-array `img[crop_slice]`, where `crop_slice`\n", " is a tuple of two `slice` objects (one per axis) — e.g.\n", " `(slice(0, 10), slice(0, 10))`.\n", "3. Compute `\"bright_fraction\"`: the fraction (a Python `float` between 0\n", " and 1) of pixels **within that crop** that are strictly greater than\n", " `threshold`.\n", "\n", "Return a dict with exactly the keys `\"mask\"`, `\"crop\"`, and\n", "`\"bright_fraction\"`.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "3ba166d2", "metadata": {}, "outputs": [], "source": [ "def analyze_image(img, threshold, crop_slice):\n", " # YOUR CODE HERE\n", " raise NotImplementedError\n" ] }, { "cell_type": "code", "execution_count": null, "id": "cecc1b7d", "metadata": {}, "outputs": [], "source": [ "test_img = np.array([\n", " [10, 10, 200, 200],\n", " [10, 10, 200, 200],\n", " [5, 5, 5, 5],\n", " [5, 5, 5, 5],\n", "])\n", "crop_slice = (slice(0, 2), slice(0, 4))\n", "result = analyze_image(test_img, 100, crop_slice)\n", "print(\"analyze_image(test_img, 100, crop_slice) ->\")\n", "print(result)\n", "\n", "expected_mask = test_img > 100\n", "expected_crop = test_img[crop_slice]\n", "expected_fraction = 0.5 # 4 of 8 crop pixels are 200\n", "\n", "_passed = (\n", " isinstance(result, dict)\n", " and set(result.keys()) == {\"mask\", \"crop\", \"bright_fraction\"}\n", " and result[\"mask\"].shape == test_img.shape\n", " and result[\"mask\"].dtype == bool\n", " and np.array_equal(result[\"mask\"], expected_mask)\n", " and np.array_equal(result[\"crop\"], expected_crop)\n", " and np.isclose(result[\"bright_fraction\"], expected_fraction)\n", ")\n", "\n", "if not _passed:\n", " print()\n", " print(\"Expected:\")\n", " print({\"mask\": expected_mask, \"crop\": expected_crop, \"bright_fraction\": expected_fraction})\n" ] }, { "cell_type": "markdown", "id": "aa25e383", "metadata": {}, "source": [ "
\n", "💡 Hint (click to expand)\n", "\n", "`mask = img > threshold`. `crop = img[crop_slice]`. For the fraction, reuse the mask but computed on the crop (or slice the mask itself with `crop_slice`) and take `.mean()` of the boolean array — the mean of a boolean array IS the fraction of `True` values.\n", "\n", "
\n" ] }, { "cell_type": "markdown", "id": "9c80b343", "metadata": {}, "source": [ "
\n", "✅ Solution (click to expand)\n", "\n", "```python\n", "def analyze_image(img, threshold, crop_slice):\n", " mask = img > threshold\n", " crop = img[crop_slice]\n", " bright_fraction = float((crop > threshold).mean())\n", " return {\"mask\": mask, \"crop\": crop, \"bright_fraction\": bright_fraction}\n", "```\n", "\n", "
\n" ] }, { "cell_type": "markdown", "id": "6d5a333a", "metadata": {}, "source": [ "## Lesson 6 — Capstone: Analyzing a Stock Price Series\n", "\n", "We'll model a daily closing-price series as a **random walk**: start at\n", "some price, then repeatedly add a small random daily change. This is\n", "synthetic data (seeded, so it's reproducible), but the *analysis* below is\n", "exactly what you'd run on a real price series.\n", "\n", "One technique here hasn't been taught yet: a **rolling mean without\n", "pandas**. `np.convolve(prices, kernel, mode=\"valid\")`, with\n", "`kernel = np.ones(window) / window`, slides a length-`window` averaging\n", "kernel across the series and keeps only the positions where the kernel\n", "fully overlaps the data (`mode=\"valid\"`) — so the result has length\n", "`len(prices) - window + 1`. It's a vectorized way to compute a moving\n", "average with zero explicit loops.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "50ca890f", "metadata": {}, "outputs": [], "source": [ "rng = np.random.default_rng(seed=42)\n", "\n", "n_days = 250\n", "daily_changes = rng.normal(loc=0.05, scale=1.0, size=n_days)\n", "prices = 100 + np.cumsum(daily_changes)\n", "\n", "plt.plot(prices)\n", "plt.title(\"Synthetic daily closing price\")\n", "plt.xlabel(\"day\")\n", "plt.ylabel(\"price\")\n", "plt.show()\n", "\n", "window = 10\n", "kernel = np.ones(window) / window\n", "rolling = np.convolve(prices, kernel, mode=\"valid\")\n", "print(\"prices:\", prices.shape, \"rolling mean:\", rolling.shape)\n" ] }, { "cell_type": "markdown", "id": "6715b579", "metadata": {}, "source": [ "### 💪 Exercise 6.1: Returns, worst day, and a rolling mean\n", "\n", "Write `analyze_prices(prices, window)` that performs three chained\n", "computations on a 1-D array of daily closing prices:\n", "\n", "1. `\"daily_returns\"`: a vectorized array of length `len(prices) - 1` where\n", " `daily_returns[i] = prices[i+1] / prices[i] - 1` (no Python loop).\n", "2. `\"worst_day_index\"` and `\"worst_day_return\"`: the index into\n", " `daily_returns` of the single worst (most negative) daily return, and\n", " that return's value.\n", "3. `\"rolling_mean\"`: the simple moving average of `prices` over `window`\n", " days, computed with `np.convolve` and `mode=\"valid\"` as described above\n", " (length `len(prices) - window + 1`).\n", "\n", "Return a dict with exactly the keys `\"daily_returns\"`, `\"worst_day_index\"`,\n", "`\"worst_day_return\"`, and `\"rolling_mean\"`.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "45c0720c", "metadata": {}, "outputs": [], "source": [ "def analyze_prices(prices, window):\n", " # YOUR CODE HERE\n", " raise NotImplementedError\n" ] }, { "cell_type": "code", "execution_count": null, "id": "4865392e", "metadata": {}, "outputs": [], "source": [ "prices = np.array([100.0, 102.0, 101.0, 95.0, 105.0, 107.0])\n", "result = analyze_prices(prices, window=3)\n", "print(\"analyze_prices(prices, window=3) ->\")\n", "print(result)\n", "\n", "expected_returns = prices[1:] / prices[:-1] - 1\n", "expected_worst_idx = int(np.argmin(expected_returns))\n", "expected_worst_return = expected_returns[expected_worst_idx]\n", "kernel = np.ones(3) / 3\n", "expected_rolling = np.convolve(prices, kernel, mode=\"valid\")\n", "\n", "_passed = (\n", " isinstance(result, dict)\n", " and set(result.keys()) == {\n", " \"daily_returns\", \"worst_day_index\", \"worst_day_return\", \"rolling_mean\"\n", " }\n", " and np.allclose(result[\"daily_returns\"], expected_returns)\n", " and result[\"worst_day_index\"] == expected_worst_idx\n", " and np.isclose(result[\"worst_day_return\"], expected_worst_return)\n", " and np.allclose(result[\"rolling_mean\"], expected_rolling)\n", " and len(result[\"rolling_mean\"]) == len(prices) - 3 + 1\n", ")\n", "\n", "if not _passed:\n", " print()\n", " print(\"Expected:\")\n", " print({\n", " \"daily_returns\": expected_returns,\n", " \"worst_day_index\": expected_worst_idx,\n", " \"worst_day_return\": expected_worst_return,\n", " \"rolling_mean\": expected_rolling,\n", " })\n" ] }, { "cell_type": "markdown", "id": "1b1d8c14", "metadata": {}, "source": [ "
\n", "💡 Hint (click to expand)\n", "\n", "`daily_returns = prices[1:] / prices[:-1] - 1`. Use `np.argmin` for the worst index, then index back into `daily_returns` for its value. For the rolling mean, build `kernel = np.ones(window) / window` and call `np.convolve(prices, kernel, mode=\"valid\")`.\n", "\n", "
\n" ] }, { "cell_type": "markdown", "id": "46031133", "metadata": {}, "source": [ "
\n", "✅ Solution (click to expand)\n", "\n", "```python\n", "def analyze_prices(prices, window):\n", " daily_returns = prices[1:] / prices[:-1] - 1\n", " worst_day_index = int(np.argmin(daily_returns))\n", " worst_day_return = float(daily_returns[worst_day_index])\n", " kernel = np.ones(window) / window\n", " rolling_mean = np.convolve(prices, kernel, mode=\"valid\")\n", " return {\n", " \"daily_returns\": daily_returns,\n", " \"worst_day_index\": worst_day_index,\n", " \"worst_day_return\": worst_day_return,\n", " \"rolling_mean\": rolling_mean,\n", " }\n", "```\n", "\n", "
\n" ] }, { "cell_type": "markdown", "id": "730236c1", "metadata": {}, "source": [ "## 🎯 Key Takeaways\n", "\n", "- `.strides` and C-vs-Fortran order explain *why* transpose is free and\n", " why some operations need `np.ascontiguousarray` first.\n", "- Avoid hidden allocation costs: preallocate instead of growing arrays,\n", " and use `out=` for in-place ops when you don't need the original.\n", "- `np.save`/`load`, `np.savez`, and `np.savetxt`/`loadtxt` cover binary,\n", " multi-array, and text persistence — all work with in-memory buffers too.\n", "- Structured arrays give you named, typed fields without needing `pandas`.\n", "- Both capstones were just Modules 1-5's tools — indexing, broadcasting,\n", " aggregation, sorting — aimed at a concrete problem.\n" ] }, { "cell_type": "markdown", "id": "5f325420", "metadata": {}, "source": [ "## 🎓 Course Complete!\n", "\n", "You've made it through the whole course:\n", "\n", "1. **NumPy Fundamentals & Array Creation** — what an array is, and every\n", " way to build one.\n", "2. **Indexing, Slicing & Views** — selecting exactly the data you want,\n", " and knowing when you get a view vs. a copy.\n", "3. **Array Operations & Broadcasting** — vectorized math and the\n", " broadcasting rules that make it work across shapes.\n", "4. **Aggregations, Statistics & Sorting** — summarizing and ordering data\n", " along any axis.\n", "5. **Linear Algebra & Random** — vectors, matrices, and reproducible\n", " randomness.\n", "6. **Performance, I/O & Mini-Projects** — memory layout, persistence, and\n", " two real mini-projects tying it all together.\n", "\n", "**Where to go next:**\n", "- **pandas** — labeled, tabular data built on top of NumPy arrays.\n", "- **matplotlib** / **seaborn** — deeper, more polished visualization.\n", "- **scipy** — advanced scientific computing (optimization, signal\n", " processing, statistics) that extends what you've learned here.\n", "\n", "Nicely done — go build something with it. 🎉\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.x" } }, "nbformat": 4, "nbformat_minor": 5 }