diff --git a/docs/foundations/big-o-notation.md b/docs/foundations/big-o-notation.md index e6bdb23..95f43ae 100644 --- a/docs/foundations/big-o-notation.md +++ b/docs/foundations/big-o-notation.md @@ -9,116 +9,754 @@ author: "ihelloeveryone" # Big-O Notation: Time Complexity Basics -## Core concept -Big-O notation is a mathematical way to describe how the running time or space usage of an algorithm grows as the input size increases, focusing on long-term growth rather than exact step counts. -It provides an implementation- and hardware-independent upper bound on how fast resource usage can grow as the input size becomes large. +## Core Concept -## The problem Big-O solves -Without a shared language like Big-O, it is difficult to compare algorithms across machines, programming languages, or implementations. -Raw timings (for example, "this runs in 10 ms") change with hardware, compiler optimizations, and constant factors, but the growth pattern with respect to input size usually does not. -Big-O gives a way to reason about scalability: how an algorithm behaves when input size grows from hundreds to millions. +Big-O notation is a mathematical language we use to describe how an algorithm's resource consumption—typically time or memory—grows as the input size increases. Think of it as a way to answer the question: "If I give this algorithm ten times more data, how much slower will it get?" -## How Big-O works -Big-O describes the dominant term of a function that counts operations (or time/space) as a function of input size, usually written as \(n\). -Formally, an algorithm with running time \(T(n)\) is in \(O(f(n))\) if there are positive constants \(c\) and \(n_0\) such that for all \(n \ge n_0\), \(T(n) \le c \cdot f(n)\). -In practice, this means: -- Model the number of basic operations as a function of \(n\). -- Drop constant factors (for example, \(3n\) becomes \(n\)). -- Keep only the highest-order term (for example, \(2n^2 + 5n + 7\) becomes \(n^2\)). +Unlike measuring execution time in seconds or milliseconds (which depends on your computer's speed, programming language, and countless other factors), Big-O gives us a universal, machine-independent way to talk about algorithmic efficiency. It focuses on the **fundamental growth pattern** rather than exact numbers. -### Logical steps for analyzing complexity -1. Identify the input and define what \(n\) represents (length of array, number of nodes, number of vertices, and so on). -2. Trace the algorithm and count how many times key operations execute in terms of \(n\). -3. Express this count as a function \(T(n)\), combining contributions from loops, branches, and function calls. -4. Simplify \(T(n)\) by removing constants and lower-order terms, then express the result using Big-O notation. +Here's the key insight: when dealing with large inputs, some terms in our time function matter much more than others. Big-O helps us identify and focus on what truly matters at scale. -## Common complexity classes +--- + +## The Problem Big-O Solves + +### Why Raw Timings Fall Short + +Imagine you write an algorithm that processes data in 10 milliseconds on your laptop. Your colleague runs the same algorithm on their server and gets 3 milliseconds. Which implementation is better? You can't tell! The difference might be due to: + +- **Hardware**: Different CPU speeds, RAM, cache sizes +- **Language**: Python vs C++ can show 10x-100x differences +- **Compiler optimizations**: Same code, different performance +- **System load**: Background processes affecting measurements + +Now imagine you need to predict: "If this works for 1,000 items in 10ms, how long for 1,000,000 items?" Raw timings can't answer this reliably. + +### The Universal Language + +Big-O notation cuts through all these variables by asking a simpler question: **"As input size grows toward infinity, what's the dominant factor affecting runtime?"** + +This gives us a common vocabulary. When someone says "this algorithm is O(n log n)," you immediately know: +- It will handle large datasets reasonably well +- Doubling the input size roughly doubles the time (plus a small logarithmic factor) +- It's faster than quadratic O(n²) algorithms for large inputs + +All of this, without running a single benchmark. + +--- + +## How Big-O Works: The Mathematics + +### Formal Definition + +Let's build this from the ground up. Suppose we have an algorithm whose runtime can be expressed as a function \(T(n)\), where \(n\) is the input size. We say that: + +\[ T(n) \text{ is } O(f(n)) \] + +if and only if there exist positive constants \(c\) and \(n_0\) such that: + +\[ T(n) \le c \cdot f(n) \text{ for all } n \ge n_0 \] + +**Let's unpack this carefully:** + +1. **\(T(n)\)** is our actual runtime function (the real cost of the algorithm) +2. **\(f(n)\)** is a simpler function we're comparing it to (like \(n\), \(n^2\), or \(n \log n\)) +3. **\(c\)** is a constant multiplier we're allowed to choose +4. **\(n_0\)** is a threshold—Big-O only cares about "sufficiently large" inputs + +**What this means in plain English:** + +"Beyond a certain input size \(n_0\), our algorithm's runtime \(T(n)\) will never grow faster than some constant multiple \(c\) of \(f(n)\)." + +The beauty is that we can pick any constants that work—Big-O is about the **growth rate shape**, not the exact values. + +### Worked Example of the Definition + +Suppose we have: + +\[ T(n) = 3n^2 + 5n + 20 \] + +Claim: \(T(n)\) is \(O(n^2)\). + +**Proof:** + +We need to find constants \(c\) and \(n_0\) such that \(3n^2 + 5n + 20 \le c \cdot n^2\) for all \(n \ge n_0\). + +Let's choose \(n_0 = 1\) and see what \(c\) we need: + +For \(n \ge 1\): +- \(5n \le 5n^2\) (since \(n \ge 1\)) +- \(20 \le 20n^2\) (since \(n \ge 1\)) + +Therefore: + +\[ +\begin{align} +T(n) &= 3n^2 + 5n + 20 \\ +&\le 3n^2 + 5n^2 + 20n^2 \\ +&= 28n^2 +\end{align} +\] + +So with \(c = 28\) and \(n_0 = 1\), we've proven \(T(n) \le 28n^2\) for all \(n \ge 1\). + +Thus, \(T(n) = O(n^2)\). The exact coefficients (3, 5, 20) don't matter—only the dominant \(n^2\) term matters for large \(n\). + +### The Practical Simplification Rules + +Instead of doing formal proofs every time, we use these shortcuts: + +**Rule 1: Drop constant factors** +- \(5n\) becomes \(n\) +- \(100n^2\) becomes \(n^2\) +- Constants multiply the function but don't change its growth shape + +**Rule 2: Keep only the highest-order term** +- \(n^2 + n\) becomes \(n^2\) (because \(n^2\) dominates \(n\) as \(n\) grows) +- \(n^3 + 100n^2 + 5000n + 1000\) becomes \(n^3\) + +**Rule 3: Drop lower-order terms** +- In \(2n^2 + 5n + 7\), when \(n = 1000\): + - \(2n^2 = 2,000,000\) + - \(5n = 5,000\) + - \(7 = 7\) +- The \(n^2\) term is 400 times larger than the \(n\) term! +- As \(n\) grows, this gap only widens + +**Why these rules work:** For large \(n\), the fastest-growing term completely overwhelms the others. + +--- + +## Logical Steps for Analyzing Complexity + +Here's how to analyze any algorithm: + +**Step 1: Identify what \(n\) represents** +- Length of an array? +- Number of nodes in a graph? +- Number of elements in a matrix? +- Be precise—this is your input size metric. + +**Step 2: Count basic operations** +- Focus on the most frequently executed operations +- Common ones: comparisons, assignments, arithmetic operations +- Express the count as a function of \(n\) + +**Step 3: Build the time function \(T(n)\)** +- Add up contributions from all parts of the algorithm +- Sequential code: \(T_1(n) + T_2(n)\) +- Nested code: \(T_1(n) \times T_2(n)\) + +**Step 4: Simplify using the rules** +- Drop constants +- Keep highest-order term +- Express in Big-O notation + +--- + +## Common Complexity Classes: A Detailed Tour + +### Visual Hierarchy (Text-Based) + +``` +Faster Slower +<------------------------------------------------------> + O(1) O(log n) O(n) O(n log n) O(n²) O(2ⁿ) + | | | | | | +Constant Log Linear Linearithmic Quad Expo +``` + +### O(1) - Constant Time + +**Definition:** Runtime doesn't depend on input size at all. + +**Mathematical form:** \(T(n) = c\) for some constant \(c\) + +**Example:** Accessing an array element by index +```python +value = array[5] # Always takes the same time, regardless of array size +``` + +**Intuition:** No matter if your array has 10 elements or 10 million, accessing `array[5]` involves: +1. Calculate memory address: `base_address + 5 × element_size` +2. Retrieve value at that address -| Notation | Name | Typical example | -|---------|-----------------|----------------------------------------------------| -| O(1) | Constant time | Accessing an element by index in an array | -| O(log n)| Logarithmic time| Binary search on a sorted array | -| O(n) | Linear time | Scanning an array once (for example, linear search)| -| O(n log n)| Quasi-linear | Efficient sorts like mergesort, heapsort | -| O(n^2) | Quadratic time | Two nested loops over the same array | -| O(2^n) | Exponential time| Brute-force subset enumeration | +Both steps take constant time. -These classes describe growth rates: for large \(n\), an O(n log n) algorithm will eventually outperform an O(n^2) algorithm, even if the quadratic algorithm is faster for very small inputs. +### O(log n) - Logarithmic Time -## Visual intuition (growth curves) -A typical way to build intuition is to plot several functions on the same graph, such as \(n\), \(n \log n\), and \(n^2\), and see how they diverge as \(n\) increases. -In the MonCSDocs media repository, this topic can be illustrated with a diagram that shows multiple complexity curves on the same axes. +**Definition:** Runtime grows logarithmically with input size. Doubling input size adds only a constant amount of time. -Example placeholder (to be provided by moncsdocs-media): +**Mathematical form:** \(T(n) = c \log n\) -![Growth curves for common complexity classes](https://media.moncsdocs.moebiusorder.com/data/foundations/complexity/big-o-growth-curves.svg) +**Where \(\log\) appears:** Typically in algorithms that repeatedly **divide** the problem in half. -> Figure: Relative growth of O(1), O(log n), O(n), O(n log n), and O(n^2) as input size increases. +**Key insight:** \(\log_2(n)\) asks "how many times can we divide \(n\) by 2 until we reach 1?" -## Mathematical examples -Consider an algorithm whose operation count can be written as: +- \(\log_2(8) = 3\) because \(8 \to 4 \to 2 \to 1\) (3 divisions) +- \(\log_2(1024) = 10\) +- \(\log_2(1{,}000{,}000) \approx 20\) -\[ T(n) = 3n^2 + 5n + 20. \] +**Example:** Binary search (explained in detail later) -For sufficiently large \(n\), the \(3n^2\) term dominates the \(5n\) and constant 20. -Therefore, the algorithm has time complexity \(O(n^2)\). +**Growth comparison:** +``` +n log₂(n) Ratio +10 3.3 — +100 6.6 2× input, 2× time +1,000 10 10× input, 3× time +1,000,000 20 1000× input, 6× time +``` + +See how slowly it grows? This is incredibly efficient. + +### O(n) - Linear Time -As another example, suppose \(T(n) = 7n + 100\). -For large \(n\), the constant 100 becomes negligible relative to \(7n\), so the running time is \(O(n)\). +**Definition:** Runtime grows proportionally to input size. Double the input, double the time. -## Worked example: simple loop +**Mathematical form:** \(T(n) = c \cdot n\) -Consider the following pseudocode that sums an array of \(n\) elements: +**Example:** Finding the maximum value in an unsorted array +```python +max_val = array[0] +for i in range(1, n): + if array[i] > max_val: + max_val = array[i] +``` + +Why O(n)? We must check every element exactly once. Can't do better without additional information. -```text -sum = 0 -for i from 0 to n - 1: - sum = sum + a[i] -return sum +**Growth:** ``` +n Operations +10 10 +100 100 +1,000 1,000 +``` +Perfectly proportional. + +### O(n log n) - Linearithmic Time -- The body of the loop runs once for each element, so the number of additions is proportional to \(n\). -- Ignoring constant-time statements before and after the loop, the total time is proportional to \(n\). +**Definition:** Runtime is \(n\) times \(\log n\)—slightly worse than linear, but much better than quadratic. -This algorithm runs in linear time, so its time complexity is \(O(n)\). +**Mathematical form:** \(T(n) = c \cdot n \log n\) -## Worked example: nested loops +**Where it appears:** Efficient sorting algorithms (mergesort, heapsort, quicksort average case) -Consider a simple double loop that compares all pairs of elements in an array: +**Intuition:** These algorithms divide the problem (\(\log n\) levels) but do linear work (\(O(n)\)) at each level. -```text -for i from 0 to n - 1: - for j from 0 to n - 1: - compare(a[i], a[j]) +**Example breakdown for mergesort:** ``` +Level 0: [8, 3, 5, 1, 9, 2, 7, 4] ← n elements + / \ +Level 1: [8,3,5,1] [9,2,7,4] ← 2 groups, n total + / \ / \ +Level 2: [8,3] [5,1] [9,2] [7,4] ← 4 groups, n total + / \ / \ / \ / \ +Level 3: [8][3][5][1] [9][2][7][4] ← 8 groups, n total + +Total levels: log₂(n) = 3 +Work per level: O(n) merging +Total: O(n log n) +``` + +**Growth:** +``` +n n log n vs n² +10 33 100 (3× better) +100 664 10,000 (15× better) +1,000 9,966 1,000,000 (100× better) +``` + +### O(n²) - Quadratic Time + +**Definition:** Runtime grows with the square of input size. + +**Mathematical form:** \(T(n) = c \cdot n^2\) + +**Where it appears:** Nested loops over the same data -- The inner loop runs \(n\) times for each of the \(n\) iterations of the outer loop. -- The comparison executes \(n \times n = n^2\) times. +**Example:** Bubble sort, checking all pairs +```python +for i in range(n): + for j in range(n): + compare(array[i], array[j]) +``` + +Outer loop: \(n\) iterations +Inner loop: \(n\) iterations for each outer iteration +Total: \(n \times n = n^2\) comparisons + +**Growth:** +``` +n n² Time if n=100 takes 1s +10 100 0.01s +100 10,000 1s +1,000 1,000,000 100s (1.7 minutes) +10,000 100,000,000 10,000s (2.7 hours) +``` + +See the explosion? This is why O(n²) algorithms don't scale. + +### O(2ⁿ) - Exponential Time + +**Definition:** Runtime **doubles** with each additional input element. + +**Mathematical form:** \(T(n) = c \cdot 2^n\) + +**Where it appears:** Brute-force solutions that try all subsets, combinations + +**Example:** Generating all subsets of a set +``` +For set {1, 2, 3}: +2³ = 8 subsets: {}, {1}, {2}, {3}, {1,2}, {1,3}, {2,3}, {1,2,3} +``` + +**Growth (the nightmare scenario):** +``` +n 2ⁿ If each step = 1 microsecond +10 1,024 0.001 seconds +20 1,048,576 1 second +30 1,073,741,824 17.9 minutes +40 1,099,511,627,776 12.7 DAYS +50 1,125,899,906,842,624 35.7 YEARS +``` -This algorithm runs in quadratic time, so its time complexity is \(O(n^2)\). +This is why we avoid exponential algorithms at all costs for large inputs. -## How to read Big-O in practice -When you see a complexity like O(n log n), interpret it as "if the input size doubles, the running time grows a bit more than linearly, but far less than quadratically." -Big-O focuses on worst-case or upper-bound behavior unless otherwise specified, which is helpful when designing systems that must remain responsive even in heavy-load scenarios. +--- -## Common misconceptions and pitfalls -- Big-O is not about exact speed; a well-optimized O(n^2) algorithm can be faster than a poorly implemented O(n log n) algorithm on small inputs. -- Big-O usually ignores constant factors and lower-order terms, but in engineering practice those constants can matter for realistic input sizes. -- Big-O by itself does not distinguish between best, average, and worst case; those need to be specified separately if they differ. +## Visual Intuition: Growth Curves (Text Representation) -## Real-world applications -Big-O notation is used to compare data structures and algorithms when designing libraries, backend services, operating systems, and more. -It informs trade-offs: for example, choosing between a hash table (average-case O(1) lookup) and a balanced tree (O(log n) lookup) depending on performance guarantees and memory constraints. +Imagine plotting these functions for \(n\) from 1 to 100: -## Practice exercises -- Analyze the complexity of an algorithm that has one loop from 0 to n, and inside that loop calls a function that runs in O(log n) time. -- Given pseudocode with three nested loops, each running from 0 to n - 1, determine the Big-O time complexity. -- For a divide-and-conquer algorithm that splits the input into two halves and recursively processes both halves with a linear-time combine step, derive the Big-O complexity. +``` +Time + | + | * O(2ⁿ) + | * + | * + | * + | * * * * * * O(n²) + | * * * + | * * * + | * * * + | * * * O(n log n) + | * * + | * * O(n) + | O(log n) + |* O(1) + +-------------------------------------------------> Input size (n) + 0 10 20 30 40 50 60 70 80 90 100 +``` -## Related topics -- Asymptotic notations beyond Big-O: Omega (lower bound) and Theta (tight bound). -- Amortized analysis for operations whose cost varies between calls. -- Space complexity, which measures how memory usage grows with \(n\). +**Key observations:** +1. O(1) is flat—no growth +2. O(log n) barely rises—very gentle +3. O(n) rises steadily at 45° +4. O(n log n) rises slightly faster than O(n) +5. O(n²) curves upward dramatically +6. O(2ⁿ) shoots off the chart almost immediately + +For \(n = 100\): +- O(1): 1 operation +- O(log n): ~7 operations +- O(n): 100 operations +- O(n log n): ~664 operations +- O(n²): 10,000 operations +- O(2ⁿ): 1,267,650,600,228,229,401,496,703,205,376 operations (good luck!) --- + +## Detailed Worked Examples + +### Example 1: Simple Loop Analysis + +**Problem:** Analyze this function that sums an array: + +```python +def sum_array(array): + total = 0 # 1 operation + for i in range(n): # loop n times + total += array[i] # 2 operations: access + addition + return total # 1 operation +``` + +**Step-by-step analysis:** + +1. **Initialization:** `total = 0` → 1 operation +2. **Loop setup:** `range(n)` → considered O(1) setup +3. **Loop body:** Executes \(n\) times + - `array[i]`: 1 array access + - `+=`: 1 addition, 1 assignment + - Total per iteration: 3 operations +4. **Return:** 1 operation + +**Total operations:** + +\[ +\begin{align} +T(n) &= 1 + n \times 3 + 1 \\ +&= 3n + 2 +\end{align} +\] + +**Apply Big-O simplification:** +- Drop constant factor 3: \(3n \to n\) +- Drop constant term 2: \(n + 2 \to n\) + +**Result:** \(O(n)\) - linear time + +**Why this makes sense:** We touch each element exactly once. Can't compute the sum without looking at all elements. + +### Example 2: Nested Loops Analysis + +**Problem:** Analyze this function that finds all pairs: + +```python +def print_all_pairs(array): + for i in range(n): # outer loop: n times + for j in range(n): # inner loop: n times + print(array[i], array[j]) +``` + +**Detailed breakdown:** + +Outer loop iteration 0: +- \(j\) goes from 0 to \(n-1\): \(n\) print statements + +Outer loop iteration 1: +- \(j\) goes from 0 to \(n-1\): \(n\) print statements + +... + +Outer loop iteration \(n-1\): +- \(j\) goes from 0 to \(n-1\): \(n\) print statements + +**Total print statements:** + +\[ +T(n) = \underbrace{n + n + n + \cdots + n}_{\text{n times}} = n \times n = n^2 +\] + +**Result:** \(O(n^2)\) - quadratic time + +**Visual representation for \(n = 4\):** + +``` +j → 0 1 2 3 +i ↓ +0 * * * * ← 4 operations +1 * * * * ← 4 operations +2 * * * * ← 4 operations +3 * * * * ← 4 operations + +Total: 4 × 4 = 16 = n² +``` + +### Example 3: Dependent Nested Loops + +**Problem:** What if the inner loop depends on the outer loop variable? + +```python +def print_triangular_pairs(array): + for i in range(n): + for j in range(i): # j goes from 0 to i-1 + print(array[i], array[j]) +``` + +**Detailed counting:** + +``` +i = 0: j loops 0 times → 0 operations +i = 1: j loops 1 time → 1 operation +i = 2: j loops 2 times → 2 operations +i = 3: j loops 3 times → 3 operations +... +i = n-1: j loops (n-1) times → (n-1) operations +``` + +**Total operations:** + +\[ +T(n) = 0 + 1 + 2 + 3 + \cdots + (n-1) = \sum_{i=0}^{n-1} i +\] + +**Using the arithmetic series formula:** + +\[ +\sum_{i=0}^{n-1} i = \frac{(n-1) \cdot n}{2} = \frac{n^2 - n}{2} = \frac{1}{2}n^2 - \frac{1}{2}n +\] + +**Apply Big-O rules:** +- Drop constant factor \(\frac{1}{2}\) +- Drop lower-order term \(n\) +- Keep dominant term \(n^2\) + +**Result:** Still \(O(n^2)\), but runs about twice as fast as the previous example in practice. + +**Key lesson:** Even though it does half the work, the growth rate shape is the same—both are quadratic. + +--- + +## Practical Interpretation + +### Reading Big-O Expressions + +When you encounter \(O(n \log n)\), here's how to think about it: + +**Doubling the input:** +- New time ≈ 2 × old time × \(\frac{\log(2n)}{\log n}\) +- Since \(\log(2n) = \log 2 + \log n \approx \log n + 1\) +- Time increases by a factor slightly larger than 2 + +**10× the input:** +- New time ≈ 10 × old time × \(\frac{\log(10n)}{\log n}\) +- Factor ≈ 10 × 1.3 = 13 + +**100× the input:** +- Factor ≈ 100 × 1.67 = 167 + +Compare to: +- O(n): factors are exactly 2, 10, 100 +- O(n²): factors are 4, 100, 10,000 + +### Worst-Case vs. Average-Case + +Big-O typically describes **worst-case** behavior unless stated otherwise: + +**Example:** Binary search +- Worst case: \(O(\log n)\) — target is not in the array +- Best case: \(O(1)\) — target is the middle element +- Average case: \(O(\log n)\) — target equally likely to be anywhere + +We usually report the worst case because it provides a **guarantee**: "The algorithm will never be slower than this." + +--- + +## Common Misconceptions (Detailed Explanations) + +### Misconception 1: "Big-O tells me exact runtime" + +**Wrong:** Big-O is about **asymptotic growth**, not absolute time. + +Consider two algorithms: +- Algorithm A: \(T_A(n) = 1000n\) +- Algorithm B: \(T_B(n) = n^2\) + +Both are \(O(n^2)\) (A is actually \(O(n)\), but it's also \(O(n^2)\) since \(n \le n^2\) for \(n \ge 1\)). + +For \(n = 500\): +- Algorithm A: \(500{,}000\) operations +- Algorithm B: \(250{,}000\) operations + +Algorithm B is faster! But for \(n = 2000\): +- Algorithm A: \(2{,}000{,}000\) operations +- Algorithm B: \(4{,}000{,}000\) operations + +Now A is faster. **Eventually**, the \(O(n)\) algorithm beats the \(O(n^2)\) one, but "eventually" might be beyond your use case. + +**Takeaway:** Big-O describes long-term growth trends, not crossover points. + +### Misconception 2: "Lower Big-O is always better" + +**Not quite:** It depends on: + +1. **Input size:** For \(n < 10\), O(n²) might beat O(n log n) due to lower constants +2. **Hidden constants:** An O(n) algorithm with huge constants might lose to an optimized O(n log n) +3. **Space-time tradeoffs:** O(1) lookup in a hash table uses O(n) space + +**Example from practice:** + +Insertion sort (\(O(n^2)\)) vs. Quicksort (\(O(n \log n)\) average) + +For small arrays (n < 10), insertion sort often wins because: +- Simple inner loop = low constants +- Good cache locality +- No recursion overhead + +Many optimized quicksort implementations switch to insertion sort for small subarrays. + +### Misconception 3: "Big-O and actual complexity are the same" + +**Clarification:** Big-O is specifically an **upper bound**. + +**Example:** Linear search is \(O(n)\), but: +- Best case (found immediately): 1 comparison +- Average case (found in middle): \(n/2\) comparisons +- Worst case (found at end or not present): \(n\) comparisons + +All of these are \(O(n)\), but they're not all equal. + +There are related notations: +- **Big-Omega (Ω):** Lower bound (best case can't be better than this) +- **Big-Theta (Θ):** Tight bound (grows exactly at this rate) + +If an algorithm is both \(O(n \log n)\) and \(\Omega(n \log n)\), we say it's \(\Theta(n \log n)\)—it grows exactly at that rate. + +--- + +## Real-World Applications + +Big-O isn't just academic—it drives critical engineering decisions: + +### Database Indexing + +**Without index:** Finding a record in a million-row table +- Linear scan: \(O(n)\) = 1,000,000 comparisons + +**With B-tree index:** Logarithmic search +- \(O(\log n)\) ≈ 20 comparisons + +**Impact:** 50,000× speedup. This is why databases can feel "instant" even with huge tables. + +### Social Networks + +**Problem:** Find mutual friends between two users + +**Naive approach:** Check all pairs +- User A has \(n\) friends, User B has \(m\) friends +- \(O(n \times m)\) comparisons +- For \(n = m = 1000\): 1,000,000 comparisons + +**Optimized approach:** Use hash sets +1. Convert A's friends to hash set: \(O(n)\) +2. For each of B's friends, check if in set: \(O(m)\) with \(O(1)\) lookups +- Total: \(O(n + m)\) +- For \(n = m = 1000\): 2,000 operations + +**Impact:** 500× speedup + +### Video Game Collision Detection + +**Naive approach:** Check every object against every other object +- \(O(n^2)\) for \(n\) objects +- For \(n = 1000\) objects: 500,000 checks per frame +- At 60 FPS: 30,000,000 checks per second + +**Spatial partitioning:** Divide world into grid cells +- Only check objects in nearby cells +- Reduces to \(O(n)\) in practice +- For \(n = 1000\): ~5,000 checks per frame +- 100× speedup = smooth gameplay + +--- + +## Practice Exercises (Detailed Solutions) + +### Exercise 1: Loop with Inner Function Call + +**Problem:** What's the complexity? + +```python +for i in range(n): + binary_search(array, i) # O(log n) operation +``` + +**Solution:** +- Outer loop: \(n\) iterations +- Each iteration: \(O(\log n)\) work +- Total: \(n \times \log n = O(n \log n)\) + +**Key principle:** Nested complexity multiplies. + +### Exercise 2: Three Nested Loops + +**Problem:** Analyze: + +```python +for i in range(n): + for j in range(n): + for k in range(n): + print(i, j, k) +``` + +**Solution:** +- Outermost loop: \(n\) iterations +- Middle loop: \(n\) iterations for each outer +- Innermost loop: \(n\) iterations for each middle +- Total: \(n \times n \times n = n^3\) +- **Result:** \(O(n^3)\) - cubic time + +**Growth:** For \(n = 100\): 1,000,000 operations. For \(n = 1000\): 1,000,000,000 operations. This doesn't scale well! + +### Exercise 3: Divide-and-Conquer Recurrence + +**Problem:** An algorithm splits input in half, recursively processes both halves, then combines results in linear time: + +``` +T(n) = 2T(n/2) + O(n) +``` + +**Solution using the Master Theorem:** + +Form: \(T(n) = aT(n/b) + f(n)\) + +Here: \(a = 2\), \(b = 2\), \(f(n) = O(n)\) + +Compute: \(n^{\log_b a} = n^{\log_2 2} = n^1 = n\) + +Since \(f(n) = \Theta(n) = \Theta(n^{\log_b a})\), we're in Case 2: + +\[ T(n) = \Theta(n^{\log_b a} \log n) = \Theta(n \log n) \] + +**Result:** \(O(n \log n)\) + +**This is mergesort!** Each level does \(O(n)\) work, there are \(\log n\) levels, so total is \(O(n \log n)\). + +--- + +## Related Topics and Extensions + +### Beyond Big-O: The Asymptotic Notation Family + +**Big-Omega (Ω) - Lower Bound:** + +\(f(n) = \Omega(g(n))\) means \(f(n)\) grows at least as fast as \(g(n)\). + +**Example:** Any comparison-based sorting algorithm is \(\Omega(n \log n)\)—you can't sort faster than this without additional assumptions. + +**Big-Theta (Θ) - Tight Bound:** + +\(f(n) = \Theta(g(n))\) means \(f(n)\) grows exactly at the same rate as \(g(n)\). + +**Example:** Mergesort is \(\Theta(n \log n)\)—its best, average, and worst cases are all \(n \log n\). + +### Amortized Analysis + +**Problem:** Some operations have varying costs. + +**Example:** Dynamic array resizing +- Most insertions: \(O(1)\) +- Occasionally (when full): \(O(n)\) to copy all elements to larger array + +**Amortized cost:** \(O(1)\) per insertion averaged over many operations. + +**Why:** The expensive \(O(n)\) operations happen so rarely that their cost "spreads out" to \(O(1)\) per operation. + +### Space Complexity + +**Same notation, different resource:** How does memory usage grow? + +**Examples:** +- Iterative binary search: \(O(1)\) space (just a few variables) +- Recursive binary search: \(O(\log n)\) space (call stack) +- Mergesort: \(O(n)\) space (needs auxiliary array) +- Quicksort: \(O(\log n)\) space average (in-place, recursion stack) + +**Tradeoff:** Often faster algorithms use more space (e.g., hash tables). + +--- + +## Summary: Key Takeaways + +1. **Big-O describes growth rates**, not exact speeds +2. **Focus on dominant terms** for large inputs +3. **Constants matter in practice**, but not in Big-O +4. **O(log n) is very fast**, O(n) is acceptable, O(n log n) is good for sorting, O(n²) struggles at scale, O(2ⁿ) is usually impractical +5. **Worst-case analysis** provides guarantees +6. **Real-world impact:** The difference between O(n²) and O(n log n) can mean seconds vs. hours on large datasets + +--- + [Back to Foundational Theory index](./README.md)