a not-too-general value backwards induction algorithm built on DBlocks by sbenthall · Pull Request #1438 · econ-ark/HARK

sbenthall · 2024-06-03T13:13:36Z

This PR aims to provide a general backwards induction algorithm that operates on a single DBlock.

It simply arranges the pieces that come with the DBlock (decision value function, arrival value function), in a sensible way, with an optimization over the action space to find the best action at each point in the state grid.

While the algorithm is not as good as FOC or EGM methods, I'm doing this for several reasons:

As a way to work out the API design for solution algorithms more generally
As a way to put pressure on the HARK 1.0 proto-language to include details that have been missing (such as bounds on actions)
As a way to demonstrate the way discretization of shocks is used for solutions, so that I can build a demo of how the same discretization can be used for solution and simulation. (This is a target for my upcoming SciPy deliverable)

The first commit of this PR just sketches the algorithm out, but it isn't tested.

One question is how to design this so that it is more agnostic to different optimizers. I know @alanlujan91 is keen on estimagic.

Tests for new functionality/models or Tests to reproduce the bug-fix in code.
Updated documentation of features that add new functionality.
Update CHANGELOG.md with major/minor changes.

sbenthall · 2024-06-03T14:33:43Z

Clarifying that by "general", I mean "not-too-general". A "not-too-general" solver, by my definition, is a solver that:

contains no specific model data
but only works when certain conditions on the model provided as input are met.

This is in contrast with a "model specific solver", which is what most HARK models currently have; these solvers contain specific model data.

sbenthall · 2024-06-04T21:16:11Z

HARK/models/consumer.py

            "b": lambda k, R, PermGroFac: k * R / PermGroFac,
            "m": lambda b, theta: b + theta,
-            "c": Control(["m"]),
+            "c": Control(["m"], upper_bound=lambda m: m),


@mnwhite @alanlujan91 I wonder what you think about this way of introducing upper/lower bound information on control variables.

I think this looks good.

If we wanted to differentiate between a fixed (real number) upper bound, and a functional upper bound, we could use the term upper_envelope for a function/lambda.

…ng test on normalized consumption block

sbenthall · 2024-06-05T20:50:09Z

This PR now includes:

setting upper and lower bounds on a control variable programmatically
passing those bounds through to the VBI solver
an automated test showing proper functionality on the normalized consumption block for the last period (solves for consume all resources).

This PR is now ready for review. Requesting review from @alanlujan91 or @mnwhite

As I mentioned in the original post for this PR, my next step, building on this PR, is a demonstration of a configuration that will use:

a 'true' model definition
a discretization of the model
using the same discretization for both solving and simulation

sbenthall · 2024-06-18T19:58:26Z

One thought on how to improve this PR:

the value backwards induction algorithm implementation currently assumes one control variable in the block ('stage'). Of course, that's not fully general.
fixing it so that it works when there are no control variables is trivial to implement, and would illustrate how the '2 or 3 value function?' question can be easily handled with contextual information. No big deal
Working with multiple control variables is a little trickier but in principle doable. I'd like to leave this out of scope for this PR.

sbenthall · 2024-11-14T15:44:06Z

Recent changes add handling and tests for cases where the stage has zero control variables.

sbenthall · 2024-11-14T15:45:02Z

This is now ready for review again.

Copilot

Pull request overview

This PR introduces a general value backwards induction (VBI) algorithm that operates on DBlock objects, enabling solution of dynamic programming problems through grid-based optimization. The implementation serves as a foundation for API design of solution algorithms and demonstrates integration with the HARK model framework.

Changes:

Adds HARK.algos.vbi module with a solve function implementing value backwards induction on DBlocks
Extends the Control class to support upper and lower bounds on control variables
Adds get_controls() method and enhances transition() method in DBlock with a screen parameter
Updates model definitions to include upper bounds on control variables

Reviewed changes

Copilot reviewed 12 out of 13 changed files in this pull request and generated 19 comments.

Show a summary per file

File	Description
HARK/algos/vbi.py	New module implementing value backwards induction algorithm with grid-based optimization
HARK/algos/tests/test_vbi.py	Test suite for VBI algorithm with multiple test cases
HARK/model.py	Extended Control class with bounds and enhanced DBlock transition method
HARK/models/consumer.py	Added upper bounds to control variables and switched to normalized blocks
HARK/models/perfect_foresight.py	Added upper bound to consumption control
HARK/models/perfect_foresight_normalized.py	Added upper bound to consumption control
HARK/models/test_models.py	Removed unused initial state "p" from test configurations
Documentation/CHANGELOG.md	Added changelog entry for VBI algorithm
Documentation/reference/index.rst	Added algos module to documentation index
Documentation/reference/tools/algos.rst	New documentation file for algos module
Documentation/reference/tools/algos/vbi.rst	New documentation file for vbi submodule
Documentation/reference/tools/index.rst.orig	Merge conflict file that should be removed

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-28T19:20:26Z

HARK/algos/vbi.py

+            # if no controls, no optimization is necessary
+            pass


When there are no control variables (len(controls) == 0), the code does nothing and proceeds with uninitialized policy_data and value_data. This will likely cause issues when trying to use these variables later. Consider either handling this case properly or raising an informative error.

Suggested change

# if no controls, no optimization is necessary

pass

# if no controls, no optimization is necessary; directly evaluate value

dr_best = {}

value = srv_function(pre_states, dr_best)

value_data.sel(**state_vals).variable.data.put(0, value)

Copilot · 2026-01-28T19:20:27Z

HARK/algos/tests/test_vbi.py

+)
+
+
+class test_vbi(unittest.TestCase):


The class name test_vbi does not follow Python naming conventions for test classes, which should use PascalCase (e.g., TestVbi or TestVBI). This inconsistency may cause issues with some test discovery tools.

Suggested change

class test_vbi(unittest.TestCase):

class TestVbi(unittest.TestCase):

Copilot · 2026-01-28T19:20:27Z

HARK/models/test_models.py

                "k": Lognormal(-6, 0),
                #'live' : 1,
-                "p": 1.0,
+                # "p": 1.0,- not needed, using normalized problem


Missing space after the comment marker. Should be "# "p": 1.0, - not needed" instead of "#"p": 1.0,- not needed" for consistency with Python style guidelines.

Suggested change

# "p": 1.0,- not needed, using normalized problem

# "p": 1.0, - not needed, using normalized problem

Copilot · 2026-01-28T19:20:27Z

HARK/algos/vbi.py

+        pre_states.update(state_vals)
+
+        # prepare function to optimize
+        def negated_value(a):  # old! (should be negative)


The comment "old! (should be negative)" is unclear and potentially confusing. If this is outdated information, it should be removed. If it's meant to explain why the function is negated, consider clarifying the comment to say something like "negated for use with minimization optimizer".

Copilot · 2026-01-28T19:20:27Z

HARK/algos/vbi.py

+
+            bounds = ((lower_bound, upper_bound),)
+
+            res = minimize(  # choice of


The comment "# choice of" on line 141 is incomplete. It appears to be a placeholder or unfinished thought that should either be completed or removed.

Suggested change

res = minimize( # choice of

res = minimize( # optimize the (bounded) scalar control using scipy.optimize.minimize

Copilot · 2026-01-28T19:20:30Z

HARK/algos/vbi.py

+    dr_from_data = {
+        c: ar_from_data(
+            policy_data
+        )  # maybe needs to be more sensitive to the information set
+        for i, c in enumerate(controls)
+    }


All controls share the same policy_data array in the decision rule construction. For problems with multiple controls, each control should have its own policy array. This will cause issues when the code is extended to handle multiple controls (currently it raises an exception for len(controls) > 1).

Suggested change

dr_from_data = {

c: ar_from_data(

policy_data

) # maybe needs to be more sensitive to the information set

for i, c in enumerate(controls)

}

if len(controls) == 0:

# no controls: empty decision rule

dr_from_data = {}

elif len(controls) == 1:

# single control: construct rule from the corresponding policy data

dr_from_data = {controls[0]: ar_from_data(policy_data)}

else:

# multiple controls are not yet supported in value backup iteration

raise Exception(

f"Value backup iteration is not yet implemented for stages with {len(controls)} > 1 control variables."

)

Copilot · 2026-01-28T19:20:30Z

Documentation/CHANGELOG.md

 - Introduces `HARK.parser' module for parsing configuration files into models [#1427](https://github.com/econ-ark/HARK/pull/1427)
 - Allows construction of shocks with arguments based on mathematical expressions [#1464](https://github.com/econ-ark/HARK/pull/1464)
 - YAML configuration file for the normalized consumption and portolio choice [#1465](https://github.com/econ-ark/HARK/pull/1465)
+- Introduces value backup algorithm on DBlocks [#1438]https://github.com/econ-ark/HARK/pull/1438)


Missing opening parenthesis in the markdown link. The link should be formatted as [#1438](https://github.com/econ-ark/HARK/pull/1438) but is currently missing the opening parenthesis before https.

Suggested change

- Introduces value backup algorithm on DBlocks [#1438]https://github.com/econ-ark/HARK/pull/1438)

- Introduces value backup algorithm on DBlocks [#1438](https://github.com/econ-ark/HARK/pull/1438)

Copilot · 2026-01-28T19:20:31Z

HARK/algos/vbi.py

+
+                dr_best = {c: get_action_rule(res.x[i]) for i, c in enumerate(controls)}
+
+                policy_data.sel(**state_vals).variable.data.put(0, res.x[0])  # ?


The question mark comment "# ?" on line 162 suggests uncertainty about this code. If there's a known issue or concern, it should be documented more clearly. If not, the comment should be removed.

Suggested change

policy_data.sel(**state_vals).variable.data.put(0, res.x[0]) # ?

policy_data.sel(**state_vals).variable.data.put(0, res.x[0])

Copilot · 2026-01-28T19:20:31Z

HARK/model.py

+    iset : list of str
        The labels of the variables that are in the information set of this control.
+
+    upper_bound : function


The docstring is incomplete. It only documents iset and upper_bound parameters, but the __init__ method also accepts a lower_bound parameter that should be documented.

Suggested change

upper_bound : function

lower_bound : function, optional

An 'equation function' which evaluates to the lower bound of the control variable.

upper_bound : function, optional

Copilot · 2026-01-28T19:20:31Z

HARK/algos/tests/test_vbi.py

+    # def setUp(self):
+    #    pass
+
+    def test_solve_block_1(self):
+        state_grid = {"m": np.linspace(0, 2, 10)}
+
+        dr, dec_vf, arr_vf = vbi.solve(block_1, lambda a: a, state_grid)
+


This comment appears to contain commented-out code.

Suggested change

# def setUp(self):

# pass

def test_solve_block_1(self):

state_grid = {"m": np.linspace(0, 2, 10)}

dr, dec_vf, arr_vf = vbi.solve(block_1, lambda a: a, state_grid)

def test_solve_block_1(self):

state_grid = {"m": np.linspace(0, 2, 10)}

dr, dec_vf, arr_vf = vbi.solve(block_1, lambda a: a, state_grid)

dr, dec_vf, arr_vf = vbi.solve(block_1, lambda a: a, state_grid)

sbenthall added 4 commits June 3, 2024 09:05

sketch of a general value backwards induction algorithm built on DBlocks

d529a44

ruff

431fc80

fixes for doc errors

fd603b5

adding algos to docs table of contents

b1daaff

sbenthall changed the title ~~a general value backwards induction algorithm built on DBlocks~~ a not-to-general value backwards induction algorithm built on DBlocks Jun 3, 2024

sbenthall changed the title ~~a not-to-general value backwards induction algorithm built on DBlocks~~ a not-too-general value backwards induction algorithm built on DBlocks Jun 3, 2024

sbenthall added 4 commits June 4, 2024 15:26

automated test for vbi solution algorithm

510acaf

Merge branch 'master' into algos-b

219b03c

changelog for vbi solver in 0.16.0 release now

7b0bf4f

first try at adding upper bound information to consumer model

0e9300c

sbenthall commented Jun 4, 2024

View reviewed changes

using lower and upper bounds given to Control in vbi algorithm; passi…

1770ab0

…ng test on normalized consumption block

sbenthall requested review from alanlujan91 and mnwhite June 5, 2024 20:45

passing disc_params to consumption block vbi solver test

aba2c55

sbenthall added 4 commits June 5, 2024 16:51

ruff fixes

e676627

fixing the general model tests -- 'screening' transition is optional

3e90f64

documentating optional screening of transition function

32eb4c1

Merge branch 'master' into algos-b

7fb5553

sbenthall added 3 commits June 18, 2024 16:00

touch ups

6ec5d5e

Merge branch 'master' into algos-b

8b43ca2

don't need to initialize income in normalized models

7040c44

sbenthall added the Status: In Progress In Progress and NOT-OK to be merged. label Jun 25, 2024

sbenthall mentioned this pull request Jul 17, 2024

Monte Carlo Simulator that accepts aggregate blocks #1471

Closed

Merge branch 'master' into algos-b

1e22c86

sbenthall added 2 commits November 14, 2024 10:42

vbi algo handling cases where stage has 0 controls

61fd2f7

rufff fixes

8f45b79

sbenthall added Status: Review Needed and removed Status: In Progress In Progress and NOT-OK to be merged. labels Nov 14, 2024

This was referenced Nov 14, 2024

standalone model configuration file with model, approximation, solution, and simulation parameters #1463

Open

Generalized backwards induction solver (Micro agent version) #938

Closed

Merge branch 'master' into algos-b

4f83723

mnwhite moved this to Needs Review in Issues & PRs May 14, 2025

mnwhite added this to Issues & PRs May 14, 2025

alanlujan91 requested a review from Copilot January 28, 2026 19:15

Copilot started reviewing on behalf of alanlujan91 January 28, 2026 19:15 View session

Copilot AI reviewed Jan 28, 2026

View reviewed changes

-            # if no controls, no optimization is necessary
-            pass
+            # if no controls, no optimization is necessary; directly evaluate value
+            dr_best = {}
+            value = srv_function(pre_states, dr_best)
+            value_data.sel(**state_vals).variable.data.put(0, value)

	class test_vbi(unittest.TestCase):
	class TestVbi(unittest.TestCase):

	# "p": 1.0,- not needed, using normalized problem
	# "p": 1.0, - not needed, using normalized problem


		bounds = ((lower_bound, upper_bound),)

		res = minimize( # choice of

	res = minimize( # choice of
	res = minimize( # optimize the (bounded) scalar control using scipy.optimize.minimize

-    dr_from_data = {
-        c: ar_from_data(
-            policy_data
-        )  # maybe needs to be more sensitive to the information set
-        for i, c in enumerate(controls)
-    }
+    if len(controls) == 0:
+        # no controls: empty decision rule
+        dr_from_data = {}
+    elif len(controls) == 1:
+        # single control: construct rule from the corresponding policy data
+        dr_from_data = {controls[0]: ar_from_data(policy_data)}
+    else:
+        # multiple controls are not yet supported in value backup iteration
+        raise Exception(
+            f"Value backup iteration is not yet implemented for stages with {len(controls)} > 1 control variables."
+        )

	- Introduces value backup algorithm on DBlocks [#1438]https://github.com/econ-ark/HARK/pull/1438)
	- Introduces value backup algorithm on DBlocks [#1438](https://github.com/econ-ark/HARK/pull/1438)


		dr_best = {c: get_action_rule(res.x[i]) for i, c in enumerate(controls)}

		policy_data.sel(**state_vals).variable.data.put(0, res.x[0]) # ?

-    upper_bound : function
+    lower_bound : function, optional
+        An 'equation function' which evaluates to the lower bound of the control variable.
+    upper_bound : function, optional

Uh oh!

Conversation

sbenthall commented Jun 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sbenthall commented Jun 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sbenthall Jun 4, 2024

Choose a reason for hiding this comment

Uh oh!

alanlujan91 Jun 4, 2024

Choose a reason for hiding this comment

Uh oh!

sbenthall commented Jun 5, 2024

Uh oh!

sbenthall commented Jun 18, 2024

Uh oh!

sbenthall commented Nov 14, 2024

Uh oh!

sbenthall commented Nov 14, 2024

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sbenthall commented Jun 3, 2024 •

edited

Loading

sbenthall commented Jun 3, 2024 •

edited

Loading