From 601e78e7101926242b919faaa1577cc6fdce3aa6 Mon Sep 17 00:00:00 2001 From: Paul Park Date: Mon, 12 Dec 2022 17:12:45 +0000 Subject: [PATCH 01/30] adding documentation for advanced end-to-end examples of probe and meter use --- docs/probe.md | 44 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100644 docs/probe.md diff --git a/docs/probe.md b/docs/probe.md new file mode 100644 index 000000000..e135b62a8 --- /dev/null +++ b/docs/probe.md @@ -0,0 +1,44 @@ +# Probes and Meters: Advanced End-to-End Examples +For an introduction to `Probe`s and `Meter`s, please refer to [Measurement Overview](./metrics.md). + +## Context +To monitor particular aspects of an `armory run` session, the user needs to know the following factors: +- What am I measuring? +- When should I measure it? +- Where should my custom monitoring script go? + +The examples in this section highlight the nuances of using `Probe`s and `Meter`s for flexible monitoring arrangements in `armory`. + +## Example 1: Model Layer Output +### User Story +I have a `PyTorchFasterRCNN` model and I am interested in output from the `relu` activation of the second `Bottleneck` of `layer4` +### Example Code +This is an example of working with a python packages/framework (i.e. `pytorch`) that comes with built-in hooking mechanisms. In the code snippet below, we are relying on an existing function `register_forward_hook` to monitor the layer of interest: +```python +from armory.scenarios.main import get as get_scenario + +s = get_scenario( + "/armory/tmp/2022-11-03T180812.020999/carla_obj_det_adversarialpatch_undefended.json", + num_eval_batches=1, +).load() + +# create the probe with "test" namespace +probe = get_probe("test") + +# define the hook to pass to "register_forward_hook" +def hook_fn(hook_module, hook_input, hook_output): + probe.update(lambda x: x.detach().cpu().numpy(), layer4_2_relu=hook_output[0][0]) # [0][0] for slicing + +s.model.model.backbone.body.layer4[2].relu.register_forward_hook(hook_fn) + +meter = Meter("layer4_2_relu", lambda x: x[0,0,:,:], "test.layer4_2_relu") +get_hub().connect_meter(meter) + +s.next() +s.run_attack() +``` +That a package provides a hooking mechanism is convenient, but the user also has to be aware of the what to pass to the hooking mechanism as well as what format to pass it in. + +Note that `pytorch` also provides other hooking functionality such as: +- `register_forward_pre_hook` +- `register_full_backward_hook` \ No newline at end of file From 3f739a447c4e0d3ca95d07620dbe504ff2260853 Mon Sep 17 00:00:00 2001 From: Paul Park Date: Mon, 12 Dec 2022 18:36:55 +0000 Subject: [PATCH 02/30] finished initial draft of Example 1 --- docs/probe.md | 57 +++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 49 insertions(+), 8 deletions(-) diff --git a/docs/probe.md b/docs/probe.md index e135b62a8..fea2ae4e1 100644 --- a/docs/probe.md +++ b/docs/probe.md @@ -11,34 +11,75 @@ The examples in this section highlight the nuances of using `Probe`s and `Meter` ## Example 1: Model Layer Output ### User Story -I have a `PyTorchFasterRCNN` model and I am interested in output from the `relu` activation of the second `Bottleneck` of `layer4` +I have a `PyTorchFasterRCNN` model and I am interested in output from the `relu` activation of the third (index 2) `Bottleneck` of `layer4` ### Example Code -This is an example of working with a python packages/framework (i.e. `pytorch`) that comes with built-in hooking mechanisms. In the code snippet below, we are relying on an existing function `register_forward_hook` to monitor the layer of interest: -```python +This is an example of working with a python package/framework (i.e. `pytorch`) that comes with built-in hooking mechanisms. In the code snippet below, we are relying on an existing function `register_forward_hook` to monitor the layer of interest: +```python showLineNumbers from armory.scenarios.main import get as get_scenario +from armory.instrument import get_probe, Meter, get_hub +# load Scenario s = get_scenario( "/armory/tmp/2022-11-03T180812.020999/carla_obj_det_adversarialpatch_undefended.json", num_eval_batches=1, ).load() -# create the probe with "test" namespace +# create Probe with "test" namespace probe = get_probe("test") # define the hook to pass to "register_forward_hook" -def hook_fn(hook_module, hook_input, hook_output): +# the signature of 3 inputs is what pytorch expects +# hook_module refers to the layer of interest, but is not explicitly referenced when passing to register_forward_hook +def hook_fn(hook_module, hook_input, hook_output): probe.update(lambda x: x.detach().cpu().numpy(), layer4_2_relu=hook_output[0][0]) # [0][0] for slicing +# register hook +# the hook_module mentioned earlier is referenced via s.model.model.backbone.body.layer4[2].relu +# the register_forward_hook method call must be passing self as a hook_module to hook_fn s.model.model.backbone.body.layer4[2].relu.register_forward_hook(hook_fn) -meter = Meter("layer4_2_relu", lambda x: x[0,0,:,:], "test.layer4_2_relu") +# create Meter for Probe with "test" namespace +meter = Meter("layer4_2_relu", lambda x: x, "test.layer4_2_relu") + +# connect Meter to Hub get_hub().connect_meter(meter) s.next() s.run_attack() ``` -That a package provides a hooking mechanism is convenient, but the user also has to be aware of the what to pass to the hooking mechanism as well as what format to pass it in. + +### Packages with Hooks +That a package provides a hooking mechanism is convenient, but the user also has to be aware of the what to pass to the hooking mechanism as well as what format to pass it in. Please reference [`pytorch` documentation](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_forward_hook) for more details regarding this example. Note that `pytorch` also provides other hooking functionality such as: - `register_forward_pre_hook` -- `register_full_backward_hook` \ No newline at end of file +- `register_full_backward_hook` + +### Probe and Meter Details +Aside the specifics of using `register_forward_hook`, consider how `Probe` and `Meter` are incorporated in this example. Recall the 4 steps for a minimal working example (in [Measurement Overview](./metrics.md)): +1. Create `Probe` via `get_probe("test")` +2. Define `Probe` actions +3. Create `Meter` with processing functions that take input from created `Probe` +4. Connect `Meter` to `Hub` via `get_hub().connect_meter(meter)` + +#### Step 1 +Note the input `"test"` that is passed in `get_probe("test")` - this needs to match with the first portion of a `.`-separated argument name `"test.layer4_2_relu"` that is passed to creating a `Meter` in [Step 3](#step3) + +#### Step 2 +The `update` method for `Probe` takes as input optional processing functions and variable names and corresponding values that are to be monitored. +- The variable name `layer4_2_relu` is how we are choosing to reference a certain value + - this needs to match with the second portion of a `.`-separated argument name `"test.layer4_2_relu"` that is passed to creating a `Meter` in [Step 3](#step3) +- `hook_output[0][0]` is the value we are interested in, which is the output from `s.model.model.backbone.body.layer4[2].relu` after a forward pass + - `[0][0]` was included to slice the output to show that it can be done, and because we know the shape of the output in advance +- `lambda x: x.detach().cpu().numpy()` is the processing function that converts `hook_output[0][0]` from a tensor to an array + +#### Step 3 +In this particular example, the `Meter` accepts 3 inputs: a meter name, a metric/function for processing, and a argument name to pass the metric/function. +- The meter name (`"layer4_2_relu"`) can be arbitrary within this context +- For the scope of this section, we only consider simple `Meter`s with the identity function as a metric i.e. `Meter` will record variables monitored by `Probe` as-is (thus `lambda x: x`) +- The argument passed to the metric/function follows a `.`-separated format (`"test.layer4_2_relu"`), which needs to be consistent with `Probe` setup: + - `test` matches input in `get_probe("test")` + - `layer4_2_relu` matches variable name in `layer4_2_relu=hook_output[0][0]` + +#### Step 4 + We don't dwell on what `armory` is doing in step 4 with `get_hub().connect_meter(meter)` other than to mention this step is necessary. \ No newline at end of file From 7db1aa4162ac91732c002bfd53efd85a2a0ab203 Mon Sep 17 00:00:00 2001 From: Paul Park Date: Tue, 13 Dec 2022 06:41:36 +0000 Subject: [PATCH 03/30] updated Example 1 with an extra step for connecting Probe, which needs more details; finished initial draft of Example 2 --- docs/probe.md | 135 ++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 125 insertions(+), 10 deletions(-) diff --git a/docs/probe.md b/docs/probe.md index fea2ae4e1..3cf47f101 100644 --- a/docs/probe.md +++ b/docs/probe.md @@ -12,9 +12,9 @@ The examples in this section highlight the nuances of using `Probe`s and `Meter` ## Example 1: Model Layer Output ### User Story I have a `PyTorchFasterRCNN` model and I am interested in output from the `relu` activation of the third (index 2) `Bottleneck` of `layer4` -### Example Code +### Example Code This is an example of working with a python package/framework (i.e. `pytorch`) that comes with built-in hooking mechanisms. In the code snippet below, we are relying on an existing function `register_forward_hook` to monitor the layer of interest: -```python showLineNumbers +```python from armory.scenarios.main import get as get_scenario from armory.instrument import get_probe, Meter, get_hub @@ -56,14 +56,15 @@ Note that `pytorch` also provides other hooking functionality such as: - `register_full_backward_hook` ### Probe and Meter Details -Aside the specifics of using `register_forward_hook`, consider how `Probe` and `Meter` are incorporated in this example. Recall the 4 steps for a minimal working example (in [Measurement Overview](./metrics.md)): +Aside the specifics of using `register_forward_hook`, consider how `Probe` and `Meter` are incorporated in this example. Recall the steps for a minimal working example (in [Measurement Overview](./metrics.md)): 1. Create `Probe` via `get_probe("test")` 2. Define `Probe` actions -3. Create `Meter` with processing functions that take input from created `Probe` -4. Connect `Meter` to `Hub` via `get_hub().connect_meter(meter)` +3. Connect `Probe` +4. Create `Meter` with processing functions that take input from created `Probe` +5. Connect `Meter` to `Hub` via `get_hub().connect_meter(meter)` #### Step 1 -Note the input `"test"` that is passed in `get_probe("test")` - this needs to match with the first portion of a `.`-separated argument name `"test.layer4_2_relu"` that is passed to creating a `Meter` in [Step 3](#step3) +Note the input `"test"` that is passed in `get_probe("test")` - this needs to match with the first portion of a `.`-separated argument name `"test.layer4_2_relu"` that is passed to creating a `Meter` in [Step 3](#step4) #### Step 2 The `update` method for `Probe` takes as input optional processing functions and variable names and corresponding values that are to be monitored. @@ -73,13 +74,127 @@ The `update` method for `Probe` takes as input optional processing functions and - `[0][0]` was included to slice the output to show that it can be done, and because we know the shape of the output in advance - `lambda x: x.detach().cpu().numpy()` is the processing function that converts `hook_output[0][0]` from a tensor to an array -#### Step 3 +#### Step 3 +This particular step is not dealt with in-depth in [Measurement Overview](./metrics.md), but requires more explanation for this section. + +#### Step 4 In this particular example, the `Meter` accepts 3 inputs: a meter name, a metric/function for processing, and a argument name to pass the metric/function. - The meter name (`"layer4_2_relu"`) can be arbitrary within this context -- For the scope of this section, we only consider simple `Meter`s with the identity function as a metric i.e. `Meter` will record variables monitored by `Probe` as-is (thus `lambda x: x`) +- For the scope of this document, we only consider simple `Meter`s with the identity function as a metric i.e. `Meter` will record variables monitored by `Probe` as-is (thus `lambda x: x`) - The argument passed to the metric/function follows a `.`-separated format (`"test.layer4_2_relu"`), which needs to be consistent with `Probe` setup: - `test` matches input in `get_probe("test")` - `layer4_2_relu` matches variable name in `layer4_2_relu=hook_output[0][0]` -#### Step 4 - We don't dwell on what `armory` is doing in step 4 with `get_hub().connect_meter(meter)` other than to mention this step is necessary. \ No newline at end of file +#### Step 5 +For the scope of this document, we don't dwell on what `armory` is doing in step 5 with `get_hub().connect_meter(meter)` other than to mention this step is necessary. + +## Example 2: Attack Artifact - Available as Output +### User Story +I am using `CARLADapricotPatch` I am interested in the patch after every iteration, which is generated by `CARLADapricotPatch._augment_images_with_patch` +### Example Code +This is an example of working with a python package/framework (i.e. `art`) that does NOT come with built-in hooking mechanisms. In the code snippet below, we define wrapper functions to wrap existing instance methods to monitor the output of interest: +```python +from armory.scenarios.main import get as get_scenario +from armory.instrument import get_probe, Meter, get_hub +import types + +def method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None): + """ + Hook target method and return the original method + If a class is passed in, hooks ALL instances of class. + If an object is passed in, only hooks the given instance. + """ + if not isinstance(obj, object): + raise ValueError(f"obj {obj} is not a class or object") + method = getattr(obj, method_name) + if not callable(method): + raise ValueError(f"obj.{method_name} attribute {method} is not callable") + wrapped = hook_wrapper( + method, pre_method_hook=pre_method_hook, post_method_hook=post_method_hook + ) + + if isinstance(obj, type): + cls = obj + setattr(cls, method_name, wrapped) + else: + setattr(obj, method_name, types.MethodType(wrapped, obj)) + + return method + +def hook_wrapper(method, pre_method_hook = None, post_method_hook = None): + def wrapped(*args, **kwargs): + return_value = method(*args[1:], **kwargs) + post_method_hook(*return_value) + return return_value + + return wrapped + +def post_method_hook(x_patch, patch_target, transformations): + probe.update(x_patch=x_patch) # adding batch dim + +# load Scenario +s = get_scenario( + "/armory/tmp/2022-11-03T180812.020999/carla_obj_det_adversarialpatch_undefended.json", + num_eval_batches=1, +).load() + +# create Probe with "hooked_method" namespace +probe = get_probe("hooked_method") + +# register hook that will update Probe +method_hook( + s.attack, "_augment_images_with_patch", post_method_hook=post_method_hook +) + +# create Meter for Probe with "hooked_method" namespace +hook_meter = Meter( + "hook_x_patch", lambda x: x, "hooked_method.x_patch" +) + +# connect Meter to Hub +get_hub().connect_meter(hook_meter) + +s.next() +s.run_attack() +``` +### Packages with NO Hooks +Unlike [Example 1](#example1), we have defined new functions to meet user needs: +- `method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None)` +- `hook_wrapper(method, pre_method_hook = None, post_method_hook = None)` +- `post_method_hook(x_patch, patch_target, transformations)` + +The steps are the same as before, with the exception that [Step 3](#step3) is more involved than Example 1. + +### Probe and Meter Details - Step 3 +The general approach for hooking a `Probe` is as follows: +1. Define the function for the `Probe` action (e.g. `post_method_hook`) +2. Wrap the method of interest (e.g. `_augment_images_with_patch`) and `post_method_hook` + 1. Define function (e.g. `hook_wrapper`) that returns another function (e.g. `wrapped`) that calls `_augment_images_with_patch` and `post_method_hook` in the desired order + 2. Assign the result of `hook_wrapper` to the original method of interest (`_augment_images_with_patch`) via `method_hook`, thus changing the behavior of the method without modifying it directly + +#### Step 3-1: `post_method_hook` +The role of the defined `post_method_hook` function is the same as that of `hook_fn` defined in [Example 1](#example1) - we are specifying the variable to update being monitored by the `Probe`. In this particular example, despite its name, it is not the name that is specifying whether the `Probe` action occurs after a method, but which input this function is assigned to when calling `method_hook`, which we will show later. + +Note the signature of `post_method_hook` with 3 arguments - this was based on the expected output of `_augment_images_with_patch` i.e. `return_value = method(*args[1:], **kwargs)`, and the anticipation that the output of `_augment_images_with_patch` would be passed to `post_method_hook` as input in `hook_wrapper` i.e. `post_method_hook(*return_value)`. + +Of those expected outputs, we are choosing to update the value assigned to argument `x_patch` of `post_method_hook` and choosing to also refer to the updated value as `x_patch` by the `Probe`, which leads to `probe.update(x_patch=x_patch)`. The `Meter` is then able to reference this as `"hooked_method.x_patch"` later on. + +For now, consider the example shown as a possible template for the user - we leave it as a template rather than add a defined function in armory such that the user can adjust as needed. + +#### Step 3-2.1: `hook_wrapper` +`hook_wrapper` determines when the `Probe` action takes place with respect to a `method` call as well as how inputs should be specified for either a `pre_method_hook` or a `post_method_hook`. Its signature as defined suggests the possibility of either, but in this example, we only specify `post_method_hook`. + +Within `hook_wrapper`, we define `wrapped`, which is where the actual order of calls for `method` and `post_method_hook` are specified, which in this case is `method` then `post_method_hook`. Again, despite the argument name, there is no reason the user cannot specify `post_method_hook` to be called before `method`, but we discourage such practices for clarity. + +Now consider the arguments involved within `hook_wrapper`. `hook_wrapper` once called, will return a function `wrapped`, which expects arguments `*args` and `**kwargs`. Because `hook_wrapper` is meant to wrap `method`, `*args` and `**kwargs` will be passed to `method` such that it performs its original function. Recall that `method` is not just any ordinary function but a method for an instance, which means that even though the first argument of `*args` will be `self` ***[why was `self` even passed to begin with???]***, the actual method call needs to exclude it, leading to `method(*args[1:], **kwargs)`. + +Also note what `wrapped` returns. Since `method` needs to maintain its original functionality, the return value for `wrapped` should also match that of `method`, thus `return return_value`. + +Last but not least, consider the `post_method_hook` call. For this example, the objective was to update a variable that is an output of `method` after each `method` call, and as mentioned in [Step 3-1](#step-3-1-post_method_hook), we pass `return_value` as-is but with a `*` to unpack the iterable, thus `post_method_hook(*return_value)`. + +We present `hook_wrapper` as another template for the user rather than a defined armory function, because, as we have alluded to before, the arrangement of a `Probe` action and instance method is heavily dependent on the user's objective and not obviously generalizable as we will see in the next example. + +#### Step 3-2.2: `method_hook` +Notice that `hook_wrapper` returns a wrapped method, but the wrapped method is not actually reassigned to the method that was meant to be wrapped. `method_hook` is defined to do just that, along with actually executing `hook_wrapper` as well for the wrapping process. + +Unlike `post_method_hook` and `hook_wrapper`, which we made a point of framing as templates, we believe `method_hook` is well-established and generalized enough to be defined as an armory function, which the user can import and use as-is. \ No newline at end of file From dc772d970c1160655d896287d8087403ea243dbc Mon Sep 17 00:00:00 2001 From: Paul Park Date: Tue, 13 Dec 2022 19:46:40 +0000 Subject: [PATCH 04/30] started documentation for Example 3 --- docs/probe.md | 112 ++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 109 insertions(+), 3 deletions(-) diff --git a/docs/probe.md b/docs/probe.md index 3cf47f101..22d7bd9c5 100644 --- a/docs/probe.md +++ b/docs/probe.md @@ -90,7 +90,7 @@ For the scope of this document, we don't dwell on what `armory` is doing in step ## Example 2: Attack Artifact - Available as Output ### User Story -I am using `CARLADapricotPatch` I am interested in the patch after every iteration, which is generated by `CARLADapricotPatch._augment_images_with_patch` +I am using `CARLADapricotPatch`, and I am interested in the patch after every iteration, which is generated by `CARLADapricotPatch._augment_images_with_patch` and returned as an output. ### Example Code This is an example of working with a python package/framework (i.e. `art`) that does NOT come with built-in hooking mechanisms. In the code snippet below, we define wrapper functions to wrap existing instance methods to monitor the output of interest: ```python @@ -195,6 +195,112 @@ Last but not least, consider the `post_method_hook` call. For this example, the We present `hook_wrapper` as another template for the user rather than a defined armory function, because, as we have alluded to before, the arrangement of a `Probe` action and instance method is heavily dependent on the user's objective and not obviously generalizable as we will see in the next example. #### Step 3-2.2: `method_hook` -Notice that `hook_wrapper` returns a wrapped method, but the wrapped method is not actually reassigned to the method that was meant to be wrapped. `method_hook` is defined to do just that, along with actually executing `hook_wrapper` as well for the wrapping process. +Notice that `hook_wrapper` returns a wrapped method, but the wrapped method is not actually reassigned to the method that was meant to be wrapped. `method_hook`, which takes an object `obj` and its associated method name `method`, is defined to do just that, along with actually executing `hook_wrapper` as well for the wrapping process. -Unlike `post_method_hook` and `hook_wrapper`, which we made a point of framing as templates, we believe `method_hook` is well-established and generalized enough to be defined as an armory function, which the user can import and use as-is. \ No newline at end of file +Unlike `post_method_hook` and `hook_wrapper`, which we made a point of framing as templates, we believe `method_hook` is well-established and generalized enough to be defined as an armory function, which the user can import and use as-is. + +## Example 3: Attack Artifact - NOT Available as Output +### User Story +I am using `CARLAAdversarialPatchPyTorch`, and I am interested in the patch after every iteration, which is generated during `CARLAAdversarialPatchPyTorch._train_step`, but NOT provided as an output. +### Example Code +Like [Example 2](#example-2-attack-artifact---available-as-output), the python package/framework (i.e. `art`) does NOT come with built-in hooking mechanisms, BUT unlike Example 2, the method of interest does NOT return the artifact of interest (`_train_step` returns `loss`) - rather, the artifact of interest is available as an attribute (`self._patch`). In the code snippet below, we adjust `post_method_hook` and `hook_wrapper` to reflect this new context: +```python +from armory.scenarios.main import get as get_scenario +from armory.instrument import get_probe, Meter, get_hub +import types + +def method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None): + """ + Hook target method and return the original method + If a class is passed in, hooks ALL instances of class. + If an object is passed in, only hooks the given instance. + """ + if not isinstance(obj, object): + raise ValueError(f"obj {obj} is not a class or object") + method = getattr(obj, method_name) + if not callable(method): + raise ValueError(f"obj.{method_name} attribute {method} is not callable") + wrapped = hook_wrapper( + method, pre_method_hook=pre_method_hook, post_method_hook=post_method_hook + ) + + if isinstance(obj, type): + cls = obj + setattr(cls, method_name, wrapped) + else: + setattr(obj, method_name, types.MethodType(wrapped, obj)) + + return method + +def hook_wrapper(method, pre_method_hook=None, post_method_hook=None): + def wrapped(*args, **kwargs): + return_value = method(*args[1:], **kwargs) + if post_method_hook is not None: + post_method_hook(*args[0]) + return return_value + + return wrapped + +def post_method_hook(obj): + probe.update(patch=obj._patch) + +# load Scenario +s = get_scenario( + "/armory/tmp/2022-11-03T180812.020999/carla_obj_det_adversarialpatch_undefended.json", + num_eval_batches=1, +).load() + +# create Probe with "hooked_method" namespace +probe = get_probe("hooked_method") + +# register hook that will update Probe +method_hook( + s.attack, "_train_step", post_method_hook=post_method_hook +) + +# create Meter for Probe with "hooked_method" namespace +hook_meter = Meter( + "hook_x_patch", lambda x: x, "hooked_method.x_patch" +) + +# connect Meter to Hub +get_hub().connect_meter(hook_meter) + +s.next() +s.run_attack() +``` +Consider the functions introduced in [Example 2](#example-2-attack-artifact---available-as-output): +- `method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None)` +- `hook_wrapper(method, pre_method_hook = None, post_method_hook = None)` +- `post_method_hook(x_patch, patch_target, transformations)` + +`method_hook` has stayed the same (which is why we define it as an armory function), but `hook_wrapper` and `post_method_hook` have changed. + +### Probe and Meter Details - Step 3 +Recall the general approach for hooking a `Probe`: +1. Define the function for the `Probe` action (e.g. `post_method_hook`) +2. Wrap the method of interest (e.g. `_train_step`) and `post_method_hook` + 1. Define function (e.g. `hook_wrapper`) that returns another function (e.g. `wrapped`) that calls `_train_step` and `post_method_hook` in the desired order + 2. Assign the result of `hook_wrapper` to the original method of interest (`_train_step`) via `method_hook`, thus changing the behavior of the method without modifying it directly ***[UNCHANGED]*** + +#### Step 3-1: `post_method_hook` +The role of the defined `post_method_hook` function is the same as that of `hook_fn` defined in [Example 1](#example1) - we are specifying the variable to update being monitored by the `Probe`. In this particular example, despite its name, it is not the name that is specifying whether the `Probe` action occurs after a method, but which input this function is assigned to when calling `method_hook`, which we will show later. + +Note the signature of `post_method_hook` with 3 arguments - this was based on the expected output of `_augment_images_with_patch` i.e. `return_value = method(*args[1:], **kwargs)`, and the anticipation that the output of `_augment_images_with_patch` would be passed to `post_method_hook` as input in `hook_wrapper` i.e. `post_method_hook(*return_value)`. + +Of those expected outputs, we are choosing to update the value assigned to argument `x_patch` of `post_method_hook` and choosing to also refer to the updated value as `x_patch` by the `Probe`, which leads to `probe.update(x_patch=x_patch)`. The `Meter` is then able to reference this as `"hooked_method.x_patch"` later on. + +For now, consider the example shown as a possible template for the user - we leave it as a template rather than add a defined function in armory such that the user can adjust as needed. + +#### Step 3-2.1: `hook_wrapper` +`hook_wrapper` determines when the `Probe` action takes place with respect to a `method` call as well as how inputs should be specified for either a `pre_method_hook` or a `post_method_hook`. Its signature as defined suggests the possibility of either, but in this example, we only specify `post_method_hook`. + +Within `hook_wrapper`, we define `wrapped`, which is where the actual order of calls for `method` and `post_method_hook` are specified, which in this case is `method` then `post_method_hook`. Again, despite the argument name, there is no reason the user cannot specify `post_method_hook` to be called before `method`, but we discourage such practices for clarity. + +Now consider the arguments involved within `hook_wrapper`. `hook_wrapper` once called, will return a function `wrapped`, which expects arguments `*args` and `**kwargs`. Because `hook_wrapper` is meant to wrap `method`, `*args` and `**kwargs` will be passed to `method` such that it performs its original function. Recall that `method` is not just any ordinary function but a method for an instance, which means that even though the first argument of `*args` will be `self` ***[why was `self` even passed to begin with???]***, the actual method call needs to exclude it, leading to `method(*args[1:], **kwargs)`. + +Also note what `wrapped` returns. Since `method` needs to maintain its original functionality, the return value for `wrapped` should also match that of `method`, thus `return return_value`. + +Last but not least, consider the `post_method_hook` call. For this example, the objective was to update a variable that is an output of `method` after each `method` call, and as mentioned in [Step 3-1](#step-3-1-post_method_hook), we pass `return_value` as-is but with a `*` to unpack the iterable, thus `post_method_hook(*return_value)`. + +We present `hook_wrapper` as another template for the user rather than a defined armory function, because, as we have alluded to before, the arrangement of a `Probe` action and instance method is heavily dependent on the user's objective and not obviously generalizable as we will see in the next example. \ No newline at end of file From 74365ededc2c7ccfbbbbd7b4af891fd2eee141a2 Mon Sep 17 00:00:00 2001 From: Paul Park Date: Wed, 14 Dec 2022 01:54:55 +0000 Subject: [PATCH 05/30] added draft of Example 3 and additional discussion of other possible use cases; draft is complete with the minimum necessary content; because code examples are from notebook experiments, need to refine content and code that can be executed via armory run with a custom script passed to a config file along with explanations of how to save any Probe\Meter outputs for end-to-end examples --- docs/probe.md | 60 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 35 insertions(+), 25 deletions(-) diff --git a/docs/probe.md b/docs/probe.md index 22d7bd9c5..fff40296c 100644 --- a/docs/probe.md +++ b/docs/probe.md @@ -123,14 +123,14 @@ def method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None): def hook_wrapper(method, pre_method_hook = None, post_method_hook = None): def wrapped(*args, **kwargs): - return_value = method(*args[1:], **kwargs) - post_method_hook(*return_value) + return_value = method(*args[1:], **kwargs) # skip self with *args[1:] + post_method_hook(*return_value) # unpack return_value with * return return_value return wrapped def post_method_hook(x_patch, patch_target, transformations): - probe.update(x_patch=x_patch) # adding batch dim + probe.update(x_patch=x_patch) # load Scenario s = get_scenario( @@ -234,9 +234,9 @@ def method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None): def hook_wrapper(method, pre_method_hook=None, post_method_hook=None): def wrapped(*args, **kwargs): - return_value = method(*args[1:], **kwargs) + return_value = method(*args[1:], **kwargs) # skip self with *args[1:] if post_method_hook is not None: - post_method_hook(*args[0]) + post_method_hook(*args[0]) # *args[0] corresponds to self with _patch attribute return return_value return wrapped @@ -260,7 +260,7 @@ method_hook( # create Meter for Probe with "hooked_method" namespace hook_meter = Meter( - "hook_x_patch", lambda x: x, "hooked_method.x_patch" + "hook_patch", lambda x: x.detach().cpu().numpy(), "hooked_method.patch" ) # connect Meter to Hub @@ -278,29 +278,39 @@ Consider the functions introduced in [Example 2](#example-2-attack-artifact---av ### Probe and Meter Details - Step 3 Recall the general approach for hooking a `Probe`: -1. Define the function for the `Probe` action (e.g. `post_method_hook`) +1. Define the function for the `Probe` action (e.g. `post_method_hook`) ***[CHANGED]*** 2. Wrap the method of interest (e.g. `_train_step`) and `post_method_hook` - 1. Define function (e.g. `hook_wrapper`) that returns another function (e.g. `wrapped`) that calls `_train_step` and `post_method_hook` in the desired order + 1. Define function (e.g. `hook_wrapper`) that returns another function (e.g. `wrapped`) that calls `_train_step` and `post_method_hook` in the desired order ***[CHANGED]*** 2. Assign the result of `hook_wrapper` to the original method of interest (`_train_step`) via `method_hook`, thus changing the behavior of the method without modifying it directly ***[UNCHANGED]*** #### Step 3-1: `post_method_hook` -The role of the defined `post_method_hook` function is the same as that of `hook_fn` defined in [Example 1](#example1) - we are specifying the variable to update being monitored by the `Probe`. In this particular example, despite its name, it is not the name that is specifying whether the `Probe` action occurs after a method, but which input this function is assigned to when calling `method_hook`, which we will show later. +The signature of `post_method_hook` now specifies a single argument `obj`, which we assume has a `_patch` attribute. Again note that this has nothing to do with the expected output of `_train_step` - we know from inspecting the `_train_step` method that a `_patch` attribute exists, which we refer to within `post_method_hook` via `obj._patch`. We are choosing to measure the value assigned to an attribute of `obj`, an input of `post_method_hook`, and also choosing to refer to the variable to be updated as `patch` by the `Probe`, which leads to `probe.update(patch=obj._patch)`. The `Meter` is then able to reference this as `"hooked_method.patch"` later on. -Note the signature of `post_method_hook` with 3 arguments - this was based on the expected output of `_augment_images_with_patch` i.e. `return_value = method(*args[1:], **kwargs)`, and the anticipation that the output of `_augment_images_with_patch` would be passed to `post_method_hook` as input in `hook_wrapper` i.e. `post_method_hook(*return_value)`. - -Of those expected outputs, we are choosing to update the value assigned to argument `x_patch` of `post_method_hook` and choosing to also refer to the updated value as `x_patch` by the `Probe`, which leads to `probe.update(x_patch=x_patch)`. The `Meter` is then able to reference this as `"hooked_method.x_patch"` later on. - -For now, consider the example shown as a possible template for the user - we leave it as a template rather than add a defined function in armory such that the user can adjust as needed. +This example is another possible template for the user, where the definition of `post_method_hook` changes depending on what the user is interested in monitoring. #### Step 3-2.1: `hook_wrapper` -`hook_wrapper` determines when the `Probe` action takes place with respect to a `method` call as well as how inputs should be specified for either a `pre_method_hook` or a `post_method_hook`. Its signature as defined suggests the possibility of either, but in this example, we only specify `post_method_hook`. - -Within `hook_wrapper`, we define `wrapped`, which is where the actual order of calls for `method` and `post_method_hook` are specified, which in this case is `method` then `post_method_hook`. Again, despite the argument name, there is no reason the user cannot specify `post_method_hook` to be called before `method`, but we discourage such practices for clarity. - -Now consider the arguments involved within `hook_wrapper`. `hook_wrapper` once called, will return a function `wrapped`, which expects arguments `*args` and `**kwargs`. Because `hook_wrapper` is meant to wrap `method`, `*args` and `**kwargs` will be passed to `method` such that it performs its original function. Recall that `method` is not just any ordinary function but a method for an instance, which means that even though the first argument of `*args` will be `self` ***[why was `self` even passed to begin with???]***, the actual method call needs to exclude it, leading to `method(*args[1:], **kwargs)`. - -Also note what `wrapped` returns. Since `method` needs to maintain its original functionality, the return value for `wrapped` should also match that of `method`, thus `return return_value`. - -Last but not least, consider the `post_method_hook` call. For this example, the objective was to update a variable that is an output of `method` after each `method` call, and as mentioned in [Step 3-1](#step-3-1-post_method_hook), we pass `return_value` as-is but with a `*` to unpack the iterable, thus `post_method_hook(*return_value)`. - -We present `hook_wrapper` as another template for the user rather than a defined armory function, because, as we have alluded to before, the arrangement of a `Probe` action and instance method is heavily dependent on the user's objective and not obviously generalizable as we will see in the next example. \ No newline at end of file +As in [Example 2](#example-2-attack-artifact---available-as-output), `method` is called before `post_method_hook` in `wrapped` of `hook_wrapper`. `return_value` is not used in any way other than being returned at the end of `wrapped` to maintain `method`'s functionality. `*arg[0]` of `*arg` however, is passed to `post_method_hook` as an input, because it refers to `self`, the instance making the `method` call. As mentioned earlier, `self` contains the `_patch` attribute, which is what `probe` is monitoring in `post_method_hook`. + +Again, this example's `hook_wrapper` is another possible template for the user, which has been defined in such as way as to accomplish the objective described in the [User Story](#user-story-2). + +#### Step 4 +A brief note on the `Meter` creation in this example: The `_patch` variable of interest is a tensor, which is why a preprocessing function is applied for the identity metric, `lambda x: x.detach().cpu().numpy()`. The chaining functions could have occurred else where in the process as well. + +## Flexible Arrangements +We have emphasized through out this section that the example functions shown (with the exception of `method_hook`) are templates, and how those functions are defined depends on the user story. Before we discuss what the user needs to consider when defining those functions, it is important that the user remembers the 5-step process for custom probes as a general framework, which may aid in any debugging efforts that may occur later. + +That said, here are the questions the user needs to consider for creating custom probes: +- What is the `method` of interest? What is the variable of interest? + - Some knowledge of the package being used is necessary +- Is there an existing hooking mechanism for the `method` of interest? + - Relying on hooks from the package provider reduces code and maintenance +- Does the `Probe` action occur before or after a `method` call? + - Define `pre_method_hook` or `post_method_hook` and arrange calls as necessary with respect to `method` call +- What should `pre_method_hook`/`post_method_hook` expect as input? + - It may be natural for a `pre_method_hook` to take the same arguments as `method` i.e. `*args` and `**kwargs` + - It may be natural for a `post_method_hook` to take the output of a `method` call i.e. `*return_value` + - If the user is interested in the state of an instance or the attributes within it pre or post `method` call, passing the instance to `pre_method_hook`/`post_method_hook` is also possible i.e. `*args[0]` +- Is `wrapped` returning `return_value`? + - `method`'s original functionality has to be maintained + +Once these questions have been answered, the user can define functions as needed to meet specific monitoring objectives. \ No newline at end of file From 28c315290faa728e9c280a1a5c1a7bd028b9fe58 Mon Sep 17 00:00:00 2001 From: Paul Park Date: Wed, 14 Dec 2022 23:58:10 +0000 Subject: [PATCH 06/30] added updates for minimal working example of probe and method hooking with user-init block that can be executed with armory run - requires necessary custom files and config not included in this repo --- armory/scenarios/scenario.py | 5 +++++ armory/utils/config_loading.py | 10 ++++++++++ docs/probe.md | 2 +- 3 files changed, 16 insertions(+), 1 deletion(-) diff --git a/armory/scenarios/scenario.py b/armory/scenarios/scenario.py index 11588dbc0..99d3afc01 100644 --- a/armory/scenarios/scenario.py +++ b/armory/scenarios/scenario.py @@ -210,6 +210,10 @@ def load_attack(self): self.use_label = use_label self.generate_kwargs = generate_kwargs + def load_user_init(self): + user_init_config = self.config["user-init"] + config_loading.load_user_init(user_init_config, self) + def load_dataset(self, eval_split_default="test"): dataset_config = self.config["dataset"] eval_split = dataset_config.get("eval_split", eval_split_default) @@ -288,6 +292,7 @@ def load(self): self.load_dataset() self.load_metrics() self.load_export_meters() + self.load_user_init() return self def evaluate_all(self): diff --git a/armory/utils/config_loading.py b/armory/utils/config_loading.py index 9ba4524d3..1907ef98e 100644 --- a/armory/utils/config_loading.py +++ b/armory/utils/config_loading.py @@ -4,6 +4,7 @@ from importlib import import_module from armory.logs import log +from armory.scenarios.scenario import Scenario # import torch before tensorflow to ensure torch.utils.data.DataLoader can utilize # all CPU resources when num_workers > 1 @@ -47,6 +48,15 @@ def load_fn(sub_config): return getattr(module, sub_config["name"]) +def load_user_init(sub_config, scenario: Scenario): + module = import_module(sub_config["module"]) + fn = getattr(module, sub_config["name"]) + # args = sub_config.get("args", []) + # kwargs = sub_config.get("kwargs", {}) + # return fn(*args, **kwargs) + return fn(scenario) + + # TODO THIS is a TERRIBLE Pattern....can we refactor? def load_dataset(dataset_config, *args, num_batches=None, check_run=False, **kwargs): """ diff --git a/docs/probe.md b/docs/probe.md index fff40296c..10d45d341 100644 --- a/docs/probe.md +++ b/docs/probe.md @@ -134,7 +134,7 @@ def post_method_hook(x_patch, patch_target, transformations): # load Scenario s = get_scenario( - "/armory/tmp/2022-11-03T180812.020999/carla_obj_det_adversarialpatch_undefended.json", + "/armory/tmp/2022-11-03T180812.020999/carla_obj_det_dpatch_undefended.json", num_eval_batches=1, ).load() From fdb42e33344ac1767d4cb62edab9dde1d83a97a2 Mon Sep 17 00:00:00 2001 From: Paul Park Date: Fri, 16 Dec 2022 02:05:58 +0000 Subject: [PATCH 07/30] Revert "added draft of Example 3 and additional discussion of other possible use cases; draft is complete with the minimum necessary content; because code examples are from notebook experiments, need to refine content and code that can be executed via armory run with a custom script passed to a config file along with explanations of how to save any Probe\Meter outputs for end-to-end examples" This reverts commit 74365ededc2c7ccfbbbbd7b4af891fd2eee141a2. --- docs/probe.md | 60 +++++++++++++++++++++------------------------------ 1 file changed, 25 insertions(+), 35 deletions(-) diff --git a/docs/probe.md b/docs/probe.md index 10d45d341..3eb3393c8 100644 --- a/docs/probe.md +++ b/docs/probe.md @@ -123,14 +123,14 @@ def method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None): def hook_wrapper(method, pre_method_hook = None, post_method_hook = None): def wrapped(*args, **kwargs): - return_value = method(*args[1:], **kwargs) # skip self with *args[1:] - post_method_hook(*return_value) # unpack return_value with * + return_value = method(*args[1:], **kwargs) + post_method_hook(*return_value) return return_value return wrapped def post_method_hook(x_patch, patch_target, transformations): - probe.update(x_patch=x_patch) + probe.update(x_patch=x_patch) # adding batch dim # load Scenario s = get_scenario( @@ -234,9 +234,9 @@ def method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None): def hook_wrapper(method, pre_method_hook=None, post_method_hook=None): def wrapped(*args, **kwargs): - return_value = method(*args[1:], **kwargs) # skip self with *args[1:] + return_value = method(*args[1:], **kwargs) if post_method_hook is not None: - post_method_hook(*args[0]) # *args[0] corresponds to self with _patch attribute + post_method_hook(*args[0]) return return_value return wrapped @@ -260,7 +260,7 @@ method_hook( # create Meter for Probe with "hooked_method" namespace hook_meter = Meter( - "hook_patch", lambda x: x.detach().cpu().numpy(), "hooked_method.patch" + "hook_x_patch", lambda x: x, "hooked_method.x_patch" ) # connect Meter to Hub @@ -278,39 +278,29 @@ Consider the functions introduced in [Example 2](#example-2-attack-artifact---av ### Probe and Meter Details - Step 3 Recall the general approach for hooking a `Probe`: -1. Define the function for the `Probe` action (e.g. `post_method_hook`) ***[CHANGED]*** +1. Define the function for the `Probe` action (e.g. `post_method_hook`) 2. Wrap the method of interest (e.g. `_train_step`) and `post_method_hook` - 1. Define function (e.g. `hook_wrapper`) that returns another function (e.g. `wrapped`) that calls `_train_step` and `post_method_hook` in the desired order ***[CHANGED]*** + 1. Define function (e.g. `hook_wrapper`) that returns another function (e.g. `wrapped`) that calls `_train_step` and `post_method_hook` in the desired order 2. Assign the result of `hook_wrapper` to the original method of interest (`_train_step`) via `method_hook`, thus changing the behavior of the method without modifying it directly ***[UNCHANGED]*** #### Step 3-1: `post_method_hook` -The signature of `post_method_hook` now specifies a single argument `obj`, which we assume has a `_patch` attribute. Again note that this has nothing to do with the expected output of `_train_step` - we know from inspecting the `_train_step` method that a `_patch` attribute exists, which we refer to within `post_method_hook` via `obj._patch`. We are choosing to measure the value assigned to an attribute of `obj`, an input of `post_method_hook`, and also choosing to refer to the variable to be updated as `patch` by the `Probe`, which leads to `probe.update(patch=obj._patch)`. The `Meter` is then able to reference this as `"hooked_method.patch"` later on. +The role of the defined `post_method_hook` function is the same as that of `hook_fn` defined in [Example 1](#example1) - we are specifying the variable to update being monitored by the `Probe`. In this particular example, despite its name, it is not the name that is specifying whether the `Probe` action occurs after a method, but which input this function is assigned to when calling `method_hook`, which we will show later. -This example is another possible template for the user, where the definition of `post_method_hook` changes depending on what the user is interested in monitoring. +Note the signature of `post_method_hook` with 3 arguments - this was based on the expected output of `_augment_images_with_patch` i.e. `return_value = method(*args[1:], **kwargs)`, and the anticipation that the output of `_augment_images_with_patch` would be passed to `post_method_hook` as input in `hook_wrapper` i.e. `post_method_hook(*return_value)`. + +Of those expected outputs, we are choosing to update the value assigned to argument `x_patch` of `post_method_hook` and choosing to also refer to the updated value as `x_patch` by the `Probe`, which leads to `probe.update(x_patch=x_patch)`. The `Meter` is then able to reference this as `"hooked_method.x_patch"` later on. + +For now, consider the example shown as a possible template for the user - we leave it as a template rather than add a defined function in armory such that the user can adjust as needed. #### Step 3-2.1: `hook_wrapper` -As in [Example 2](#example-2-attack-artifact---available-as-output), `method` is called before `post_method_hook` in `wrapped` of `hook_wrapper`. `return_value` is not used in any way other than being returned at the end of `wrapped` to maintain `method`'s functionality. `*arg[0]` of `*arg` however, is passed to `post_method_hook` as an input, because it refers to `self`, the instance making the `method` call. As mentioned earlier, `self` contains the `_patch` attribute, which is what `probe` is monitoring in `post_method_hook`. - -Again, this example's `hook_wrapper` is another possible template for the user, which has been defined in such as way as to accomplish the objective described in the [User Story](#user-story-2). - -#### Step 4 -A brief note on the `Meter` creation in this example: The `_patch` variable of interest is a tensor, which is why a preprocessing function is applied for the identity metric, `lambda x: x.detach().cpu().numpy()`. The chaining functions could have occurred else where in the process as well. - -## Flexible Arrangements -We have emphasized through out this section that the example functions shown (with the exception of `method_hook`) are templates, and how those functions are defined depends on the user story. Before we discuss what the user needs to consider when defining those functions, it is important that the user remembers the 5-step process for custom probes as a general framework, which may aid in any debugging efforts that may occur later. - -That said, here are the questions the user needs to consider for creating custom probes: -- What is the `method` of interest? What is the variable of interest? - - Some knowledge of the package being used is necessary -- Is there an existing hooking mechanism for the `method` of interest? - - Relying on hooks from the package provider reduces code and maintenance -- Does the `Probe` action occur before or after a `method` call? - - Define `pre_method_hook` or `post_method_hook` and arrange calls as necessary with respect to `method` call -- What should `pre_method_hook`/`post_method_hook` expect as input? - - It may be natural for a `pre_method_hook` to take the same arguments as `method` i.e. `*args` and `**kwargs` - - It may be natural for a `post_method_hook` to take the output of a `method` call i.e. `*return_value` - - If the user is interested in the state of an instance or the attributes within it pre or post `method` call, passing the instance to `pre_method_hook`/`post_method_hook` is also possible i.e. `*args[0]` -- Is `wrapped` returning `return_value`? - - `method`'s original functionality has to be maintained - -Once these questions have been answered, the user can define functions as needed to meet specific monitoring objectives. \ No newline at end of file +`hook_wrapper` determines when the `Probe` action takes place with respect to a `method` call as well as how inputs should be specified for either a `pre_method_hook` or a `post_method_hook`. Its signature as defined suggests the possibility of either, but in this example, we only specify `post_method_hook`. + +Within `hook_wrapper`, we define `wrapped`, which is where the actual order of calls for `method` and `post_method_hook` are specified, which in this case is `method` then `post_method_hook`. Again, despite the argument name, there is no reason the user cannot specify `post_method_hook` to be called before `method`, but we discourage such practices for clarity. + +Now consider the arguments involved within `hook_wrapper`. `hook_wrapper` once called, will return a function `wrapped`, which expects arguments `*args` and `**kwargs`. Because `hook_wrapper` is meant to wrap `method`, `*args` and `**kwargs` will be passed to `method` such that it performs its original function. Recall that `method` is not just any ordinary function but a method for an instance, which means that even though the first argument of `*args` will be `self` ***[why was `self` even passed to begin with???]***, the actual method call needs to exclude it, leading to `method(*args[1:], **kwargs)`. + +Also note what `wrapped` returns. Since `method` needs to maintain its original functionality, the return value for `wrapped` should also match that of `method`, thus `return return_value`. + +Last but not least, consider the `post_method_hook` call. For this example, the objective was to update a variable that is an output of `method` after each `method` call, and as mentioned in [Step 3-1](#step-3-1-post_method_hook), we pass `return_value` as-is but with a `*` to unpack the iterable, thus `post_method_hook(*return_value)`. + +We present `hook_wrapper` as another template for the user rather than a defined armory function, because, as we have alluded to before, the arrangement of a `Probe` action and instance method is heavily dependent on the user's objective and not obviously generalizable as we will see in the next example. \ No newline at end of file From 48234e17492547297e5b4b250e841e8f4de5b130 Mon Sep 17 00:00:00 2001 From: Paul Park Date: Wed, 14 Dec 2022 01:54:55 +0000 Subject: [PATCH 08/30] added draft of Example 3 and additional discussion of other possible use cases; draft is complete with the minimum necessary content; because code examples are from notebook experiments, need to refine content and code that can be executed via armory run with a custom script passed to a config file along with explanations of how to save any Probe\Meter outputs for end-to-end examples --- docs/probe.md | 60 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 35 insertions(+), 25 deletions(-) diff --git a/docs/probe.md b/docs/probe.md index 3eb3393c8..10d45d341 100644 --- a/docs/probe.md +++ b/docs/probe.md @@ -123,14 +123,14 @@ def method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None): def hook_wrapper(method, pre_method_hook = None, post_method_hook = None): def wrapped(*args, **kwargs): - return_value = method(*args[1:], **kwargs) - post_method_hook(*return_value) + return_value = method(*args[1:], **kwargs) # skip self with *args[1:] + post_method_hook(*return_value) # unpack return_value with * return return_value return wrapped def post_method_hook(x_patch, patch_target, transformations): - probe.update(x_patch=x_patch) # adding batch dim + probe.update(x_patch=x_patch) # load Scenario s = get_scenario( @@ -234,9 +234,9 @@ def method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None): def hook_wrapper(method, pre_method_hook=None, post_method_hook=None): def wrapped(*args, **kwargs): - return_value = method(*args[1:], **kwargs) + return_value = method(*args[1:], **kwargs) # skip self with *args[1:] if post_method_hook is not None: - post_method_hook(*args[0]) + post_method_hook(*args[0]) # *args[0] corresponds to self with _patch attribute return return_value return wrapped @@ -260,7 +260,7 @@ method_hook( # create Meter for Probe with "hooked_method" namespace hook_meter = Meter( - "hook_x_patch", lambda x: x, "hooked_method.x_patch" + "hook_patch", lambda x: x.detach().cpu().numpy(), "hooked_method.patch" ) # connect Meter to Hub @@ -278,29 +278,39 @@ Consider the functions introduced in [Example 2](#example-2-attack-artifact---av ### Probe and Meter Details - Step 3 Recall the general approach for hooking a `Probe`: -1. Define the function for the `Probe` action (e.g. `post_method_hook`) +1. Define the function for the `Probe` action (e.g. `post_method_hook`) ***[CHANGED]*** 2. Wrap the method of interest (e.g. `_train_step`) and `post_method_hook` - 1. Define function (e.g. `hook_wrapper`) that returns another function (e.g. `wrapped`) that calls `_train_step` and `post_method_hook` in the desired order + 1. Define function (e.g. `hook_wrapper`) that returns another function (e.g. `wrapped`) that calls `_train_step` and `post_method_hook` in the desired order ***[CHANGED]*** 2. Assign the result of `hook_wrapper` to the original method of interest (`_train_step`) via `method_hook`, thus changing the behavior of the method without modifying it directly ***[UNCHANGED]*** #### Step 3-1: `post_method_hook` -The role of the defined `post_method_hook` function is the same as that of `hook_fn` defined in [Example 1](#example1) - we are specifying the variable to update being monitored by the `Probe`. In this particular example, despite its name, it is not the name that is specifying whether the `Probe` action occurs after a method, but which input this function is assigned to when calling `method_hook`, which we will show later. +The signature of `post_method_hook` now specifies a single argument `obj`, which we assume has a `_patch` attribute. Again note that this has nothing to do with the expected output of `_train_step` - we know from inspecting the `_train_step` method that a `_patch` attribute exists, which we refer to within `post_method_hook` via `obj._patch`. We are choosing to measure the value assigned to an attribute of `obj`, an input of `post_method_hook`, and also choosing to refer to the variable to be updated as `patch` by the `Probe`, which leads to `probe.update(patch=obj._patch)`. The `Meter` is then able to reference this as `"hooked_method.patch"` later on. -Note the signature of `post_method_hook` with 3 arguments - this was based on the expected output of `_augment_images_with_patch` i.e. `return_value = method(*args[1:], **kwargs)`, and the anticipation that the output of `_augment_images_with_patch` would be passed to `post_method_hook` as input in `hook_wrapper` i.e. `post_method_hook(*return_value)`. - -Of those expected outputs, we are choosing to update the value assigned to argument `x_patch` of `post_method_hook` and choosing to also refer to the updated value as `x_patch` by the `Probe`, which leads to `probe.update(x_patch=x_patch)`. The `Meter` is then able to reference this as `"hooked_method.x_patch"` later on. - -For now, consider the example shown as a possible template for the user - we leave it as a template rather than add a defined function in armory such that the user can adjust as needed. +This example is another possible template for the user, where the definition of `post_method_hook` changes depending on what the user is interested in monitoring. #### Step 3-2.1: `hook_wrapper` -`hook_wrapper` determines when the `Probe` action takes place with respect to a `method` call as well as how inputs should be specified for either a `pre_method_hook` or a `post_method_hook`. Its signature as defined suggests the possibility of either, but in this example, we only specify `post_method_hook`. - -Within `hook_wrapper`, we define `wrapped`, which is where the actual order of calls for `method` and `post_method_hook` are specified, which in this case is `method` then `post_method_hook`. Again, despite the argument name, there is no reason the user cannot specify `post_method_hook` to be called before `method`, but we discourage such practices for clarity. - -Now consider the arguments involved within `hook_wrapper`. `hook_wrapper` once called, will return a function `wrapped`, which expects arguments `*args` and `**kwargs`. Because `hook_wrapper` is meant to wrap `method`, `*args` and `**kwargs` will be passed to `method` such that it performs its original function. Recall that `method` is not just any ordinary function but a method for an instance, which means that even though the first argument of `*args` will be `self` ***[why was `self` even passed to begin with???]***, the actual method call needs to exclude it, leading to `method(*args[1:], **kwargs)`. - -Also note what `wrapped` returns. Since `method` needs to maintain its original functionality, the return value for `wrapped` should also match that of `method`, thus `return return_value`. - -Last but not least, consider the `post_method_hook` call. For this example, the objective was to update a variable that is an output of `method` after each `method` call, and as mentioned in [Step 3-1](#step-3-1-post_method_hook), we pass `return_value` as-is but with a `*` to unpack the iterable, thus `post_method_hook(*return_value)`. - -We present `hook_wrapper` as another template for the user rather than a defined armory function, because, as we have alluded to before, the arrangement of a `Probe` action and instance method is heavily dependent on the user's objective and not obviously generalizable as we will see in the next example. \ No newline at end of file +As in [Example 2](#example-2-attack-artifact---available-as-output), `method` is called before `post_method_hook` in `wrapped` of `hook_wrapper`. `return_value` is not used in any way other than being returned at the end of `wrapped` to maintain `method`'s functionality. `*arg[0]` of `*arg` however, is passed to `post_method_hook` as an input, because it refers to `self`, the instance making the `method` call. As mentioned earlier, `self` contains the `_patch` attribute, which is what `probe` is monitoring in `post_method_hook`. + +Again, this example's `hook_wrapper` is another possible template for the user, which has been defined in such as way as to accomplish the objective described in the [User Story](#user-story-2). + +#### Step 4 +A brief note on the `Meter` creation in this example: The `_patch` variable of interest is a tensor, which is why a preprocessing function is applied for the identity metric, `lambda x: x.detach().cpu().numpy()`. The chaining functions could have occurred else where in the process as well. + +## Flexible Arrangements +We have emphasized through out this section that the example functions shown (with the exception of `method_hook`) are templates, and how those functions are defined depends on the user story. Before we discuss what the user needs to consider when defining those functions, it is important that the user remembers the 5-step process for custom probes as a general framework, which may aid in any debugging efforts that may occur later. + +That said, here are the questions the user needs to consider for creating custom probes: +- What is the `method` of interest? What is the variable of interest? + - Some knowledge of the package being used is necessary +- Is there an existing hooking mechanism for the `method` of interest? + - Relying on hooks from the package provider reduces code and maintenance +- Does the `Probe` action occur before or after a `method` call? + - Define `pre_method_hook` or `post_method_hook` and arrange calls as necessary with respect to `method` call +- What should `pre_method_hook`/`post_method_hook` expect as input? + - It may be natural for a `pre_method_hook` to take the same arguments as `method` i.e. `*args` and `**kwargs` + - It may be natural for a `post_method_hook` to take the output of a `method` call i.e. `*return_value` + - If the user is interested in the state of an instance or the attributes within it pre or post `method` call, passing the instance to `pre_method_hook`/`post_method_hook` is also possible i.e. `*args[0]` +- Is `wrapped` returning `return_value`? + - `method`'s original functionality has to be maintained + +Once these questions have been answered, the user can define functions as needed to meet specific monitoring objectives. \ No newline at end of file From 0e42c434d05560913b68ad56003504d8301fe02f Mon Sep 17 00:00:00 2001 From: Paul Park Date: Fri, 16 Dec 2022 02:21:03 +0000 Subject: [PATCH 09/30] Revert "added updates for minimal working example of probe and method hooking with user-init block that can be executed with armory run - requires necessary custom files and config not included in this repo" This reverts commit 28c315290faa728e9c280a1a5c1a7bd028b9fe58. Reverting this commit based on discussion with Lucas and the need to focus on Probe examples that work under the assumption that the user is actively changing code to add Probes for a model or attack, as opposed to hooking a Probe to an existing model or attack without modifying the respective code directly - aside from updating the docs (I think at this point I'll rename the existing Probe.md as a different use case and add another doc just for the use case that I dicussed with Lucas), this will not change the fact that the user needs to use the user-init block (I may need to do a pull since the user-init implementation doesn't exist here yet - I've definitely seen it in the repo) with a Meter defined for the Probes that will be added, and it will also not change the need for perhaps a new Meter class that will export probe variables to a pickle file depending on what the user is trying to save (e.g. patch data or image with patch data) rather than a json file. There is also the question of how to deal with size restrictions placed on a Meter (unclear where the restrictions are actually defined - I only know that it exists because of armory logs and the fact that the variable doesn't get saved to the json file when I see size restriction logs). An additional side note regarding the user-init block for the case that we do decide to add capabilities for Probe hooking without modifying the model or attack code: The way I got it to work in this commit (which will be reverted) is to load the user-init block at the end of all other loads (e.g. model, attack, dataset, metrics etc.) because an instance of a model or attack had to be available to hook to, which I accomplished by passing the Scenario (which contains the model and attack as an attribute) as an argument to a user init function. The existing user-init feature (again not in this branch currently, so will require a git pull and merge), assumes the user-init load happens prior to any other loads, which means none of the instances will be available just yet. Hopefully there will be no issue with this when defining a Meter to accompany any Probe the user may have added to a model or attack code, since the Meter can exist separately from a Probe instance that the Meter will get updates from. --- armory/scenarios/scenario.py | 5 ----- armory/utils/config_loading.py | 10 ---------- docs/probe.md | 2 +- 3 files changed, 1 insertion(+), 16 deletions(-) diff --git a/armory/scenarios/scenario.py b/armory/scenarios/scenario.py index 99d3afc01..11588dbc0 100644 --- a/armory/scenarios/scenario.py +++ b/armory/scenarios/scenario.py @@ -210,10 +210,6 @@ def load_attack(self): self.use_label = use_label self.generate_kwargs = generate_kwargs - def load_user_init(self): - user_init_config = self.config["user-init"] - config_loading.load_user_init(user_init_config, self) - def load_dataset(self, eval_split_default="test"): dataset_config = self.config["dataset"] eval_split = dataset_config.get("eval_split", eval_split_default) @@ -292,7 +288,6 @@ def load(self): self.load_dataset() self.load_metrics() self.load_export_meters() - self.load_user_init() return self def evaluate_all(self): diff --git a/armory/utils/config_loading.py b/armory/utils/config_loading.py index 1907ef98e..9ba4524d3 100644 --- a/armory/utils/config_loading.py +++ b/armory/utils/config_loading.py @@ -4,7 +4,6 @@ from importlib import import_module from armory.logs import log -from armory.scenarios.scenario import Scenario # import torch before tensorflow to ensure torch.utils.data.DataLoader can utilize # all CPU resources when num_workers > 1 @@ -48,15 +47,6 @@ def load_fn(sub_config): return getattr(module, sub_config["name"]) -def load_user_init(sub_config, scenario: Scenario): - module = import_module(sub_config["module"]) - fn = getattr(module, sub_config["name"]) - # args = sub_config.get("args", []) - # kwargs = sub_config.get("kwargs", {}) - # return fn(*args, **kwargs) - return fn(scenario) - - # TODO THIS is a TERRIBLE Pattern....can we refactor? def load_dataset(dataset_config, *args, num_batches=None, check_run=False, **kwargs): """ diff --git a/docs/probe.md b/docs/probe.md index 10d45d341..fff40296c 100644 --- a/docs/probe.md +++ b/docs/probe.md @@ -134,7 +134,7 @@ def post_method_hook(x_patch, patch_target, transformations): # load Scenario s = get_scenario( - "/armory/tmp/2022-11-03T180812.020999/carla_obj_det_dpatch_undefended.json", + "/armory/tmp/2022-11-03T180812.020999/carla_obj_det_adversarialpatch_undefended.json", num_eval_batches=1, ).load() From dbbd8c96b4cc952b9cd7d44007aed58109488950 Mon Sep 17 00:00:00 2001 From: Paul Park Date: Fri, 16 Dec 2022 02:53:59 +0000 Subject: [PATCH 10/30] made a copy of updated probe.md (probe_hooking_notebook.md) to save as an example for probe hooking within a jupyter notebook; will update probe.md to address the user case discussed with Lucas --- docs/probe.md | 10 +- docs/probe_hooking_notebook.md | 316 +++++++++++++++++++++++++++++++++ 2 files changed, 321 insertions(+), 5 deletions(-) create mode 100644 docs/probe_hooking_notebook.md diff --git a/docs/probe.md b/docs/probe.md index fff40296c..d70a26065 100644 --- a/docs/probe.md +++ b/docs/probe.md @@ -1,4 +1,4 @@ -# Probes and Meters: Advanced End-to-End Examples +# Probes and Meters: Advanced Hooking Examples for Notebooks For an introduction to `Probe`s and `Meter`s, please refer to [Measurement Overview](./metrics.md). ## Context @@ -7,7 +7,7 @@ To monitor particular aspects of an `armory run` session, the user needs to know - When should I measure it? - Where should my custom monitoring script go? -The examples in this section highlight the nuances of using `Probe`s and `Meter`s for flexible monitoring arrangements in `armory`. +The examples in this section highlight the nuances of using `Probe`s and `Meter`s for flexible monitoring arrangements in `armory`. In particular, we assume the user is working from a Jupyter notebook and wishes to use `Probe`s and `Meter`s to monitor a model or attack ***without*** modifying the existing code. Documentation and development for enabling the functionality as discribed here for an end-to-end execution with `armory run` using `user-init` blocks is under future consideration. ## Example 1: Model Layer Output ### User Story @@ -20,7 +20,7 @@ from armory.instrument import get_probe, Meter, get_hub # load Scenario s = get_scenario( - "/armory/tmp/2022-11-03T180812.020999/carla_obj_det_adversarialpatch_undefended.json", + "/armory/scenario_configs/eval6/carla_overhead_object_detection/carla_obj_det_adversarialpatch_undefended.json", num_eval_batches=1, ).load() @@ -134,7 +134,7 @@ def post_method_hook(x_patch, patch_target, transformations): # load Scenario s = get_scenario( - "/armory/tmp/2022-11-03T180812.020999/carla_obj_det_adversarialpatch_undefended.json", + "/armory/scenario_configs/eval6/carla_overhead_object_detection/carla_obj_det_dpatch_undefended.json", num_eval_batches=1, ).load() @@ -246,7 +246,7 @@ def post_method_hook(obj): # load Scenario s = get_scenario( - "/armory/tmp/2022-11-03T180812.020999/carla_obj_det_adversarialpatch_undefended.json", + "/armory/scenario_configs/eval6/carla_overhead_object_detection/carla_obj_det_adversarialpatch_undefended.json", num_eval_batches=1, ).load() diff --git a/docs/probe_hooking_notebook.md b/docs/probe_hooking_notebook.md new file mode 100644 index 000000000..d70a26065 --- /dev/null +++ b/docs/probe_hooking_notebook.md @@ -0,0 +1,316 @@ +# Probes and Meters: Advanced Hooking Examples for Notebooks +For an introduction to `Probe`s and `Meter`s, please refer to [Measurement Overview](./metrics.md). + +## Context +To monitor particular aspects of an `armory run` session, the user needs to know the following factors: +- What am I measuring? +- When should I measure it? +- Where should my custom monitoring script go? + +The examples in this section highlight the nuances of using `Probe`s and `Meter`s for flexible monitoring arrangements in `armory`. In particular, we assume the user is working from a Jupyter notebook and wishes to use `Probe`s and `Meter`s to monitor a model or attack ***without*** modifying the existing code. Documentation and development for enabling the functionality as discribed here for an end-to-end execution with `armory run` using `user-init` blocks is under future consideration. + +## Example 1: Model Layer Output +### User Story +I have a `PyTorchFasterRCNN` model and I am interested in output from the `relu` activation of the third (index 2) `Bottleneck` of `layer4` +### Example Code +This is an example of working with a python package/framework (i.e. `pytorch`) that comes with built-in hooking mechanisms. In the code snippet below, we are relying on an existing function `register_forward_hook` to monitor the layer of interest: +```python +from armory.scenarios.main import get as get_scenario +from armory.instrument import get_probe, Meter, get_hub + +# load Scenario +s = get_scenario( + "/armory/scenario_configs/eval6/carla_overhead_object_detection/carla_obj_det_adversarialpatch_undefended.json", + num_eval_batches=1, +).load() + +# create Probe with "test" namespace +probe = get_probe("test") + +# define the hook to pass to "register_forward_hook" +# the signature of 3 inputs is what pytorch expects +# hook_module refers to the layer of interest, but is not explicitly referenced when passing to register_forward_hook +def hook_fn(hook_module, hook_input, hook_output): + probe.update(lambda x: x.detach().cpu().numpy(), layer4_2_relu=hook_output[0][0]) # [0][0] for slicing + +# register hook +# the hook_module mentioned earlier is referenced via s.model.model.backbone.body.layer4[2].relu +# the register_forward_hook method call must be passing self as a hook_module to hook_fn +s.model.model.backbone.body.layer4[2].relu.register_forward_hook(hook_fn) + +# create Meter for Probe with "test" namespace +meter = Meter("layer4_2_relu", lambda x: x, "test.layer4_2_relu") + +# connect Meter to Hub +get_hub().connect_meter(meter) + +s.next() +s.run_attack() +``` + +### Packages with Hooks +That a package provides a hooking mechanism is convenient, but the user also has to be aware of the what to pass to the hooking mechanism as well as what format to pass it in. Please reference [`pytorch` documentation](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_forward_hook) for more details regarding this example. + +Note that `pytorch` also provides other hooking functionality such as: +- `register_forward_pre_hook` +- `register_full_backward_hook` + +### Probe and Meter Details +Aside the specifics of using `register_forward_hook`, consider how `Probe` and `Meter` are incorporated in this example. Recall the steps for a minimal working example (in [Measurement Overview](./metrics.md)): +1. Create `Probe` via `get_probe("test")` +2. Define `Probe` actions +3. Connect `Probe` +4. Create `Meter` with processing functions that take input from created `Probe` +5. Connect `Meter` to `Hub` via `get_hub().connect_meter(meter)` + +#### Step 1 +Note the input `"test"` that is passed in `get_probe("test")` - this needs to match with the first portion of a `.`-separated argument name `"test.layer4_2_relu"` that is passed to creating a `Meter` in [Step 3](#step4) + +#### Step 2 +The `update` method for `Probe` takes as input optional processing functions and variable names and corresponding values that are to be monitored. +- The variable name `layer4_2_relu` is how we are choosing to reference a certain value + - this needs to match with the second portion of a `.`-separated argument name `"test.layer4_2_relu"` that is passed to creating a `Meter` in [Step 3](#step3) +- `hook_output[0][0]` is the value we are interested in, which is the output from `s.model.model.backbone.body.layer4[2].relu` after a forward pass + - `[0][0]` was included to slice the output to show that it can be done, and because we know the shape of the output in advance +- `lambda x: x.detach().cpu().numpy()` is the processing function that converts `hook_output[0][0]` from a tensor to an array + +#### Step 3 +This particular step is not dealt with in-depth in [Measurement Overview](./metrics.md), but requires more explanation for this section. + +#### Step 4 +In this particular example, the `Meter` accepts 3 inputs: a meter name, a metric/function for processing, and a argument name to pass the metric/function. +- The meter name (`"layer4_2_relu"`) can be arbitrary within this context +- For the scope of this document, we only consider simple `Meter`s with the identity function as a metric i.e. `Meter` will record variables monitored by `Probe` as-is (thus `lambda x: x`) +- The argument passed to the metric/function follows a `.`-separated format (`"test.layer4_2_relu"`), which needs to be consistent with `Probe` setup: + - `test` matches input in `get_probe("test")` + - `layer4_2_relu` matches variable name in `layer4_2_relu=hook_output[0][0]` + +#### Step 5 +For the scope of this document, we don't dwell on what `armory` is doing in step 5 with `get_hub().connect_meter(meter)` other than to mention this step is necessary. + +## Example 2: Attack Artifact - Available as Output +### User Story +I am using `CARLADapricotPatch`, and I am interested in the patch after every iteration, which is generated by `CARLADapricotPatch._augment_images_with_patch` and returned as an output. +### Example Code +This is an example of working with a python package/framework (i.e. `art`) that does NOT come with built-in hooking mechanisms. In the code snippet below, we define wrapper functions to wrap existing instance methods to monitor the output of interest: +```python +from armory.scenarios.main import get as get_scenario +from armory.instrument import get_probe, Meter, get_hub +import types + +def method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None): + """ + Hook target method and return the original method + If a class is passed in, hooks ALL instances of class. + If an object is passed in, only hooks the given instance. + """ + if not isinstance(obj, object): + raise ValueError(f"obj {obj} is not a class or object") + method = getattr(obj, method_name) + if not callable(method): + raise ValueError(f"obj.{method_name} attribute {method} is not callable") + wrapped = hook_wrapper( + method, pre_method_hook=pre_method_hook, post_method_hook=post_method_hook + ) + + if isinstance(obj, type): + cls = obj + setattr(cls, method_name, wrapped) + else: + setattr(obj, method_name, types.MethodType(wrapped, obj)) + + return method + +def hook_wrapper(method, pre_method_hook = None, post_method_hook = None): + def wrapped(*args, **kwargs): + return_value = method(*args[1:], **kwargs) # skip self with *args[1:] + post_method_hook(*return_value) # unpack return_value with * + return return_value + + return wrapped + +def post_method_hook(x_patch, patch_target, transformations): + probe.update(x_patch=x_patch) + +# load Scenario +s = get_scenario( + "/armory/scenario_configs/eval6/carla_overhead_object_detection/carla_obj_det_dpatch_undefended.json", + num_eval_batches=1, +).load() + +# create Probe with "hooked_method" namespace +probe = get_probe("hooked_method") + +# register hook that will update Probe +method_hook( + s.attack, "_augment_images_with_patch", post_method_hook=post_method_hook +) + +# create Meter for Probe with "hooked_method" namespace +hook_meter = Meter( + "hook_x_patch", lambda x: x, "hooked_method.x_patch" +) + +# connect Meter to Hub +get_hub().connect_meter(hook_meter) + +s.next() +s.run_attack() +``` +### Packages with NO Hooks +Unlike [Example 1](#example1), we have defined new functions to meet user needs: +- `method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None)` +- `hook_wrapper(method, pre_method_hook = None, post_method_hook = None)` +- `post_method_hook(x_patch, patch_target, transformations)` + +The steps are the same as before, with the exception that [Step 3](#step3) is more involved than Example 1. + +### Probe and Meter Details - Step 3 +The general approach for hooking a `Probe` is as follows: +1. Define the function for the `Probe` action (e.g. `post_method_hook`) +2. Wrap the method of interest (e.g. `_augment_images_with_patch`) and `post_method_hook` + 1. Define function (e.g. `hook_wrapper`) that returns another function (e.g. `wrapped`) that calls `_augment_images_with_patch` and `post_method_hook` in the desired order + 2. Assign the result of `hook_wrapper` to the original method of interest (`_augment_images_with_patch`) via `method_hook`, thus changing the behavior of the method without modifying it directly + +#### Step 3-1: `post_method_hook` +The role of the defined `post_method_hook` function is the same as that of `hook_fn` defined in [Example 1](#example1) - we are specifying the variable to update being monitored by the `Probe`. In this particular example, despite its name, it is not the name that is specifying whether the `Probe` action occurs after a method, but which input this function is assigned to when calling `method_hook`, which we will show later. + +Note the signature of `post_method_hook` with 3 arguments - this was based on the expected output of `_augment_images_with_patch` i.e. `return_value = method(*args[1:], **kwargs)`, and the anticipation that the output of `_augment_images_with_patch` would be passed to `post_method_hook` as input in `hook_wrapper` i.e. `post_method_hook(*return_value)`. + +Of those expected outputs, we are choosing to update the value assigned to argument `x_patch` of `post_method_hook` and choosing to also refer to the updated value as `x_patch` by the `Probe`, which leads to `probe.update(x_patch=x_patch)`. The `Meter` is then able to reference this as `"hooked_method.x_patch"` later on. + +For now, consider the example shown as a possible template for the user - we leave it as a template rather than add a defined function in armory such that the user can adjust as needed. + +#### Step 3-2.1: `hook_wrapper` +`hook_wrapper` determines when the `Probe` action takes place with respect to a `method` call as well as how inputs should be specified for either a `pre_method_hook` or a `post_method_hook`. Its signature as defined suggests the possibility of either, but in this example, we only specify `post_method_hook`. + +Within `hook_wrapper`, we define `wrapped`, which is where the actual order of calls for `method` and `post_method_hook` are specified, which in this case is `method` then `post_method_hook`. Again, despite the argument name, there is no reason the user cannot specify `post_method_hook` to be called before `method`, but we discourage such practices for clarity. + +Now consider the arguments involved within `hook_wrapper`. `hook_wrapper` once called, will return a function `wrapped`, which expects arguments `*args` and `**kwargs`. Because `hook_wrapper` is meant to wrap `method`, `*args` and `**kwargs` will be passed to `method` such that it performs its original function. Recall that `method` is not just any ordinary function but a method for an instance, which means that even though the first argument of `*args` will be `self` ***[why was `self` even passed to begin with???]***, the actual method call needs to exclude it, leading to `method(*args[1:], **kwargs)`. + +Also note what `wrapped` returns. Since `method` needs to maintain its original functionality, the return value for `wrapped` should also match that of `method`, thus `return return_value`. + +Last but not least, consider the `post_method_hook` call. For this example, the objective was to update a variable that is an output of `method` after each `method` call, and as mentioned in [Step 3-1](#step-3-1-post_method_hook), we pass `return_value` as-is but with a `*` to unpack the iterable, thus `post_method_hook(*return_value)`. + +We present `hook_wrapper` as another template for the user rather than a defined armory function, because, as we have alluded to before, the arrangement of a `Probe` action and instance method is heavily dependent on the user's objective and not obviously generalizable as we will see in the next example. + +#### Step 3-2.2: `method_hook` +Notice that `hook_wrapper` returns a wrapped method, but the wrapped method is not actually reassigned to the method that was meant to be wrapped. `method_hook`, which takes an object `obj` and its associated method name `method`, is defined to do just that, along with actually executing `hook_wrapper` as well for the wrapping process. + +Unlike `post_method_hook` and `hook_wrapper`, which we made a point of framing as templates, we believe `method_hook` is well-established and generalized enough to be defined as an armory function, which the user can import and use as-is. + +## Example 3: Attack Artifact - NOT Available as Output +### User Story +I am using `CARLAAdversarialPatchPyTorch`, and I am interested in the patch after every iteration, which is generated during `CARLAAdversarialPatchPyTorch._train_step`, but NOT provided as an output. +### Example Code +Like [Example 2](#example-2-attack-artifact---available-as-output), the python package/framework (i.e. `art`) does NOT come with built-in hooking mechanisms, BUT unlike Example 2, the method of interest does NOT return the artifact of interest (`_train_step` returns `loss`) - rather, the artifact of interest is available as an attribute (`self._patch`). In the code snippet below, we adjust `post_method_hook` and `hook_wrapper` to reflect this new context: +```python +from armory.scenarios.main import get as get_scenario +from armory.instrument import get_probe, Meter, get_hub +import types + +def method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None): + """ + Hook target method and return the original method + If a class is passed in, hooks ALL instances of class. + If an object is passed in, only hooks the given instance. + """ + if not isinstance(obj, object): + raise ValueError(f"obj {obj} is not a class or object") + method = getattr(obj, method_name) + if not callable(method): + raise ValueError(f"obj.{method_name} attribute {method} is not callable") + wrapped = hook_wrapper( + method, pre_method_hook=pre_method_hook, post_method_hook=post_method_hook + ) + + if isinstance(obj, type): + cls = obj + setattr(cls, method_name, wrapped) + else: + setattr(obj, method_name, types.MethodType(wrapped, obj)) + + return method + +def hook_wrapper(method, pre_method_hook=None, post_method_hook=None): + def wrapped(*args, **kwargs): + return_value = method(*args[1:], **kwargs) # skip self with *args[1:] + if post_method_hook is not None: + post_method_hook(*args[0]) # *args[0] corresponds to self with _patch attribute + return return_value + + return wrapped + +def post_method_hook(obj): + probe.update(patch=obj._patch) + +# load Scenario +s = get_scenario( + "/armory/scenario_configs/eval6/carla_overhead_object_detection/carla_obj_det_adversarialpatch_undefended.json", + num_eval_batches=1, +).load() + +# create Probe with "hooked_method" namespace +probe = get_probe("hooked_method") + +# register hook that will update Probe +method_hook( + s.attack, "_train_step", post_method_hook=post_method_hook +) + +# create Meter for Probe with "hooked_method" namespace +hook_meter = Meter( + "hook_patch", lambda x: x.detach().cpu().numpy(), "hooked_method.patch" +) + +# connect Meter to Hub +get_hub().connect_meter(hook_meter) + +s.next() +s.run_attack() +``` +Consider the functions introduced in [Example 2](#example-2-attack-artifact---available-as-output): +- `method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None)` +- `hook_wrapper(method, pre_method_hook = None, post_method_hook = None)` +- `post_method_hook(x_patch, patch_target, transformations)` + +`method_hook` has stayed the same (which is why we define it as an armory function), but `hook_wrapper` and `post_method_hook` have changed. + +### Probe and Meter Details - Step 3 +Recall the general approach for hooking a `Probe`: +1. Define the function for the `Probe` action (e.g. `post_method_hook`) ***[CHANGED]*** +2. Wrap the method of interest (e.g. `_train_step`) and `post_method_hook` + 1. Define function (e.g. `hook_wrapper`) that returns another function (e.g. `wrapped`) that calls `_train_step` and `post_method_hook` in the desired order ***[CHANGED]*** + 2. Assign the result of `hook_wrapper` to the original method of interest (`_train_step`) via `method_hook`, thus changing the behavior of the method without modifying it directly ***[UNCHANGED]*** + +#### Step 3-1: `post_method_hook` +The signature of `post_method_hook` now specifies a single argument `obj`, which we assume has a `_patch` attribute. Again note that this has nothing to do with the expected output of `_train_step` - we know from inspecting the `_train_step` method that a `_patch` attribute exists, which we refer to within `post_method_hook` via `obj._patch`. We are choosing to measure the value assigned to an attribute of `obj`, an input of `post_method_hook`, and also choosing to refer to the variable to be updated as `patch` by the `Probe`, which leads to `probe.update(patch=obj._patch)`. The `Meter` is then able to reference this as `"hooked_method.patch"` later on. + +This example is another possible template for the user, where the definition of `post_method_hook` changes depending on what the user is interested in monitoring. + +#### Step 3-2.1: `hook_wrapper` +As in [Example 2](#example-2-attack-artifact---available-as-output), `method` is called before `post_method_hook` in `wrapped` of `hook_wrapper`. `return_value` is not used in any way other than being returned at the end of `wrapped` to maintain `method`'s functionality. `*arg[0]` of `*arg` however, is passed to `post_method_hook` as an input, because it refers to `self`, the instance making the `method` call. As mentioned earlier, `self` contains the `_patch` attribute, which is what `probe` is monitoring in `post_method_hook`. + +Again, this example's `hook_wrapper` is another possible template for the user, which has been defined in such as way as to accomplish the objective described in the [User Story](#user-story-2). + +#### Step 4 +A brief note on the `Meter` creation in this example: The `_patch` variable of interest is a tensor, which is why a preprocessing function is applied for the identity metric, `lambda x: x.detach().cpu().numpy()`. The chaining functions could have occurred else where in the process as well. + +## Flexible Arrangements +We have emphasized through out this section that the example functions shown (with the exception of `method_hook`) are templates, and how those functions are defined depends on the user story. Before we discuss what the user needs to consider when defining those functions, it is important that the user remembers the 5-step process for custom probes as a general framework, which may aid in any debugging efforts that may occur later. + +That said, here are the questions the user needs to consider for creating custom probes: +- What is the `method` of interest? What is the variable of interest? + - Some knowledge of the package being used is necessary +- Is there an existing hooking mechanism for the `method` of interest? + - Relying on hooks from the package provider reduces code and maintenance +- Does the `Probe` action occur before or after a `method` call? + - Define `pre_method_hook` or `post_method_hook` and arrange calls as necessary with respect to `method` call +- What should `pre_method_hook`/`post_method_hook` expect as input? + - It may be natural for a `pre_method_hook` to take the same arguments as `method` i.e. `*args` and `**kwargs` + - It may be natural for a `post_method_hook` to take the output of a `method` call i.e. `*return_value` + - If the user is interested in the state of an instance or the attributes within it pre or post `method` call, passing the instance to `pre_method_hook`/`post_method_hook` is also possible i.e. `*args[0]` +- Is `wrapped` returning `return_value`? + - `method`'s original functionality has to be maintained + +Once these questions have been answered, the user can define functions as needed to meet specific monitoring objectives. \ No newline at end of file From e6e13da366cfa5682c8eee5acc490e3677903191 Mon Sep 17 00:00:00 2001 From: Paul Park Date: Fri, 16 Dec 2022 17:53:53 +0000 Subject: [PATCH 11/30] completed initial draft of probe.md with new examples; need to test and also add a section on exporting to pickle file --- docs/probe.md | 403 +++++++++++++++++--------------------------------- 1 file changed, 132 insertions(+), 271 deletions(-) diff --git a/docs/probe.md b/docs/probe.md index d70a26065..ceb596ecd 100644 --- a/docs/probe.md +++ b/docs/probe.md @@ -1,4 +1,4 @@ -# Probes and Meters: Advanced Hooking Examples for Notebooks +# Probes and Meters: Examples for Adding to Code For an introduction to `Probe`s and `Meter`s, please refer to [Measurement Overview](./metrics.md). ## Context @@ -7,310 +7,171 @@ To monitor particular aspects of an `armory run` session, the user needs to know - When should I measure it? - Where should my custom monitoring script go? -The examples in this section highlight the nuances of using `Probe`s and `Meter`s for flexible monitoring arrangements in `armory`. In particular, we assume the user is working from a Jupyter notebook and wishes to use `Probe`s and `Meter`s to monitor a model or attack ***without*** modifying the existing code. Documentation and development for enabling the functionality as discribed here for an end-to-end execution with `armory run` using `user-init` blocks is under future consideration. +We assume the user is modifying code for models or attacks and wishes to use `Probe`s and `Meter`s to monitor certain variables within the code. + +Recall the steps for a minimal working example (in [Measurement Overview](./metrics.md)): +1. Create `Probe` via `get_probe(name)` +2. Place `Probe` actions +3. Create `Meter` with processing functions that take input from created `Probe` +4. Connect `Meter` to `Hub` via `get_hub().connect_meter(meter)` + +The examples will show how each of these steps are accomplished. ## Example 1: Model Layer Output ### User Story -I have a `PyTorchFasterRCNN` model and I am interested in output from the `relu` activation of the third (index 2) `Bottleneck` of `layer4` -### Example Code -This is an example of working with a python package/framework (i.e. `pytorch`) that comes with built-in hooking mechanisms. In the code snippet below, we are relying on an existing function `register_forward_hook` to monitor the layer of interest: +I am interested in layer output from the second `relu` activation of a `forward` method located in `armory/baseline_models/pytorch/cifar.py`. +### `Probe` Example Code +The code below is an example of how to accomplish steps 1 and 2 (note the lines of code with `# added` comments at the end) for a model code that the user is modifying. ```python -from armory.scenarios.main import get as get_scenario -from armory.instrument import get_probe, Meter, get_hub - -# load Scenario -s = get_scenario( - "/armory/scenario_configs/eval6/carla_overhead_object_detection/carla_obj_det_adversarialpatch_undefended.json", - num_eval_batches=1, -).load() +""" +CNN model for 32x32x3 image classification +""" +from typing import Optional -# create Probe with "test" namespace -probe = get_probe("test") +import torch +import torch.nn as nn +import torch.nn.functional as F +from art.estimators.classification import PyTorchClassifier -# define the hook to pass to "register_forward_hook" -# the signature of 3 inputs is what pytorch expects -# hook_module refers to the layer of interest, but is not explicitly referenced when passing to register_forward_hook -def hook_fn(hook_module, hook_input, hook_output): - probe.update(lambda x: x.detach().cpu().numpy(), layer4_2_relu=hook_output[0][0]) # [0][0] for slicing +from armory.instrument import get_probe # added +probe = get_probe("my_model") # added -# register hook -# the hook_module mentioned earlier is referenced via s.model.model.backbone.body.layer4[2].relu -# the register_forward_hook method call must be passing self as a hook_module to hook_fn -s.model.model.backbone.body.layer4[2].relu.register_forward_hook(hook_fn) +DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu") -# create Meter for Probe with "test" namespace -meter = Meter("layer4_2_relu", lambda x: x, "test.layer4_2_relu") -# connect Meter to Hub -get_hub().connect_meter(meter) +class Net(nn.Module): + """ + This is a simple CNN for CIFAR-10 and does not achieve SotA performance + """ -s.next() -s.run_attack() + def __init__(self) -> None: + super(Net, self).__init__() + self.conv1 = nn.Conv2d(3, 4, 5, 1) + self.conv2 = nn.Conv2d(4, 10, 5, 1) + self.fc1 = nn.Linear(250, 100) + self.fc2 = nn.Linear(100, 10) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + x = x.permute(0, 3, 1, 2) # from NHWC to NCHW + x = self.conv1(x) + x = F.relu(x) + x = F.max_pool2d(x, 2) + x = self.conv2(x) + x = F.relu(x) + x_out = x.detach().cpu().numpy() # added + probe.update(layer_output=x_out) # added + x = F.max_pool2d(x, 2) + x = torch.flatten(x, 1) + x = self.fc1(x) + x = F.relu(x) + x = self.fc2(x) + output = F.log_softmax(x, dim=1) + return output + +... ``` -### Packages with Hooks -That a package provides a hooking mechanism is convenient, but the user also has to be aware of the what to pass to the hooking mechanism as well as what format to pass it in. Please reference [`pytorch` documentation](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_forward_hook) for more details regarding this example. -Note that `pytorch` also provides other hooking functionality such as: -- `register_forward_pre_hook` -- `register_full_backward_hook` - -### Probe and Meter Details -Aside the specifics of using `register_forward_hook`, consider how `Probe` and `Meter` are incorporated in this example. Recall the steps for a minimal working example (in [Measurement Overview](./metrics.md)): -1. Create `Probe` via `get_probe("test")` -2. Define `Probe` actions -3. Connect `Probe` -4. Create `Meter` with processing functions that take input from created `Probe` -5. Connect `Meter` to `Hub` via `get_hub().connect_meter(meter)` #### Step 1 -Note the input `"test"` that is passed in `get_probe("test")` - this needs to match with the first portion of a `.`-separated argument name `"test.layer4_2_relu"` that is passed to creating a `Meter` in [Step 3](#step4) +After importing `get_probe`, `probe = get_probe("my_model")` creates a `Probe` object with the name `"my_model"`, which is what the user can refer to later to apply processing functions through a `Meter` object. #### Step 2 -The `update` method for `Probe` takes as input optional processing functions and variable names and corresponding values that are to be monitored. -- The variable name `layer4_2_relu` is how we are choosing to reference a certain value - - this needs to match with the second portion of a `.`-separated argument name `"test.layer4_2_relu"` that is passed to creating a `Meter` in [Step 3](#step3) -- `hook_output[0][0]` is the value we are interested in, which is the output from `s.model.model.backbone.body.layer4[2].relu` after a forward pass - - `[0][0]` was included to slice the output to show that it can be done, and because we know the shape of the output in advance -- `lambda x: x.detach().cpu().numpy()` is the processing function that converts `hook_output[0][0]` from a tensor to an array +`x_out = x.detach().cpu().numpy()` is taking the layer output of interest (second `relu` activation output) and converting the tensor to `numpy` array on the CPU, which will be passed to `probe`. An updated value of `x_out` is stored in `layer_output` via `probe.update(layer_output=x_out)`. Like the `Probe` name `"my_model"`, `layer_output` can be referenced by the user later to apply additional processing functions through a `Meter` object. -#### Step 3 -This particular step is not dealt with in-depth in [Measurement Overview](./metrics.md), but requires more explanation for this section. +### `Meter` Example Code +Now that a `Probe` instance has been created, we need to create a `Meter` object to accept any updated values from `Probe` and apply further processing that the user desires. Suppose the user created a script located at `armory/user_init.py`: +```python +from armory.instrument import get_hub, Meter -#### Step 4 +def user_init_function(): + meter = Meter( + "my_layer_identity", lambda x: x, "my_model.layer_output" + ) + get_hub().connect_meter(meter) +``` +#### Step 3 In this particular example, the `Meter` accepts 3 inputs: a meter name, a metric/function for processing, and a argument name to pass the metric/function. -- The meter name (`"layer4_2_relu"`) can be arbitrary within this context +- The meter name (`"my_layer_identity"`) can be arbitrary within this context - For the scope of this document, we only consider simple `Meter`s with the identity function as a metric i.e. `Meter` will record variables monitored by `Probe` as-is (thus `lambda x: x`) -- The argument passed to the metric/function follows a `.`-separated format (`"test.layer4_2_relu"`), which needs to be consistent with `Probe` setup: - - `test` matches input in `get_probe("test")` - - `layer4_2_relu` matches variable name in `layer4_2_relu=hook_output[0][0]` +- The argument passed to the metric/function follows a `.`-separated format (`"my_model.layer_output"`), which needs to be consistent with `Probe` setup earlier: + - `my_model` matches input in `probe = get_probe("my_model")` + - `layer_output` matches variable name in `probe.update(layer_output=x_out)` -#### Step 5 -For the scope of this document, we don't dwell on what `armory` is doing in step 5 with `get_hub().connect_meter(meter)` other than to mention this step is necessary. +#### Step 4 +For the scope of this document, we don't dwell on what `armory` is doing with `get_hub().connect_meter(meter)` other than to mention this step is necessary for establishing the connection between `meter` created in `armory/user_init.py` and `probe` created in the modified version of `armory/baseline_models/pytorch/cifar.py`. + +### Config Setup +Last but not least, the config file passed to `armory run` needs to updated for these changes to take effect. Assuming the `"model"` block has been changed appropriately, the other block that needs to be added is `"user-init"`: +```json +... + "model": { + ... + }, + "user-init": { + "module": "user_init", + "name": "user_init_function" + }, +... +``` +This will prompt armory to run `user_init_function` in `user_init.py` before anything else is loaded for the scenario. -## Example 2: Attack Artifact - Available as Output +## Example 2: Attack Output ### User Story -I am using `CARLADapricotPatch`, and I am interested in the patch after every iteration, which is generated by `CARLADapricotPatch._augment_images_with_patch` and returned as an output. -### Example Code -This is an example of working with a python package/framework (i.e. `art`) that does NOT come with built-in hooking mechanisms. In the code snippet below, we define wrapper functions to wrap existing instance methods to monitor the output of interest: +I defined a custom attack with `CARLADapricotPatch` in `armory/custom_attack.py`, and I am interested in the patch after ***every iteration***, which is generated by `CARLADapricotPatch._augment_images_with_patch` and returned as an output. +### `Probe` Example Code ```python -from armory.scenarios.main import get as get_scenario -from armory.instrument import get_probe, Meter, get_hub -import types - -def method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None): - """ - Hook target method and return the original method - If a class is passed in, hooks ALL instances of class. - If an object is passed in, only hooks the given instance. - """ - if not isinstance(obj, object): - raise ValueError(f"obj {obj} is not a class or object") - method = getattr(obj, method_name) - if not callable(method): - raise ValueError(f"obj.{method_name} attribute {method} is not callable") - wrapped = hook_wrapper( - method, pre_method_hook=pre_method_hook, post_method_hook=post_method_hook - ) +from armory.art_experimental.attacks.carla_obj_det_patch import CARLADapricotPatch +from armory.instrument import get_probe +probe = get_probe("my_attack") - if isinstance(obj, type): - cls = obj - setattr(cls, method_name, wrapped) - else: - setattr(obj, method_name, types.MethodType(wrapped, obj)) +class CustomAttack(CARLADapricotPatch): + def _augment_images_with_patch(self, **kwargs): + return_value = super()._augment_images_with_patch(**kwargs) + x_patch, patch_target, transformations = return_value + probe.update(attack_output=x_patch) - return method - -def hook_wrapper(method, pre_method_hook = None, post_method_hook = None): - def wrapped(*args, **kwargs): - return_value = method(*args[1:], **kwargs) # skip self with *args[1:] - post_method_hook(*return_value) # unpack return_value with * return return_value - - return wrapped - -def post_method_hook(x_patch, patch_target, transformations): - probe.update(x_patch=x_patch) - -# load Scenario -s = get_scenario( - "/armory/scenario_configs/eval6/carla_overhead_object_detection/carla_obj_det_dpatch_undefended.json", - num_eval_batches=1, -).load() - -# create Probe with "hooked_method" namespace -probe = get_probe("hooked_method") - -# register hook that will update Probe -method_hook( - s.attack, "_augment_images_with_patch", post_method_hook=post_method_hook -) - -# create Meter for Probe with "hooked_method" namespace -hook_meter = Meter( - "hook_x_patch", lambda x: x, "hooked_method.x_patch" -) - -# connect Meter to Hub -get_hub().connect_meter(hook_meter) - -s.next() -s.run_attack() ``` -### Packages with NO Hooks -Unlike [Example 1](#example1), we have defined new functions to meet user needs: -- `method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None)` -- `hook_wrapper(method, pre_method_hook = None, post_method_hook = None)` -- `post_method_hook(x_patch, patch_target, transformations)` - -The steps are the same as before, with the exception that [Step 3](#step3) is more involved than Example 1. - -### Probe and Meter Details - Step 3 -The general approach for hooking a `Probe` is as follows: -1. Define the function for the `Probe` action (e.g. `post_method_hook`) -2. Wrap the method of interest (e.g. `_augment_images_with_patch`) and `post_method_hook` - 1. Define function (e.g. `hook_wrapper`) that returns another function (e.g. `wrapped`) that calls `_augment_images_with_patch` and `post_method_hook` in the desired order - 2. Assign the result of `hook_wrapper` to the original method of interest (`_augment_images_with_patch`) via `method_hook`, thus changing the behavior of the method without modifying it directly - -#### Step 3-1: `post_method_hook` -The role of the defined `post_method_hook` function is the same as that of `hook_fn` defined in [Example 1](#example1) - we are specifying the variable to update being monitored by the `Probe`. In this particular example, despite its name, it is not the name that is specifying whether the `Probe` action occurs after a method, but which input this function is assigned to when calling `method_hook`, which we will show later. - -Note the signature of `post_method_hook` with 3 arguments - this was based on the expected output of `_augment_images_with_patch` i.e. `return_value = method(*args[1:], **kwargs)`, and the anticipation that the output of `_augment_images_with_patch` would be passed to `post_method_hook` as input in `hook_wrapper` i.e. `post_method_hook(*return_value)`. - -Of those expected outputs, we are choosing to update the value assigned to argument `x_patch` of `post_method_hook` and choosing to also refer to the updated value as `x_patch` by the `Probe`, which leads to `probe.update(x_patch=x_patch)`. The `Meter` is then able to reference this as `"hooked_method.x_patch"` later on. - -For now, consider the example shown as a possible template for the user - we leave it as a template rather than add a defined function in armory such that the user can adjust as needed. - -#### Step 3-2.1: `hook_wrapper` -`hook_wrapper` determines when the `Probe` action takes place with respect to a `method` call as well as how inputs should be specified for either a `pre_method_hook` or a `post_method_hook`. Its signature as defined suggests the possibility of either, but in this example, we only specify `post_method_hook`. - -Within `hook_wrapper`, we define `wrapped`, which is where the actual order of calls for `method` and `post_method_hook` are specified, which in this case is `method` then `post_method_hook`. Again, despite the argument name, there is no reason the user cannot specify `post_method_hook` to be called before `method`, but we discourage such practices for clarity. - -Now consider the arguments involved within `hook_wrapper`. `hook_wrapper` once called, will return a function `wrapped`, which expects arguments `*args` and `**kwargs`. Because `hook_wrapper` is meant to wrap `method`, `*args` and `**kwargs` will be passed to `method` such that it performs its original function. Recall that `method` is not just any ordinary function but a method for an instance, which means that even though the first argument of `*args` will be `self` ***[why was `self` even passed to begin with???]***, the actual method call needs to exclude it, leading to `method(*args[1:], **kwargs)`. - -Also note what `wrapped` returns. Since `method` needs to maintain its original functionality, the return value for `wrapped` should also match that of `method`, thus `return return_value`. - -Last but not least, consider the `post_method_hook` call. For this example, the objective was to update a variable that is an output of `method` after each `method` call, and as mentioned in [Step 3-1](#step-3-1-post_method_hook), we pass `return_value` as-is but with a `*` to unpack the iterable, thus `post_method_hook(*return_value)`. - -We present `hook_wrapper` as another template for the user rather than a defined armory function, because, as we have alluded to before, the arrangement of a `Probe` action and instance method is heavily dependent on the user's objective and not obviously generalizable as we will see in the next example. - -#### Step 3-2.2: `method_hook` -Notice that `hook_wrapper` returns a wrapped method, but the wrapped method is not actually reassigned to the method that was meant to be wrapped. `method_hook`, which takes an object `obj` and its associated method name `method`, is defined to do just that, along with actually executing `hook_wrapper` as well for the wrapping process. +#### Step 1 +This step is the same as before, except `Probe` name is set to`"my_attack"`, which is what the user can refer to later to apply processing functions through a `Meter` object. -Unlike `post_method_hook` and `hook_wrapper`, which we made a point of framing as templates, we believe `method_hook` is well-established and generalized enough to be defined as an armory function, which the user can import and use as-is. +#### Step 2 +The only difference between `CustomAttack` and `CARLADapricotPatch` is that `_augment_images_with_patch` has been redefined to call on `CARLADapricotPatch._augment_images_with_patch` and then have `probe` update the value for `x_patch` that results from that call. An updated value of `x_patch` is stored in `attack_output` via `probe.update(attack_output=x_patch)`. Like the `Probe` name `"my_attack"`, `attack_output` can be referenced by the user later to apply additional processing functions through a `Meter` object. -## Example 3: Attack Artifact - NOT Available as Output -### User Story -I am using `CARLAAdversarialPatchPyTorch`, and I am interested in the patch after every iteration, which is generated during `CARLAAdversarialPatchPyTorch._train_step`, but NOT provided as an output. -### Example Code -Like [Example 2](#example-2-attack-artifact---available-as-output), the python package/framework (i.e. `art`) does NOT come with built-in hooking mechanisms, BUT unlike Example 2, the method of interest does NOT return the artifact of interest (`_train_step` returns `loss`) - rather, the artifact of interest is available as an attribute (`self._patch`). In the code snippet below, we adjust `post_method_hook` and `hook_wrapper` to reflect this new context: +### `Meter` Example Code +Now that a `Probe` instance has been created, we need to create a `Meter` object to accept any updated values from `Probe` and apply further processing that the user desires. Suppose the user created a script located at `armory/user_init.py`: ```python -from armory.scenarios.main import get as get_scenario -from armory.instrument import get_probe, Meter, get_hub -import types +from armory.instrument import get_hub, Meter -def method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None): - """ - Hook target method and return the original method - If a class is passed in, hooks ALL instances of class. - If an object is passed in, only hooks the given instance. - """ - if not isinstance(obj, object): - raise ValueError(f"obj {obj} is not a class or object") - method = getattr(obj, method_name) - if not callable(method): - raise ValueError(f"obj.{method_name} attribute {method} is not callable") - wrapped = hook_wrapper( - method, pre_method_hook=pre_method_hook, post_method_hook=post_method_hook +def user_init_function(): + meter = Meter( + "my_attack_identity", lambda x: x, "my_attack.attack_output" ) - - if isinstance(obj, type): - cls = obj - setattr(cls, method_name, wrapped) - else: - setattr(obj, method_name, types.MethodType(wrapped, obj)) - - return method - -def hook_wrapper(method, pre_method_hook=None, post_method_hook=None): - def wrapped(*args, **kwargs): - return_value = method(*args[1:], **kwargs) # skip self with *args[1:] - if post_method_hook is not None: - post_method_hook(*args[0]) # *args[0] corresponds to self with _patch attribute - return return_value - - return wrapped - -def post_method_hook(obj): - probe.update(patch=obj._patch) - -# load Scenario -s = get_scenario( - "/armory/scenario_configs/eval6/carla_overhead_object_detection/carla_obj_det_adversarialpatch_undefended.json", - num_eval_batches=1, -).load() - -# create Probe with "hooked_method" namespace -probe = get_probe("hooked_method") - -# register hook that will update Probe -method_hook( - s.attack, "_train_step", post_method_hook=post_method_hook -) - -# create Meter for Probe with "hooked_method" namespace -hook_meter = Meter( - "hook_patch", lambda x: x.detach().cpu().numpy(), "hooked_method.patch" -) - -# connect Meter to Hub -get_hub().connect_meter(hook_meter) - -s.next() -s.run_attack() + get_hub().connect_meter(meter) ``` -Consider the functions introduced in [Example 2](#example-2-attack-artifact---available-as-output): -- `method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None)` -- `hook_wrapper(method, pre_method_hook = None, post_method_hook = None)` -- `post_method_hook(x_patch, patch_target, transformations)` - -`method_hook` has stayed the same (which is why we define it as an armory function), but `hook_wrapper` and `post_method_hook` have changed. - -### Probe and Meter Details - Step 3 -Recall the general approach for hooking a `Probe`: -1. Define the function for the `Probe` action (e.g. `post_method_hook`) ***[CHANGED]*** -2. Wrap the method of interest (e.g. `_train_step`) and `post_method_hook` - 1. Define function (e.g. `hook_wrapper`) that returns another function (e.g. `wrapped`) that calls `_train_step` and `post_method_hook` in the desired order ***[CHANGED]*** - 2. Assign the result of `hook_wrapper` to the original method of interest (`_train_step`) via `method_hook`, thus changing the behavior of the method without modifying it directly ***[UNCHANGED]*** - -#### Step 3-1: `post_method_hook` -The signature of `post_method_hook` now specifies a single argument `obj`, which we assume has a `_patch` attribute. Again note that this has nothing to do with the expected output of `_train_step` - we know from inspecting the `_train_step` method that a `_patch` attribute exists, which we refer to within `post_method_hook` via `obj._patch`. We are choosing to measure the value assigned to an attribute of `obj`, an input of `post_method_hook`, and also choosing to refer to the variable to be updated as `patch` by the `Probe`, which leads to `probe.update(patch=obj._patch)`. The `Meter` is then able to reference this as `"hooked_method.patch"` later on. - -This example is another possible template for the user, where the definition of `post_method_hook` changes depending on what the user is interested in monitoring. - -#### Step 3-2.1: `hook_wrapper` -As in [Example 2](#example-2-attack-artifact---available-as-output), `method` is called before `post_method_hook` in `wrapped` of `hook_wrapper`. `return_value` is not used in any way other than being returned at the end of `wrapped` to maintain `method`'s functionality. `*arg[0]` of `*arg` however, is passed to `post_method_hook` as an input, because it refers to `self`, the instance making the `method` call. As mentioned earlier, `self` contains the `_patch` attribute, which is what `probe` is monitoring in `post_method_hook`. - -Again, this example's `hook_wrapper` is another possible template for the user, which has been defined in such as way as to accomplish the objective described in the [User Story](#user-story-2). +#### Step 3 +As before, the `Meter` accepts 3 inputs: a meter name, a metric/function for processing, and a argument name to pass the metric/function. +- The meter name (`"my_attack_identity"`) can be arbitrary within this context +- Again, `Meter` will record variables monitored by `Probe` as-is (thus `lambda x: x`) +- The argument passed to the metric/function follows a `.`-separated format (`"my_attack.attack_output"`), which needs to be consistent with `Probe` setup earlier: + - `my_attack` matches input in `probe = get_probe("my_attack")` + - `attack_output` matches variable name in `probe.update(attack_output=x_patch)` #### Step 4 -A brief note on the `Meter` creation in this example: The `_patch` variable of interest is a tensor, which is why a preprocessing function is applied for the identity metric, `lambda x: x.detach().cpu().numpy()`. The chaining functions could have occurred else where in the process as well. - -## Flexible Arrangements -We have emphasized through out this section that the example functions shown (with the exception of `method_hook`) are templates, and how those functions are defined depends on the user story. Before we discuss what the user needs to consider when defining those functions, it is important that the user remembers the 5-step process for custom probes as a general framework, which may aid in any debugging efforts that may occur later. - -That said, here are the questions the user needs to consider for creating custom probes: -- What is the `method` of interest? What is the variable of interest? - - Some knowledge of the package being used is necessary -- Is there an existing hooking mechanism for the `method` of interest? - - Relying on hooks from the package provider reduces code and maintenance -- Does the `Probe` action occur before or after a `method` call? - - Define `pre_method_hook` or `post_method_hook` and arrange calls as necessary with respect to `method` call -- What should `pre_method_hook`/`post_method_hook` expect as input? - - It may be natural for a `pre_method_hook` to take the same arguments as `method` i.e. `*args` and `**kwargs` - - It may be natural for a `post_method_hook` to take the output of a `method` call i.e. `*return_value` - - If the user is interested in the state of an instance or the attributes within it pre or post `method` call, passing the instance to `pre_method_hook`/`post_method_hook` is also possible i.e. `*args[0]` -- Is `wrapped` returning `return_value`? - - `method`'s original functionality has to be maintained - -Once these questions have been answered, the user can define functions as needed to meet specific monitoring objectives. \ No newline at end of file +Again, `get_hub().connect_meter(meter)` is necessary for establishing the connection between `meter` created in `armory/user_init.py` and `probe` created in `armory/custom_attack.py`. + +### Config Setup +Last but not least, the config file passed to `armory run` needs to updated for these changes to take effect. Assuming the `"attack"` block has been changed appropriately, the other block that needs to be added is `"user-init"`: +```json +... + "attack": { + ... + }, + "user-init": { + "module": "user_init", + "name": "user_init_function" + }, +... +``` +This will prompt armory to run `user_init_function` in `user_init.py` before anything else is loaded for the scenario. \ No newline at end of file From d0c53e0525561eb642189ffa7e1780cb566c0872 Mon Sep 17 00:00:00 2001 From: Paul Park Date: Fri, 16 Dec 2022 19:42:24 +0000 Subject: [PATCH 12/30] tested examples with armory run and corrected user_init block name for config documentation --- docs/probe.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/probe.md b/docs/probe.md index ceb596ecd..a0bd437e5 100644 --- a/docs/probe.md +++ b/docs/probe.md @@ -102,13 +102,13 @@ In this particular example, the `Meter` accepts 3 inputs: a meter name, a metric For the scope of this document, we don't dwell on what `armory` is doing with `get_hub().connect_meter(meter)` other than to mention this step is necessary for establishing the connection between `meter` created in `armory/user_init.py` and `probe` created in the modified version of `armory/baseline_models/pytorch/cifar.py`. ### Config Setup -Last but not least, the config file passed to `armory run` needs to updated for these changes to take effect. Assuming the `"model"` block has been changed appropriately, the other block that needs to be added is `"user-init"`: +Last but not least, the config file passed to `armory run` needs to updated for these changes to take effect. Assuming the `"model"` block has been changed appropriately, the other block that needs to be added is `"user_init"`: ```json ... "model": { ... }, - "user-init": { + "user_init": { "module": "user_init", "name": "user_init_function" }, @@ -162,13 +162,13 @@ As before, the `Meter` accepts 3 inputs: a meter name, a metric/function for pro Again, `get_hub().connect_meter(meter)` is necessary for establishing the connection between `meter` created in `armory/user_init.py` and `probe` created in `armory/custom_attack.py`. ### Config Setup -Last but not least, the config file passed to `armory run` needs to updated for these changes to take effect. Assuming the `"attack"` block has been changed appropriately, the other block that needs to be added is `"user-init"`: +Last but not least, the config file passed to `armory run` needs to updated for these changes to take effect. Assuming the `"attack"` block has been changed appropriately, the other block that needs to be added is `"user_init"`: ```json ... "attack": { ... }, - "user-init": { + "user_init": { "module": "user_init", "name": "user_init_function" }, From 8a4b9fff907de7dd2a40cf94ee8d63cca51e2eff Mon Sep 17 00:00:00 2001 From: Paul Park Date: Fri, 16 Dec 2022 19:52:48 +0000 Subject: [PATCH 13/30] removed code and documentation referring to hook methods defined for Probe - these changes were made based on consensus reached with David and Lucas that documentation showing how Probes can be used within existing packages or code is more sustainable for maintenance in the long run rather than having to define functions to fit hooking mechanisms that may or may not exist in other packages into the Probe framework, which will require updates should any of the hooks defined in those packages change --- armory/instrument/instrument.py | 53 --------------------------------- docs/metrics.md | 16 ---------- 2 files changed, 69 deletions(-) diff --git a/armory/instrument/instrument.py b/armory/instrument/instrument.py index f18e19906..c489f1470 100644 --- a/armory/instrument/instrument.py +++ b/armory/instrument/instrument.py @@ -118,59 +118,6 @@ def update(self, *preprocessing, **named_values): # Push to sink self.sink.update(name, value) - def hook(self, module, *preprocessing, input=None, output=None, mode="pytorch"): - if mode == "pytorch": - return self.hook_torch(module, *preprocessing, input=input, output=output) - elif mode == "tf": - return self.hook_tf(module, *preprocessing, input=input, output=output) - raise ValueError(f"mode {mode} not in ('pytorch', 'tf')") - - def hook_tf(self, module, *preprocessing, input=None, output=None): - raise NotImplementedError("hooking not ready for tensorflow") - # NOTE: - # https://discuss.pytorch.org/t/get-the-activations-of-the-second-to-last-layer/55629/6 - # TensorFlow hooks - # https://www.tensorflow.org/api_docs/python/tf/estimator/SessionRunHook - # https://github.com/tensorflow/tensorflow/issues/33478 - # https://github.com/tensorflow/tensorflow/issues/33129 - # https://stackoverflow.com/questions/48966281/get-intermediate-output-from-keras-tensorflow-during-prediction - # https://stackoverflow.com/questions/59493222/access-output-of-intermediate-layers-in-tensor-flow-2-0-in-eager-mode/60945216#60945216 - - def hook_torch(self, module, *preprocessing, input=None, output=None): - if not hasattr(module, "register_forward_hook"): - raise ValueError( - f"module {module} does not have method 'register_forward_hook'. Is it a torch.nn.Module?" - ) - if input == "" or (input is not None and not isinstance(input, str)): - raise ValueError(f"input {input} must be None or a non-empty string") - if output == "" or (output is not None and not isinstance(output, str)): - raise ValueError(f"output {output} must be None or a non-empty string") - if input is None and output is None: - raise ValueError("input and output cannot both be None") - if module in self._hooks: - raise ValueError(f"module {module} is already hooked") - - def hook_fn(hook_module, hook_input, hook_output): - del hook_module - key_values = {} - if input is not None: - key_values[input] = hook_input - if output is not None: - key_values[output] = hook_output - self.update(*preprocessing, **key_values) - - hook = module.register_forward_hook(hook_fn) - self._hooks[module] = (hook, "pytorch") - - def unhook(self, module): - hook, mode = self._hooks.pop(module) - if mode == "pytorch": - hook.remove() - elif mode == "tf": - raise NotImplementedError() - else: - raise ValueError(f"mode {mode} not in ('pytorch', 'tf')") - class MockSink: """ diff --git a/docs/metrics.md b/docs/metrics.md index 4f5d18729..1ef3af611 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -427,22 +427,6 @@ probe.update(func1, func2, func3, my_var=y) ``` will publish the value `func3(func2(func1(y)))`. -#### Hooking - -Probes can also hook models to enable capturing values without modifying the target code. -Currently, hooking is only implemented for PyTorch, but TensorFlow is on the roadmap. - -To hook a model module, you can use the `hook` function. -For instance, -```python -# probe.hook(module, *preprocessing, input=None, output=None) -probe.hook(convnet.layer1[0].conv2, lambda x: x.detach().cpu().numpy(), output="b") -``` -This essentially wraps the `probe.update` call with a hooking function. -This is intended for usage that cannot or does not modify the target codebase. - -More general hooking (e.g., for python methods) is TBD. - #### Interactive Testing An easy way to test probe outputs is to set the probe to a `MockSink` interface. From 89d200a63ee9cc4c7e417bc781ceab000dd8791f Mon Sep 17 00:00:00 2001 From: Paul Park Date: Fri, 16 Dec 2022 20:53:22 +0000 Subject: [PATCH 14/30] resolving pr comments --- docs/{probe.md => instrumentation_example.md} | 30 +++++++++---------- 1 file changed, 15 insertions(+), 15 deletions(-) rename docs/{probe.md => instrumentation_example.md} (88%) diff --git a/docs/probe.md b/docs/instrumentation_example.md similarity index 88% rename from docs/probe.md rename to docs/instrumentation_example.md index a0bd437e5..fbe356faf 100644 --- a/docs/probe.md +++ b/docs/instrumentation_example.md @@ -1,4 +1,4 @@ -# Probes and Meters: Examples for Adding to Code +# Armory Instrumentation Examples: Capture Artifacts from Existing Code For an introduction to `Probe`s and `Meter`s, please refer to [Measurement Overview](./metrics.md). ## Context @@ -7,9 +7,9 @@ To monitor particular aspects of an `armory run` session, the user needs to know - When should I measure it? - Where should my custom monitoring script go? -We assume the user is modifying code for models or attacks and wishes to use `Probe`s and `Meter`s to monitor certain variables within the code. +We assume the user is capturing artifacts from the model or attack and wishes to use `Probe`s and `Meter`s to monitor certain variables within the code. -Recall the steps for a minimal working example (in [Measurement Overview](./metrics.md)): +Recall the steps for a minimal working example (in [Measurement Overview](./metrics.md#instrumentation)): 1. Create `Probe` via `get_probe(name)` 2. Place `Probe` actions 3. Create `Meter` with processing functions that take input from created `Probe` @@ -17,7 +17,7 @@ Recall the steps for a minimal working example (in [Measurement Overview](./metr The examples will show how each of these steps are accomplished. -## Example 1: Model Layer Output +## Example 1: Measuring a Model Layer's Output ### User Story I am interested in layer output from the second `relu` activation of a `forward` method located in `armory/baseline_models/pytorch/cifar.py`. ### `Probe` Example Code @@ -84,15 +84,15 @@ Now that a `Probe` instance has been created, we need to create a `Meter` object ```python from armory.instrument import get_hub, Meter -def user_init_function(): +def set_up_meter(): meter = Meter( - "my_layer_identity", lambda x: x, "my_model.layer_output" + "my_arbitrary_meter_name", lambda x: x, "my_model.layer_output" ) get_hub().connect_meter(meter) ``` #### Step 3 In this particular example, the `Meter` accepts 3 inputs: a meter name, a metric/function for processing, and a argument name to pass the metric/function. -- The meter name (`"my_layer_identity"`) can be arbitrary within this context +- The meter name (`"my_arbitrary_meter_name"`) can be arbitrary within this context - For the scope of this document, we only consider simple `Meter`s with the identity function as a metric i.e. `Meter` will record variables monitored by `Probe` as-is (thus `lambda x: x`) - The argument passed to the metric/function follows a `.`-separated format (`"my_model.layer_output"`), which needs to be consistent with `Probe` setup earlier: - `my_model` matches input in `probe = get_probe("my_model")` @@ -110,13 +110,13 @@ Last but not least, the config file passed to `armory run` needs to updated for }, "user_init": { "module": "user_init", - "name": "user_init_function" + "name": "set_up_meter" }, ... ``` -This will prompt armory to run `user_init_function` in `user_init.py` before anything else is loaded for the scenario. +This will prompt armory to run `set_up_meter` in `user_init.py` before anything else is loaded for the scenario. -## Example 2: Attack Output +## Example 2: Measuring Attack Artifact ### User Story I defined a custom attack with `CARLADapricotPatch` in `armory/custom_attack.py`, and I am interested in the patch after ***every iteration***, which is generated by `CARLADapricotPatch._augment_images_with_patch` and returned as an output. ### `Probe` Example Code @@ -144,15 +144,15 @@ Now that a `Probe` instance has been created, we need to create a `Meter` object ```python from armory.instrument import get_hub, Meter -def user_init_function(): +def set_up_meter(): meter = Meter( - "my_attack_identity", lambda x: x, "my_attack.attack_output" + "my_arbitrary_meter_name", lambda x: x, "my_attack.attack_output" ) get_hub().connect_meter(meter) ``` #### Step 3 As before, the `Meter` accepts 3 inputs: a meter name, a metric/function for processing, and a argument name to pass the metric/function. -- The meter name (`"my_attack_identity"`) can be arbitrary within this context +- The meter name (`"my_arbitrary_meter_name"`) can be arbitrary within this context - Again, `Meter` will record variables monitored by `Probe` as-is (thus `lambda x: x`) - The argument passed to the metric/function follows a `.`-separated format (`"my_attack.attack_output"`), which needs to be consistent with `Probe` setup earlier: - `my_attack` matches input in `probe = get_probe("my_attack")` @@ -170,8 +170,8 @@ Last but not least, the config file passed to `armory run` needs to updated for }, "user_init": { "module": "user_init", - "name": "user_init_function" + "name": "set_up_meter" }, ... ``` -This will prompt armory to run `user_init_function` in `user_init.py` before anything else is loaded for the scenario. \ No newline at end of file +This will prompt armory to run `set_up_meter` in `user_init.py` before anything else is loaded for the scenario. \ No newline at end of file From f402f30dcdb2bd703292e701b768a892ec24aa5d Mon Sep 17 00:00:00 2001 From: Paul Park Date: Sat, 17 Dec 2022 01:55:22 +0000 Subject: [PATCH 15/30] added new section for saving outputs to pickle files --- docs/instrumentation_example.md | 58 ++++++++++++++++++++++++++++++++- 1 file changed, 57 insertions(+), 1 deletion(-) diff --git a/docs/instrumentation_example.md b/docs/instrumentation_example.md index fbe356faf..c3982b729 100644 --- a/docs/instrumentation_example.md +++ b/docs/instrumentation_example.md @@ -174,4 +174,60 @@ Last but not least, the config file passed to `armory run` needs to updated for }, ... ``` -This will prompt armory to run `set_up_meter` in `user_init.py` before anything else is loaded for the scenario. \ No newline at end of file +This will prompt armory to run `set_up_meter` in `user_init.py` before anything else is loaded for the scenario. + +## Saving Results +By default, outputs from `Meter`s will be saved to the output `json` file after `armory run`. Whether this suffices for the user depends on what the user is trying to measure. + +Users who have tried the examples in this document, however, may run into some of the following warning logs: +> 2022-12-16 19:34:36 30s WARNING armory.instrument.instrument:_write:856 record (name=my_arbitrary_meter_name, batch=0, result=...) size > max_record_size 1048576. Dropping. + +This is because of `Meter`'s default settings, which has a size limit for each record. That the outputs exceed a size limit also suggests that a `json` file may not be the best file type to save to. To override these behaviors, we need a new `Meter` subclass to work with our examples that does not have a size limit and will save to another filetype such as a `pkl` file. Below is an updated `user_init.py` for Example 2 with a new `PickleMeter` class: +```python +from armory.instrument import get_hub, Meter +import pickle + +class PickleMeter(Meter): + def __init__( + self, + name, + output_dir, + x, + max_batches=None, + ): + """ + :param name (string): name given to PickleMeter + :param output_dir (string): + :param x (string): .-separated string of probe name and variable name e.g. "scenario.x" + :param max_batches (int or None): maximum number of batches to export + """ + metric_args = [x] + super().__init__(name, lambda x: x, *metric_args) + + self.max_batches = max_batches + self.metric_args = metric_args + self.output_dir = output_dir + self.iter = 0 + + if not os.path.exists(self.output_dir): + os.mkdir(self.output_dir) + + def measure(self, clear_values=True): + self.is_ready(raise_error=True) + batch_num, batch_data = self.arg_batch_indices[0], self.values[0] + if self.max_batches is not None and batch_num >= self.max_batches: + return + + with open(os.path.join(self.output_dir, f"{self.name}_batch_{batch_num}_iter_{self.iter}.pkl"), "wb") as f: + pickle.dump(batch_data, f) + self.iter += 1 + if clear_values: + self.clear() + self.never_measured = False + +def set_up_meter(): + meter = PickleMeter( + "my_arbitrary_meter_name", get_hub().export_dir, "my_attack.attack_output" + ) + get_hub().connect_meter(meter, use_default_writers=False) +``` \ No newline at end of file From dd462e2b87ce1977d8c82fbab4e9254b8c87639b Mon Sep 17 00:00:00 2001 From: Paul Park Date: Mon, 19 Dec 2022 17:16:29 +0000 Subject: [PATCH 16/30] added reference links to user_init block in scenarios.md --- docs/instrumentation_example.md | 8 ++++---- docs/metrics.md | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/instrumentation_example.md b/docs/instrumentation_example.md index c3982b729..f975d7bb4 100644 --- a/docs/instrumentation_example.md +++ b/docs/instrumentation_example.md @@ -80,7 +80,7 @@ After importing `get_probe`, `probe = get_probe("my_model")` creates a `Probe` o `x_out = x.detach().cpu().numpy()` is taking the layer output of interest (second `relu` activation output) and converting the tensor to `numpy` array on the CPU, which will be passed to `probe`. An updated value of `x_out` is stored in `layer_output` via `probe.update(layer_output=x_out)`. Like the `Probe` name `"my_model"`, `layer_output` can be referenced by the user later to apply additional processing functions through a `Meter` object. ### `Meter` Example Code -Now that a `Probe` instance has been created, we need to create a `Meter` object to accept any updated values from `Probe` and apply further processing that the user desires. Suppose the user created a script located at `armory/user_init.py`: +Now that a `Probe` instance has been created, we need to create a `Meter` object to accept any updated values from `Probe` and apply further processing that the user desires. Suppose the user created a script located at `armory/user_init.py` (Please refer to [User Initialization](./scenarios.md#user-initialization) for more details about using the `user_init` block): ```python from armory.instrument import get_hub, Meter @@ -102,7 +102,7 @@ In this particular example, the `Meter` accepts 3 inputs: a meter name, a metric For the scope of this document, we don't dwell on what `armory` is doing with `get_hub().connect_meter(meter)` other than to mention this step is necessary for establishing the connection between `meter` created in `armory/user_init.py` and `probe` created in the modified version of `armory/baseline_models/pytorch/cifar.py`. ### Config Setup -Last but not least, the config file passed to `armory run` needs to updated for these changes to take effect. Assuming the `"model"` block has been changed appropriately, the other block that needs to be added is `"user_init"`: +Last but not least, the config file passed to `armory run` needs to be updated for these changes to take effect. Assuming the `"model"` block has been changed appropriately, the other block that needs to be added is `"user_init"` (please refer to [User Initialization](./scenarios.md#user-initialization) for more details about using the `user_init` block): ```json ... "model": { @@ -140,7 +140,7 @@ This step is the same as before, except `Probe` name is set to`"my_attack"`, whi The only difference between `CustomAttack` and `CARLADapricotPatch` is that `_augment_images_with_patch` has been redefined to call on `CARLADapricotPatch._augment_images_with_patch` and then have `probe` update the value for `x_patch` that results from that call. An updated value of `x_patch` is stored in `attack_output` via `probe.update(attack_output=x_patch)`. Like the `Probe` name `"my_attack"`, `attack_output` can be referenced by the user later to apply additional processing functions through a `Meter` object. ### `Meter` Example Code -Now that a `Probe` instance has been created, we need to create a `Meter` object to accept any updated values from `Probe` and apply further processing that the user desires. Suppose the user created a script located at `armory/user_init.py`: +Now that a `Probe` instance has been created, we need to create a `Meter` object to accept any updated values from `Probe` and apply further processing that the user desires. Suppose the user created a script located at `armory/user_init.py` (Please refer to [User Initialization](./scenarios.md#user-initialization) for more details about using the `user_init` block): ```python from armory.instrument import get_hub, Meter @@ -162,7 +162,7 @@ As before, the `Meter` accepts 3 inputs: a meter name, a metric/function for pro Again, `get_hub().connect_meter(meter)` is necessary for establishing the connection between `meter` created in `armory/user_init.py` and `probe` created in `armory/custom_attack.py`. ### Config Setup -Last but not least, the config file passed to `armory run` needs to updated for these changes to take effect. Assuming the `"attack"` block has been changed appropriately, the other block that needs to be added is `"user_init"`: +Last but not least, the config file passed to `armory run` needs to be updated for these changes to take effect. Assuming the `"attack"` block has been changed appropriately, the other block that needs to be added is `"user_init"` (please refer to [User Initialization](./scenarios.md#user-initialization) for more details about using the `user_init` block): ```json ... "attack": { diff --git a/docs/metrics.md b/docs/metrics.md index 1ef3af611..69acc5411 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -350,7 +350,7 @@ assert results == [7, 11] Since these all use a global Hub object, it doesn't matter which python files they are instantatied in. Probe should be instantiated in the file or class you are trying to measure. -Meters and writers can be instantiated in your initial setup, and can be connected before probes are constructed. +Meters and writers can be instantiated in your initial setup (please refer to [User Initialization](./scenarios.md#user-initialization) for more details about using the `user_init` block), and can be connected before probes are constructed. #### Direct Recording From 1371a49b177edf3d24eb8c4e288a8aee137b7e9c Mon Sep 17 00:00:00 2001 From: Paul Park Date: Mon, 19 Dec 2022 18:59:41 +0000 Subject: [PATCH 17/30] added line number references for code blocks --- docs/instrumentation_example.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/instrumentation_example.md b/docs/instrumentation_example.md index f975d7bb4..480109a5c 100644 --- a/docs/instrumentation_example.md +++ b/docs/instrumentation_example.md @@ -74,10 +74,10 @@ class Net(nn.Module): #### Step 1 -After importing `get_probe`, `probe = get_probe("my_model")` creates a `Probe` object with the name `"my_model"`, which is what the user can refer to later to apply processing functions through a `Meter` object. +After importing `get_probe` in line 11, `probe = get_probe("my_model")` in line 12 creates a `Probe` object with the name `"my_model"`, which is what the user can refer to later to apply processing functions through a `Meter` object. #### Step 2 -`x_out = x.detach().cpu().numpy()` is taking the layer output of interest (second `relu` activation output) and converting the tensor to `numpy` array on the CPU, which will be passed to `probe`. An updated value of `x_out` is stored in `layer_output` via `probe.update(layer_output=x_out)`. Like the `Probe` name `"my_model"`, `layer_output` can be referenced by the user later to apply additional processing functions through a `Meter` object. +`x_out = x.detach().cpu().numpy()` in line 36 is taking the layer output of interest (second `relu` activation output) and converting the tensor to `numpy` array on the CPU, which will be passed to `probe`. An updated value of `x_out` is stored in `layer_output` via `probe.update(layer_output=x_out)` in line 37. Like the `Probe` name `"my_model"`, `layer_output` can be referenced by the user later to apply additional processing functions through a `Meter` object. ### `Meter` Example Code Now that a `Probe` instance has been created, we need to create a `Meter` object to accept any updated values from `Probe` and apply further processing that the user desires. Suppose the user created a script located at `armory/user_init.py` (Please refer to [User Initialization](./scenarios.md#user-initialization) for more details about using the `user_init` block): @@ -91,7 +91,7 @@ def set_up_meter(): get_hub().connect_meter(meter) ``` #### Step 3 -In this particular example, the `Meter` accepts 3 inputs: a meter name, a metric/function for processing, and a argument name to pass the metric/function. +In this particular example, the `Meter` accepts 3 inputs (line 5): a meter name, a metric/function for processing, and a argument name to pass the metric/function. - The meter name (`"my_arbitrary_meter_name"`) can be arbitrary within this context - For the scope of this document, we only consider simple `Meter`s with the identity function as a metric i.e. `Meter` will record variables monitored by `Probe` as-is (thus `lambda x: x`) - The argument passed to the metric/function follows a `.`-separated format (`"my_model.layer_output"`), which needs to be consistent with `Probe` setup earlier: @@ -99,7 +99,7 @@ In this particular example, the `Meter` accepts 3 inputs: a meter name, a metric - `layer_output` matches variable name in `probe.update(layer_output=x_out)` #### Step 4 -For the scope of this document, we don't dwell on what `armory` is doing with `get_hub().connect_meter(meter)` other than to mention this step is necessary for establishing the connection between `meter` created in `armory/user_init.py` and `probe` created in the modified version of `armory/baseline_models/pytorch/cifar.py`. +For the scope of this document, we don't dwell on what `armory` is doing with `get_hub().connect_meter(meter)` in line 7 other than to mention this step is necessary for establishing the connection between `meter` created in `armory/user_init.py` and `probe` created in the modified version of `armory/baseline_models/pytorch/cifar.py`. ### Config Setup Last but not least, the config file passed to `armory run` needs to be updated for these changes to take effect. Assuming the `"model"` block has been changed appropriately, the other block that needs to be added is `"user_init"` (please refer to [User Initialization](./scenarios.md#user-initialization) for more details about using the `user_init` block): @@ -134,10 +134,10 @@ class CustomAttack(CARLADapricotPatch): return return_value ``` #### Step 1 -This step is the same as before, except `Probe` name is set to`"my_attack"`, which is what the user can refer to later to apply processing functions through a `Meter` object. +This step is the same as before, except `Probe` name is set to`"my_attack"` in line 3, which is what the user can refer to later to apply processing functions through a `Meter` object. #### Step 2 -The only difference between `CustomAttack` and `CARLADapricotPatch` is that `_augment_images_with_patch` has been redefined to call on `CARLADapricotPatch._augment_images_with_patch` and then have `probe` update the value for `x_patch` that results from that call. An updated value of `x_patch` is stored in `attack_output` via `probe.update(attack_output=x_patch)`. Like the `Probe` name `"my_attack"`, `attack_output` can be referenced by the user later to apply additional processing functions through a `Meter` object. +The only difference between `CustomAttack` and `CARLADapricotPatch` is that `_augment_images_with_patch` in line 6 has been redefined to call on `CARLADapricotPatch._augment_images_with_patch` and then have `probe` update the value for `x_patch` that results from that call (line 7-9). An updated value of `x_patch` is stored in `attack_output` via `probe.update(attack_output=x_patch)`. Like the `Probe` name `"my_attack"`, `attack_output` can be referenced by the user later to apply additional processing functions through a `Meter` object. ### `Meter` Example Code Now that a `Probe` instance has been created, we need to create a `Meter` object to accept any updated values from `Probe` and apply further processing that the user desires. Suppose the user created a script located at `armory/user_init.py` (Please refer to [User Initialization](./scenarios.md#user-initialization) for more details about using the `user_init` block): @@ -151,7 +151,7 @@ def set_up_meter(): get_hub().connect_meter(meter) ``` #### Step 3 -As before, the `Meter` accepts 3 inputs: a meter name, a metric/function for processing, and a argument name to pass the metric/function. +As before, the `Meter` accepts 3 inputs (line 5): a meter name, a metric/function for processing, and a argument name to pass the metric/function. - The meter name (`"my_arbitrary_meter_name"`) can be arbitrary within this context - Again, `Meter` will record variables monitored by `Probe` as-is (thus `lambda x: x`) - The argument passed to the metric/function follows a `.`-separated format (`"my_attack.attack_output"`), which needs to be consistent with `Probe` setup earlier: @@ -159,7 +159,7 @@ As before, the `Meter` accepts 3 inputs: a meter name, a metric/function for pro - `attack_output` matches variable name in `probe.update(attack_output=x_patch)` #### Step 4 -Again, `get_hub().connect_meter(meter)` is necessary for establishing the connection between `meter` created in `armory/user_init.py` and `probe` created in `armory/custom_attack.py`. +Again, `get_hub().connect_meter(meter)` in line 7 is necessary for establishing the connection between `meter` created in `armory/user_init.py` and `probe` created in `armory/custom_attack.py`. ### Config Setup Last but not least, the config file passed to `armory run` needs to be updated for these changes to take effect. Assuming the `"attack"` block has been changed appropriately, the other block that needs to be added is `"user_init"` (please refer to [User Initialization](./scenarios.md#user-initialization) for more details about using the `user_init` block): From ee52a9e7306a9864632f445c312be211ae8991e6 Mon Sep 17 00:00:00 2001 From: Paul Park Date: Mon, 19 Dec 2022 19:35:44 +0000 Subject: [PATCH 18/30] updated file name and title --- .../{instrumentation_example.md => instrumentation_examples.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename docs/{instrumentation_example.md => instrumentation_examples.md} (99%) diff --git a/docs/instrumentation_example.md b/docs/instrumentation_examples.md similarity index 99% rename from docs/instrumentation_example.md rename to docs/instrumentation_examples.md index 480109a5c..3a65dafcb 100644 --- a/docs/instrumentation_example.md +++ b/docs/instrumentation_examples.md @@ -1,4 +1,4 @@ -# Armory Instrumentation Examples: Capture Artifacts from Existing Code +# Armory Instrumentation Examples: Measuring Experiment Artifacts Using Probes and Meters For an introduction to `Probe`s and `Meter`s, please refer to [Measurement Overview](./metrics.md). ## Context From 72577b757add29675fa83fcb08083ae9e22d6f28 Mon Sep 17 00:00:00 2001 From: Paul Park Date: Mon, 19 Dec 2022 22:07:18 +0000 Subject: [PATCH 19/30] updated documentation to use a new Writer class for saving to a pkl file, which is more intuitive than overwriting the measure method for a Meter class, which is the format of ExportMeter --- docs/instrumentation_examples.md | 53 ++++++++++---------------------- 1 file changed, 17 insertions(+), 36 deletions(-) diff --git a/docs/instrumentation_examples.md b/docs/instrumentation_examples.md index 3a65dafcb..09141dc7f 100644 --- a/docs/instrumentation_examples.md +++ b/docs/instrumentation_examples.md @@ -182,52 +182,33 @@ By default, outputs from `Meter`s will be saved to the output `json` file after Users who have tried the examples in this document, however, may run into some of the following warning logs: > 2022-12-16 19:34:36 30s WARNING armory.instrument.instrument:_write:856 record (name=my_arbitrary_meter_name, batch=0, result=...) size > max_record_size 1048576. Dropping. -This is because of `Meter`'s default settings, which has a size limit for each record. That the outputs exceed a size limit also suggests that a `json` file may not be the best file type to save to. To override these behaviors, we need a new `Meter` subclass to work with our examples that does not have a size limit and will save to another filetype such as a `pkl` file. Below is an updated `user_init.py` for Example 2 with a new `PickleMeter` class: +Outputs are saved to a `json` file because of a default `ResultWriter` class tied to the `Meter` class, which has a `max_record_size` limit for each record. Any record that exceeds `max_record_size` will not save to the `json` file. That the outputs exceed a size limit also suggests that a `json` file may not be the best file type to save to. To work around these behaviors, we can define a new `Writer` subclass (`ResultWriter` is also a `Writer` subclass) to work with our examples that does not have a size limit and will save to another filetype such as a `pkl` file. Below is an updated `user_init.py` for Example 2 with a new `PickleWriter` class and `set_up_meter_writer` function that will be executed with the `user_init` block: ```python -from armory.instrument import get_hub, Meter +from armory.instrument import get_hub, Meter, Writer import pickle -class PickleMeter(Meter): - def __init__( - self, - name, - output_dir, - x, - max_batches=None, - ): - """ - :param name (string): name given to PickleMeter - :param output_dir (string): - :param x (string): .-separated string of probe name and variable name e.g. "scenario.x" - :param max_batches (int or None): maximum number of batches to export - """ - metric_args = [x] - super().__init__(name, lambda x: x, *metric_args) - - self.max_batches = max_batches - self.metric_args = metric_args +class PickleWriter(Writer): + def __init__(self, output_dir): + super().__init__() self.output_dir = output_dir self.iter = 0 - + self.batch = 0 if not os.path.exists(self.output_dir): os.mkdir(self.output_dir) - def measure(self, clear_values=True): - self.is_ready(raise_error=True) - batch_num, batch_data = self.arg_batch_indices[0], self.values[0] - if self.max_batches is not None and batch_num >= self.max_batches: - return - - with open(os.path.join(self.output_dir, f"{self.name}_batch_{batch_num}_iter_{self.iter}.pkl"), "wb") as f: - pickle.dump(batch_data, f) + def _write(self, name, batch, result): + if batch != self.batch: + self.batch = batch + self.iter = 0 + with open(os.path.join(self.output_dir, f"{name}_batch_{batch}_iter_{self.iter}.pkl"), "wb") as f: + pickle.dump(result, f) self.iter += 1 - if clear_values: - self.clear() - self.never_measured = False -def set_up_meter(): - meter = PickleMeter( - "my_arbitrary_meter_name", get_hub().export_dir, "my_attack.attack_output" +def set_up_meter_writer(): + meter = Meter( + "my_attack_identity", lambda x: x, "my_attack.attack_output" ) + writer = PickleWriter(output_dir = get_hub().export_dir) + meter.add_writer(writer) get_hub().connect_meter(meter, use_default_writers=False) ``` \ No newline at end of file From dd8feb7f24d07f13c36e83b128617b58d65baab9 Mon Sep 17 00:00:00 2001 From: Paul Park Date: Tue, 20 Dec 2022 19:05:30 +0000 Subject: [PATCH 20/30] updated documentation and tested example using a defined ImageWriter class to save Probe values as images --- docs/instrumentation_examples.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/instrumentation_examples.md b/docs/instrumentation_examples.md index 09141dc7f..8ae801066 100644 --- a/docs/instrumentation_examples.md +++ b/docs/instrumentation_examples.md @@ -182,33 +182,33 @@ By default, outputs from `Meter`s will be saved to the output `json` file after Users who have tried the examples in this document, however, may run into some of the following warning logs: > 2022-12-16 19:34:36 30s WARNING armory.instrument.instrument:_write:856 record (name=my_arbitrary_meter_name, batch=0, result=...) size > max_record_size 1048576. Dropping. -Outputs are saved to a `json` file because of a default `ResultWriter` class tied to the `Meter` class, which has a `max_record_size` limit for each record. Any record that exceeds `max_record_size` will not save to the `json` file. That the outputs exceed a size limit also suggests that a `json` file may not be the best file type to save to. To work around these behaviors, we can define a new `Writer` subclass (`ResultWriter` is also a `Writer` subclass) to work with our examples that does not have a size limit and will save to another filetype such as a `pkl` file. Below is an updated `user_init.py` for Example 2 with a new `PickleWriter` class and `set_up_meter_writer` function that will be executed with the `user_init` block: +Outputs are saved to a `json` file because of a default `ResultWriter` class tied to the `Meter` class, which has a `max_record_size` limit for each record. Any record that exceeds `max_record_size` will not save to the `json` file. That the outputs exceed a size limit also suggests that a `json` file may not be the best file type to save to. To work around these behaviors, we can define a new `Writer` subclass (`ResultWriter` is also a `Writer` subclass) to work with our examples that does not have a size limit and will save to another filetype, such as a `png` file, since we are saving data for an image. Below is an updated `user_init.py` for Example 2 with a new `ImageWriter` class, which uses the `export` method of `ImageClassificationExporter` to save an image, and a `set_up_meter_writer` function that will be executed with the `user_init` block: ```python from armory.instrument import get_hub, Meter, Writer -import pickle +from armory.instrument.export import ImageClassificationExporter -class PickleWriter(Writer): +class ImageWriter(Writer): def __init__(self, output_dir): super().__init__() self.output_dir = output_dir - self.iter = 0 + self.iter_step = 0 self.batch = 0 - if not os.path.exists(self.output_dir): - os.mkdir(self.output_dir) + self.exporter = ImageClassificationExporter(self.output_dir) def _write(self, name, batch, result): if batch != self.batch: self.batch = batch - self.iter = 0 - with open(os.path.join(self.output_dir, f"{name}_batch_{batch}_iter_{self.iter}.pkl"), "wb") as f: - pickle.dump(result, f) - self.iter += 1 + self.iter_step = 0 + basename = f"{name}_batch_{batch}_iter_{self.iter_step}" + # assume single image per batch: result[0] + self.exporter.export(x_i = result[0], basename = basename) + self.iter_step += 1 def set_up_meter_writer(): meter = Meter( "my_attack_identity", lambda x: x, "my_attack.attack_output" ) - writer = PickleWriter(output_dir = get_hub().export_dir) + writer = ImageWriter(output_dir = get_hub().export_dir) meter.add_writer(writer) get_hub().connect_meter(meter, use_default_writers=False) ``` \ No newline at end of file From c0f278010d6e9e25837df82391f20c98164c74dc Mon Sep 17 00:00:00 2001 From: Paul Park Date: Thu, 22 Dec 2022 17:36:31 +0000 Subject: [PATCH 21/30] removing unit tests involving Probe hook method --- tests/unit/test_instrument.py | 43 ----------------------------------- 1 file changed, 43 deletions(-) diff --git a/tests/unit/test_instrument.py b/tests/unit/test_instrument.py index d8dfe1952..6ce2faeda 100644 --- a/tests/unit/test_instrument.py +++ b/tests/unit/test_instrument.py @@ -112,10 +112,6 @@ def not_implemented(*args, **kwargs): probe.update(not_implemented, x=1) sink._is_measuring = True - jax_model = None - with pytest.raises(ValueError): - probe.hook(jax_model, mode="jax") - def get_pytorch_model(): # Taken from https://pytorch.org/docs/stable/generated/torch.nn.Module.html @@ -135,45 +131,6 @@ def forward(self, x): return Model() -@pytest.mark.docker_required -def test_probe_pytorch_hook(): - import torch - - instrument.del_globals() - sink = HelperSink() - probe = instrument.Probe("model", sink=sink) - model = get_pytorch_model() - probe.hook(model.conv1, lambda x: x.detach().cpu().numpy(), output="b") - - key = "model.b" - assert key not in sink.probe_variables - - x1 = torch.rand((1, 1, 28, 28)) - model(x1) - b1 = sink.probe_variables["model.b"] - assert b1.shape == (1, 20, 24, 24) - x2 = torch.rand((1, 1, 28, 28)) - model(x2) - b2 = sink.probe_variables["model.b"] - assert b2.shape == (1, 20, 24, 24) - assert not (b1 == b2).all() - probe.unhook(model.conv1) - # probe is unhooked, no update should occur - model(x1) - b3 = sink.probe_variables["model.b"] - assert b3 is b2 - - -@pytest.mark.docker_required -def test_probe_tensorflow_hook(): - # Once implemented, update test - instrument.del_globals() - probe = instrument.get_probe() - with pytest.raises(NotImplementedError): - tf_model = None - probe.hook(tf_model, mode="tf") - - def test_process_meter_arg(): for arg in ("x", "scenario.x"): assert instrument.process_meter_arg(arg) == (arg, None) From c24837a7c23c618b44c1e94de78c8b6ba7f8c87b Mon Sep 17 00:00:00 2001 From: Paul Park Date: Thu, 22 Dec 2022 17:51:16 +0000 Subject: [PATCH 22/30] resolved some PR comments --- docs/instrumentation_examples.md | 14 +++----------- docs/metrics.md | 1 - 2 files changed, 3 insertions(+), 12 deletions(-) diff --git a/docs/instrumentation_examples.md b/docs/instrumentation_examples.md index 8ae801066..f3fbeb053 100644 --- a/docs/instrumentation_examples.md +++ b/docs/instrumentation_examples.md @@ -1,13 +1,5 @@ # Armory Instrumentation Examples: Measuring Experiment Artifacts Using Probes and Meters -For an introduction to `Probe`s and `Meter`s, please refer to [Measurement Overview](./metrics.md). - -## Context -To monitor particular aspects of an `armory run` session, the user needs to know the following factors: -- What am I measuring? -- When should I measure it? -- Where should my custom monitoring script go? - -We assume the user is capturing artifacts from the model or attack and wishes to use `Probe`s and `Meter`s to monitor certain variables within the code. +For an introduction to `Probe`s and `Meter`s, please refer to [Measurement Overview](./metrics.md#instrumentation). We assume the user is capturing artifacts from the model or attack and wishes to use `Probe`s and `Meter`s to monitor certain variables within the code. Recall the steps for a minimal working example (in [Measurement Overview](./metrics.md#instrumentation)): 1. Create `Probe` via `get_probe(name)` @@ -19,7 +11,7 @@ The examples will show how each of these steps are accomplished. ## Example 1: Measuring a Model Layer's Output ### User Story -I am interested in layer output from the second `relu` activation of a `forward` method located in `armory/baseline_models/pytorch/cifar.py`. +I am interested in the layer output from the second `relu` activation of a `forward` method located in `armory/baseline_models/pytorch/cifar.py`. ### `Probe` Example Code The code below is an example of how to accomplish steps 1 and 2 (note the lines of code with `# added` comments at the end) for a model code that the user is modifying. ```python @@ -80,7 +72,7 @@ After importing `get_probe` in line 11, `probe = get_probe("my_model")` in line `x_out = x.detach().cpu().numpy()` in line 36 is taking the layer output of interest (second `relu` activation output) and converting the tensor to `numpy` array on the CPU, which will be passed to `probe`. An updated value of `x_out` is stored in `layer_output` via `probe.update(layer_output=x_out)` in line 37. Like the `Probe` name `"my_model"`, `layer_output` can be referenced by the user later to apply additional processing functions through a `Meter` object. ### `Meter` Example Code -Now that a `Probe` instance has been created, we need to create a `Meter` object to accept any updated values from `Probe` and apply further processing that the user desires. Suppose the user created a script located at `armory/user_init.py` (Please refer to [User Initialization](./scenarios.md#user-initialization) for more details about using the `user_init` block): +Now that a `Probe` instance has been created, we need to create a `Meter` object to accept any updated values from `Probe` and apply further processing that the user desires. We can create the `Meter` in a function added to a local Python script we'll name `user_init.py`. In [Config Setup](#config-setup) shortly below, we'll show how to ensure this code is run during scenario initialization. ```python from armory.instrument import get_hub, Meter diff --git a/docs/metrics.md b/docs/metrics.md index 69acc5411..a4d0577c3 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -24,7 +24,6 @@ Desired metrics and flags are placed under the key `"metric"` dictionary in the } ``` The `perturbation` and `task` fields can be null, a single string, or a list of strings. -Strings must be a valid armory metric from `armory.utils.metrics`, which are also described in the Metrics section below. The perturbation metrics measure the difference between the benign and adversarial inputs `x`. The task metrics measure the task performance on the predicted value w.r.t the true value `y`, for both benign and adversarial inputs. If task metrics take keyword arguments, such as `"iou_threshold"`, these can be (optionally) added a list of kwarg dicts. From 9d35fa8caea7333efe069667070623e48140e6ed Mon Sep 17 00:00:00 2001 From: Paul Park Date: Thu, 22 Dec 2022 17:57:59 +0000 Subject: [PATCH 23/30] shorter code snippet to resolve PR comment --- docs/instrumentation_examples.md | 23 ++--------------------- 1 file changed, 2 insertions(+), 21 deletions(-) diff --git a/docs/instrumentation_examples.md b/docs/instrumentation_examples.md index f3fbeb053..dc4225323 100644 --- a/docs/instrumentation_examples.md +++ b/docs/instrumentation_examples.md @@ -18,30 +18,13 @@ The code below is an example of how to accomplish steps 1 and 2 (note the lines """ CNN model for 32x32x3 image classification """ -from typing import Optional - -import torch -import torch.nn as nn -import torch.nn.functional as F -from art.estimators.classification import PyTorchClassifier +... from armory.instrument import get_probe # added probe = get_probe("my_model") # added -DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu") - - class Net(nn.Module): - """ - This is a simple CNN for CIFAR-10 and does not achieve SotA performance - """ - - def __init__(self) -> None: - super(Net, self).__init__() - self.conv1 = nn.Conv2d(3, 4, 5, 1) - self.conv2 = nn.Conv2d(4, 10, 5, 1) - self.fc1 = nn.Linear(250, 100) - self.fc2 = nn.Linear(100, 10) + ... def forward(self, x: torch.Tensor) -> torch.Tensor: x = x.permute(0, 3, 1, 2) # from NHWC to NCHW @@ -63,8 +46,6 @@ class Net(nn.Module): ... ``` - - #### Step 1 After importing `get_probe` in line 11, `probe = get_probe("my_model")` in line 12 creates a `Probe` object with the name `"my_model"`, which is what the user can refer to later to apply processing functions through a `Meter` object. From 445a2ab4dd0a35e07c6f6a2c906ac19902b78594 Mon Sep 17 00:00:00 2001 From: Paul Park Date: Thu, 22 Dec 2022 18:04:49 +0000 Subject: [PATCH 24/30] removed line references since Github Markdown does not support adding line numbers to code blocks --- docs/instrumentation_examples.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/instrumentation_examples.md b/docs/instrumentation_examples.md index dc4225323..ddb40d635 100644 --- a/docs/instrumentation_examples.md +++ b/docs/instrumentation_examples.md @@ -47,10 +47,10 @@ class Net(nn.Module): ``` #### Step 1 -After importing `get_probe` in line 11, `probe = get_probe("my_model")` in line 12 creates a `Probe` object with the name `"my_model"`, which is what the user can refer to later to apply processing functions through a `Meter` object. +After importing `get_probe`, `probe = get_probe("my_model")` creates a `Probe` object with the name `"my_model"`, which is what the user can refer to later to apply processing functions through a `Meter` object. #### Step 2 -`x_out = x.detach().cpu().numpy()` in line 36 is taking the layer output of interest (second `relu` activation output) and converting the tensor to `numpy` array on the CPU, which will be passed to `probe`. An updated value of `x_out` is stored in `layer_output` via `probe.update(layer_output=x_out)` in line 37. Like the `Probe` name `"my_model"`, `layer_output` can be referenced by the user later to apply additional processing functions through a `Meter` object. +`x_out = x.detach().cpu().numpy()` is taking the layer output of interest (second `relu` activation output) and converting the tensor to `numpy` array on the CPU, which will be passed to `probe`. An updated value of `x_out` is stored in `layer_output` via `probe.update(layer_output=x_out)`. Like the `Probe` name `"my_model"`, `layer_output` can be referenced by the user later to apply additional processing functions through a `Meter` object. ### `Meter` Example Code Now that a `Probe` instance has been created, we need to create a `Meter` object to accept any updated values from `Probe` and apply further processing that the user desires. We can create the `Meter` in a function added to a local Python script we'll name `user_init.py`. In [Config Setup](#config-setup) shortly below, we'll show how to ensure this code is run during scenario initialization. @@ -64,7 +64,7 @@ def set_up_meter(): get_hub().connect_meter(meter) ``` #### Step 3 -In this particular example, the `Meter` accepts 3 inputs (line 5): a meter name, a metric/function for processing, and a argument name to pass the metric/function. +In this particular example, the `Meter` accepts 3 inputs: a meter name, a metric/function for processing, and a argument name to pass the metric/function. - The meter name (`"my_arbitrary_meter_name"`) can be arbitrary within this context - For the scope of this document, we only consider simple `Meter`s with the identity function as a metric i.e. `Meter` will record variables monitored by `Probe` as-is (thus `lambda x: x`) - The argument passed to the metric/function follows a `.`-separated format (`"my_model.layer_output"`), which needs to be consistent with `Probe` setup earlier: @@ -72,7 +72,7 @@ In this particular example, the `Meter` accepts 3 inputs (line 5): a meter name, - `layer_output` matches variable name in `probe.update(layer_output=x_out)` #### Step 4 -For the scope of this document, we don't dwell on what `armory` is doing with `get_hub().connect_meter(meter)` in line 7 other than to mention this step is necessary for establishing the connection between `meter` created in `armory/user_init.py` and `probe` created in the modified version of `armory/baseline_models/pytorch/cifar.py`. +For the scope of this document, we don't dwell on what `armory` is doing with `get_hub().connect_meter(meter)` other than to mention this step is necessary for establishing the connection between `meter` created in `armory/user_init.py` and `probe` created in the modified version of `armory/baseline_models/pytorch/cifar.py`. ### Config Setup Last but not least, the config file passed to `armory run` needs to be updated for these changes to take effect. Assuming the `"model"` block has been changed appropriately, the other block that needs to be added is `"user_init"` (please refer to [User Initialization](./scenarios.md#user-initialization) for more details about using the `user_init` block): @@ -107,10 +107,10 @@ class CustomAttack(CARLADapricotPatch): return return_value ``` #### Step 1 -This step is the same as before, except `Probe` name is set to`"my_attack"` in line 3, which is what the user can refer to later to apply processing functions through a `Meter` object. +This step is the same as before, except `Probe` name is set to`"my_attack"`, which is what the user can refer to later to apply processing functions through a `Meter` object. #### Step 2 -The only difference between `CustomAttack` and `CARLADapricotPatch` is that `_augment_images_with_patch` in line 6 has been redefined to call on `CARLADapricotPatch._augment_images_with_patch` and then have `probe` update the value for `x_patch` that results from that call (line 7-9). An updated value of `x_patch` is stored in `attack_output` via `probe.update(attack_output=x_patch)`. Like the `Probe` name `"my_attack"`, `attack_output` can be referenced by the user later to apply additional processing functions through a `Meter` object. +The only difference between `CustomAttack` and `CARLADapricotPatch` is that `_augment_images_with_patch` has been redefined to call on `CARLADapricotPatch._augment_images_with_patch` and then have `probe` update the value for `x_patch` that results from that call. An updated value of `x_patch` is stored in `attack_output` via `probe.update(attack_output=x_patch)`. Like the `Probe` name `"my_attack"`, `attack_output` can be referenced by the user later to apply additional processing functions through a `Meter` object. ### `Meter` Example Code Now that a `Probe` instance has been created, we need to create a `Meter` object to accept any updated values from `Probe` and apply further processing that the user desires. Suppose the user created a script located at `armory/user_init.py` (Please refer to [User Initialization](./scenarios.md#user-initialization) for more details about using the `user_init` block): @@ -124,7 +124,7 @@ def set_up_meter(): get_hub().connect_meter(meter) ``` #### Step 3 -As before, the `Meter` accepts 3 inputs (line 5): a meter name, a metric/function for processing, and a argument name to pass the metric/function. +As before, the `Meter` accepts 3 inputs: a meter name, a metric/function for processing, and a argument name to pass the metric/function. - The meter name (`"my_arbitrary_meter_name"`) can be arbitrary within this context - Again, `Meter` will record variables monitored by `Probe` as-is (thus `lambda x: x`) - The argument passed to the metric/function follows a `.`-separated format (`"my_attack.attack_output"`), which needs to be consistent with `Probe` setup earlier: @@ -132,7 +132,7 @@ As before, the `Meter` accepts 3 inputs (line 5): a meter name, a metric/functio - `attack_output` matches variable name in `probe.update(attack_output=x_patch)` #### Step 4 -Again, `get_hub().connect_meter(meter)` in line 7 is necessary for establishing the connection between `meter` created in `armory/user_init.py` and `probe` created in `armory/custom_attack.py`. +Again, `get_hub().connect_meter(meter)` is necessary for establishing the connection between `meter` created in `armory/user_init.py` and `probe` created in `armory/custom_attack.py`. ### Config Setup Last but not least, the config file passed to `armory run` needs to be updated for these changes to take effect. Assuming the `"attack"` block has been changed appropriately, the other block that needs to be added is `"user_init"` (please refer to [User Initialization](./scenarios.md#user-initialization) for more details about using the `user_init` block): From 8f57948a574cbd1f3038e338cae71f4e90a93009 Mon Sep 17 00:00:00 2001 From: Paul Park Date: Thu, 22 Dec 2022 18:09:59 +0000 Subject: [PATCH 25/30] updated documentation for consistency between examples --- docs/instrumentation_examples.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/instrumentation_examples.md b/docs/instrumentation_examples.md index ddb40d635..37bee5a39 100644 --- a/docs/instrumentation_examples.md +++ b/docs/instrumentation_examples.md @@ -113,7 +113,7 @@ This step is the same as before, except `Probe` name is set to`"my_attack"`, whi The only difference between `CustomAttack` and `CARLADapricotPatch` is that `_augment_images_with_patch` has been redefined to call on `CARLADapricotPatch._augment_images_with_patch` and then have `probe` update the value for `x_patch` that results from that call. An updated value of `x_patch` is stored in `attack_output` via `probe.update(attack_output=x_patch)`. Like the `Probe` name `"my_attack"`, `attack_output` can be referenced by the user later to apply additional processing functions through a `Meter` object. ### `Meter` Example Code -Now that a `Probe` instance has been created, we need to create a `Meter` object to accept any updated values from `Probe` and apply further processing that the user desires. Suppose the user created a script located at `armory/user_init.py` (Please refer to [User Initialization](./scenarios.md#user-initialization) for more details about using the `user_init` block): +As in [Example 1](#meter-example-code), we need to create a `Meter` object to accept any updated values from `Probe` and apply further processing that the user desires. We can create the `Meter` in a function added to a local Python script `user_init.py`. In [Config Setup](#config-setup-1) shortly below, we'll show how to ensure this code is run during scenario initialization. ```python from armory.instrument import get_hub, Meter From 8cf857809537d6b77217215fdea14dc339ad2ee1 Mon Sep 17 00:00:00 2001 From: Paul Park Date: Thu, 22 Dec 2022 18:19:09 +0000 Subject: [PATCH 26/30] resolved PR comment on whether model and attack block in config file should be changed for custom models and attacks; renamed user_init.py to user_init_script.py to disambiguate from user_init block in config file --- docs/instrumentation_examples.md | 28 +++++++++++----------------- 1 file changed, 11 insertions(+), 17 deletions(-) diff --git a/docs/instrumentation_examples.md b/docs/instrumentation_examples.md index 37bee5a39..cd320553b 100644 --- a/docs/instrumentation_examples.md +++ b/docs/instrumentation_examples.md @@ -53,7 +53,7 @@ After importing `get_probe`, `probe = get_probe("my_model")` creates a `Probe` o `x_out = x.detach().cpu().numpy()` is taking the layer output of interest (second `relu` activation output) and converting the tensor to `numpy` array on the CPU, which will be passed to `probe`. An updated value of `x_out` is stored in `layer_output` via `probe.update(layer_output=x_out)`. Like the `Probe` name `"my_model"`, `layer_output` can be referenced by the user later to apply additional processing functions through a `Meter` object. ### `Meter` Example Code -Now that a `Probe` instance has been created, we need to create a `Meter` object to accept any updated values from `Probe` and apply further processing that the user desires. We can create the `Meter` in a function added to a local Python script we'll name `user_init.py`. In [Config Setup](#config-setup) shortly below, we'll show how to ensure this code is run during scenario initialization. +Now that a `Probe` instance has been created, we need to create a `Meter` object to accept any updated values from `Probe` and apply further processing that the user desires. We can create the `Meter` in a function added to a local Python script we'll name `user_init_script.py`. In [Config Setup](#config-setup) shortly below, we'll show how to ensure this code is run during scenario initialization. ```python from armory.instrument import get_hub, Meter @@ -72,22 +72,19 @@ In this particular example, the `Meter` accepts 3 inputs: a meter name, a metric - `layer_output` matches variable name in `probe.update(layer_output=x_out)` #### Step 4 -For the scope of this document, we don't dwell on what `armory` is doing with `get_hub().connect_meter(meter)` other than to mention this step is necessary for establishing the connection between `meter` created in `armory/user_init.py` and `probe` created in the modified version of `armory/baseline_models/pytorch/cifar.py`. +For the scope of this document, we don't dwell on what `armory` is doing with `get_hub().connect_meter(meter)` other than to mention this step is necessary for establishing the connection between `meter` created in `armory/user_init_script.py` and `probe` created in the modified version of `armory/baseline_models/pytorch/cifar.py`. ### Config Setup -Last but not least, the config file passed to `armory run` needs to be updated for these changes to take effect. Assuming the `"model"` block has been changed appropriately, the other block that needs to be added is `"user_init"` (please refer to [User Initialization](./scenarios.md#user-initialization) for more details about using the `user_init` block): +Last but not least, the config file passed to `armory run` needs to be updated for these changes to take effect, which is accomplished by adding the `"user_init"` block (please refer to [User Initialization](./scenarios.md#user-initialization) for more details): ```json ... - "model": { - ... - }, "user_init": { - "module": "user_init", + "module": "user_init_script", "name": "set_up_meter" }, ... ``` -This will prompt armory to run `set_up_meter` in `user_init.py` before anything else is loaded for the scenario. +This will prompt armory to run `set_up_meter` in `user_init_script.py` before anything else is loaded for the scenario. ## Example 2: Measuring Attack Artifact ### User Story @@ -113,7 +110,7 @@ This step is the same as before, except `Probe` name is set to`"my_attack"`, whi The only difference between `CustomAttack` and `CARLADapricotPatch` is that `_augment_images_with_patch` has been redefined to call on `CARLADapricotPatch._augment_images_with_patch` and then have `probe` update the value for `x_patch` that results from that call. An updated value of `x_patch` is stored in `attack_output` via `probe.update(attack_output=x_patch)`. Like the `Probe` name `"my_attack"`, `attack_output` can be referenced by the user later to apply additional processing functions through a `Meter` object. ### `Meter` Example Code -As in [Example 1](#meter-example-code), we need to create a `Meter` object to accept any updated values from `Probe` and apply further processing that the user desires. We can create the `Meter` in a function added to a local Python script `user_init.py`. In [Config Setup](#config-setup-1) shortly below, we'll show how to ensure this code is run during scenario initialization. +As in [Example 1](#meter-example-code), we need to create a `Meter` object to accept any updated values from `Probe` and apply further processing that the user desires. We can create the `Meter` in a function added to a local Python script `user_init_script.py`. In [Config Setup](#config-setup-1) shortly below, we'll show how to ensure this code is run during scenario initialization. ```python from armory.instrument import get_hub, Meter @@ -132,22 +129,19 @@ As before, the `Meter` accepts 3 inputs: a meter name, a metric/function for pro - `attack_output` matches variable name in `probe.update(attack_output=x_patch)` #### Step 4 -Again, `get_hub().connect_meter(meter)` is necessary for establishing the connection between `meter` created in `armory/user_init.py` and `probe` created in `armory/custom_attack.py`. +Again, `get_hub().connect_meter(meter)` is necessary for establishing the connection between `meter` created in `armory/user_init_script.py` and `probe` created in `armory/custom_attack.py`. ### Config Setup -Last but not least, the config file passed to `armory run` needs to be updated for these changes to take effect. Assuming the `"attack"` block has been changed appropriately, the other block that needs to be added is `"user_init"` (please refer to [User Initialization](./scenarios.md#user-initialization) for more details about using the `user_init` block): +Last but not least, the config file passed to `armory run` needs to be updated for these changes to take effect, which is accomplished by adding the `"user_init"` block (please refer to [User Initialization](./scenarios.md#user-initialization) for more details): ```json ... - "attack": { - ... - }, "user_init": { - "module": "user_init", + "module": "user_init_script", "name": "set_up_meter" }, ... ``` -This will prompt armory to run `set_up_meter` in `user_init.py` before anything else is loaded for the scenario. +This will prompt armory to run `set_up_meter` in `user_init_script.py` before anything else is loaded for the scenario. ## Saving Results By default, outputs from `Meter`s will be saved to the output `json` file after `armory run`. Whether this suffices for the user depends on what the user is trying to measure. @@ -155,7 +149,7 @@ By default, outputs from `Meter`s will be saved to the output `json` file after Users who have tried the examples in this document, however, may run into some of the following warning logs: > 2022-12-16 19:34:36 30s WARNING armory.instrument.instrument:_write:856 record (name=my_arbitrary_meter_name, batch=0, result=...) size > max_record_size 1048576. Dropping. -Outputs are saved to a `json` file because of a default `ResultWriter` class tied to the `Meter` class, which has a `max_record_size` limit for each record. Any record that exceeds `max_record_size` will not save to the `json` file. That the outputs exceed a size limit also suggests that a `json` file may not be the best file type to save to. To work around these behaviors, we can define a new `Writer` subclass (`ResultWriter` is also a `Writer` subclass) to work with our examples that does not have a size limit and will save to another filetype, such as a `png` file, since we are saving data for an image. Below is an updated `user_init.py` for Example 2 with a new `ImageWriter` class, which uses the `export` method of `ImageClassificationExporter` to save an image, and a `set_up_meter_writer` function that will be executed with the `user_init` block: +Outputs are saved to a `json` file because of a default `ResultWriter` class tied to the `Meter` class, which has a `max_record_size` limit for each record. Any record that exceeds `max_record_size` will not save to the `json` file. That the outputs exceed a size limit also suggests that a `json` file may not be the best file type to save to. To work around these behaviors, we can define a new `Writer` subclass (`ResultWriter` is also a `Writer` subclass) to work with our examples that does not have a size limit and will save to another filetype, such as a `png` file, since we are saving data for an image. Below is an updated `user_init_script.py` for Example 2 with a new `ImageWriter` class, which uses the `export` method of `ImageClassificationExporter` to save an image, and a `set_up_meter_writer` function that will be executed with the `user_init` block: ```python from armory.instrument import get_hub, Meter, Writer from armory.instrument.export import ImageClassificationExporter From f1fbb0c3142af00d1bf6b94cc0e29edc433c607c Mon Sep 17 00:00:00 2001 From: Paul Park Date: Thu, 22 Dec 2022 18:30:27 +0000 Subject: [PATCH 27/30] resolved PR comment by changing ImageClassificationExporter to ObjectDetectionExporter --- docs/instrumentation_examples.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/instrumentation_examples.md b/docs/instrumentation_examples.md index cd320553b..4aac6dad7 100644 --- a/docs/instrumentation_examples.md +++ b/docs/instrumentation_examples.md @@ -152,7 +152,7 @@ Users who have tried the examples in this document, however, may run into some o Outputs are saved to a `json` file because of a default `ResultWriter` class tied to the `Meter` class, which has a `max_record_size` limit for each record. Any record that exceeds `max_record_size` will not save to the `json` file. That the outputs exceed a size limit also suggests that a `json` file may not be the best file type to save to. To work around these behaviors, we can define a new `Writer` subclass (`ResultWriter` is also a `Writer` subclass) to work with our examples that does not have a size limit and will save to another filetype, such as a `png` file, since we are saving data for an image. Below is an updated `user_init_script.py` for Example 2 with a new `ImageWriter` class, which uses the `export` method of `ImageClassificationExporter` to save an image, and a `set_up_meter_writer` function that will be executed with the `user_init` block: ```python from armory.instrument import get_hub, Meter, Writer -from armory.instrument.export import ImageClassificationExporter +from armory.instrument.export import ObjectDetectionExporter class ImageWriter(Writer): def __init__(self, output_dir): @@ -160,7 +160,7 @@ class ImageWriter(Writer): self.output_dir = output_dir self.iter_step = 0 self.batch = 0 - self.exporter = ImageClassificationExporter(self.output_dir) + self.exporter = ObjectDetectionExporter(self.output_dir) def _write(self, name, batch, result): if batch != self.batch: @@ -168,7 +168,7 @@ class ImageWriter(Writer): self.iter_step = 0 basename = f"{name}_batch_{batch}_iter_{self.iter_step}" # assume single image per batch: result[0] - self.exporter.export(x_i = result[0], basename = basename) + self.exporter.export(x = result[0], basename = basename) self.iter_step += 1 def set_up_meter_writer(): From 30a5b921d8b7c16576fcb14df68c2a801b7410b7 Mon Sep 17 00:00:00 2001 From: Paul Park Date: Thu, 22 Dec 2022 18:49:24 +0000 Subject: [PATCH 28/30] updated variable names and added comments for clarity --- docs/instrumentation_examples.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/instrumentation_examples.md b/docs/instrumentation_examples.md index 4aac6dad7..4f3c7f17e 100644 --- a/docs/instrumentation_examples.md +++ b/docs/instrumentation_examples.md @@ -159,17 +159,17 @@ class ImageWriter(Writer): super().__init__() self.output_dir = output_dir self.iter_step = 0 - self.batch = 0 + self.current_batch_index = 0 self.exporter = ObjectDetectionExporter(self.output_dir) def _write(self, name, batch, result): - if batch != self.batch: - self.batch = batch - self.iter_step = 0 + if batch != self.current_batch_index: + self.current_batch_index = batch # we are on a new batch + self.iter_step = 0 # restart iter_step count basename = f"{name}_batch_{batch}_iter_{self.iter_step}" # assume single image per batch: result[0] self.exporter.export(x = result[0], basename = basename) - self.iter_step += 1 + self.iter_step += 1 # increment iter_step def set_up_meter_writer(): meter = Meter( From b282870ddc1a08926913efa5f22d63201f1b6c0c Mon Sep 17 00:00:00 2001 From: Paul Park Date: Thu, 22 Dec 2022 18:51:11 +0000 Subject: [PATCH 29/30] removing hooking example documentation for now --- docs/probe_hooking_notebook.md | 316 --------------------------------- 1 file changed, 316 deletions(-) delete mode 100644 docs/probe_hooking_notebook.md diff --git a/docs/probe_hooking_notebook.md b/docs/probe_hooking_notebook.md deleted file mode 100644 index d70a26065..000000000 --- a/docs/probe_hooking_notebook.md +++ /dev/null @@ -1,316 +0,0 @@ -# Probes and Meters: Advanced Hooking Examples for Notebooks -For an introduction to `Probe`s and `Meter`s, please refer to [Measurement Overview](./metrics.md). - -## Context -To monitor particular aspects of an `armory run` session, the user needs to know the following factors: -- What am I measuring? -- When should I measure it? -- Where should my custom monitoring script go? - -The examples in this section highlight the nuances of using `Probe`s and `Meter`s for flexible monitoring arrangements in `armory`. In particular, we assume the user is working from a Jupyter notebook and wishes to use `Probe`s and `Meter`s to monitor a model or attack ***without*** modifying the existing code. Documentation and development for enabling the functionality as discribed here for an end-to-end execution with `armory run` using `user-init` blocks is under future consideration. - -## Example 1: Model Layer Output -### User Story -I have a `PyTorchFasterRCNN` model and I am interested in output from the `relu` activation of the third (index 2) `Bottleneck` of `layer4` -### Example Code -This is an example of working with a python package/framework (i.e. `pytorch`) that comes with built-in hooking mechanisms. In the code snippet below, we are relying on an existing function `register_forward_hook` to monitor the layer of interest: -```python -from armory.scenarios.main import get as get_scenario -from armory.instrument import get_probe, Meter, get_hub - -# load Scenario -s = get_scenario( - "/armory/scenario_configs/eval6/carla_overhead_object_detection/carla_obj_det_adversarialpatch_undefended.json", - num_eval_batches=1, -).load() - -# create Probe with "test" namespace -probe = get_probe("test") - -# define the hook to pass to "register_forward_hook" -# the signature of 3 inputs is what pytorch expects -# hook_module refers to the layer of interest, but is not explicitly referenced when passing to register_forward_hook -def hook_fn(hook_module, hook_input, hook_output): - probe.update(lambda x: x.detach().cpu().numpy(), layer4_2_relu=hook_output[0][0]) # [0][0] for slicing - -# register hook -# the hook_module mentioned earlier is referenced via s.model.model.backbone.body.layer4[2].relu -# the register_forward_hook method call must be passing self as a hook_module to hook_fn -s.model.model.backbone.body.layer4[2].relu.register_forward_hook(hook_fn) - -# create Meter for Probe with "test" namespace -meter = Meter("layer4_2_relu", lambda x: x, "test.layer4_2_relu") - -# connect Meter to Hub -get_hub().connect_meter(meter) - -s.next() -s.run_attack() -``` - -### Packages with Hooks -That a package provides a hooking mechanism is convenient, but the user also has to be aware of the what to pass to the hooking mechanism as well as what format to pass it in. Please reference [`pytorch` documentation](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_forward_hook) for more details regarding this example. - -Note that `pytorch` also provides other hooking functionality such as: -- `register_forward_pre_hook` -- `register_full_backward_hook` - -### Probe and Meter Details -Aside the specifics of using `register_forward_hook`, consider how `Probe` and `Meter` are incorporated in this example. Recall the steps for a minimal working example (in [Measurement Overview](./metrics.md)): -1. Create `Probe` via `get_probe("test")` -2. Define `Probe` actions -3. Connect `Probe` -4. Create `Meter` with processing functions that take input from created `Probe` -5. Connect `Meter` to `Hub` via `get_hub().connect_meter(meter)` - -#### Step 1 -Note the input `"test"` that is passed in `get_probe("test")` - this needs to match with the first portion of a `.`-separated argument name `"test.layer4_2_relu"` that is passed to creating a `Meter` in [Step 3](#step4) - -#### Step 2 -The `update` method for `Probe` takes as input optional processing functions and variable names and corresponding values that are to be monitored. -- The variable name `layer4_2_relu` is how we are choosing to reference a certain value - - this needs to match with the second portion of a `.`-separated argument name `"test.layer4_2_relu"` that is passed to creating a `Meter` in [Step 3](#step3) -- `hook_output[0][0]` is the value we are interested in, which is the output from `s.model.model.backbone.body.layer4[2].relu` after a forward pass - - `[0][0]` was included to slice the output to show that it can be done, and because we know the shape of the output in advance -- `lambda x: x.detach().cpu().numpy()` is the processing function that converts `hook_output[0][0]` from a tensor to an array - -#### Step 3 -This particular step is not dealt with in-depth in [Measurement Overview](./metrics.md), but requires more explanation for this section. - -#### Step 4 -In this particular example, the `Meter` accepts 3 inputs: a meter name, a metric/function for processing, and a argument name to pass the metric/function. -- The meter name (`"layer4_2_relu"`) can be arbitrary within this context -- For the scope of this document, we only consider simple `Meter`s with the identity function as a metric i.e. `Meter` will record variables monitored by `Probe` as-is (thus `lambda x: x`) -- The argument passed to the metric/function follows a `.`-separated format (`"test.layer4_2_relu"`), which needs to be consistent with `Probe` setup: - - `test` matches input in `get_probe("test")` - - `layer4_2_relu` matches variable name in `layer4_2_relu=hook_output[0][0]` - -#### Step 5 -For the scope of this document, we don't dwell on what `armory` is doing in step 5 with `get_hub().connect_meter(meter)` other than to mention this step is necessary. - -## Example 2: Attack Artifact - Available as Output -### User Story -I am using `CARLADapricotPatch`, and I am interested in the patch after every iteration, which is generated by `CARLADapricotPatch._augment_images_with_patch` and returned as an output. -### Example Code -This is an example of working with a python package/framework (i.e. `art`) that does NOT come with built-in hooking mechanisms. In the code snippet below, we define wrapper functions to wrap existing instance methods to monitor the output of interest: -```python -from armory.scenarios.main import get as get_scenario -from armory.instrument import get_probe, Meter, get_hub -import types - -def method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None): - """ - Hook target method and return the original method - If a class is passed in, hooks ALL instances of class. - If an object is passed in, only hooks the given instance. - """ - if not isinstance(obj, object): - raise ValueError(f"obj {obj} is not a class or object") - method = getattr(obj, method_name) - if not callable(method): - raise ValueError(f"obj.{method_name} attribute {method} is not callable") - wrapped = hook_wrapper( - method, pre_method_hook=pre_method_hook, post_method_hook=post_method_hook - ) - - if isinstance(obj, type): - cls = obj - setattr(cls, method_name, wrapped) - else: - setattr(obj, method_name, types.MethodType(wrapped, obj)) - - return method - -def hook_wrapper(method, pre_method_hook = None, post_method_hook = None): - def wrapped(*args, **kwargs): - return_value = method(*args[1:], **kwargs) # skip self with *args[1:] - post_method_hook(*return_value) # unpack return_value with * - return return_value - - return wrapped - -def post_method_hook(x_patch, patch_target, transformations): - probe.update(x_patch=x_patch) - -# load Scenario -s = get_scenario( - "/armory/scenario_configs/eval6/carla_overhead_object_detection/carla_obj_det_dpatch_undefended.json", - num_eval_batches=1, -).load() - -# create Probe with "hooked_method" namespace -probe = get_probe("hooked_method") - -# register hook that will update Probe -method_hook( - s.attack, "_augment_images_with_patch", post_method_hook=post_method_hook -) - -# create Meter for Probe with "hooked_method" namespace -hook_meter = Meter( - "hook_x_patch", lambda x: x, "hooked_method.x_patch" -) - -# connect Meter to Hub -get_hub().connect_meter(hook_meter) - -s.next() -s.run_attack() -``` -### Packages with NO Hooks -Unlike [Example 1](#example1), we have defined new functions to meet user needs: -- `method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None)` -- `hook_wrapper(method, pre_method_hook = None, post_method_hook = None)` -- `post_method_hook(x_patch, patch_target, transformations)` - -The steps are the same as before, with the exception that [Step 3](#step3) is more involved than Example 1. - -### Probe and Meter Details - Step 3 -The general approach for hooking a `Probe` is as follows: -1. Define the function for the `Probe` action (e.g. `post_method_hook`) -2. Wrap the method of interest (e.g. `_augment_images_with_patch`) and `post_method_hook` - 1. Define function (e.g. `hook_wrapper`) that returns another function (e.g. `wrapped`) that calls `_augment_images_with_patch` and `post_method_hook` in the desired order - 2. Assign the result of `hook_wrapper` to the original method of interest (`_augment_images_with_patch`) via `method_hook`, thus changing the behavior of the method without modifying it directly - -#### Step 3-1: `post_method_hook` -The role of the defined `post_method_hook` function is the same as that of `hook_fn` defined in [Example 1](#example1) - we are specifying the variable to update being monitored by the `Probe`. In this particular example, despite its name, it is not the name that is specifying whether the `Probe` action occurs after a method, but which input this function is assigned to when calling `method_hook`, which we will show later. - -Note the signature of `post_method_hook` with 3 arguments - this was based on the expected output of `_augment_images_with_patch` i.e. `return_value = method(*args[1:], **kwargs)`, and the anticipation that the output of `_augment_images_with_patch` would be passed to `post_method_hook` as input in `hook_wrapper` i.e. `post_method_hook(*return_value)`. - -Of those expected outputs, we are choosing to update the value assigned to argument `x_patch` of `post_method_hook` and choosing to also refer to the updated value as `x_patch` by the `Probe`, which leads to `probe.update(x_patch=x_patch)`. The `Meter` is then able to reference this as `"hooked_method.x_patch"` later on. - -For now, consider the example shown as a possible template for the user - we leave it as a template rather than add a defined function in armory such that the user can adjust as needed. - -#### Step 3-2.1: `hook_wrapper` -`hook_wrapper` determines when the `Probe` action takes place with respect to a `method` call as well as how inputs should be specified for either a `pre_method_hook` or a `post_method_hook`. Its signature as defined suggests the possibility of either, but in this example, we only specify `post_method_hook`. - -Within `hook_wrapper`, we define `wrapped`, which is where the actual order of calls for `method` and `post_method_hook` are specified, which in this case is `method` then `post_method_hook`. Again, despite the argument name, there is no reason the user cannot specify `post_method_hook` to be called before `method`, but we discourage such practices for clarity. - -Now consider the arguments involved within `hook_wrapper`. `hook_wrapper` once called, will return a function `wrapped`, which expects arguments `*args` and `**kwargs`. Because `hook_wrapper` is meant to wrap `method`, `*args` and `**kwargs` will be passed to `method` such that it performs its original function. Recall that `method` is not just any ordinary function but a method for an instance, which means that even though the first argument of `*args` will be `self` ***[why was `self` even passed to begin with???]***, the actual method call needs to exclude it, leading to `method(*args[1:], **kwargs)`. - -Also note what `wrapped` returns. Since `method` needs to maintain its original functionality, the return value for `wrapped` should also match that of `method`, thus `return return_value`. - -Last but not least, consider the `post_method_hook` call. For this example, the objective was to update a variable that is an output of `method` after each `method` call, and as mentioned in [Step 3-1](#step-3-1-post_method_hook), we pass `return_value` as-is but with a `*` to unpack the iterable, thus `post_method_hook(*return_value)`. - -We present `hook_wrapper` as another template for the user rather than a defined armory function, because, as we have alluded to before, the arrangement of a `Probe` action and instance method is heavily dependent on the user's objective and not obviously generalizable as we will see in the next example. - -#### Step 3-2.2: `method_hook` -Notice that `hook_wrapper` returns a wrapped method, but the wrapped method is not actually reassigned to the method that was meant to be wrapped. `method_hook`, which takes an object `obj` and its associated method name `method`, is defined to do just that, along with actually executing `hook_wrapper` as well for the wrapping process. - -Unlike `post_method_hook` and `hook_wrapper`, which we made a point of framing as templates, we believe `method_hook` is well-established and generalized enough to be defined as an armory function, which the user can import and use as-is. - -## Example 3: Attack Artifact - NOT Available as Output -### User Story -I am using `CARLAAdversarialPatchPyTorch`, and I am interested in the patch after every iteration, which is generated during `CARLAAdversarialPatchPyTorch._train_step`, but NOT provided as an output. -### Example Code -Like [Example 2](#example-2-attack-artifact---available-as-output), the python package/framework (i.e. `art`) does NOT come with built-in hooking mechanisms, BUT unlike Example 2, the method of interest does NOT return the artifact of interest (`_train_step` returns `loss`) - rather, the artifact of interest is available as an attribute (`self._patch`). In the code snippet below, we adjust `post_method_hook` and `hook_wrapper` to reflect this new context: -```python -from armory.scenarios.main import get as get_scenario -from armory.instrument import get_probe, Meter, get_hub -import types - -def method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None): - """ - Hook target method and return the original method - If a class is passed in, hooks ALL instances of class. - If an object is passed in, only hooks the given instance. - """ - if not isinstance(obj, object): - raise ValueError(f"obj {obj} is not a class or object") - method = getattr(obj, method_name) - if not callable(method): - raise ValueError(f"obj.{method_name} attribute {method} is not callable") - wrapped = hook_wrapper( - method, pre_method_hook=pre_method_hook, post_method_hook=post_method_hook - ) - - if isinstance(obj, type): - cls = obj - setattr(cls, method_name, wrapped) - else: - setattr(obj, method_name, types.MethodType(wrapped, obj)) - - return method - -def hook_wrapper(method, pre_method_hook=None, post_method_hook=None): - def wrapped(*args, **kwargs): - return_value = method(*args[1:], **kwargs) # skip self with *args[1:] - if post_method_hook is not None: - post_method_hook(*args[0]) # *args[0] corresponds to self with _patch attribute - return return_value - - return wrapped - -def post_method_hook(obj): - probe.update(patch=obj._patch) - -# load Scenario -s = get_scenario( - "/armory/scenario_configs/eval6/carla_overhead_object_detection/carla_obj_det_adversarialpatch_undefended.json", - num_eval_batches=1, -).load() - -# create Probe with "hooked_method" namespace -probe = get_probe("hooked_method") - -# register hook that will update Probe -method_hook( - s.attack, "_train_step", post_method_hook=post_method_hook -) - -# create Meter for Probe with "hooked_method" namespace -hook_meter = Meter( - "hook_patch", lambda x: x.detach().cpu().numpy(), "hooked_method.patch" -) - -# connect Meter to Hub -get_hub().connect_meter(hook_meter) - -s.next() -s.run_attack() -``` -Consider the functions introduced in [Example 2](#example-2-attack-artifact---available-as-output): -- `method_hook(obj, method_name, pre_method_hook=None, post_method_hook=None)` -- `hook_wrapper(method, pre_method_hook = None, post_method_hook = None)` -- `post_method_hook(x_patch, patch_target, transformations)` - -`method_hook` has stayed the same (which is why we define it as an armory function), but `hook_wrapper` and `post_method_hook` have changed. - -### Probe and Meter Details - Step 3 -Recall the general approach for hooking a `Probe`: -1. Define the function for the `Probe` action (e.g. `post_method_hook`) ***[CHANGED]*** -2. Wrap the method of interest (e.g. `_train_step`) and `post_method_hook` - 1. Define function (e.g. `hook_wrapper`) that returns another function (e.g. `wrapped`) that calls `_train_step` and `post_method_hook` in the desired order ***[CHANGED]*** - 2. Assign the result of `hook_wrapper` to the original method of interest (`_train_step`) via `method_hook`, thus changing the behavior of the method without modifying it directly ***[UNCHANGED]*** - -#### Step 3-1: `post_method_hook` -The signature of `post_method_hook` now specifies a single argument `obj`, which we assume has a `_patch` attribute. Again note that this has nothing to do with the expected output of `_train_step` - we know from inspecting the `_train_step` method that a `_patch` attribute exists, which we refer to within `post_method_hook` via `obj._patch`. We are choosing to measure the value assigned to an attribute of `obj`, an input of `post_method_hook`, and also choosing to refer to the variable to be updated as `patch` by the `Probe`, which leads to `probe.update(patch=obj._patch)`. The `Meter` is then able to reference this as `"hooked_method.patch"` later on. - -This example is another possible template for the user, where the definition of `post_method_hook` changes depending on what the user is interested in monitoring. - -#### Step 3-2.1: `hook_wrapper` -As in [Example 2](#example-2-attack-artifact---available-as-output), `method` is called before `post_method_hook` in `wrapped` of `hook_wrapper`. `return_value` is not used in any way other than being returned at the end of `wrapped` to maintain `method`'s functionality. `*arg[0]` of `*arg` however, is passed to `post_method_hook` as an input, because it refers to `self`, the instance making the `method` call. As mentioned earlier, `self` contains the `_patch` attribute, which is what `probe` is monitoring in `post_method_hook`. - -Again, this example's `hook_wrapper` is another possible template for the user, which has been defined in such as way as to accomplish the objective described in the [User Story](#user-story-2). - -#### Step 4 -A brief note on the `Meter` creation in this example: The `_patch` variable of interest is a tensor, which is why a preprocessing function is applied for the identity metric, `lambda x: x.detach().cpu().numpy()`. The chaining functions could have occurred else where in the process as well. - -## Flexible Arrangements -We have emphasized through out this section that the example functions shown (with the exception of `method_hook`) are templates, and how those functions are defined depends on the user story. Before we discuss what the user needs to consider when defining those functions, it is important that the user remembers the 5-step process for custom probes as a general framework, which may aid in any debugging efforts that may occur later. - -That said, here are the questions the user needs to consider for creating custom probes: -- What is the `method` of interest? What is the variable of interest? - - Some knowledge of the package being used is necessary -- Is there an existing hooking mechanism for the `method` of interest? - - Relying on hooks from the package provider reduces code and maintenance -- Does the `Probe` action occur before or after a `method` call? - - Define `pre_method_hook` or `post_method_hook` and arrange calls as necessary with respect to `method` call -- What should `pre_method_hook`/`post_method_hook` expect as input? - - It may be natural for a `pre_method_hook` to take the same arguments as `method` i.e. `*args` and `**kwargs` - - It may be natural for a `post_method_hook` to take the output of a `method` call i.e. `*return_value` - - If the user is interested in the state of an instance or the attributes within it pre or post `method` call, passing the instance to `pre_method_hook`/`post_method_hook` is also possible i.e. `*args[0]` -- Is `wrapped` returning `return_value`? - - `method`'s original functionality has to be maintained - -Once these questions have been answered, the user can define functions as needed to meet specific monitoring objectives. \ No newline at end of file From 6de62f22a0ffbd9da8388f827164000f67402cb0 Mon Sep 17 00:00:00 2001 From: Paul Park Date: Thu, 22 Dec 2022 21:35:39 +0000 Subject: [PATCH 30/30] missed substitution --- docs/instrumentation_examples.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/instrumentation_examples.md b/docs/instrumentation_examples.md index 4f3c7f17e..5488920c6 100644 --- a/docs/instrumentation_examples.md +++ b/docs/instrumentation_examples.md @@ -149,7 +149,7 @@ By default, outputs from `Meter`s will be saved to the output `json` file after Users who have tried the examples in this document, however, may run into some of the following warning logs: > 2022-12-16 19:34:36 30s WARNING armory.instrument.instrument:_write:856 record (name=my_arbitrary_meter_name, batch=0, result=...) size > max_record_size 1048576. Dropping. -Outputs are saved to a `json` file because of a default `ResultWriter` class tied to the `Meter` class, which has a `max_record_size` limit for each record. Any record that exceeds `max_record_size` will not save to the `json` file. That the outputs exceed a size limit also suggests that a `json` file may not be the best file type to save to. To work around these behaviors, we can define a new `Writer` subclass (`ResultWriter` is also a `Writer` subclass) to work with our examples that does not have a size limit and will save to another filetype, such as a `png` file, since we are saving data for an image. Below is an updated `user_init_script.py` for Example 2 with a new `ImageWriter` class, which uses the `export` method of `ImageClassificationExporter` to save an image, and a `set_up_meter_writer` function that will be executed with the `user_init` block: +Outputs are saved to a `json` file because of a default `ResultWriter` class tied to the `Meter` class, which has a `max_record_size` limit for each record. Any record that exceeds `max_record_size` will not save to the `json` file. That the outputs exceed a size limit also suggests that a `json` file may not be the best file type to save to. To work around these behaviors, we can define a new `Writer` subclass (`ResultWriter` is also a `Writer` subclass) to work with our examples that does not have a size limit and will save to another filetype, such as a `png` file, since we are saving data for an image. Below is an updated `user_init_script.py` for Example 2 with a new `ImageWriter` class, which uses the `export` method of `ObjectDetectionExporter` to save an image, and a `set_up_meter_writer` function that will be executed with the `user_init` block: ```python from armory.instrument import get_hub, Meter, Writer from armory.instrument.export import ObjectDetectionExporter