Instrumentation Examples with Probes, Meters, and Writers#1821
Merged
lcadalzo merged 32 commits intotwosixlabs:developfrom Dec 22, 2022
Merged
Instrumentation Examples with Probes, Meters, and Writers#1821lcadalzo merged 32 commits intotwosixlabs:developfrom
lcadalzo merged 32 commits intotwosixlabs:developfrom
Conversation
…s more details; finished initial draft of Example 2
…use cases; draft is complete with the minimum necessary content; because code examples are from notebook experiments, need to refine content and code that can be executed via armory run with a custom script passed to a config file along with explanations of how to save any Probe\Meter outputs for end-to-end examples
… with user-init block that can be executed with armory run - requires necessary custom files and config not included in this repo
…ossible use cases; draft is complete with the minimum necessary content; because code examples are from notebook experiments, need to refine content and code that can be executed via armory run with a custom script passed to a config file along with explanations of how to save any Probe\Meter outputs for end-to-end examples" This reverts commit 74365ed.
…use cases; draft is complete with the minimum necessary content; because code examples are from notebook experiments, need to refine content and code that can be executed via armory run with a custom script passed to a config file along with explanations of how to save any Probe\Meter outputs for end-to-end examples
… hooking with user-init block that can be executed with armory run - requires necessary custom files and config not included in this repo" This reverts commit 28c3152. Reverting this commit based on discussion with Lucas and the need to focus on Probe examples that work under the assumption that the user is actively changing code to add Probes for a model or attack, as opposed to hooking a Probe to an existing model or attack without modifying the respective code directly - aside from updating the docs (I think at this point I'll rename the existing Probe.md as a different use case and add another doc just for the use case that I dicussed with Lucas), this will not change the fact that the user needs to use the user-init block (I may need to do a pull since the user-init implementation doesn't exist here yet - I've definitely seen it in the repo) with a Meter defined for the Probes that will be added, and it will also not change the need for perhaps a new Meter class that will export probe variables to a pickle file depending on what the user is trying to save (e.g. patch data or image with patch data) rather than a json file. There is also the question of how to deal with size restrictions placed on a Meter (unclear where the restrictions are actually defined - I only know that it exists because of armory logs and the fact that the variable doesn't get saved to the json file when I see size restriction logs). An additional side note regarding the user-init block for the case that we do decide to add capabilities for Probe hooking without modifying the model or attack code: The way I got it to work in this commit (which will be reverted) is to load the user-init block at the end of all other loads (e.g. model, attack, dataset, metrics etc.) because an instance of a model or attack had to be available to hook to, which I accomplished by passing the Scenario (which contains the model and attack as an attribute) as an argument to a user init function. The existing user-init feature (again not in this branch currently, so will require a git pull and merge), assumes the user-init load happens prior to any other loads, which means none of the instances will be available just yet. Hopefully there will be no issue with this when defining a Meter to accompany any Probe the user may have added to a model or attack code, since the Meter can exist separately from a Probe instance that the Meter will get updates from.
…s an example for probe hooking within a jupyter notebook; will update probe.md to address the user case discussed with Lucas
…nd also add a section on exporting to pickle file
…r config documentation
…Probe - these changes were made based on consensus reached with David and Lucas that documentation showing how Probes can be used within existing packages or code is more sustainable for maintenance in the long run rather than having to define functions to fit hooking mechanisms that may or may not exist in other packages into the Probe framework, which will require updates should any of the hooks defined in those packages change
lcadalzo
reviewed
Dec 16, 2022
lcadalzo
reviewed
Dec 16, 2022
lcadalzo
reviewed
Dec 16, 2022
lcadalzo
reviewed
Dec 16, 2022
lcadalzo
reviewed
Dec 16, 2022
lcadalzo
reviewed
Dec 16, 2022
lcadalzo
reviewed
Dec 16, 2022
lcadalzo
reviewed
Dec 16, 2022
lcadalzo
reviewed
Dec 16, 2022
lcadalzo
reviewed
Dec 16, 2022
lcadalzo
reviewed
Dec 16, 2022
lcadalzo
reviewed
Dec 19, 2022
Contributor
|
@ppark-twosixtech |
lcadalzo
reviewed
Dec 21, 2022
Contributor
|
this predates your PR, but in Given our now support for custom metrics, this would be a good time to remove this |
lcadalzo
reviewed
Dec 21, 2022
lcadalzo
reviewed
Dec 21, 2022
lcadalzo
reviewed
Dec 21, 2022
lcadalzo
reviewed
Dec 21, 2022
lcadalzo
reviewed
Dec 21, 2022
lcadalzo
reviewed
Dec 21, 2022
lcadalzo
reviewed
Dec 21, 2022
lcadalzo
reviewed
Dec 21, 2022
lcadalzo
reviewed
Dec 21, 2022
lcadalzo
reviewed
Dec 21, 2022
lcadalzo
reviewed
Dec 21, 2022
lcadalzo
reviewed
Dec 21, 2022
… line numbers to code blocks
…should be changed for custom models and attacks; renamed user_init.py to user_init_script.py to disambiguate from user_init block in config file
Contributor
Author
resolved with c24837a |
lcadalzo
reviewed
Dec 22, 2022
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR was the outcome of discussions on how to make
Probes andMeters more approachable to the user, with the conclusion that more documentation with specific examples would be more beneficial then defining hooking methods in theProbeclass for different packages (i.e.pytorch,art, etc.). We determined that defining hooking methods in this manner would not be sustainable since they would have to be maintained by the GARD program should any of those packages change in the future. The templates introduced in this documentation show a flexible approach in howProbes andMeters can fit into other packages/frameworks, and avoids the need to figure out how other packages need to fit intoarmory. Code and documentation referring to any defined hooking methods forProbes have been removed as well.