[Distributed DP] Cannot capture a result of an unsupported type NoneType #67
Description
Under the directory distributed-dp, I ran the command
bazel run :fl_run -- \
--task=emnist_character \
--server_optimizer=sgd \
--server_learning_rate=1 \
--server_sgd_momentum=0.0 \
--client_optimizer=sgd \
--client_learning_rate=0.01 \
--client_sgd_momentum=0.9 \
--client_batch_size=20 \
--clients_per_round=100 \
--experiment_name=my_emnist_test \
--epsilon=6 \
--num_bits=20 \
--l2_norm_clip=1 \
--k_stddevs=3 \
--client_epochs_per_round=2 \
--dp_mechanism=dskellam \
--total_rounds=50 \
--logtostderr > log.txt 2>&1In the log which I used to collect redirected messages, I obtained
Loading:
Loading: 0 packages loaded
DEBUG: Rule 'rules_python' indicated that a canonical reproducible form can be obtained by modifying arguments commit = "a0fbf98d4e3a232144df4d0d80b577c7a693b570", shallow_since = "1586444447 +0200" and dropping ["tag"]
DEBUG: Repository rules_python instantiated at:
/data/samuel/federated/WORKSPACE:5:15: in <toplevel>
Repository rule git_repository defined at:
/data/samuel/.cache/bazel/_bazel_zjiangaj/3fe67cea0bf4ee0dd0c8eb14b19e1272/external/bazel_tools/tools/build_defs/repo/git.bzl:199:33: in <toplevel>
Analyzing: target //distributed_dp:fl_run (0 packages loaded, 0 targets configured)
DEBUG: Rule 'rules_license' indicated that a canonical reproducible form can be obtained by modifying arguments commit = "e3bdc544ef373156da36638b774d65ff2d978bfa", shallow_since = "1667362800 -0400" and dropping ["tag"]
DEBUG: Repository rules_license instantiated at:
/data/samuel/federated/WORKSPACE:14:15: in <toplevel>
Repository rule git_repository defined at:
/data/samuel/.cache/bazel/_bazel_zjiangaj/3fe67cea0bf4ee0dd0c8eb14b19e1272/external/bazel_tools/tools/build_defs/repo/git.bzl:199:33: in <toplevel>
INFO: Analyzed target //distributed_dp:fl_run (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
[0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt
Target //distributed_dp:fl_run up-to-date:
bazel-bin/distributed_dp/fl_run
INFO: Elapsed time: 4.312s, Critical Path: 0.02s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/distributed_dp/fl_run '--task=emnist_character' '--server_optimizer=sgd' '--server_learning_rate=1' '--server_sgd_momentum=0.0' '--client_optimizer=sgd' '--client_learning_rate=0.01' '--client_sgd_momentum=0.9' '--client_batch_size=20' '--clients_per_round=100' '--experiment_name=my_emnist_test' '--epsilon=6' '--num_bits=20' '--l2_norm_clip=1' '--k_stddevs=3' '--client_epochs_per_round=2' '--dp_mechanism=dskellam' '--total_rounds=50' --logtostderr
INFO: Build completed successfully, 1 total action
I0317 04:07:43.299935 140328278546240 sql_client_data.py:127] Loaded 3400 client ids from SQL database.
I0317 04:07:50.917061 140328278546240 sql_client_data.py:127] Loaded 3400 client ids from SQL database.
I0317 04:07:51.318575 140328278546240 keras_utils.py:362] Adding default num_examples metric to model
I0317 04:07:51.318807 140328278546240 keras_utils.py:365] Adding default num_batches metric to model
I0317 04:07:51.492904 140328278546240 keras_utils.py:362] Adding default num_examples metric to model
I0317 04:07:51.493148 140328278546240 keras_utils.py:365] Adding default num_batches metric to model
I0317 04:07:51.502441 140328278546240 fl_utils.py:72] Shared DP Parameters:
I0317 04:07:51.502845 140328278546240 fl_utils.py:73] {'clip': 1.0,
'delta': 0.0002941176470588235,
'dim': 1018174,
'epsilon': 6.0,
'mechanism': 'dskellam',
'num_clients': 3400,
'num_clients_per_round': 100,
'num_rounds': 50,
'sampling_rate': 1.0}
I0317 04:07:52.706335 140328278546240 fl_utils.py:152] dskellam parameters:
I0317 04:07:52.706791 140328278546240 fl_utils.py:153] {'beta': 0.6065306597126334,
'bits': 20,
'dim': 1018174,
'gamma': 2.8479735556541443e-05,
'inflated_l2': 1.000120752105709,
'k_stddevs': 3,
'local_stddev': 0.49762363478987853,
'mechanism': 'dskellam',
'noise_mult_clip': 4.976236347898785,
'noise_mult_inflated': 4.975635529431367,
'padded_dim': 1048576.0,
'scale': 35112.68558005667}
I0317 04:07:52.706896 140328278546240 ddpquery_utils.py:44] Conditional rounding set to True (beta = 0.606531)
I0317 04:07:52.983747 140328278546240 keras_utils.py:362] Adding default num_examples metric to model
I0317 04:07:52.983982 140328278546240 keras_utils.py:365] Adding default num_batches metric to model
Traceback (most recent call last):
File "/data/samuel/.cache/bazel/_bazel_zjiangaj/3fe67cea0bf4ee0dd0c8eb14b19e1272/execroot/org_federated_research/bazel-out/k8-opt/bin/distributed_dp/fl_run.runfiles/org_federated_research/distributed_dp/fl_run.py", line 304, in <module>
app.run(main)
File "/data/samuel/anaconda3/envs/google/lib/python3.9/site-packages/absl/app.py", line 312, in run
_run_main(main, args)
File "/data/samuel/anaconda3/envs/google/lib/python3.9/site-packages/absl/app.py", line 258, in _run_main
sys.exit(main(argv))
File "/data/samuel/.cache/bazel/_bazel_zjiangaj/3fe67cea0bf4ee0dd0c8eb14b19e1272/execroot/org_federated_research/bazel-out/k8-opt/bin/distributed_dp/fl_run.runfiles/org_federated_research/distributed_dp/fl_run.py", line 246, in main
iterative_process = tff.learning.algorithms.build_unweighted_fed_avg(
File "/data/samuel/anaconda3/envs/google/lib/python3.9/site-packages/tensorflow_federated/python/learning/algorithms/fed_avg.py", line 324, in build_unweighted_fed_avg
return build_weighted_fed_avg(
File "/data/samuel/anaconda3/envs/google/lib/python3.9/site-packages/tensorflow_federated/python/learning/algorithms/fed_avg.py", line 198, in build_weighted_fed_avg
aggregator = model_aggregator.create(model_update_type,
File "/data/samuel/anaconda3/envs/google/lib/python3.9/site-packages/tensorflow_federated/python/aggregators/factory_utils.py", line 52, in create
aggregator = self._factory.create(value_type)
File "/data/samuel/anaconda3/envs/google/lib/python3.9/site-packages/tensorflow_federated/python/aggregators/mean.py", line 200, in create
value_sum_process = self._value_sum_factory.create(value_type)
File "/data/samuel/anaconda3/envs/google/lib/python3.9/site-packages/tensorflow_federated/python/aggregators/robust.py", line 378, in create
inner_agg_process = inner_agg_factory.create(value_type)
File "/data/samuel/anaconda3/envs/google/lib/python3.9/site-packages/tensorflow_federated/python/aggregators/differential_privacy.py", line 335, in create
get_noised_result = computations.tf_computation(
File "/data/samuel/anaconda3/envs/google/lib/python3.9/site-packages/tensorflow_federated/python/core/impl/wrappers/computation_wrapper.py", line 496, in __call__
wrapped_func = self._strategy(
File "/data/samuel/anaconda3/envs/google/lib/python3.9/site-packages/tensorflow_federated/python/core/impl/wrappers/computation_wrapper.py", line 237, in __call__
return wrapped_fn_generator.send(result)
File "/data/samuel/anaconda3/envs/google/lib/python3.9/site-packages/tensorflow_federated/python/core/impl/wrappers/computation_wrapper.py", line 80, in _wrap_concrete
concrete_fn = generator.send(result)
File "/data/samuel/anaconda3/envs/google/lib/python3.9/site-packages/tensorflow_federated/python/core/impl/wrappers/computation_wrapper_instances.py", line 63, in _tf_wrapper_fn
comp_pb, extra_type_spec = tf_serializer.send(result)
File "/data/samuel/anaconda3/envs/google/lib/python3.9/site-packages/tensorflow_federated/python/core/impl/tensorflow_context/tensorflow_serialization.py", line 111, in tf_computation_serializer
result_type, result_binding = tensorflow_utils.capture_result_from_graph(
File "/data/samuel/anaconda3/envs/google/lib/python3.9/site-packages/tensorflow_federated/python/core/impl/utils/tensorflow_utils.py", line 295, in capture_result_from_graph
element_type_binding_pairs = [
File "/data/samuel/anaconda3/envs/google/lib/python3.9/site-packages/tensorflow_federated/python/core/impl/utils/tensorflow_utils.py", line 296, in <listcomp>
capture_result_from_graph(e, graph) for e in result
File "/data/samuel/anaconda3/envs/google/lib/python3.9/site-packages/tensorflow_federated/python/core/impl/utils/tensorflow_utils.py", line 328, in capture_result_from_graph
raise UnsupportedGraphResultError(
tensorflow_federated.python.core.impl.utils.tensorflow_utils.UnsupportedGraphResultError: Cannot capture a result of an unsupported type NoneType.
By the way, I am using Python 3.9.7, with pip installed packages (only ones related to tensorflow is listed)
tensorboard 2.8.0
tensorboard-data-server 0.6.1
tensorboard-plugin-wit 1.8.1
tensorflow 2.8.4
tensorflow-addons 0.19.0
tensorflow-datasets 4.5.2
tensorflow-estimator 2.8.0
tensorflow-federated 0.24.0
tensorflow-io-gcs-filesystem 0.31.0
tensorflow-metadata 1.12.0
tensorflow-model-optimization 0.7.3
tensorflow-privacy 0.8.0
tensorflow-probability 0.15.0
Could anyone help me with this issue, if you can successfully run the above command? Thank you in advance.