Dev add bert automated test by luqiang-guo · Pull Request #203 · Oneflow-Inc/OneFlow-Benchmark

luqiang-guo · 2021-06-22T03:21:24Z

Add docker image multi-machine startup script and bert multi-machine execution script.

util.py Add GPU memory for printing. analysis.py Analyze log files to get loss and GPU memory. result_analysis.py Compare the two results to generate a png image. stitching_pic.py Stitching pictures. train_perbert.sh Training script. train_perbert_list.sh Training list.

This reverts commit 647dbd6.

util.py Add GPU memory for printing. analysis.py Analyze log files to get loss and GPU memory. result_analysis.py Compare the two results to generate a png image. stitching_pic.py Stitching pictures. train_perbert.sh Training script. train_perbert_list.sh Training list.

jackalcooper · 2021-06-22T03:33:14Z

能不能把那些bash脚本改成python的，复杂的bash脚本很难维护

jackalcooper · 2021-06-22T03:33:41Z

这个测试能做到无人值守吗，就是脚本自己能判断精度和性能是否达标

luqiang-guo · 2021-06-22T03:36:14Z

试能做到无人值守吗，就是脚本自己能判断精度和

可以做到自己判断精度，而且还可以出一个分析报告。

luqiang-guo · 2021-06-22T03:38:58Z

能不能把那些bash脚本改成python的，复杂的bash脚本很难维护

之前试着改过python的方式，但是尝试失败了。主要问题是我需要非阻塞的在其他机上执行启动脚本的命令。

jackalcooper · 2021-06-22T05:00:52Z

能不能把那些bash脚本改成python的，复杂的bash脚本很难维护

之前试着改过python的方式，但是尝试失败了。主要问题是我需要非阻塞的在其他机上执行启动脚本的命令。

用 subprocess 模块的poll 或者 async await 都可以做到在python里面的非阻塞

luqiang-guo · 2021-06-22T06:36:03Z

能不能把那些bash脚本改成python的，复杂的bash脚本很难维护

之前试着改过python的方式，但是尝试失败了。主要问题是我需要非阻塞的在其他机上执行启动脚本的命令。

用 subprocess 模块的poll 或者 async await 都可以做到在python里面的非阻塞

好的，我后面用async await 修改

ouyangyu · 2021-06-22T07:06:56Z

LanguageModeling/BERT/tools/stitching_pic.py

+if __name__ == "__main__":
+    main()
+
+


空行处理一下

ouyangyu · 2021-06-22T07:07:20Z

LanguageModeling/BERT/train_perbert.sh

+PRINT_ITER=1
+ITER_NUM=130
+
+NODE_IPS='10.10.0.2','10.10.0.3','10.10.0.4','10.10.0.5'


这个ip地址改成10.11.0.2

ouyangyu · 2021-06-22T07:07:37Z

LanguageModeling/BERT/train_perbert.sh

+
+NODE_IPS='10.10.0.2','10.10.0.3','10.10.0.4','10.10.0.5'
+
+# INIT_MODEL=/opt/initial_model 


ouyangyu · 2021-06-22T07:09:34Z

LanguageModeling/BERT/train_perbert_list.sh

+
+# ###############################################################################
+# #                                f32  adam
+# ###############################################################################


这个显得有点臃肿，能通过传参数不？

ouyangyu · 2021-06-22T07:10:01Z

LanguageModeling/BERT/tools/result_analysis.py

@@ -0,0 +1,134 @@
+import json
+import matplotlib.pyplot as plt


建议加requirements.txt

ouyangyu · 2021-06-22T07:11:56Z

LanguageModeling/BERT/oneflow_auto_bert.sh

+PYTHON="python3.8"
+DOCKER_USER=root
+
+multi_machine() 


用subprocess 或async处理下吧

ouyangyu · 2021-06-22T07:12:15Z

LanguageModeling/BERT/oneflow_auto_bert.sh

+    # param 5  
+    IS_F32=$5
+
+    declare -a host_list=("10.10.0.2" "10.10.0.3" "10.10.0.4" "10.10.0.5")


ouyangyu · 2021-06-22T07:12:58Z

LanguageModeling/BERT/oneflow_auto_bert.sh

+########################################################################################
+ALL_NODES=4
+
+declare -a host_list=("10.11.0.2" "10.11.0.3" "10.11.0.4" "10.11.0.5")


上面不是定义了？

ouyangyu · 2021-06-22T07:13:22Z

LanguageModeling/BERT/oneflow_auto_bert.sh

+
+#######################################################################################
+#                                2   run   single
+########################################################################################


这里应该也能通过传参解决

ouyangyu · 2021-06-22T07:13:44Z

LanguageModeling/BERT/run_squad.sh

@@ -0,0 +1,74 @@
+BENCH_ROOT_DIR=/home/oyy/workspace/OneFlow-Benchmark/LanguageModeling/BERT


ouyangyu and others added 18 commits January 12, 2021 15:29

bert pretraining adam squad sh

05ddf09

args num_accumulation_steps

de98fac

fix args

89b4e05

fix batch_size

ed66b48

bert lamb

0e4d46b

lamb weight decay

52adf9f

Revert "run_pretraining.py Add some parameters."

a4299c0

This reverts commit 647dbd6.

Organize files and move them to the tools file directory

728dd68

Modify path

7f98272

conda environment multi-machine automatic car market

e0093dd

Add multi-machine bert automated test

494da78

Add automated test script

389af5d

Modify the automatic test script

8f09974

Add multi-machine docker script

a4cc1eb

Add close multi-machine docker script

8c32fa4

megre test_oneflow_release

1a00c7d

luqiang-guo requested a review from ouyangyu June 22, 2021 03:21

luqiang-guo added documentation Improvements or additions to documentation enhancement New feature or request and removed documentation Improvements or additions to documentation labels Jun 22, 2021

merge master

3ac9581

Merge branch 'test_oneflow_release' into dev_add_bert_automated_test

b859111

luqiang-guo added documentation Improvements or additions to documentation and removed enhancement New feature or request documentation Improvements or additions to documentation labels Jun 22, 2021

ouyangyu reviewed Jun 22, 2021

View reviewed changes

luqiang-guo added 6 commits June 22, 2021 16:46

Modify small details

2d36b9a

oneflow_auto_bert.sh can replace train_perbert_list.sh

638d576

modified oneflow_auto_bert.sh ip

4b217e9

Fix merge errors

45d9ace

Modify the bert automatic test script

bb3d0bf

Fix merge errors

d81c54a


		NODE_IPS='10.10.0.2','10.10.0.3','10.10.0.4','10.10.0.5'

		# INIT_MODEL=/opt/initial_model

		@@ -0,0 +1,74 @@
		BENCH_ROOT_DIR=/home/oyy/workspace/OneFlow-Benchmark/LanguageModeling/BERT

Conversation

luqiang-guo commented Jun 22, 2021

Uh oh!

jackalcooper commented Jun 22, 2021

Uh oh!

jackalcooper commented Jun 22, 2021

Uh oh!

luqiang-guo commented Jun 22, 2021

Uh oh!

luqiang-guo commented Jun 22, 2021

Uh oh!

jackalcooper commented Jun 22, 2021

Uh oh!

luqiang-guo commented Jun 22, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants