-
Notifications
You must be signed in to change notification settings - Fork 1
Expand file tree
/
Copy pathatom2.xml
More file actions
230 lines (124 loc) · 84.4 KB
/
atom2.xml
File metadata and controls
230 lines (124 loc) · 84.4 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
<title>Wade Deng's Blog</title>
<subtitle>move from cnblogs.com/no7dw</subtitle>
<link href="/atom.xml" rel="self"/>
<link href="http://deng.io/"/>
<updated>2019-09-10T14:04:33.378Z</updated>
<id>http://deng.io/</id>
<author>
<name>Wade Deng</name>
</author>
<generator uri="http://hexo.io/">Hexo</generator>
<entry>
<title></title>
<link href="http://deng.io/2019/09/10/MongoDB-compound-index/"/>
<id>http://deng.io/2019/09/10/MongoDB-compound-index/</id>
<published>2019-09-10T14:02:04.812Z</published>
<updated>2019-09-10T14:04:33.378Z</updated>
<content type="html"><![CDATA[<p>根据典型碰到的场景，来做几个实验：<br>这里创建了个loans collection。简化只有100条数据。这个是借贷的表有 _id, userId, status(借贷状态), amount(金额). </p><p><strong><em>看完 这个实验后， 你会明白了 {userId:1, status:1}, vs {status:1,userId:1} 的差别</em></strong></p><p>PS：这个case 里面其实status 区分度不高，不应该建立的，这里只是作为实例展示。</p><p>总结：</p><ul><li>注意使用上 使用频率上 区分高的/常用的在前面</li><li>如果需要减少索引以节省memory/提高修改数据的性能的话，可以保留区分度高，常用的，去除区分度不高，不常用的索引。</li></ul><p>实验如下：</p><blockquote><p>db.loans.count()<br>100</p></blockquote><blockquote><p>db.loans.find({ “userId” : “59e022d33f239800129c61c7”, “status” : “repayed”, }).explain()<br>{<br> “queryPlanner” : {<br> “plannerVersion” : 1,<br> “namespace” : “cashLoan.loans”,<br> “indexFilterSet” : false,<br> “parsedQuery” : {<br> “$and” : [<br> {<br> “status” : {<br> “$eq” : “repayed”<br> }<br> },<br> {<br> “userId” : {<br> “$eq” : “59e022d33f239800129c61c7”<br> }<br> }<br> ]<br> },<br> “queryHash” : “15D5A9A1”,<br> “planCacheKey” : “15D5A9A1”,<br> “winningPlan” : {<br> “stage” : “COLLSCAN”,<br> “filter” : {<br> “$and” : [<br> {<br> “status” : {<br> “$eq” : “repayed”<br> }<br> },<br> {<br> “userId” : {<br> “$eq” : “59e022d33f239800129c61c7”<br> }<br> }<br> ]<br> },<br> “direction” : “forward”<br> },<br> “rejectedPlans” : [ ]<br> },<br> “ok” : 1<br>}</p></blockquote><p><strong><em>注意上面 COLLSCAN 全表扫描了。因为没有索引。</em></strong><br>next 我们分别建立几个索引</p><p><strong><em>step 1 先建立 {userId:1, status:1}</em></strong></p><blockquote><p>db.loans.createIndex({userId:1, status:1})<br>{<br> “createdCollectionAutomatically” : false,<br> “numIndexesBefore” : 1,<br> “numIndexesAfter” : 2,<br> “ok” : 1<br>}</p></blockquote><blockquote><p>db.loans.find({ “userId” : “59e022d33f239800129c61c7”, “status” : “repayed”, }).explain()<br>{<br> “queryPlanner” : {<br> “plannerVersion” : 1,<br> “namespace” : “cashLoan.loans”,<br> “indexFilterSet” : false,<br> “parsedQuery” : {<br> “$and” : [<br> {<br> “status” : {<br> “$eq” : “repayed”<br> }<br> },<br> {<br> “userId” : {<br> “$eq” : “59e022d33f239800129c61c7”<br> }<br> }<br> ]<br> },<br> “queryHash” : “15D5A9A1”,<br> “planCacheKey” : “BB87F2BA”,<br> “winningPlan” : {<br> “stage” : “FETCH”,<br> “inputStage” : {<br> “stage” : “IXSCAN”,<br> “keyPattern” : {<br> “userId” : 1,<br> “status” : 1<br> },<br> “indexName” : “userId_1_status_1”,<br> “isMultiKey” : false,<br> “multiKeyPaths” : {<br> “userId” : [ ],<br> “status” : [ ]<br> },<br> “isUnique” : false,<br> “isSparse” : false,<br> “isPartial” : false,<br> “indexVersion” : 2,<br> “direction” : “forward”,<br> “indexBounds” : {<br> “userId” : [<br> “[\”59e022d33f239800129c61c7\”, \”59e022d33f239800129c61c7\”]”<br> ],<br> “status” : [<br> “[\”repayed\”, \”repayed\”]”<br> ]<br> }<br> }<br> },<br> “rejectedPlans” : [ ]<br> },<br> “ok” : 1<br>}</p></blockquote><p><strong><em>如愿命中 {userId:1, status:1} 作为 winning plan</em></strong></p><p><strong><em>step2 再建立个典型的索引 userId</em></strong></p><blockquote><p>db.loans.createIndex({userId:1})<br>{<br> “createdCollectionAutomatically” : false,<br> “numIndexesBefore” : 2,<br> “numIndexesAfter” : 3,<br> “ok” : 1<br>}</p></blockquote><blockquote><p>db.loans.find({ “userId” : “59e022d33f239800129c61c7”, “status” : “repayed”, }).explain()<br>{<br> “queryPlanner” : {<br> “plannerVersion” : 1,<br> “namespace” : “cashLoan.loans”,<br> “indexFilterSet” : false,<br> “parsedQuery” : {<br> “$and” : [<br> {<br> “status” : {<br> “$eq” : “repayed”<br> }<br> },<br> {<br> “userId” : {<br> “$eq” : “59e022d33f239800129c61c7”<br> }<br> }<br> ]<br> },<br> “queryHash” : “15D5A9A1”,<br> “planCacheKey” : “1B1A4861”,<br> “winningPlan” : {<br> “stage” : “FETCH”,<br> “inputStage” : {<br> “stage” : “IXSCAN”,<br> “keyPattern” : {<br> “userId” : 1,<br> “status” : 1<br> },<br> “indexName” : “userId_1_status_1”,<br> “isMultiKey” : false,<br> “multiKeyPaths” : {<br> “userId” : [ ],<br> “status” : [ ]<br> },<br> “isUnique” : false,<br> “isSparse” : false,<br> “isPartial” : false,<br> “indexVersion” : 2,<br> “direction” : “forward”,<br> “indexBounds” : {<br> “userId” : [<br> “[\”59e022d33f239800129c61c7\”, \”59e022d33f239800129c61c7\”]”<br> ],<br> “status” : [<br> “[\”repayed\”, \”repayed\”]”<br> ]<br> }<br> }<br> },<br> “rejectedPlans” : [<br> {<br> “stage” : “FETCH”,<br> “filter” : {<br> “status” : {<br> “$eq” : “repayed”<br> }<br> },<br> “inputStage” : {<br> “stage” : “IXSCAN”,<br> “keyPattern” : {<br> “userId” : 1<br> },<br> “indexName” : “userId_1”,<br> “isMultiKey” : false,<br> “multiKeyPaths” : {<br> “userId” : [ ]<br> },<br> “isUnique” : false,<br> “isSparse” : false,<br> “isPartial” : false,<br> “indexVersion” : 2,<br> “direction” : “forward”,<br> “indexBounds” : {<br> “userId” : [<br> “[\”59e022d33f239800129c61c7\”, \”59e022d33f239800129c61c7\”]”<br> ]<br> }<br> }<br> }<br> ]<br> },<br> “ok” : 1<br>}</p></blockquote><p><strong><em>留意到 DB 检测到 {userId:1, status:1} 为更优执行的方案</em></strong></p><blockquote><p>db.loans.find({ “userId” : “59e022d33f239800129c61c7” }).explain()<br>{<br> “queryPlanner” : {<br> “plannerVersion” : 1,<br> “namespace” : “cashLoan.loans”,<br> “indexFilterSet” : false,<br> “parsedQuery” : {<br> “userId” : {<br> “$eq” : “59e022d33f239800129c61c7”<br> }<br> },<br> “queryHash” : “B1777DBA”,<br> “planCacheKey” : “1F09D68E”,<br> “winningPlan” : {<br> “stage” : “FETCH”,<br> “inputStage” : {<br> “stage” : “IXSCAN”,<br> “keyPattern” : {<br> “userId” : 1<br> },<br> “indexName” : “userId_1”,<br> “isMultiKey” : false,<br> “multiKeyPaths” : {<br> “userId” : [ ]<br> },<br> “isUnique” : false,<br> “isSparse” : false,<br> “isPartial” : false,<br> “indexVersion” : 2,<br> “direction” : “forward”,<br> “indexBounds” : {<br> “userId” : [<br> “[\”59e022d33f239800129c61c7\”, \”59e022d33f239800129c61c7\”]”<br> ]<br> }<br> }<br> },<br> “rejectedPlans” : [<br> {<br> “stage” : “FETCH”,<br> “inputStage” : {<br> “stage” : “IXSCAN”,<br> “keyPattern” : {<br> “userId” : 1,<br> “status” : 1<br> },<br> “indexName” : “userId_1_status_1”,<br> “isMultiKey” : false,<br> “multiKeyPaths” : {<br> “userId” : [ ],<br> “status” : [ ]<br> },<br> “isUnique” : false,<br> “isSparse” : false,<br> “isPartial” : false,<br> “indexVersion” : 2,<br> “direction” : “forward”,<br> “indexBounds” : {<br> “userId” : [<br> “[\”59e022d33f239800129c61c7\”, \”59e022d33f239800129c61c7\”]”<br> ],<br> “status” : [<br> “[MinKey, MaxKey]”<br> ]<br> }<br> }<br> }<br> ]<br> },<br> “ok” : 1<br>}</p></blockquote><p><strong><em>留意到 DB 检测到 {userId:1} 为更优执行的方案，嗯~，如我们所料</em></strong></p><blockquote><p>db.loans.find({ “status” : “repayed” }).explain()<br>{<br> “queryPlanner” : {<br> “plannerVersion” : 1,<br> “namespace” : “cashLoan.loans”,<br> “indexFilterSet” : false,<br> “parsedQuery” : {<br> “status” : {<br> “$eq” : “repayed”<br> }<br> },<br> “queryHash” : “E6304EB6”,<br> “planCacheKey” : “7A94191B”,<br> “winningPlan” : {<br> “stage” : “COLLSCAN”,<br> “filter” : {<br> “status” : {<br> “$eq” : “repayed”<br> }<br> },<br> “direction” : “forward”<br> },<br> “rejectedPlans” : [ ]<br> },<br> “ok” : 1<br>}</p></blockquote><p><strong><em>有趣的部分： status 不命中索引， 全表扫描 </em></strong><br>接下来，我加了个sort </p><blockquote><p>db.loans.find({ “userId” : “59e022d33f239800129c61c7” }).sort({status:1}).explain()<br>{<br> “queryPlanner” : {<br> “plannerVersion” : 1,<br> “namespace” : “cashLoan.loans”,<br> “indexFilterSet” : false,<br> “parsedQuery” : {<br> “userId” : {<br> “$eq” : “59e022d33f239800129c61c7”<br> }<br> },<br> “queryHash” : “F5ABB1AA”,<br> “planCacheKey” : “764CBAA8”,<br> “winningPlan” : {<br> “stage” : “FETCH”,<br> “inputStage” : {<br> “stage” : “IXSCAN”,<br> “keyPattern” : {<br> “userId” : 1,<br> “status” : 1<br> },<br> “indexName” : “userId_1_status_1”,<br> “isMultiKey” : false,<br> “multiKeyPaths” : {<br> “userId” : [ ],<br> “status” : [ ]<br> },<br> “isUnique” : false,<br> “isSparse” : false,<br> “isPartial” : false,<br> “indexVersion” : 2,<br> “direction” : “forward”,<br> “indexBounds” : {<br> “userId” : [<br> “[\”59e022d33f239800129c61c7\”, \”59e022d33f239800129c61c7\”]”<br> ],<br> “status” : [<br> “[MinKey, MaxKey]”<br> ]<br> }<br> }<br> },<br> “rejectedPlans” : [<br> {<br> “stage” : “SORT”,<br> “sortPattern” : {<br> “status” : 1<br> },<br> “inputStage” : {<br> “stage” : “SORT_KEY_GENERATOR”,<br> “inputStage” : {<br> “stage” : “FETCH”,<br> “inputStage” : {<br> “stage” : “IXSCAN”,<br> “keyPattern” : {<br> “userId” : 1<br> },<br> “indexName” : “userId_1”,<br> “isMultiKey” : false,<br> “multiKeyPaths” : {<br> “userId” : [ ]<br> },<br> “isUnique” : false,<br> “isSparse” : false,<br> “isPartial” : false,<br> “indexVersion” : 2,<br> “direction” : “forward”,<br> “indexBounds” : {<br> “userId” : [<br> “[\”59e022d33f239800129c61c7\”, \”59e022d33f239800129c61c7\”]”<br> ]<br> }<br> }<br> }<br> }<br> }<br> ]<br> },<br> “ok” : 1<br>}</p></blockquote><p><strong><em>有趣的部分： status 不命中索引 </em></strong></p><blockquote><p>db.loans.find({ “status” : “repayed”,”userId” : “59e022d33f239800129c61c7”, }).explain()<br>{<br> “queryPlanner” : {<br> “plannerVersion” : 1,<br> “namespace” : “cashLoan.loans”,<br> “indexFilterSet” : false,<br> “parsedQuery” : {<br> “$and” : [<br> {<br> “status” : {<br> “$eq” : “repayed”<br> }<br> },<br> {<br> “userId” : {<br> “$eq” : “59e022d33f239800129c61c7”<br> }<br> }<br> ]<br> },<br> “queryHash” : “15D5A9A1”,<br> “planCacheKey” : “1B1A4861”,<br> “winningPlan” : {<br> “stage” : “FETCH”,<br> “inputStage” : {<br> “stage” : “IXSCAN”,<br> “keyPattern” : {<br> “userId” : 1,<br> “status” : 1<br> },<br> “indexName” : “userId_1_status_1”,<br> “isMultiKey” : false,<br> “multiKeyPaths” : {<br> “userId” : [ ],<br> “status” : [ ]<br> },<br> “isUnique” : false,<br> “isSparse” : false,<br> “isPartial” : false,<br> “indexVersion” : 2,<br> “direction” : “forward”,<br> “indexBounds” : {<br> “userId” : [<br> “[\”59e022d33f239800129c61c7\”, \”59e022d33f239800129c61c7\”]”<br> ],<br> “status” : [<br> “[\”repayed\”, \”repayed\”]”<br> ]<br> }<br> }<br> },<br> “rejectedPlans” : [<br> {<br> “stage” : “FETCH”,<br> “filter” : {<br> “status” : {<br> “$eq” : “repayed”<br> }<br> },<br> “inputStage” : {<br> “stage” : “IXSCAN”,<br> “keyPattern” : {<br> “userId” : 1<br> },<br> “indexName” : “userId_1”,<br> “isMultiKey” : false,<br> “multiKeyPaths” : {<br> “userId” : [ ]<br> },<br> “isUnique” : false,<br> “isSparse” : false,<br> “isPartial” : false,<br> “indexVersion” : 2,<br> “direction” : “forward”,<br> “indexBounds” : {<br> “userId” : [<br> “[\”59e022d33f239800129c61c7\”, \”59e022d33f239800129c61c7\”]”<br> ]<br> }<br> }<br> }<br> ]<br> },<br> “ok” : 1<br>}</p></blockquote><p><strong><em>命中索引， 跟 query 的各个字段顺序不相关，如我们猜测</em></strong></p><p><strong><em>有趣部分再来， 我们删掉索引{userId:1}</em></strong></p><blockquote><p>db.loans.dropIndex({“userId”:1})<br>{ “nIndexesWas” : 3, “ok” : 1 }</p></blockquote><blockquote><p>db.loans.find({“userId” : “59e022d33f239800129c61c7”, }).explain()<br>{<br> “queryPlanner” : {<br> “plannerVersion” : 1,<br> “namespace” : “cashLoan.loans”,<br> “indexFilterSet” : false,<br> “parsedQuery” : {<br> “userId” : {<br> “$eq” : “59e022d33f239800129c61c7”<br> }<br> },<br> “queryHash” : “B1777DBA”,<br> “planCacheKey” : “5776AB9C”,<br> “winningPlan” : {<br> “stage” : “FETCH”,<br> “inputStage” : {<br> “stage” : “IXSCAN”,<br> “keyPattern” : {<br> “userId” : 1,<br> “status” : 1<br> },<br> “indexName” : “userId_1_status_1”,<br> “isMultiKey” : false,<br> “multiKeyPaths” : {<br> “userId” : [ ],<br> “status” : [ ]<br> },<br> “isUnique” : false,<br> “isSparse” : false,<br> “isPartial” : false,<br> “indexVersion” : 2,<br> “direction” : “forward”,<br> “indexBounds” : {<br> “userId” : [<br> “[\”59e022d33f239800129c61c7\”, \”59e022d33f239800129c61c7\”]”<br> ],<br> “status” : [<br> “[MinKey, MaxKey]”<br> ]<br> }<br> }<br> },<br> “rejectedPlans” : [ ]<br> },<br> “ok” : 1<br>}</p></blockquote><p><strong><em>DB 执行分析器觉得索引{userId:1, status:1} 能更优</em></strong></p><p><strong><em>没有命中复合索引 ，这个是因为status 不是 leading field</em></strong></p><blockquote><p>db.loans.find({ “status” : “repayed” }).explain()<br>{<br> “queryPlanner” : {<br> “plannerVersion” : 1,<br> “namespace” : “cashLoan.loans”,<br> “indexFilterSet” : false,<br> “parsedQuery” : {<br> “status” : {<br> “$eq” : “repayed”<br> }<br> },<br> “queryHash” : “E6304EB6”,<br> “planCacheKey” : “7A94191B”,<br> “winningPlan” : {<br> “stage” : “COLLSCAN”,<br> “filter” : {<br> “status” : {<br> “$eq” : “repayed”<br> }<br> },<br> “direction” : “forward”<br> },<br> “rejectedPlans” : [ ]<br> },<br> “ok” : 1<br>}</p></blockquote><p><strong><em>再换个角度sort 一遍， 与前面query & sort 互换 ，之前是</em></strong></p><blockquote><p>db.loans.find({userId:1}).sort({ “status” : “repayed” })<br>看看有啥不一样？</p></blockquote><blockquote><p>db.loans.find({ “status” : “repayed” }).sort({userId:1}).explain()<br>{<br> “queryPlanner” : {<br> “plannerVersion” : 1,<br> “namespace” : “cashLoan.loans”,<br> “indexFilterSet” : false,<br> “parsedQuery” : {<br> “status” : {<br> “$eq” : “repayed”<br> }<br> },<br> “queryHash” : “56EA6313”,<br> “planCacheKey” : “2CFCDA7F”,<br> “winningPlan” : {<br> “stage” : “FETCH”,<br> “filter” : {<br> “status” : {<br> “$eq” : “repayed”<br> }<br> },<br> “inputStage” : {<br> “stage” : “IXSCAN”,<br> “keyPattern” : {<br> “userId” : 1,<br> “status” : 1<br> },<br> “indexName” : “userId_1_status_1”,<br> “isMultiKey” : false,<br> “multiKeyPaths” : {<br> “userId” : [ ],<br> “status” : [ ]<br> },<br> “isUnique” : false,<br> “isSparse” : false,<br> “isPartial” : false,<br> “indexVersion” : 2,<br> “direction” : “forward”,<br> “indexBounds” : {<br> “userId” : [<br> “[MinKey, MaxKey]”<br> ],<br> “status” : [<br> “[MinKey, MaxKey]”<br> ]<br> }<br> }<br> },<br> “rejectedPlans” : [ ]<br> },<br> “ok” : 1<br>}</p></blockquote><p><strong><em>如猜测，命中索引</em></strong></p><p>再来玩1玩，确认下<strong><em>leading filed</em></strong>试验：</p><blockquote><p>db.loans.dropIndex(“userId_1_status_1”)<br>{ “nIndexesWas” : 2, “ok” : 1 }</p></blockquote><blockquote><p>db.loans.getIndexes()<br>[<br> {<br> “v” : 2,<br> “key” : {<br> “_id” : 1<br> },<br> “name” : “_id_”,<br> “ns” : “cashLoan.loans”<br> }<br>]</p></blockquote><blockquote><p>db.loans.createIndex({status:1, userId:1})<br>{<br> “createdCollectionAutomatically” : false,<br> “numIndexesBefore” : 1,<br> “numIndexesAfter” : 2,<br> “ok” : 1<br>}</p></blockquote><blockquote><p>db.loans.getIndexes()<br>[<br> {<br> “v” : 2,<br> “key” : {<br> “_id” : 1<br> },<br> “name” : “_id_”,<br> “ns” : “cashLoan.loans”<br> },<br> {<br> “v” : 2,<br> “key” : {<br> “status” : 1,<br> “userId” : 1<br> },<br> “name” : “status_1_userId_1”,<br> “ns” : “cashLoan.loans”<br> }<br>]</p></blockquote><blockquote><p>db.loans.find({ “status” : “repayed” }).explain()<br>{<br> “queryPlanner” : {<br> “plannerVersion” : 1,<br> “namespace” : “cashLoan.loans”,<br> “indexFilterSet” : false,<br> “parsedQuery” : {<br> “status” : {<br> “$eq” : “repayed”<br> }<br> },<br> “queryHash” : “E6304EB6”,<br> “planCacheKey” : “7A94191B”,<br> “winningPlan” : {<br> “stage” : “FETCH”,<br> “inputStage” : {<br> “stage” : “IXSCAN”,<br> “keyPattern” : {<br> “status” : 1,<br> “userId” : 1<br> },<br> “indexName” : “status_1_userId_1”,<br> “isMultiKey” : false,<br> “multiKeyPaths” : {<br> “status” : [ ],<br> “userId” : [ ]<br> },<br> “isUnique” : false,<br> “isSparse” : false,<br> “isPartial” : false,<br> “indexVersion” : 2,<br> “direction” : “forward”,<br> “indexBounds” : {<br> “status” : [<br> “[\”repayed\”, \”repayed\”]”<br> ],<br> “userId” : [<br> “[MinKey, MaxKey]”<br> ]<br> }<br> }<br> },<br> “rejectedPlans” : [ ]<br> },<br> “ok” : 1<br>}</p></blockquote><blockquote><p>db.loans.getIndexes()<br>[<br> {<br> “v” : 2,<br> “key” : {<br> “_id” : 1<br> },<br> “name” : “_id_”,<br> “ns” : “cashLoan.loans”<br> },<br> {<br> “v” : 2,<br> “key” : {<br> “status” : 1,<br> “userId” : 1<br> },<br> “name” : “status_1_userId_1”,<br> “ns” : “cashLoan.loans”<br> }<br>]</p></blockquote><blockquote><p>db.loans.find({“userId” : “59e022d33f239800129c61c7”, }).explain()<br>{<br> “queryPlanner” : {<br> “plannerVersion” : 1,<br> “namespace” : “cashLoan.loans”,<br> “indexFilterSet” : false,<br> “parsedQuery” : {<br> “userId” : {<br> “$eq” : “59e022d33f239800129c61c7”<br> }<br> },<br> “queryHash” : “B1777DBA”,<br> “planCacheKey” : “5776AB9C”,<br> “winningPlan” : {<br> “stage” : “COLLSCAN”,<br> “filter” : {<br> “userId” : {<br> “$eq” : “59e022d33f239800129c61c7”<br> }<br> },<br> “direction” : “forward”<br> },<br> “rejectedPlans” : [ ]<br> },<br> “ok” : 1<br>}</p></blockquote>]]></content>
<summary type="html">
<p>根据典型碰到的场景，来做几个实验：<br>这里创建了个loans collection。简化只有100条数据。这个是借贷的表有 _id, userId, status(借贷状态), amount(金额). </p>
<p><strong><em>看完 这个实验后， 你会明白
</summary>
</entry>
<entry>
<title></title>
<link href="http://deng.io/2019/06/22/MongoDB-Best-Proactices-About-Index/"/>
<id>http://deng.io/2019/06/22/MongoDB-Best-Proactices-About-Index/</id>
<published>2019-06-22T15:31:17.686Z</published>
<updated>2019-09-10T15:11:21.129Z</updated>
<content type="html"><![CDATA[<h3 id="MongoDB-索引的最佳实践"><a href="#MongoDB-索引的最佳实践" class="headerlink" title="MongoDB 索引的最佳实践"></a>MongoDB 索引的最佳实践</h3><p>大部分开发者都知道加索引会快。但实际过程中，我们常碰到一些疑问&困难：</p><ul><li>我们查询的字段会各种case都有，是不是各个涉及查询的字段都要加索引？</li><li>复合索引和单字段怎么选择，都加还是每一个的单个字段就好了？</li><li>加索引有没有副作用？</li><li>索引都加了，但还是不够快？怎么办？</li></ul><p>本文尝试去解释索引的基本知识&解答上述的疑问。</p><h4 id="索引究竟是什么东西？"><a href="#索引究竟是什么东西？" class="headerlink" title="索引究竟是什么东西？"></a>索引究竟是什么东西？</h4><p>大部分开发者接触索引，大概知道索引类似书的目录，你要找到想要的内容，通过目录找到限定的关键字，进而找到对应的章节的pageno，再找到具体的内容。<br>在数据结构里面，最简单的索引实现类似hashmap，通过关键字key，映射到具体的位置，找到具体的内容。但除了hash的方式，还有多种的方式实现索引。</p><h4 id="索引的多种实现方式以及特色"><a href="#索引的多种实现方式以及特色" class="headerlink" title="索引的多种实现方式以及特色"></a>索引的多种实现方式以及特色</h4><p>hash / b-tree / b+-tree<br>redis HSET / MongoDB&PostgreSQL / MySQL </p><p>hashmap</p><p><img src="http://www.51code.com/uploads/allimg/190315/4_1453444561.jpg" alt="hashmap"></p><p>一图见b-tree & b+-tree 差别</p><p><img src="https://i.stack.imgur.com/l6UyF.png" alt="b-tree & b+-tree"></p><ul><li>b+-tree 叶子存数据，非叶子存索引，不存数据，叶子间有link</li><li>b-tree 非叶子可存数据</li></ul><p>算法查找复杂度上来说<br>hash 接近O(1)<br>b-tree O(1)~ O(Log(n))更快的平均查找时间，不稳定的查询时间<br>b+ tree O(Log(n)) 连续数据， 查询的稳定性</p><p>至于为何MongoDB 的实现选择b-tree 而非 b+-tree ？<br>网上多篇文章有阐述，非本文重点。</p><h4 id="数据-amp-索引的存储"><a href="#数据-amp-索引的存储" class="headerlink" title="数据&索引的存储"></a>数据&索引的存储</h4><p><img src="https://slideplayer.com/slide/13450455/80/images/18/Ensure+indexes+fit+in+RAM.jpg" alt="storage"><br>index 尽量存储在内存， data 其次。<br>注意只保留必要的index，内存尽量用在刀刃上。<br>如果index memory 都接近占满memory，那么就很容易读到disk，速度就下来了。</p><h4 id="知道索引的实现-amp-存储原理后的思考"><a href="#知道索引的实现-amp-存储原理后的思考" class="headerlink" title="知道索引的实现&存储原理后的思考"></a>知道索引的实现&存储原理后的思考</h4><p> insert/update/delete 会触发rebalance tree – 所以，增删改数据，索引会触发修改，性能会有损耗<br><strong><em>索引不是越多越好</em></strong><br>既然如此，选择哪些字段作为索引呢？当查询用到这些条件，怎么办？<br>拿一个最简单的hashmap来讲，为什么复杂度不是O(1)，而是所谓接近 O(1)。因为有key 冲突/重复啊~。DB 去找的时候，key 冲突的数据一大堆的话，还是得轮着继续找。<br>上面的b-tree 看键(key)的选择也是如此。<br>因此一个大部分开发经常犯的错就是<strong><em>对没有区分度的key 建索引</em></strong>。<br>例如：很多就只有集中类别的 type/status 的 documents count 达几十万以上的collection — 通常这种索引没什么帮助。</p><h4 id="复合索引"><a href="#复合索引" class="headerlink" title="复合索引"></a>复合索引</h4><p>复合索引不是越多越好<br>如果不想多建多余的索引，开发的同事在复合 & 单个字段 有时选择上挺纠结的。</p><p>根据典型碰到的场景，来做几个实验：<br>这里创建了个loans collection。简化只有100条数据。这个是借贷的表有 _id, userId, status(借贷状态), amount(金额). </p><h3 id="MongoDB-索引的最佳实践-1"><a href="#MongoDB-索引的最佳实践-1" class="headerlink" title="MongoDB 索引的最佳实践"></a>MongoDB 索引的最佳实践</h3><p>大部分开发者都知道加索引会快。但实际过程中，我们常碰到一些疑问&困难：</p><ul><li>我们查询的字段会各种case都有，是不是各个涉及查询的字段都要加索引？</li><li>复合索引和单字段怎么选择，都加还是每一个的单个字段就好了？</li><li>加索引有没有副作用？</li><li>索引都加了，但还是不够快？怎么办？</li></ul><p>本文尝试去解释索引的基本知识&解答上述的疑问。</p><h4 id="索引究竟是什么东西？-1"><a href="#索引究竟是什么东西？-1" class="headerlink" title="索引究竟是什么东西？"></a>索引究竟是什么东西？</h4><p>大部分开发者接触索引，大概知道索引类似书的目录，你要找到想要的内容，通过目录找到限定的关键字，进而找到对应的章节的pageno，再找到具体的内容。<br>在数据结构里面，最简单的索引实现类似hashmap，通过关键字key，映射到具体的位置，找到具体的内容。但除了hash的方式，还有多种的方式实现索引。</p><h4 id="索引的多种实现方式以及特色-1"><a href="#索引的多种实现方式以及特色-1" class="headerlink" title="索引的多种实现方式以及特色"></a>索引的多种实现方式以及特色</h4><p>hash / b-tree / b+-tree<br>redis HSET / MongoDB&PostgreSQL / MySQL </p><p>hashmap</p><p><img src="http://www.51code.com/uploads/allimg/190315/4_1453444561.jpg" alt="hashmap"></p><p>一图见b-tree & b+-tree 差别</p><p><img src="https://i.stack.imgur.com/l6UyF.png" alt="b-tree & b+-tree"></p><ul><li>b+-tree 叶子存数据，非叶子存索引，不存数据，叶子间有link</li><li>b-tree 非叶子可存数据</li></ul><p>算法查找复杂度上来说<br>hash 接近O(1)<br>b-tree O(1)~ O(Log(n))更快的平均查找时间，不稳定的查询时间<br>b+ tree O(Log(n)) 连续数据， 查询的稳定性</p><p>至于为何MongoDB 的实现选择b-tree 而非 b+-tree ？<br>网上多篇文章有阐述，非本文重点。</p><h4 id="数据-amp-索引的存储-1"><a href="#数据-amp-索引的存储-1" class="headerlink" title="数据&索引的存储"></a>数据&索引的存储</h4><p><img src="https://slideplayer.com/slide/13450455/80/images/18/Ensure+indexes+fit+in+RAM.jpg" alt="storage"><br>index 尽量存储在内存， data 其次。<br>注意只保留必要的index，内存尽量用在刀刃上。<br>如果index memory 都接近占满memory，那么就很容易读到disk，速度就下来了。</p><h4 id="知道索引的实现-amp-存储原理后的思考-1"><a href="#知道索引的实现-amp-存储原理后的思考-1" class="headerlink" title="知道索引的实现&存储原理后的思考"></a>知道索引的实现&存储原理后的思考</h4><p> insert/update/delete 会触发rebalance tree – 所以，增删改数据，索引会触发修改，性能会有损耗<br><strong><em>索引不是越多越好</em></strong><br>既然如此，选择哪些字段作为索引呢？当查询用到这些条件，怎么办？<br>拿一个最简单的hashmap来讲，为什么复杂度不是O(1)，而是所谓接近 O(1)。因为有key 冲突/重复啊~。DB 去找的时候，key 冲突的数据一大堆的话，还是得轮着继续找。<br>上面的b-tree 看键(key)的选择也是如此。<br>因此一个大部分开发经常犯的错就是<strong><em>对没有区分度的key 建索引</em></strong>。<br>例如：很多就只有集中类别的 type/status 的 documents count 达几十万以上的collection — 通常这种索引没什么帮助。</p><h4 id="复合索引-1"><a href="#复合索引-1" class="headerlink" title="复合索引"></a>复合索引</h4><p>复合索引不是越多越好<br>如果不想多建多余的索引，开发的同事在复合 & 单个字段 有时选择上挺纠结的。</p><p>根据典型碰到的场景，来做几个<a href="http://www.deng.io/index.php/2019/09/10/mongodb-compound-index/" target="_blank" rel="noopener">实验</a>：<br>这里创建了个loans collection。简化只有100条数据。这个是借贷的表有 _id, userId, status(借贷状态), amount(金额). </p><p><strong><em>看完 这个实验， 明白了 {userId:1, status:1}, vs {status:1,userId:1} 的差别了吗?</em></strong></p><p>PS：这个case 里面其实status 区分度不高，不应该建立的，这里只是作为实例展示。</p><p>总结：</p><ul><li>注意使用上 使用频率上 区分高的/常用的在前面</li><li>如果需要减少索引以节省memory/提高修改数据的性能的话，可以保留区分度高，常用的，去除区分度不高，不常用的索引。</li></ul><h3 id="学会用explain（）验证分析性能"><a href="#学会用explain（）验证分析性能" class="headerlink" title="学会用explain（）验证分析性能"></a>学会用explain（）验证分析性能</h3><p>DB 一般都有执行器优化的分析，MySQL & MongoDB 都是 用explain 来做分析。<br>语法上MySQL ：</p><blockquote><p>explain your_sql </p></blockquote><p>MongoDB:</p><blockquote><p>yoursql.explain()</p></blockquote><p> 至于怎么看，这里不展开，网上可以搜到各种文章。</p><h4 id="总结典型：理想的查询"><a href="#总结典型：理想的查询" class="headerlink" title="总结典型：理想的查询"></a>总结典型：理想的查询</h4><p>结合explain 的指标，他们通常是多个的混合</p><ul><li>IXSCAN : 索引命中</li><li>Limit : 带limit</li><li>Projection : 相当于非 select * </li><li>Docs Size less is better </li><li>Docs Examined less is better </li><li>nReturned=totalDocsExamined=totalKeysExamined </li><li>SORT in index ：sort 也是命中索引，否则，需要拿到数据后，再执行一遍排序</li><li>Limit Array elements ： 限定数组返回的条数，数组也不应该太多数据，否则schema 设计不合理</li></ul><p>文末，还有一个最开头一个问题没回答，如果我的索引改加的都加了，还不够快。怎么办？留个悬念，之后再写一篇。</p><p>PS：这个case 里面其实status 区分度不高，不应该建立的，这里只是作为实例展示。</p><p>总结：</p><ul><li>注意使用上 使用频率上 区分高的/常用的在前面</li><li>如果需要减少索引以节省memory/提高修改数据的性能的话，可以保留区分度高，常用的，去除区分度不高，不常用的索引。</li></ul><h3 id="学会用explain（）验证分析性能-1"><a href="#学会用explain（）验证分析性能-1" class="headerlink" title="学会用explain（）验证分析性能"></a>学会用explain（）验证分析性能</h3><p>DB 一般都有执行器优化的分析，MySQL & MongoDB 都是 用explain 来做分析。<br>语法上MySQL ：</p><blockquote><p>explain your_sql </p></blockquote><p>MongoDB:</p><blockquote><p>yoursql.explain()</p></blockquote><p> 至于怎么看，这里不展开，网上可以搜到各种文章。</p><h4 id="总结典型：理想的查询-1"><a href="#总结典型：理想的查询-1" class="headerlink" title="总结典型：理想的查询"></a>总结典型：理想的查询</h4><p>结合explain 的指标，他们通常是多个的混合</p><ul><li>IXSCAN : 索引命中</li><li>Limit : 带limit</li><li>Projection : 相当于非 select * </li><li>Docs Size less is better </li><li>Docs Examined less is better </li><li>nReturned=totalDocsExamined=totalKeysExamined </li><li>SORT in index ：sort 也是命中索引，否则，需要拿到数据后，再执行一遍排序</li><li>Limit Array elements ： 限定数组返回的条数，数组也不应该太多数据，否则schema 设计不合理</li></ul><p>文末，还有一个最开头一个问题没回答，如果我的索引改加的都加了，还不够快。怎么办？留个悬念，之后再写一篇。</p>]]></content>
<summary type="html">
<h3 id="MongoDB-索引的最佳实践"><a href="#MongoDB-索引的最佳实践" class="headerlink" title="MongoDB 索引的最佳实践"></a>MongoDB 索引的最佳实践</h3><p>大部分开发者都知道加索引会快。但实际
</summary>
</entry>
<entry>
<title></title>
<link href="http://deng.io/2019/06/10/network-error/"/>
<id>http://deng.io/2019/06/10/network-error/</id>
<published>2019-06-10T07:35:50.544Z</published>
<updated>2019-06-10T07:35:50.544Z</updated>
<content type="html"><</code></pre>]]></content>
<summary type="html">
<h3 id="common-network-error"><a href="#common-network-error" class="headerlink" title="common network error"></a>common network error</h3><
</summary>
</entry>
<entry>
<title></title>
<link href="http://deng.io/2019/04/18/mongo-transaction/"/>
<id>http://deng.io/2019/04/18/mongo-transaction/</id>
<published>2019-04-17T16:28:48.554Z</published>
<updated>2019-05-17T10:09:57.821Z</updated>
<content type="html"><![CDATA[<h3 id="mongodb-transaction"><a href="#mongodb-transaction" class="headerlink" title="mongodb transaction"></a>mongodb transaction</h3><h4 id="init-replset-mongodb-env"><a href="#init-replset-mongodb-env" class="headerlink" title="init replset mongodb env"></a>init replset mongodb env</h4><p>$mkdir db1 && mkdir db2 && mkdir db3<br>$mongod –port 27017 –dbpath ./db –replSet rstest1<br>$mongod –port 37017 –dbpath ./db2 –replSet rstest1<br>$mongod –port 47017 –dbpath ./db3 –replSet rstest1</p><p>$mongo –port 27017</p><p><code>rs.initiate( { _id : "rstest1", members: [ { _id: 0, host: "127.0.0.1:27017" }, { _id: 1, host: "127.0.0.1:37017" }, { _id: 1, host: "127.0.0.1:47017" }, ] })</code></p><figure class="highlight javascript"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">const</span> mongoose = <span class="built_in">require</span>(<span class="string">'mongoose'</span>)</span><br><span class="line"><span class="keyword">const</span> mongoUri = <span class="string">'mongodb://localhost:27017,localhost:37017,localhost:47017/test'</span></span><br><span class="line"><span class="keyword">const</span> client = mongoose.createConnection(mongoUri, { <span class="attr">replicaSet</span>: <span class="string">'rstest'</span> })</span><br><span class="line"><span class="keyword">let</span> Acc = client.model(<span class="string">'Account'</span>, <span class="keyword">new</span> mongoose.Schema({</span><br><span class="line"> name: <span class="built_in">String</span>,</span><br><span class="line"> balance: <span class="built_in">Number</span></span><br><span class="line">}))</span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">function</span> <span class="title">initTest</span>(<span class="params"></span>) </span>{</span><br><span class="line"> <span class="keyword">await</span> Acc.create({ <span class="string">'name'</span>: <span class="string">'James'</span>, <span class="string">'balance'</span>: <span class="number">3000</span> })</span><br><span class="line"> <span class="keyword">await</span> Acc.create({ <span class="string">'name'</span>: <span class="string">'Wade'</span>, <span class="string">'balance'</span>: <span class="number">0</span> })</span><br><span class="line">}</span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">function</span> <span class="title">afterWork</span>(<span class="params"></span>) </span>{</span><br><span class="line"> <span class="keyword">await</span> Acc.remove({ <span class="string">'name'</span>: <span class="string">'James'</span> })</span><br><span class="line"> <span class="keyword">await</span> Acc.remove({ <span class="string">'name'</span>: <span class="string">'Wade'</span> })</span><br><span class="line">}</span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">function</span> <span class="title">transferTest</span>(<span class="params">transfer</span>) </span>{</span><br><span class="line"> <span class="keyword">let</span> session = <span class="keyword">await</span> Acc.startSession()</span><br><span class="line"> session.startTransaction()</span><br><span class="line"> <span class="keyword">try</span> {</span><br><span class="line"> <span class="keyword">const</span> opts = { session, <span class="attr">new</span>: <span class="literal">true</span> }</span><br><span class="line"> <span class="keyword">let</span> a = <span class="keyword">await</span> Acc.findOneAndUpdate({</span><br><span class="line"> <span class="string">'name'</span>: <span class="string">'James'</span></span><br><span class="line"> }, {</span><br><span class="line"> $inc: { <span class="string">'balance'</span>: -transfer }</span><br><span class="line"> }, opts)</span><br><span class="line"> <span class="built_in">console</span>.log(a.toObject());</span><br><span class="line"> <span class="keyword">let</span> b = <span class="keyword">await</span> Acc.findOneAndUpdate({</span><br><span class="line"> <span class="string">'name'</span>: <span class="string">'Wade'</span>,</span><br><span class="line"> }, {</span><br><span class="line"> $inc: { <span class="string">'balance'</span>: transfer }</span><br><span class="line"> }, opts)</span><br><span class="line"> <span class="built_in">console</span>.log(b.toObject());</span><br><span class="line"> <span class="keyword">await</span> session.commitTransaction()</span><br><span class="line"> } <span class="keyword">catch</span> (err) {</span><br><span class="line"> session.abortTransaction()</span><br><span class="line"> <span class="built_in">console</span>.error(err)</span><br><span class="line"> } <span class="keyword">finally</span> {</span><br><span class="line"> session.endSession()</span><br><span class="line"> }</span><br><span class="line"></span><br><span class="line">}</span><br><span class="line"><span class="keyword">async</span> <span class="function"><span class="keyword">function</span> <span class="title">test</span>(<span class="params"></span>) </span>{</span><br><span class="line"> <span class="keyword">await</span> afterWork()</span><br><span class="line"> <span class="keyword">await</span> initTest()</span><br><span class="line"> <span class="keyword">await</span> transferTest(<span class="number">100</span>)</span><br><span class="line"> <span class="comment">// await afterWork()</span></span><br><span class="line">}</span><br><span class="line">test()</span><br></pre></td></tr></table></figure><p>set a debug point or sleep point at<br><figure class="highlight css"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="selector-tag">console</span><span class="selector-class">.log</span>(<span class="selector-tag">a</span><span class="selector-class">.toObject</span>());</span><br></pre></td></tr></table></figure></p><p>to stop the process<br>then run code to read them<br>`<br> { “_id” : ObjectId(“5cb752de57e8791edf030e9d”), “name” : “James”, “balance” : 3000, “<strong>v” : 0 }<br> { “_id” : ObjectId(“5cb752de57e8791edf030e9e”), “name” : “Wade”, “balance” : 0, “</strong>v” : 0 }</p><p>`<br>it works fine</p><p>then run :<br>`<br> rstest:PRIMARY> db.accounts.update({“name”:”Wade”}, { $inc:{‘balance’:10}})<br>WriteResult({ “nMatched” : 1, “nUpserted” : 0, “nModified” : 1 })</p><p>`</p><p>this will return immediately, when the debug point continue, it ran into exception</p><p><code>null: MongoError: WriteConflict MongoError: Attempted illegal state transition from [TRANSACTION_ABORTED] to [TRANSACTION_ABORTED] ....</code></p><h3 id="PAY-attention"><a href="#PAY-attention" class="headerlink" title="PAY attention"></a>PAY attention</h3><p><img src="https://cdn-images-1.medium.com/max/1600/1*F_a3X-box50qZOllKb-D1g.jpeg" alt="write concern & read concern"></p><p>In MongoDB, clients can see the results of writes before the writes are durable:<br>clients using “local” <strong><em>by default</em></strong> can see the result of a write operation <strong><em>before</em></strong> the write operation is acknowledged to the issuing client.<br><a href="https://docs.mongodb.com/v3.2/core/read-isolation-consistency-recency/" target="_blank" rel="noopener">read isolation</a></p><p>Thus , in the case of we need to <strong><em>read own writes</em></strong> we need to read concern set “majority”, wirte concern “majority”</p><h4 id="how-to-config-this"><a href="#how-to-config-this" class="headerlink" title="how to config this"></a>how to config this</h4><p><a href="https://docs.mongodb.com/manual/reference/read-concern/" target="_blank" rel="noopener">further info read-concern</a></p><p><a href="https://docs.mongodb.com/manual/reference/read-concern-majority/#read-your-own-writes" target="_blank" rel="noopener">read-your-own-writes</a></p><p>read concern : you can disable read concern by setting either:</p><ul><li>– enableMajorityReadConcern command line option to false.</li><li>replication.enableMajorityReadConcern configuration file setting to false</li></ul><p>write concern : your code write-concern option, example :</p><figure class="highlight yaml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"> <span class="string">const</span> <span class="string">schema</span> <span class="string">=</span> <span class="string">new</span> <span class="string">Schema({</span> <span class="attr">name:</span> <span class="string">String</span> <span class="string">},</span> <span class="string">{</span></span><br><span class="line"> <span class="attr">writeConcern:</span> <span class="string">{</span></span><br><span class="line"> <span class="attr">w:</span> <span class="string">'majority'</span><span class="string">,</span></span><br><span class="line"> <span class="attr">j:</span> <span class="literal">true</span></span><br><span class="line"> <span class="string">}</span></span><br><span class="line"><span class="string">});</span></span><br></pre></td></tr></table></figure><p><a href="https://medium.com/@sj82516/mongodb-isolation-%E8%88%87-transaction-132ab29731c2" target="_blank" rel="noopener">more ref</a></p>]]></content>
<summary type="html">
<h3 id="mongodb-transaction"><a href="#mongodb-transaction" class="headerlink" title="mongodb transaction"></a>mongodb transaction</h3><h4 i
</summary>
</entry>
<entry>
<title></title>
<link href="http://deng.io/2019/03/14/learning%20CFA%20as%20software%20developer/"/>
<id>http://deng.io/2019/03/14/learning CFA as software developer/</id>
<published>2019-03-14T08:26:03.475Z</published>
<updated>2019-04-10T04:14:20.410Z</updated>
<content type="html"><![CDATA[<h3 id="learning-CFA-as-software-developer"><a href="#learning-CFA-as-software-developer" class="headerlink" title="learning CFA as software developer"></a>learning CFA as software developer</h3><p>irr<br>pv<br>Unrealized Gains and Losses<br>收益率return rate<br><img src="https://wade-blog.oss-cn-shenzhen.aliyuncs.com/Screen%20Shot%202019-04-10%20at%2012.12.33%20PM.png" alt="sample"><br>应计/应收<br>清分Clearing/清算Settlement/结算Settlement of Accounts </p><p>清分 Clearing对交易数据依据机构和交易类型进行分类汇总，并计算结算金额的过程。<br>清算 Settlement指根据清分结果对交易数据进行净额轧差和提交并完成资金划拨的全过程。<br>结算 Settlement of Accounts指完成客户账户间资金划拨的全过程。</p><p>清分=记账<br>清算=算账<br>结算=转账</p><p><img src="https://www.zhihu.com/question/19892912" alt="ref"><br><img src="https://wade-blog.oss-cn-shenzhen.aliyuncs.com/Screen%20Shot%202019-04-10%20at%2012.01.58%20PM.png" alt="example"></p>]]></content>
<summary type="html">
<h3 id="learning-CFA-as-software-developer"><a href="#learning-CFA-as-software-developer" class="headerlink" title="learning CFA as software
</summary>
</entry>
<entry>
<title></title>
<link href="http://deng.io/2019/03/14/exchange%20trading%20system%20design/"/>
<id>http://deng.io/2019/03/14/exchange trading system design/</id>
<published>2019-03-14T07:23:24.715Z</published>
<updated>2019-03-21T03:44:00.718Z</updated>
<content type="html"><![CDATA[<h3 id="exchange-trading-system-design-交易所系统设计"><a href="#exchange-trading-system-design-交易所系统设计" class="headerlink" title="exchange trading system design 交易所系统设计"></a>exchange trading system design 交易所系统设计</h3><h4 id="model关系"><a href="#model关系" class="headerlink" title="model关系"></a>model关系</h4><p>用户 1 –》n 账户 account<br>账户 1 –》n 订单 order<br>订单 1 –》n 账单流水ledger</p><h4 id="model设计关键"><a href="#model设计关键" class="headerlink" title="model设计关键"></a>model设计关键</h4><p>订单order:<br>type 类型 :</p><ul><li>market order</li><li>limit order<br>方向 side:</li><li>buy </li><li>sell<br>ID:</li><li>订单id order_id</li><li>用户设置的订单ID（系统foreign key to join）client_id<br>状态 status：<ul><li>submitted（始态）</li><li>open</li><li>filled（终态）</li><li>partial-filled</li><li>canceled（终态）</li><li>partial-canceled（终态）</li></ul></li></ul><p>账单流水ledger：</p><ul><li>balance 余额</li><li>order_id 关联foreign key</li><li>amount 变动的量</li><li>currency 币种(数字货币&外汇比较重要)</li><li>type 流水来源/类型<ul><li>trade 交易</li><li>transfer 转账</li><li>fee 手续费</li><li>rebate 返佣</li></ul></li></ul><p>账户</p><ul><li>ID 账户ID</li><li>balance 余额</li><li>hold 冻结</li><li>available 可用余额</li><li>currency 币种</li><li>type/name 用于区分用途</li></ul><p>钱包账户– 可用理解成总账户,与对外的流水入口。譬如充值、提现关联.<br>参考 okex 账户体系<br> <img src="https://support.okex.com/hc/article_attachments/360002553091/mceclip2.png" alt></p><p>每次的流水都记录当前的balance</p><p>holding:持仓，指数量(股数)，跟market price不挂钩</p><p>portfolio</p><p>product & instrument</p><h4 id="rest-api-设计"><a href="#rest-api-设计" class="headerlink" title="rest api 设计"></a>rest api 设计</h4><ul><li>version control v1 v2 v3 in url<br>2018-09-12T07:46:19.435ZGET/api/spot/v3/accounts/btc/ledger?limit=1&from=2&to=4 check <best practices for designing web api></best></li><li></li></ul><p>注意所有的数字在传输的时候都是 string，防止数据在加密过程中容易出错，并保留固定位数。（防止 2.00 与 2.000 有差别）</p><h4 id="performance-metric"><a href="#performance-metric" class="headerlink" title="performance metric"></a>performance metric</h4><ul><li>return</li><li>exposure</li><li>unrealized return</li></ul><h4 id="错误码"><a href="#错误码" class="headerlink" title="错误码"></a>错误码</h4><h4 id="安全"><a href="#安全" class="headerlink" title="安全"></a>安全</h4><h4 id="交易撮合"><a href="#交易撮合" class="headerlink" title="交易撮合"></a>交易撮合</h4><p>关于时间 UTC String(Thu, 14 Mar 2019 08:06:39 GMT) vs timestamp (1552550799091) vs ISODate (2019-03-14T08:06:39.091Z)<br>建议 ISODate</p><ul><li>优点：比较直观的是ISODate , 通用，便于时区识别 </li><li>缺点：排序稍差于 timestamp</li></ul><h4 id="币币交易"><a href="#币币交易" class="headerlink" title="币币交易"></a>币币交易</h4><h4 id="期货-合约交易"><a href="#期货-合约交易" class="headerlink" title="期货/合约交易"></a>期货/合约交易</h4><h3 id="数据"><a href="#数据" class="headerlink" title="数据"></a>数据</h3><h4 id="数据存储："><a href="#数据存储：" class="headerlink" title="数据存储："></a>数据存储：</h4><ul><li>区分热数据、冷数据<ul><li>热数据：最近的、要实时快的</li><li>冷数据：稍久的，允许稍长一点的时延</li></ul></li></ul><h4 id="kline-的数据存储"><a href="#kline-的数据存储" class="headerlink" title="kline 的数据存储"></a>kline 的数据存储</h4><ul><li>按频率区分开表格存 </li><li>不用筛选，快速</li></ul><h4 id="数据真的太多了怎么办"><a href="#数据真的太多了怎么办" class="headerlink" title="数据真的太多了怎么办"></a>数据真的太多了怎么办</h4><ul><li>某日清算正确后，可以将balance 等关键信息汇总至某日，作为snapshot。以后基于这个数据去计算</li><li>某日清算正确前的数据，把数据存到冷存储，如磁带机(多份)</li></ul><h4 id="架构"><a href="#架构" class="headerlink" title="架构"></a>架构</h4><p>Kong 分发不同的交易币种种,不同的交易产品(期货合约/币币交易/法币/其他API)</p><p>`<br> $ curl -I <a href="https://www.okex.com" target="_blank" rel="noopener">https://www.okex.com</a><br> HTTP/1.1 200 Tunnel established</p><p> HTTP/1.1 200<br> Content-Type: text/html;charset=UTF-8<br> Content-Length: 45825<br> Connection: keep-alive<br> Vary: Accept-Encoding<br> X-RateLimit-Limit-second: 50<br> X-RateLimit-Remaining-second: 49<br> X-RateLimit-Limit-minute: 100<br> X-RateLimit-Remaining-minute: 99<br> X-Content-Type-Options: nosniff<br> X-XSS-Protection: 1; mode=block<br> Cache-Control: no-cache, no-store, max-age=0, must-revalidate<br> Pragma: no-cache<br> Expires: 0<br> X-Frame-Options: SAMEORIGIN<br> Set-Cookie: locale=en_US; Max-Age=604800; Expires=Thu, 28-Mar-2019 03:39:44 GMT; Path=/<br> Content-Language: en-US<br> Date: Thu, 21 Mar 2019 03:39:44 GMT<br> X-Kong-Upstream-Latency: 13<br> X-Kong-Proxy-Latency: 0<br> Via: kong/0.12.3</p><p>`</p><p>容量估计：</p><ul><li>带宽</li><li>数据</li><li>系统压力</li></ul><p>关键子系统概述:</p><ul><li>行情系统</li><li>交易系统</li><li>风控系统</li><li>营销系统</li></ul><p>量化策略系统</p><ul><li>行情系统</li><li>交易系统</li><li>风控系统</li><li>策略系统</li></ul><p>ref :</p><p><a href="https://www.okex.com/docs/zh" target="_blank" rel="noopener">okex api</a></p><p><a href="https://github.com/huobiapi/API_Docs/wiki/REST_api_reference" target="_blank" rel="noopener">huobi api</a></p>]]></content>
<summary type="html">
<h3 id="exchange-trading-system-design-交易所系统设计"><a href="#exchange-trading-system-design-交易所系统设计" class="headerlink" title="exchange trading
</summary>
</entry>
<entry>
<title></title>
<link href="http://deng.io/2019/03/10/ipfs/"/>
<id>http://deng.io/2019/03/10/ipfs/</id>
<published>2019-03-10T13:48:07.419Z</published>
<updated>2019-03-10T14:03:23.836Z</updated>
<content type="html"><</code></pre><p><a href="http://localhost:5001/webui" target="_blank" rel="noopener">http://localhost:5001/webui</a></p><p>peers</p><p> <img src="http://wade-blog.oss-cn-shenzhen.aliyuncs.com/fdfc58ab-4b9c-4eb2-8373-f9fad8e63547.png" alt="peers"></p><p>耗费网络的ipfs 节点<br><img src="http://wade-blog.oss-cn-shenzhen.aliyuncs.com/9bf33608-fc78-4cd1-bc9f-db1f7db6c09b.png" alt="network-usage"></p><h4 id="基本操作："><a href="#基本操作：" class="headerlink" title="基本操作："></a>基本操作：</h4><ul><li>添加文件：<ul><li>ipfs add file path</li><li><img src="http://wade-blog.oss-cn-shenzhen.aliyuncs.com/da7539c4-f792-4c10-9200-6666be4346b5.png" alt="add file"></li></ul></li><li>获取<ul><li>ipfs cat /ipfs/hash </li><li>ipfs cat /ipfs/QmVg7JXmiZy8YRXMKS2VvmXzmPAgRo7KQvajsuVjtGfcB3 > 1.pdf</li><li>web 方式</li><li><img src="http://wade-blog.oss-cn-shenzhen.aliyuncs.com/6570d24c-0fe9-4e52-938e-b028493f0671.png" alt="web"></li><li>wget <a href="https://ipfs.io/ipfs/QmciDozPmgVpRKNuuGtvT72o1BExSKE7SWFWczvyMfmM4d" target="_blank" rel="noopener">https://ipfs.io/ipfs/QmciDozPmgVpRKNuuGtvT72o1BExSKE7SWFWczvyMfmM4d</a></li><li><img src="http://wade-blog.oss-cn-shenzhen.aliyuncs.com/f6cede87-644d-42b4-8419-acbfb77ebff2.png" alt="wget"></li></ul></li></ul><h4 id="空间问题"><a href="#空间问题" class="headerlink" title="空间问题"></a>空间问题</h4><ul><li>pin</li><li>那么删除呢？<ul><li><a href="https://github.com/ipfs/faq/issues/9" target="_blank" rel="noopener">https://github.com/ipfs/faq/issues/9</a> </li><li><a href="https://discuss.ipfs.io/t/how-can-i-delete-a-file-from-ipfs/1556" target="_blank" rel="noopener">https://discuss.ipfs.io/t/how-can-i-delete-a-file-from-ipfs/1556</a> <a href="https://stackoverflow.com/questions/43118022/how-do-i-unpin-and-remove-all-ipfs-content-from-my-machine" target="_blank" rel="noopener">https://stackoverflow.com/questions/43118022/how-do-i-unpin-and-remove-all-ipfs-content-from-my-machine</a></li><li>ipfs pin rm $YOUR_HASH</li><li>ipfs repo gc</li><li>你只能删你本地的数据，你关不了别的节点，尤其是别的节点的pin 数据<br><img src="http://wade-blog.oss-cn-shenzhen.aliyuncs.com/5944a978-0b39-4762-a81f-d0fad6920358.png" alt="rm"></li></ul></li></ul><h5 id="GC"><a href="#GC" class="headerlink" title="GC"></a>GC</h5><p>GC打出的log ，这里面包含其他的节点的别人的数据，我还是能在本地看到这些数据，<br><img src="http://wade-blog.oss-cn-shenzhen.aliyuncs.com/b4e0d583-a8e1-499f-9ad7-5b2c9cacda22.png" alt="gc"><br>递归形：除了直接的内容访问外，还有一种特殊的，recursive<br><img src="http://wade-blog.oss-cn-shenzhen.aliyuncs.com/a581b227-b6ed-46e4-a35e-7d69b0035649.png" alt="recursive"></p><p>担心硬盘爆：</p><ul><li>config 路径: ~/.ipfs/config </li><li>Datastore.StorageMax 默认10GB 限制max storage</li><li>StorageGCWatermark:90 90%存储空间使用了的时候触发GC</li></ul><p>因为存储空间有限，不是每个人都自愿无偿的贡献存储空间的，所以数据不保证永久存储，想保证永久，要pin serivce<br>（<a href="https://docs.ipfs.io/guides/concepts/pinning/）" target="_blank" rel="noopener">https://docs.ipfs.io/guides/concepts/pinning/）</a> ，以避免重要的存储数据被触发delete 掉。</p><p>进一步了解关于pin机制： <a href="https://discuss.ipfs.io/t/trying-to-better-understand-the-pinning-concept/754/2" target="_blank" rel="noopener">https://discuss.ipfs.io/t/trying-to-better-understand-the-pinning-concept/754/2</a><br>总结：</p><ul><li>pin的内容会告诉自己的节点不要进入GC 删除他</li><li>对pin的操作控制不同步到其他节点，其他人的节点爱咋地咋地</li><li>自己的节点add的，会自动pin （Objects added through ipfs add are pinned recursively by default.）</li><li>如果都pin了，应该是根据访问量、陈旧度来决定GC</li></ul><h4 id="ipfs-vs-BitTorrent-几大区别点"><a href="#ipfs-vs-BitTorrent-几大区别点" class="headerlink" title="ipfs vs BitTorrent 几大区别点"></a>ipfs vs BitTorrent 几大区别点</h4><ul><li>BT 依赖的是torrent 文件， ipfs 仅通过hash 找到文件</li><li>ipfs 不仅仅支持文件的下载，日后是一种完善的文件协议</li><li>参考git支持，通过hash 避免重复数据的存储，区分不同的版本 </li><li>对于web 应用来说，ipfs 是一种cache system<br><a href="https://medium.com/@kidinamoto/ipfs-vs-bittorrent-9f1c3adb8fcd" target="_blank" rel="noopener">https://medium.com/@kidinamoto/ipfs-vs-bittorrent-9f1c3adb8fcd</a><br><a href="https://github.com/ipfs/notes/issues/208" target="_blank" rel="noopener">https://github.com/ipfs/notes/issues/208</a><br><a href="https://github.com/ipfs/faq/issues/17" target="_blank" rel="noopener">https://github.com/ipfs/faq/issues/17</a></li></ul><h4 id="suck-part"><a href="#suck-part" class="headerlink" title="suck part:"></a>suck part:</h4><ul><li>ipfs add file 后， 后缀都会丢失</li><li>有时 ，下载很慢 slow / cannot download </li><li>$ wget <a href="https://ipfs.io/ipfs/QmYx4BnhnLXeMWF5mKu16fJgUBiVP7ECXh7qcsUZnXiRxc" target="_blank" rel="noopener">https://ipfs.io/ipfs/QmYx4BnhnLXeMWF5mKu16fJgUBiVP7ECXh7qcsUZnXiRxc</a><br>–2018-10-22 12:15:00– <a href="https://ipfs.io/ipfs/QmYx4BnhnLXeMWF5mKu16fJgUBiVP7ECXh7qcsUZnXiRxc" target="_blank" rel="noopener">https://ipfs.io/ipfs/QmYx4BnhnLXeMWF5mKu16fJgUBiVP7ECXh7qcsUZnXiRxc</a><br>Resolving ipfs.io (ipfs.io)… 209.94.78.81, 209.94.90.1, 209.94.78.80, …<br>Connecting to ipfs.io (ipfs.io)|209.94.78.81|:443… connected.<br>HTTP request sent, awaiting response…<br>Read error (Connection timed out) in headers.<br>Retrying.</li></ul><p>–2018-10-22 12:30:01– (try: 2) <a href="https://ipfs.io/ipfs/QmYx4BnhnLXeMWF5mKu16fJgUBiVP7ECXh7qcsUZnXiRxc" target="_blank" rel="noopener">https://ipfs.io/ipfs/QmYx4BnhnLXeMWF5mKu16fJgUBiVP7ECXh7qcsUZnXiRxc</a><br>Connecting to ipfs.io (ipfs.io)|209.94.78.81|:443… connected.<br>HTTP request sent, awaiting response…</p><p> 但 official资源比较快，如：<br><a href="https://ipfs.io/ipfs/QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG" target="_blank" rel="noopener">https://ipfs.io/ipfs/QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG</a> </p><ul><li>猜想：可能需要与是否永久存储的问题相关</li><li>你关闭了节点daemon 后一段时间(1天以上)，再访问自己的资源，发现不能访问；启动回daemon，1分钟内又可以访问了；再紧接着断开daemon ，disable local cache of chrome ，也还是能访问;后续又停了后，又不能访问了。 — 应该是ipfs.io 帮助cache 了，但没多少节点真正原意pin 并记录，可能要不在search，要不就找不到记录的节点。— 这个部分要深入看代码才知道（可能相关访问频率、访问时间的cache 机制在控制）。</li><li>相对于传统cdn无权限访问控制</li></ul><h4 id="其他"><a href="#其他" class="headerlink" title="其他"></a>其他</h4><p>注意通过浏览器获取回来还是经过了ipfs.io 的服务器，背后应该是经过gateway ，然后通过ipfs node 获取回来</p><p>more need to dive into：</p><ul><li>public key usage </li><li>对于可变内容， 参考：<a href="https://docs.ipfs.io/guides/concepts/ipns/" target="_blank" rel="noopener">https://docs.ipfs.io/guides/concepts/ipns/</a> </li><li>IPFS 未来要支持Namecoin ，那么代表传统DNS、ICANN 在网络中的工作角色会被干掉/替换。</li><li>FileCoin 对比<ul><li>Storj</li><li>Fcoin</li><li>Ulord</li><li>Burst</li></ul></li></ul><h3 id="FAQ-："><a href="#FAQ-：" class="headerlink" title="FAQ ："></a>FAQ ：</h3><p>是否add file 就永久存储：</p><ul><li>content storage not forever , then who decide long persistence –> Filecoin <a href="http://www.infoq.com/cn/articles/how-ipfs-is-disrupting-the-web" target="_blank" rel="noopener">http://www.infoq.com/cn/articles/how-ipfs-is-disrupting-the-web</a> </li><li>IPFS doesn’t solve the persistence problem for you, the only way currently to ensure that your files will exist is to pin them on an IPFS node, which means you need pin rights on that node. Run your own node, there are a few services out there that you can pay to pin content, find a node that will volunteer to pin your content, or wait for Filecoin which solves the problem by allowing you to pay Filecoin for persistence.</li></ul><p>激励机制：</p><ul><li>符合存储证明的获得token奖励<ul><li>反欺诈机制（防止只存一段时间就删）：<ul><li>隔断时间检查文件是否存在<ul><li>太频繁—导致消耗资源</li><li>太少频率–导致欺诈概率上升</li><li>Ref：<br><a href="http://liyuechun.org/2017/09/18/ipfs-blockchain/" target="_blank" rel="noopener">http://liyuechun.org/2017/09/18/ipfs-blockchain/</a></li></ul></li></ul></li></ul></li></ul>]]></content>
<summary type="html">
<h3 id="IPFS-探索"><a href="#IPFS-探索" class="headerlink" title="IPFS 探索"></a>IPFS 探索</h3><p>比特币当前是用于存金融交易数据，有leveldb 存关键小的交易数据。那么我们的文件，譬如一个网站里
</summary>
</entry>
<entry>
<title></title>
<link href="http://deng.io/2019/03/05/consul/"/>
<id>http://deng.io/2019/03/05/consul/</id>
<published>2019-03-05T02:31:51.283Z</published>
<updated>2019-03-25T02:01:24.146Z</updated>
<content type="html"><![CDATA[<h3 id="consul-amp-registrator-amp-consul-template-使用"><a href="#consul-amp-registrator-amp-consul-template-使用" class="headerlink" title="consul & registrator & consul-template 使用"></a>consul & registrator & consul-template 使用</h3><p>参考这里的文章：<br><a href="https://www.jianshu.com/p/a4c04a3eeb57" target="_blank" rel="noopener">https://www.jianshu.com/p/a4c04a3eeb57</a></p><p>docker-compose.yml</p><figure class="highlight"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br></pre></td><td class="code"><pre><span class="line"><span class="attribute">version</span>: '3'</span><br><span class="line"></span><br><span class="line"><span class="attribute">services:</span></span><br><span class="line"> web:</span><br><span class="line"> image: liberalman/helloworld:latest</span><br><span class="line"> environment:</span><br><span class="line"> SERVICE_80_NAME: my-web-server</span><br><span class="line"> SERVICE_TAGS: backend-1</span><br><span class="line"> MY_HOST: host-1</span><br><span class="line"> ports:</span><br><span class="line"> - "80"</span><br><span class="line"></span><br><span class="line"> lb:</span><br><span class="line"> image: liberalman/nginx-consul-template:latest</span><br><span class="line"> hostname: lb</span><br><span class="line"> links:</span><br><span class="line"> - consulserver:consul</span><br><span class="line"> ports:</span><br><span class="line"> - "80:80"</span><br><span class="line"></span><br><span class="line"> consulserver:</span><br><span class="line"> image: progrium/consul:latest</span><br><span class="line"> environment:</span><br><span class="line"> SERVICE_TAGS: consul servers</span><br><span class="line"> hostname: consulserver</span><br><span class="line"> ports:</span><br><span class="line"> - "8300"</span><br><span class="line"> - "8400"</span><br><span class="line"> - "8500:8500"</span><br><span class="line"> - "53"</span><br><span class="line"> command: -server -ui-dir /ui -data-dir /tmp/consul -bootstrap-expect 1</span><br><span class="line"></span><br><span class="line"> registrator:</span><br><span class="line"> image: gliderlabs/registrator:master</span><br><span class="line"> hostname: registrator</span><br><span class="line"> links:</span><br><span class="line"> - consulserver:consul</span><br><span class="line"> volumes:</span><br><span class="line"> - "/var/run/docker.sock:/tmp/docker.sock"</span><br><span class="line"> command: -internal consul://consul:8500</span><br></pre></td></tr></table></figure><p><code>docker-compose up</code></p><p><code>docker-compose up --scale web=3</code><br><img src="https://wade-blog.oss-cn-shenzhen.aliyuncs.com/nginx.gif" alt="执行效果"></p><p>停掉其中一个container ，看看发生什么事情导致能够检测到节点挂掉，并不转发流量？<br>关键处：订阅了状态的结果后，动态改变了nginx的配置。效果：</p><p><img src="https://wade-blog.oss-cn-shenzhen.aliyuncs.com/consule.gif" alt="template"></p><p>单机的架构、原理如下：<br><img src="http://wade-blog.oss-cn-shenzhen.aliyuncs.com/1551845679501.png" alt="Alt text"><br>or 参考：<br><img src="http://wade-blog.oss-cn-shenzhen.aliyuncs.com/1551846959382.png" alt="Alt text"></p><p>consul 只是其中一种注册中心的实现，registrator support zk&consule/etcd/skydns2 </p><h3 id="registrator-主动检查"><a href="#registrator-主动检查" class="headerlink" title="registrator 主动检查"></a>registrator 主动检查</h3><p><a href="https://github.com/gliderlabs/registrator/blob/master/registrator.go" target="_blank" rel="noopener">https://github.com/gliderlabs/registrator/blob/master/registrator.go</a><br>根据tick 的到来，registrator.go 做几件事情：</p><p><img src="http://wade-blog.oss-cn-shenzhen.aliyuncs.com/1551840835089.png" alt="registrator.go"></p><ul><li>resync interval 到达，扫描， 同步容器状态 refresh ：<ul><li>alive 的container，</li><li>delete 到consul </li></ul></li><li>refresh interval 到达，sync :<ul><li>list 当前的container</li><li>移除不健康的container</li><li>deregister 不健康的container</li></ul></li></ul><p> 看到这里，这种基于registrator的主动检测，解耦服务于注册中心的代码</p><p><a href="https://medium.com/eleven-labs/consul-service-discovery-and-failure-detection-64b06a5cbce6" target="_blank" rel="noopener">ref</a></p><h3 id="分布式的配置"><a href="#分布式的配置" class="headerlink" title="分布式的配置"></a>分布式的配置</h3><p>每个宿主机都要配置 registrator ，因为只能同一台宿主机去扫描container 的状态，注意红框处：</p><p><img src="http://wade-blog.oss-cn-shenzhen.aliyuncs.com/1551845649823.png" alt="Alt text"></p><p>集群的架构</p><p><img src="https://www.consul.io/assets/images/consul-arch-420ce04a.png" alt="architecture"></p><h3 id="consul-vs-zk-etcd"><a href="#consul-vs-zk-etcd" class="headerlink" title="consul vs zk , etcd"></a>consul vs zk , etcd</h3><p><a href="https://www.consul.io/intro/vs/zookeeper.html" target="_blank" rel="noopener">https://www.consul.io/intro/vs/zookeeper.html</a></p><h3 id="vs-traditional-nginx-lua-jenkins"><a href="#vs-traditional-nginx-lua-jenkins" class="headerlink" title="vs traditional: nginx + lua + jenkins"></a>vs traditional: nginx + lua + jenkins</h3><p>传统的发布模式：</p><ul><li>服务需要配置健康检查</li><li>添加新的服务节点要运维人为操作下到nginx 分发</li><li>lua 要配置好蓝绿发布状态的控制</li><li>如果传统的配置中心：<ul><li>服务要配置注册中心 register deregister，耦合</li></ul></li></ul><p>而现在，只需要简单的配置增加服务节点，就好了</p><h3 id="websocket-https"><a href="#websocket-https" class="headerlink" title="websocket/https"></a>websocket/https</h3><p>ref :<br><a href="https://www.jianshu.com/p/a4c04a3eeb57" target="_blank" rel="noopener">https://www.jianshu.com/p/a4c04a3eeb57</a><br><a href="https://segmentfault.com/a/1190000005026022" target="_blank" rel="noopener">https://segmentfault.com/a/1190000005026022</a><br><a href="https://segmentfault.com/a/1190000013720661" target="_blank" rel="noopener">https://segmentfault.com/a/1190000013720661</a><br><a href="https://segmentfault.com/a/1190000005040914" target="_blank" rel="noopener">https://segmentfault.com/a/1190000005040914</a><br><a href="https://learn.hashicorp.com/consul/getting-started/checks.html" target="_blank" rel="noopener">https://learn.hashicorp.com/consul/getting-started/checks.html</a><br><a href="https://www.consul.io/docs/agent/checks.html" target="_blank" rel="noopener">https://www.consul.io/docs/agent/checks.html</a><br><a href="https://my.oschina.net/guol/blog/353101" target="_blank" rel="noopener">https://my.oschina.net/guol/blog/353101</a><br><a href="https://www.jianshu.com/p/f8746b81d65d" target="_blank" rel="noopener">https://www.jianshu.com/p/f8746b81d65d</a><br><a href="https://medium.com/eleven-labs/consul-service-discovery-and-failure-detection-64b06a5cbce6" target="_blank" rel="noopener">https://medium.com/eleven-labs/consul-service-discovery-and-failure-detection-64b06a5cbce6</a><br><a href="https://www.cnblogs.com/zhangdk/p/Registrator_reference.html" target="_blank" rel="noopener">https://www.cnblogs.com/zhangdk/p/Registrator_reference.html</a><br><a href="https://kevinguo.me/2017/09/01/docker-consul-consul-template-registrator-nginx/#nginx-with-consul-template" target="_blank" rel="noopener">https://kevinguo.me/2017/09/01/docker-consul-consul-template-registrator-nginx/#nginx-with-consul-template</a><br><a href="https://juejin.im/post/5b59d64b6fb9a04fd1602017" target="_blank" rel="noopener">https://juejin.im/post/5b59d64b6fb9a04fd1602017</a></p>]]></content>
<summary type="html">
<h3 id="consul-amp-registrator-amp-consul-template-使用"><a href="#consul-amp-registrator-amp-consul-template-使用" class="headerlink" title="co
</summary>
</entry>
<entry>
<title>golang error handling</title>
<link href="http://deng.io/2018/11/26/golang-error-handling/"/>
<id>http://deng.io/2018/11/26/golang-error-handling/</id>
<published>2018-11-26T08:25:39.640Z</published>
<updated>2018-11-26T08:25:39.640Z</updated>
<content type="html"><![CDATA[<h3 id="golang-error-handling"><a href="#golang-error-handling" class="headerlink" title="golang error handling"></a>golang error handling</h3><p><a href="https://blog.golang.org/error-handling-and-go" target="_blank" rel="noopener">https://blog.golang.org/error-handling-and-go</a></p><p><a href="https://github.com/astaxie/build-web-application-with-golang/blob/master/zh/11.1.md" target="_blank" rel="noopener">https://github.com/astaxie/build-web-application-with-golang/blob/master/zh/11.1.md</a><br><a href="https://github.com/gopherchina/conference" target="_blank" rel="noopener">https://github.com/gopherchina/conference</a></p><p><a href="https://github.com/gopherchina/conference/blob/master/2018/1.3%20Rethinking%20Errors%20for%20Go%202.pdf" target="_blank" rel="noopener">https://github.com/gopherchina/conference/blob/master/2018/1.3%20Rethinking%20Errors%20for%20Go%202.pdf</a></p><p><a href="https://www.dotconferences.com/2017/11/marcel-van-lohuizen-rethinking-errors-in-go" target="_blank" rel="noopener">https://www.dotconferences.com/2017/11/marcel-van-lohuizen-rethinking-errors-in-go</a></p>]]></content>
<summary type="html">
<h3 id="golang-error-handling"><a href="#golang-error-handling" class="headerlink" title="golang error handling"></a>golang error handling</
</summary>
</entry>
<entry>
<title>package dependency manager</title>
<link href="http://deng.io/2018/11/26/golang-pkg/"/>
<id>http://deng.io/2018/11/26/golang-pkg/</id>
<published>2018-11-26T07:17:57.879Z</published>
<updated>2018-11-26T07:17:57.879Z</updated>
<content type="html"><![CDATA[<h3 id="package-dependency-manager"><a href="#package-dependency-manager" class="headerlink" title="package dependency manager"></a>package dependency manager</h3><p><a href="https://github.com/golang/go/wiki/PackageManagementTools" target="_blank" rel="noopener">我们有很多的管理工具：</a></p><p>dep , godep, govendor ,glide </p><p>其中比较好的是<br>govendor, glide</p><p>好的指需要做到：</p><ul><li>每个项目是独立的</li><li>依赖管理是清晰、容易管理的</li></ul><p>glide 的yaml 是比较清晰and 好管理的<br>在不同项目的不同版本上 gobendor , glide 都比较好的做到了，各自目录下面都有vendor 存储对应的依赖<br>类似npm</p><p>但都没有解决国内加速访问的问题<br>npm 可以设cnpm<br>maven 可以添加 conf/settings.xml 配置</p><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag"><<span class="name">mirror</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">id</span>></span>nexus-aliyun<span class="tag"></<span class="name">id</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">mirrorOf</span>></span>*<span class="tag"></<span class="name">mirrorOf</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>Nexus aliyun<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">url</span>></span>http://maven.aliyun.com/nexus/content/groups/public<span class="tag"></<span class="name">url</span>></span></span><br><span class="line"><span class="tag"></<span class="name">mirror</span>></span></span><br></pre></td></tr></table></figure><p>or<br><figure class="highlight xml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="tag"><<span class="name">mirror</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">id</span>></span>CN<span class="tag"></<span class="name">id</span>></span> </span><br><span class="line"> <span class="tag"><<span class="name">name</span>></span>OSChina Central<span class="tag"></<span class="name">name</span>></span></span><br><span class="line"> <span class="tag"><<span class="name">url</span>></span>http://maven.oschina.net/content/groups/public/<span class="tag"></<span class="name">url</span>></span> </span><br><span class="line"> <span class="tag"><<span class="name">mirrorOf</span>></span>central<span class="tag"></<span class="name">mirrorOf</span>></span> </span><br><span class="line"><span class="tag"></<span class="name">mirror</span>></span></span><br></pre></td></tr></table></figure></p><p><a href="https://yanyiwu.com/work/2014/10/01/node-vs-go-about-packagemanager.html" target="_blank" rel="noopener">ref</a><br><a href="https://blog.csdn.net/jettery/article/details/78104601?utm_source=blogxgwz2" target="_blank" rel="noopener">ref</a></p>]]></content>
<summary type="html">
<h3 id="package-dependency-manager"><a href="#package-dependency-manager" class="headerlink" title="package dependency manager"></a>package
</summary>
</entry>
</feed>