-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathREADME
More file actions
204 lines (140 loc) · 7.22 KB
/
README
File metadata and controls
204 lines (140 loc) · 7.22 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
******************************************************************************************
C programs for rectified-wire networks
Veit Elser, April 2018
******************************************************************************************
reference: "Monotone Learning with Rectifier Networks", V. Elser, D. Schmidt & J. Yedidia
arXiv:1805.03963
bug reports to ve10@cornell.edu
******************************************************************************************
rainman.c : sequential deactivation (SDA) algorithm for training rectified-wire networks
expander.c : generates sparse expander networks, for input to rainman
******************************************************************************************
compile:
gcc -O2 rainman.c -lm -o rainman
gcc -O2 expander.c -o expander
******************************************************************************************
usage:
./expander insize outsize hidlayers growth [netfile]
insize: number of input nodes
outsize: number of classes
hidlayers: number of hidden layers
growth: layer-to-layer growth factor (integer)
netfile: destination of network
When expander is run without a netfile argument, it returns the number of edges and no
network file is written.
./rainman trainfile testfile netfile trainstop trainbatch testbatch &
trainfile: data file used for training
testfile: data file used for testing
netfile: network file, such as generated by expander
trainstop: stop training after this many data items, read cyclically from trainfile
trainbatch: number of data between each line-entry of logfile
testbatch: number of testdata items to use for testing (if less than size of testdata)
outputs:
netfile, with bias values for highest achieved classification accuracy
logfile, named netfile.log
******************************************************************************************
Examples of data files
market_train:
1
100
16 2
0 10 16 26 33 10 16 22 31 39 49 55 34 9 15 22 0
49 58 65 40 48 24 0 9 15 24 31 40 46 51 59 37 0
15 40 34 24 45 39 33 26 19 10 0 25 20 13 36 29 1
27 20 41 34 59 52 74 64 59 52 46 36 26 17 9 0 1
23 16 6 0 24 14 35 60 55 48 41 36 29 50 44 34 1
74 50 25 0 9 19 28 37 45 20 31 38 47 55 62 70 0
...
The 1 in the first line announces the data is in analog format. Data vector components
range from 0 to the number in the second line, 100.
The third line gives the number of vector components (16) and the number of classes (2).
For analog data files, the insize parameter of expander (number of input nodes) must be
twice the number of vector components (32 for this file).
Starting with the 4th line are the data (16 analog components followed by the class
label). Data with c classes must have integer labels 0, 1, ... , c-1.
When the trainstop parameter of rainman exceeds the number of data items, training loops
back to line 4 when the end of trainfile is reached.
markov4test:
4
ABCD
20 2
DBDDBDBDBDCCCCDBDBDB 0
BABADCDADCDDBABABADC 1
DABDBADBBCCCCAAAABAA 0
ABACAABABABBADDCACDC 1
DDBCCBDDADDBCDBDDBAB 0
BDDBDBCBDCCACDBCCBDB 0
...
In this file the 4 (>1) in the first line annouces the data is symbolic with 4 symbols.
The particular symbols (ASCII characters) are given in the second line
From the 3rd line we see that each data item is a string of 20 symbols and there are two
classes. For processing symbolic data the insize parameter of expander must be the product
of the number of symbols and the string length (80 for this file).
Data begin in the 4th line: a character string followed by the integer class label.
******************************************************************************************
Example of a netfile
To create a network with 2 hidden layers and growth factor 10 for the market_train data
(above), one would run
./expander 32 2 2 10 2_10
where 2_10 was given as the name of the network file.
2_10:
2
32 320 3200 2
13440
28 32 0.
9 33 0.
3 34 0.
19 35 0.
31 36 0.
26 37 0.
...
The 2 in the first line gives the number of hidden layers. This is followed by the number
of nodes in each layer, beginning with the input layer and ending with the output layer.
The number of edges is given in the third line (13440).
Nodes are labeled with consecutive integers, and consecutively within layers. There is a
line for each edge comprising the two node labels and the bias value. Although the latter
are 0., these numbers are ignored by rainman (it initializes all bias parameters with 0.).
The random number generator of expander is seeded with time(0).
******************************************************************************************
Example training run and logfile
To train network 2_10 (created by expander, above) on the market_train data, and test
against the market_test data, one could run
./rainman market_train market_test 2_10 200000 10000 10000 &
Since market_train has 10^5 items, this will cycle through two epochs of training. After
every 10^4 items, a new line of output is appended to logfile.
2_10.log :
trainfile: market_train
testfile: market_test
data false iter aveiter true zero act 0 act 1 act 2
1.00e+04 3.52e+03 6.84e+03 1.94 70.38 0.00 9.02e-01 8.24e-01 2.98e-01
2.00e+04 5.46e+03 9.99e+03 1.63 88.16 0.00 8.93e-01 7.89e-01 1.44e-01
3.00e+04 6.55e+03 1.16e+04 1.52 93.47 0.00 8.89e-01 7.72e-01 9.60e-02
4.00e+04 6.99e+03 1.23e+04 1.39 97.67 0.00 8.88e-01 7.67e-01 8.34e-02
5.00e+04 7.24e+03 1.26e+04 1.30 97.84 0.00 8.87e-01 7.64e-01 7.79e-02
6.00e+04 7.42e+03 1.28e+04 1.33 97.24 0.00 8.87e-01 7.62e-01 7.44e-02
7.00e+04 7.58e+03 1.30e+04 1.26 98.51 0.00 8.86e-01 7.61e-01 7.18e-02
8.00e+04 7.68e+03 1.31e+04 1.22 98.05 0.00 8.86e-01 7.60e-01 7.01e-02
9.00e+04 7.78e+03 1.33e+04 1.20 98.78 0.00 8.86e-01 7.59e-01 6.83e-02
1.00e+05 7.84e+03 1.33e+04 1.33 98.47 0.00 8.85e-01 7.58e-01 6.73e-02
1.10e+05 7.90e+03 1.34e+04 1.23 99.11 0.00 8.85e-01 7.58e-01 6.65e-02
1.20e+05 7.94e+03 1.35e+04 1.10 99.00 0.00 8.85e-01 7.58e-01 6.61e-02
1.30e+05 7.99e+03 1.35e+04 1.19 99.17 0.00 8.85e-01 7.57e-01 6.53e-02
1.40e+05 8.02e+03 1.36e+04 1.14 98.97 0.00 8.85e-01 7.57e-01 6.49e-02
1.50e+05 8.05e+03 1.36e+04 1.19 99.17 0.00 8.85e-01 7.57e-01 6.46e-02
1.60e+05 8.07e+03 1.36e+04 1.12 99.11 0.00 8.85e-01 7.56e-01 6.41e-02
1.70e+05 8.08e+03 1.36e+04 1.12 99.24 0.00 8.85e-01 7.56e-01 6.40e-02
1.80e+05 8.08e+03 1.36e+04 1.00 99.26 0.00 8.85e-01 7.56e-01 6.40e-02
1.90e+05 8.09e+03 1.36e+04 1.00 99.31 0.00 8.85e-01 7.56e-01 6.39e-02
2.00e+05 8.10e+03 1.36e+04 1.00 99.23 0.00 8.85e-01 7.56e-01 6.39e-02
best: 99.31
total false: 8095
total iterations: 13640
data: net training items processed
false: net false classifications (number of times SDA is invoked)
iter: net SDA iterations
aveiter: average number of SDA iterations (per misclassification) in the current batch
true: classification accuracy
zero: percentage of tests where the wrong class node has value zero
act0: activation percentage in the current batch, averaged over edges in first layer
...
act2: activation percentage in the current batch, averaged over edges in last layer