Runtime Error when compiling with -O2 or higher

[jhennrich]

I managed to build kaldi for android (ndk r16b, api 21, arm) using your dockerfile and also crosscompiling on my linux_x86_64 machine following your blog post. However I get runtime errors when I start onlinedecoding. This happens only when compiling with "-O2" or "-O3". With "-O0" or "-O1" everything works like a charm. The behavior is the same with the docker-build and locally cross-compiled version.

This is the error:
ERROR: FstHeader::Read: Bad FST header: HCLG.fst
ERROR ([5.4.153~1-3f4f4]:ReadFstKaldiGeneric():kaldi-fst-io.cc:53) Reading FST: error reading FST header from HCLG.fst

Digging into the problem I found that kaldi::Input input stream is corrupted: (again, works with -O0/O1, so the fst-file should be alright)

kaldi::Input ki("HCLG.fst");
ki.Stream().good() // false
ki.Stream().eof() // false
ki.Stream().fail() // true
ki.Stream().bad() // true

Any Idea what might be the problem?

[jcsilva]

Hi, thanks for your report.
Could you please detail your environment? I mean, which Clang version you used? Which Kaldi commit you compiled, ...
And also, how did you test it? You used a real device or did you use the emulator? Which Android version?
For now, I don't have any idea about what is the problem, but I can try to figure it out. I'm not working with it anymore, but I'll try to build the same scenario you used and investigate this issue.

[jhennrich]

I attached my version of your dockerfile: docker.zip. I made some adjustments:

• OpenBLAS is in /tools
• everything is built during container runtime, not on container build time
• I had to use a different openfst mirror, openfst.cs.nyu.edu is down

The kaldi commit is 04b1f7d6658bc035df93d53cb424edc127fab819
NDK Toolchain: android-ndk-r16b-linux-x86_64

I tested on real devices: Google Pixel 1 (Android 8.0), Asus P027 Tablet (Android 7.0), Samsung S3 Mini (Android 5.0.2)

I did not attach a minimal code that produces the error, but let me know if you have troubles reproducing the error then I will see if I can come up with something. Thanks for looking into this

[jcsilva]

Hi,

just to update you. I'm trying to find out the problem, but until now I didn't succeed.

I start to think about a possible bug in clang. I saw other problems related to optimizations O2 and O3 in clang-3.5. As Ubuntu 16.04 uses clang-3.8 (and currently clang is in version 6), I think it could be a bug with clang. But, until now, I can't afirm it. If possible, I'd suggest you to change the Ubuntu version in your Dockerfile to 18.04 and test if the problem was solved.

[jhennrich]

Thanks for looking into it!
I get the same behavior when I build locally on Ubuntu 18.04 with Clang 6

[johnburkey]

oh yeah this is my bug too, and yeah real devices here too. Im Ubuntu 16.04 and latest clang as well. Does the workaround of lower optimization actually work ?

[jhennrich]

Yes, for me it worked

[johnburkey]

did u just selectively de-O a few files ? Like sounds like its the stream creator file.? Do you remember which ones?

[johnburkey]

and thanks!

[jhennrich]

No, I did not put more effort into it yet