Random Tool Change (2 into 1 add-on)

[HippoDan]

Printing from Octoprint frequently finds an extruder switch at random, uncommanded. The forum also reports this problem on any USB connection, not just Octoprint. I'm currently testing this from the SD card, which does not seem to have the same problem.

Firmware downloaded and flashed 10/25/17 using Arduino IDE 1.6.1.

[teejaydub]

It seems like printing from the SD card is a decent workaround for that problem if printing from the SD card fits with your workflow. It doesn't for me, so, working on Ryan's assessment that the problem is caused by too much RAM being used by the second extruder, I tried disabling SD card support instead:

teejaydub@cea255d

(of course you also need to set NUM_EXTRUDER to 2, but I wanted to leave my repo ready for a pull request)

That frees up quite a bit of RAM - it goes down from 89% usage with one extruder + SD card support to 81% with two extruders + no SD card support.

Time will tell if this fixes the random extruder switches, but I got through one print OK and will report back as I get more experience with it.

[louvanna]

Using cea255d firmware on RostockMax V3 -dual, and have the same RAM %'s using IDE Arduino 1.8.4. I am 0 for 2 attempts where the controller will stall early into a print and eventually go into reset. No joy.
There is one other comment elsewhere about memory overflow "Array memory overrun fix and ui_lang.h inclusion fix. #14" , but have yet to incorporate that yet.

[teejaydub]

Hm, stalling and resetting could be a different problem? I haven't seen that at all.

[louvanna]

I am using OctoPrint as the USB controller. Clarification, the LCD initially shows Extruder0 , as intended. The job is only T0, so the random extruder change is still present. The delta stalls after the LCD changes to Extruder1, but the extruder stepper (1) is now active. So really the delta has stalled, but the firmware is still executing something.

[dbfrompw]

On the Rostock V2, if set to 2 extruders, if you change to 2nd extruder, it locks up and needs rebooting. V91 works but cant get inductive probe to work.

[louvanna]

A rollback to v2 / v91, would mean forgoing the HE280 accelerometer, nope.
Update, it works somewhat.
After deep thought, in the fog the 2017/2018 seasonal flu, I forgot to erase the EEProm prior to the new upload, and recalib.
Remedying this omission, I can now print for an hour+ before a random tool change away from T0. Too bad my print is 7 hrs. While it printed with an empty T1, I manually simply moved my filament spool from T0 to T1. Many layers and hours later, the printer was found cold/idle, with its head buried at the print surface, and the LCD display line, 'Print Killed'.

Machine: dual 'Y" extruder kit, Rostock Max v3, firmware cea255d plus pull #14,
Controller: OctoPrint 1.3.6 as the USB controller on RPI.

[EliW]

Chiming in here: There is an identical/similar bug being discussed over on the OctoPrint Github (because I filed it a while back not knowing about this one):
OctoPrint/OctoPrint#3031

Basically I'm having this exact same issue. However, with a couple twists:

• I'd been printing fine for months with a dual extruder over USB, when connected to a desktop and S3D. I only started to have this random extruder-swap when moving to using OctoPi
• I've recently had a different fault while debugging the extruder swap. Where some garbled communication caused an issue 15hrs into a 16hr print (after I'd originally found a workaround of 'print only on T1' which got me past the 1hr point. But it seems, only to a 15hr point.

[HippoDan]

I found s solution for this a while back. Cut the data rate in half. Simply reducing the baud rate in octoprint to 115000 or whatever it is, made the problem go away entirely. It would take a lot longer to transfer files to the sd card through octoprint, but I never do that, and print speed is unaffected.

I have successfully run a 38 hour print with that configuration.

[EliW]

So: New problems from this weekend (and this morning)

I've been printing 24/7 in preparation for an event. And twice now I've had extruder changes while printing from an SD card. The latest one this morning.

So now I'm completely flummoxed as to what to do. I did still have Octoprint connected via USB. But just hanging out monitoring. So if just 'being connected' is enough to trigger the bug ... OR ... It has nothing to do with Octoprint at all.

I'm seriously considering just reflashing my firmware to single extrusion and considering the whole dual extrusion addon to be complete junk because of this :(

And I was really looking forward to making more dual extrusion awesomeness....

[teejaydub]

I haven't been looking into this much, because I have had persistent stringing problems, making dual extrusion spectacularly crappy even without random tool changes. BUT I just wanted to put this out there: it smells a lot like a buffer overflow I recently found in a totally different project. I would like to get some time to investigate what happens when the firmware runs out of RAM for incoming data - in particular, does something get overwritten, by a class C array overrun or some other way?

Just in case someone else has the time to pursue that angle before I do.

I'm not sure how that would explain the issue when printing from SD.

[EliW]

I dunno, buffer flow sounds like a legit issue here (and why I prefer coding in higher level languages now-a-days) ... Even on SD card, just perhaps for different reasons.

If you look in the other thread at my output, you'll see in one case where the Rostock sent a very garbled message, which totally looks like a buffer overflow situation (or at least a non-cleared buffer). So that's not a bad theory.

while USB is attached, and connected, it's still constantly communicating with the Octoprint, sending the 'SD printin byte' commands, and temperature reports at times.

I will also add, that the models that this started happening to me on, also happen to be very complicated models. IE: When I was printing single extrusion AFTER upgraded to the Y and it was working, it was over USB via S3D ... and I had zero issues.

But those, while sometimes LARGE prints (12"x12"x12") ... were very simple prints. Lots of straight lines, box shapes, etc. So the GCODE was actually very small.

When I started seeing hte problems was not just when I switched to Octoprint, but also I started printing wargaming terrain for an upcoming con that I'm running. And now these was things like 'stone walls' where there isn't a single flat surface. Looking at the Gcode, it's HUGE, as it's a constant mess of "go 0.1mm this direction, now 0.1mm that direction, now 0.1mm that directino' ... to get the 'rippling' effect of stone.

Also FWIW: The two I've had that switched while printing on the SD card. Both did it at an obvious point of 'change' in the model. That may just be coincidence. But basically one was between two layers of brick perfectly. The other was right before a line of skull/wings happened, again, perfectly. May just be coincidence.

[teejaydub]

That makes sense - the printer controller (Octoprint for you, MatterControl for me) throttles what it sends to the printer firmware, but it's doing it heuristically, not with any kind of flow control. When there's more data per layer (or per mm extruded, or whatever it's looking at), the firmware has more of a chance of getting behind, and when that buffer fills up, something bad happens. Reducing the baud rate acts as another throttling heuristic.

Looking through the firmware source, it looks like G-code commands are buffered, and if the buffer is full, it stops reading from the serial port - see the beginning of GCode::readFromSerial(). I wonder if, when it's full for too long, the Arduino serial buffer fills up because it's not getting polled, and then when we resume reading, either the Arduino has wrapped to the beginning of the serial buffer, or ignored incoming characters from the end - either way, commands get garbled. And a single "move" command getting garbled wouldn't produce much of a noticeable problem, but a "change tool" is more noticeable.

In other words, they've handled an overflow condition in the parsed G-code command buffer, but while waiting for that condition to clear, the Arduino serial buffer overflows, and then there's no provision to resync, resulting in serial data getting dropped mid-command.

So maybe instead of "if the command buffer is full, do nothing," that first check should be more like:

1. if the command buffer is full, set a new inputOverflow flag, and also set the existing waitUntilAllCommandsAreParsed flag (which really means "wait until all commands are executed").
2. waitUntilAllCommandsAreParsed will then drain commands without processing input
3. when the command buffer is empty, waitUntilAllCommandsAreParsed is already cleared; additionally, at that time, if inputOverflow is set, call requestResend(), which resets serial input completely and resyncs.

I'll try this when I can, but anyone else is welcome to beat me to it!

[teejaydub]

I implemented that inputOverflow flag in my fork: https://github.com/teejaydub/Firmware. It compiles, but my printer's hot end is out of commission so I can't stress-test it. If anyone would like to give it a whirl, let me know how it goes!