One way to fix this is to check a boolean associated with each directory's state, and to rotate the arrow to a consistent state in onBindViewHolder().