With quaternions you can do same thing, but with small correction: 1. Negate quaternion before averaging if its dot product with previous sum is negative. 2. Normalize average quaternion, a the end of averaging, if your library works with unit quaternions.
The average quaternion will represent approximately average rotation (max error about 5 degree).
WARNING: Average matrix of different orientations can be broken if rotations too different.