The “dots” are called noise or variance and appear when you want to numerically integrate a function by random sampling it. One way to reduce it is to render it into a buffer and blur it. But I have no idea what the state-of-art approach for AO currently is.

When raytracing, you sample each lights, giving a dot free result right? But when sampling for ambient occlusion, the shadows have dots like in a path tracer and takes a long time to converge. How do you fix that so it appears smooth?

Thanks