Without the filter fabric, the fines will migrate to the stone and you will have clogged up stone and poor draining. Ideally, if you can layer the gradient, going from course to fine in sucessive layers, you could get away without filter cloth, but that is a lot of work, and a lot of material. I suggest you use the filter fabric but then put a layer of coarse builders sand over it. The idea is not to have an attractive growth medium where roots want to migrate to. None the less, the roots will eventually win anyway.