Hi, What version of FlatCAM are you using?
8.5, since the newer version did some funky things when I attempted to use negative tool size (can't recall what it was, but I think it simply hung altogether)
You could have reported if there was an issue. It would have helped... But right now negative tool dias work ok. I've made a GIF and posted it in a topic bellow, named: "flatCAM and opensuse Tumbleweed". Regarding FlatCAM 8.5 I'm sorry but I can't help. I'm immersed in the features of beta and I find it harder and harder to remember what was in 8.5, what was in development 8.5, what actually I've tried to add (pull requests) to dev version of 8.5. Also before beta (which started to happen at the start of this year) I already worked for a long time on the so called FlatCAM Nightly's ... just to add to the confusion of what is and what is not. :)
In any case, due of some other reason (removal of silkscreen that is overlapped over soldermask) I've just added this feature to the FlatCAM beta. It will be available as Substract Tool in the future release of FlatCAM beta 8.915. Substract Tool will work on Gerber objects but also in Geometry objects. Even more, the multigeo Geometry objects( geometries resulted from Paint or NCC Tools) can be used as targets, too.
Something like this: https://gifyu.com/image/93Xz