It sounds more like you have "group only" access. This sets the no enry lines to about 50m high above the terrain limit. If your home, or part of it, is above this limit, people can still enter.
This is different to actually banning somebody outright, in which case, the ban lines now go 5000m high.
Suggested fix: get a security orb and set it to protect your land. ergo, problem solved.