Yes, size matters. The larger your mesh is, the more LODs are going to be ignored by the LI calculation. The radius of the bounding box (think of a sphere encompassing the mesh) is what determines which of the lower LODs is being dropped.
If said radius is larger than 5.43 meters, the lowest LOD will be ignored.
If said radius is larger than 10.86 meters, the low, and lowest LOD will be ignored.
If said radius is larger than 43.44 meters, othe lowest, low, and medium LOD will be ignored.
The formula to get the radius is sqrt(x*x + y*y + z*z) / 2
Also, it's always a good idea to keep larger meshes as lowpoly as possible, and add window frames, and any detail as a separate mesh object.