Deeding land means the owner of the land is a group. Setting land to a group means the avatar is the owner of the land, but shares land with a group.
I'm not sure why you need to deed land to a group. Deeding land allows you to own 10% more of land for the group, but it doesn't give you more prims. In the long run, if you can't buy that 10% more of land in the same region, it's not necessary to deed the land, because you will not have more prims.
Here is the thing. If an avatar owns 5120sqm of land in 1 region has the same prims than a 5120sqm of land deeded to a group in the same region.
Deeding land to a group would allow you to buy an extra 512sqm (in your tier level) of land and if you use that land in the same region, then having more prims.
If the system returned your objects it's probably because you had "auto return" enabled in the land, and objects which are not rezzed in the same group. As an avatar, the system never returns the objects you own, but when land is deeded, the owner of the land is not the avatar: it's the group. So weird things can happen.
Set all your objects to the same group that will own the land before messing up with land.
You have only deeded the 1024sqm plot of land. You still have to deed your 4096sqm plot of land.
You have deeded the 1024 sqm plot of land. (set all your objects to the same group that will own the land )
Deed the 4096sqm plot of land to the group. That will give you a contribution of 3723sqm of land.
Now you will have 931sqm + 3723sqm land donated to the group.
Buy another 512sqm plot of land in the same region by deeding it to the group. Don't use "buy land", but use "Buy land for group". 465sqm will be donated to the group.
That will give you 931sqm + 3723sqm + 465 sqm = 5120 sqm
Remember that deeding land doesn't give you more prims, owning more land give you more prims. !