Blatantly stolen image...
Pretty much one half of the bearing will feel loose when it comes out, you'll notice the shaft is worn on one half and not the other if this is the problem you've having. Notice the lip at the top and bottom caused by the wear.
Hopefully just the nut hasn't been tightened up or something else though, otherwise it'll be a case of taking the bearing back apart.