Most programs start at 620, but FHA and VA loans can go lower with strong compensating factors.