Don't use the " sign when adding measurements. The system recognizes the numbers as being in inches.
I am not sure that is your issue but you may want to make sure your dropdown says package or thick envelope. If you choose letter or large envelope there will likely be other restrictions to the dimensions.
alternatively you can try leaving out the dimensions