It's interesting to know the driver aren't unloaded. So I did a couple  
of tests with rmmod:

Test 1:
1. Plug in the USB device
2. Run test application, baud rate is successfully set
3. Plug out the USB device, lsmod shows usbserial and ftdi_sio
4. rmmod ftdi_sio
5. Plug in the USB device
6. Run test application, baud rate not set in the USB device

Test 2:
1. Plug in the USB device
2. Run test application, baud rate is successfully set
3. Plug out the USB device, lsmod shows usbserial and ftdi_sio
4. rmmod ftdi_sio
5. rmmod usbserial
6. Plug in the USB device
7. Run test application, baud rate is successfully set

Thus if I remove both ftdi_sio and usbserial before I plug the USB  
device back in, the problem is solved. Maybe usbserial is causing the  
problem?


Quoting Greg KH [off-list ref]:

On Sat, Sep 08, 2012 at 07:35:16PM +0200, peter.sheeren@axiris.be wrote:

quoted

I'm sorry, I've no experience with compiling a driver for Linux.

Q: When I plug in the USB device without even using it, should
ftdi_sio and usbserial remain loaded after I unplug the USB device?

Yes, no one unloads modules after devices are removed, as you really
don't know if the driver is still needed or not.

But, this shouldn't be an issue for you, when you replug the device in,
it should be reset, with no internal, or kernel, state as to what it is
in.  You have to always explicitly set the line settings up again,
otherwise you have no idea what it is running at.

greg k-h