I have found a couple of solutions to this on the web, but nothing like this.
We have SharePoint Server 2010 and Office 2010. I was browsing to a file located on My Site, but other documents . Excel or Word would open and then i would be told:
Could not open filename.doc
You would then click ok, the message disappeared and you would not have an open document.
In our case, it was nothing to do with IE security settings, or Office 2010 protected mode.
We had been trialling SharePoint WorkSpace with a copy f my My Site attached so that I could work on my MySite documents at home. The copy that was stored on my computer had not been updated as I had disabled the option to start SharePoint WorkSpace after I was happy that it would work.
The fix for me therefore was to either delete the local sync copy or to have it update the local copy.
Back to be able to get my work.