So I've put some thought into this one.
Consider two sentences:
"Linux is an open-source operating system."
"Red Hat is an open-source company."
Now, the first sentence is Chicago Manual of Style correct. In the sentence, "open" and "source" are not separately modifying the noun "system". This construct is to be contrasted, for example, with "The Pacific is a deep blue ocean" in which "deep" and "blue" are adjectives that indepently modify "ocean". Chicago Manual dictates that in such circumstances, the dependent modifiers form a joint adjectival phrase, and hence must be joined by a hyphen.
That's a great theory, but it doesn't help us with the second sentence above. Red Hat is not, in fact, an open-source company. Neither is VA Linux, or Sun, or Microsoft, or any other major corporation that I'm aware of. The "source code" to Red Hat's business practices is not open. Red Hat's books are not open. In no sense could the company itself be described as an open-source company. It deals in open-source products, but the company itself is not an open-source company.
Suppose we rewrite these two sentences as:
"Linux is an Open Source operating system."
"Red Hat is an Open Source company."
Let me take the latter sentence first this time. What are we changing by
removing the hyphen and adding capitalization? Well, compare the usage
with these two sentences:
"Gore is a Democratic candidate."
"Gore is a democratic candidate."
I suppose that both of these are correct, but clearly the first sentence is what people typically have in mind. They are using "Democratic" in a titular rather than adjectival sense. In other words, it is appropriate to bestow the title of Democratic on Gore as a candidate. Democrats are a formally recognized group, and Gore is a member of that group. What makes Democrats a formally recognized group is that there's a clear definition specifying whether one is or is not a Democrat.
Open Source as a movement has this kind of status. There is a formal definition specifying whether a piece of software counts as Open Source or not. This is a black and white matter. There are also companies that promote the distribution of open-source software, and companies that do not. I think we, as a movement, have been able to successfully defend this distinction as unambiguous as well, despite the efforts of Sun and other companies to blur the distinction.
So I would conclude that it is appropriate to use the titular sense of "Open Source", but not the adjectival sense of "open-source" in describing Red Hat.
What about Linux, the operating system? I suppose that the adjectival sense is a more appropriate descriptor here, but both senses are appropriate, and understandable.
So the most correct usage would probably be:
"Linux is an open-source operating system."
"Red Hat is an Open Source company."
We actually tried to stick to this convention for a while at O'Reilly. It was a disaster. First of all, the dual usage is often confusing and distracting to the reader, unless you lay out the whole tedious line of reasoning I've just given. Second, even our copy editors couldn't grasp the appropriate rules of usage in every case. The problem is that appropriate usage becomes contextual: you have to actually know something about software companies and software to tell whether the titular or adjectival sense is correct in any given case. A great many copy editors hate having to actually know something about what they are copy editing.
We concluded that, in the interest of clear communication and clean copy editing, we were going to have to pick one usage and stick to it. And if you have to pick one usage and stick to it in all cases, then the appropriate usage to pick is the titular sense of the phrase.
So it is "Open Source." Not "open-source." Every time.