one observation i want to make concerns the appropriate reliability coefficient. you seem to have settled on kappa. i am suggesting to you that this is an inappropriate statistics, at least in the domain of content analysis or the coding of qualitative data more generally, as it excludes unreliabilities due to unequal coder preferences for categories (manifest in unequal marginal frequencies of a contingency table). i have written about kappa's inconsistent definition. in its denominator kappa conforms to association/correlation statistics, e,g. chi square, while in its numerator it conforms to agreement statistics, e.g. scott's pi. this makes kappa hybrid whose values are difficult to interprete. alpha includes this source of unreliability. you may notice that when margins are unequal, kappa is always larger than scott's pi, the difference being due to unreliabilities hat kappa ignores but pi includes. alpha is not a derivative or form of kappa, as you state. if you want to relate it to another coefficient: when sample sizes are very large, the nominal and two coder form of alpha equals scott's pi. historically, pi preceeded kappa (and alpha). if you have large numbers of two coder and nominal data, pi is preferrable to kappa. in all other situations i have to recommend alpha.