dfs_information_theory.pl -- Information theory

Information theoretic metrics for vectors, and for formulas given a set of models.

 dfs_surprisal(+Vector1, +Vector2, -Surprisal) is det

 dfs_surprisal(+Formula1, +Formula2, +ModelSet, -Surprisal) is det

dfs_surprisal(+Formula1, +Formula2, +ModelMatrix, -Surprisal) is det

Surprisal of P given Q, where P and Q are either vectors or formulas.

surprisal(P,Q) = -log Pr(P|Q)

 dfs_entropy(+Vector, -Entropy) is det

 dfs_entropy(+Formula, +ModelSet, -Entropy) is det

dfs_entropy(+Formula, +ModelMatrix, -Entropy) is det

Entropy of P, where P is either a vector or a formula.

H(P) = - sum_{s in S} Pr(s|P) * log Pr(s|P)

where the set S consists of all possible points in the DFS space that are fully specified with respect to the atomic propositions; that is, each point s in S constitutes a unique logical combination of all atomic propostions.

 dfs_delta_entropy(+Vector1, +Vector2, -DeltaH) is det

 dfs_delta_entropy(+Formula1, +Formula2, +ModelSet, -DeltaH) is det

dfs_delta_entropy(+Formula1, +Formula2, +ModelMatrix, -DeltaH) is det

Entropy delta of P given Q, where P and Q are either vectors or formulas.

DH(P,Q) = H(Q) - H(P)

 dfs_syntactic_surprisal(+Word, +Prefix, -Surprisal) is det

Syntactic surprisal of a word given a prefix.

S(w_i+1) = -log(P(w_i+1|w_1...i))
         = log(P(w_1...i)) - log(P(w_1...i+1))
         = log(freq(w_1...i)) - log(freq(w_1...i+1))

 dfs_syntactic_entropy(+Prefix, -Entropy) is det

Syntactic entropy for a prefix.

H(w_i) = -sum_(w_1...i,w_i+1...n)
         Pr(w_1...i,w_i+1...n|w_1...i)
         * log(Pr(w_1...i,w_i+1...n|w_1...i))

 dfs_syntactic_delta_entropy(+Word, +Prefix, -DeltaH) is det

Syntactic entropy delta for a word given a prefix.

DH(w_i+1) = H(w_i) - H(w_i+1)

 dfs_semantic_surprisal(+Word, +Prefix, +ModelSet, -Surprisal) is det

Semantic suprisal for a word given a prefix.

S(w_i+1) = -log(Pr(v(w_1...i+1)|w_1...i))

where v(w_1...i) is the disjunction of all semantics consistent with the prefix w_1...w_i.

 dfs_semantic_entropy(+Prefix, +ModelSet, -Entropy) is det

Semantic entropy for a prefix.

H(w_i) = - sum_(foreach s in S) Pr(s|v(w_1...i))
         * log(Pr(s|v(w_1...i)))

where v(w_1...i) is the disjunction of all semantics consistent with the prefix w_1...w_i.

 dfs_semantic_delta_entropy(+Word, +Prefix, +ModelSet, -DeltaH) is det

Semantic entropy delta for a word given a prefix.

DH(w_i+1) = H(w_i) - H(w_i+1)