The ability of a machine learning system to know what it knows is one of the proposed Revised Criteria for Machine Intelligence. It is also one of the desiderata of Carbonell's Life Long Learning Manifesto. In HAT-AI, one way to add introspection to a neural network is to add a Judgment Node that judges the performance of an existing node in the network.
A judgment node for each output node will enable a neural network to superficially satisfy the requirement "know what it knows and know what it does not yet know." However, true introspection requrires more. People are not perfect at introspection. A person often does not know how they know something. Similarly, a machine learning system may satisfy the criterion for machine intelligence without understanding what each inner layer node is trying to achieve.
However, such understanding is clearly a desriable goal. The tools Knowledge Sharing and Judgment Nodes can facilitate this goal. Knowledge sharing faciliates the propagation of interpretability from output nodes and other nodes with confirmed interpreations to inner nodes in the same or other networks. A judgment node may provide knowledge of what a node knows and what it does not yet know. However, training a judgment node requires knowing the objective of the node to be judged.
Generally, there is no explicit objective for an inner layer node. However, an implicit objective may be inferring, although this implicit objective may change during the course of the training. During each epoch of training, for each datum, back propagation is used to compute the derivative of the network objective with respect to the activation value of the node including the effect of any rgularization. Comparing the sign of this derviative and the activation level of the node may be used to determine whether an inner layer node meets its implicit objective for a datum.
Let a theshold value be set for the activation value of each inner node. This threshold value may be determined by a hyperparameter, or it may be set by a Human + AI learning management system. Often it will simply be the infection point or point of discontinuity in the activation function. A node meets its implicit objective for a datum if an unupdate based on the derivative computed for that datum would move the activation value away from the threshold, which indicates current activation value is on the desired side of the threshold. On the other hand, if an update based on the datum would move the activation value towards the threshold, that is a move toward the other side of the threshold. In the first case, the current activation of the node meets its implicit objective. In the second case the current activation of the node does not meet its implicit objective.
To reduce confusion with the performance on the objective of the overall network, how well a node meets its implicit local objective will be referred to as the "satifaction" of the node rather than as its "performance"
Knowing the implicit objective of an inner layer node may be useful in seveal ways:
by James K Baker and Bradley J Baker