Welcome back to the AI Bayeslab Statistics series. Today, let's dive deeper with a theory derived from effect size. We discussed the effect size formula of a two-population t-test in the last post, "Meta-analysis: Transitioning from Effect Size to Omega Squared (ω2)". And the formula for it is:
Thus, we could derive a functional decision-making theory from it, known as Signal Detection Theory. Perhaps someone has heard of it before, but the real-world usefulness of this theory often goes unnoticed. Through this post, I aim to promote this theory significantly in terms of its application, particularly in communication, management, and life decision-making.
1.What is signal detection theory?
So, What is Signal Detection Theory(SDT)? Let's start with a familiar visualization: two normality curves representing the distributions of two populations. For example, compare this year's and last year's test scores. What if we replace them with two populations, one containing valid information and the other lacking valid information?
What are you going to reel in your mind? What thoughts are you fishing for?
→ Type I error VS Type II error, right?
We also call the Type II error a false alarm(FA) and the Type I error a miss(M) We denote a signal that contains valid information as an SN(signal with noise), as we said before, sampling error is everywhere, and a signal that lacks valid information as an N(pure noise).
2.Example of signal detection theory
Perhaps it remains abstract, so let's give everyone a trope to grasp it easily. Let's take an example of team management or the management of public opinion. Let's combine a conventional Chinese tale to understand the theory:
The tale is "Zou Ji's Advice for King Qi to Accept Criticism." It illustrates how effective governance provides a competitive edge. You might also view it as a management story share.
The Story is：
Zou Ji was more than eight chi tall and had an imposing and magnificent build. One morning, as he dressed, he looked at himself in the mirror and asked his wife, “Who do you think is more handsome, Mr. Xu from North City or me?” Mr. Xu from North City was widely recognized as the most handsome man in the Qi state. His wife replied, “You are more handsome. Mr. Xu is not your match.” Zou, unconvinced, turned to his concubine and asked, “Who do you think is more handsome, Mr. Xu from North City or me?” His concubine responded, “You are. Mr. Xu is not as handsome as you are.” The next day, a friend came to visit him. During their conversation, Zou asked, ‘Who do you think is more handsome, Mr. Xu from North City or me?” The visitor said, “Mr. Xu is not as handsome as you are.” The day after, Mr. Xu from North City visited him. Zou seized the opportunity to examine Mr. Xu closely and concluded that he was not equal to Mr. Xu. Looking in the mirror once more, he became even more convinced that he should not be compared with Mr. Xu. That night, he thought, “My wife said I was more handsome because she was partial to me. My concubine said I was more handsome because she admired me. My friend said I was handsome because he had a favor to ask of me.”
He then approached the court to seek an audience with King Wei of Qi. “I, your Majesty’s humble servant, know for certain,” he said to the king, “that I am not as handsome as Mr. Xu from North City. But my wife said I was because she was partial to me. My concubine said so because she is in awe of me. My friend said so because he had a favor to ask of me. The Qi state now possesses a territory extending a thousand miles and one hundred and twenty cities and towns. All the palace ladies and attendants are partial to you in Your Majesty's court. The high officials in your Majesty’s court are all in awe of you, and all the people in the state have favors to ask of your Majesty. So, it is evident that Your Majesty has been seriously impeded from knowing the truth.
The king replied, “Excellent!” He then issued a decree stating, “All those officials and people who can admonish me in person regarding my mistakes will receive the highest rewards. Those who do so in writing will receive moderate rewards. Those who publicly voice their complaints about my misdoings will receive small rewards.” Shortly after the decree was announced to the public, the court became as crowded as a marketplace with officials and citizens who came to offer their remonstrations. A few months later, the number of attendees gradually dwindled. After a year, no one came anymore because they had nothing to complain about.
The states of Yan, Zhao, Han, and Wei, upon hearing what had transpired in Qi, all rushed to the Qi court to pay their respects to King Wei. Thus, His Majesty achieved victory over other states without resorting to military means, demonstrating how well-managed internal affairs can lead to triumph over other nations without military action.
Note:
Feet: The ancient Chinese "feet" differ from the modern "feet." During the Warring States period, one " feet" was approximately 23 centimeters. Therefore, "eight feet or more" is about 184 centimeters.
Concubine: A concubine is historically defined as a person in a long-term sexual relationship with someone they cannot fully marry due to social, legal, or religious barriers. This practice was common in various cultures, including China, where it persisted until it was banned in 1949. The status and rights of concubines varied widely depending on the cultural and legal context.
3.Story and Theory - A Detailed Analysis
In signal detection theory, "signal" (S) and "noise" (N) are fundamental concepts. A signal is the desired information to be identified, while noise is the distracting information that should be ignored.
In this story, we can relate them as follows:
Signal (S):
The facts and truths, such as the King of Qi's understanding of the country's real state, and the actual impact of his decisions.
Noise (N):
The biased and flattering information, for example, the compliments from Zou Ji’s wife, concubine, and friend, were influenced by personal fondness, admiration, or favors and did not truly reflect his appearance.
Zou Ji’s thought process and actions involve identifying and managing signals and noise. He realized that praise from different people might be tainted by noise and not represent accurate information. By understanding this, he advised the king to be aware of potential noise issues and take measures (like rewarding honesty) to enhance signal recognition, reducing noise interference in decision-making. This helped the king govern better, achieving a way to rule without military force.
In the realm of management or public governance, no nightmare can be worse for any leader than Anderson's fairy tale, "The Emperor's New Clothes" (Kejserens nye Klæder).
4.About: Signal Detection Theory
Signal detection theory is a statistical approach to decision-making that focuses on distinguishing between subjective response tendencies and an individual's actual sensitivity.
Noise is an unavoidable aspect of the system that cannot be eliminated; therefore, signal detection aims to identify the signal against the noise backdrop. For example:
When designing a new product, the users' motivations, attitudes, and response preferences can complicate the measurement framework used to assess the impact of test conditions or product design variations.
If you manage a team, effective team management hinges on the daily communications received by team members, which are influenced by motivation, their environment, and the official task details. This former aspect, influenced by motivation and the environment, often makes the team execute ineffectively, limiting overall effectiveness.
As noted earlier, signal detection theory is often underappreciated in practice. After highlighting its importance, let's explore other relevant statistical concepts and the core principles of the theory. In the following section, we will also introduce an important classical data visualization, the ROC Curve.
The context preview of the following section：
The foundational principle of the signal detection theory
What is the likelihood ratio?
What is the report criterion?
What is the d'(d prime)？
What are the four components of signal detection theory?
The payoff matrix
What is the ROC curve used for?
ROC curve graph
What is the purpose of the ROC analysis?
A simple case involving the ROC curve alongside leadership.
……
Stay tuned, subscribe to Bayeslab, and let everyone master the wisdom of statistics at a low cost with the AI Agent Online tool.