Did Clyburn Help Biden in South Carolina?

By: Dr. Ikjyot Singh Kohli

The conventional wisdom by the political pundits/analysts who are seeking to explain Joe Biden’s massive win in the 2020 South Carolina primary is that Jim Clyburn’s endorsement was the sole reason why Biden won. (Here is just one article describing this.)

I wanted to analyze the data behind this and actually measure the effect of the Clyburn effect. Clyburn formally endorsed Biden on February 26, 2020.

Using extensive polling data from RealClearPolitics, I looked at Biden’s margin of victory according to various polling samples before the Clyburn endorsement. I used Kernel Density Estimation to form the following probability density function of Biden’s predicted margin of victory (as a percentage/popular vote) in the 2020 South Carolina Primary:

Assuming this probability density function has the form p(x), we notice some interesting properties:

  • The Expected Margin of Victory for Biden is given by: \int x p(x) dx. Using numerical integration, we find that this is \int x p(x) dx = 18.513 \%. The error in this prediction is given by var(x) = \int x^2 p(x) dx - (\int x p(x) dx)^2 = 107.79. This means that the predicted Biden margin of victory is 18.51 \pm 10.382. Clearly, the higher bound of this prediction is 28.89%. That is, according to the data before Clyburn’s endorsement, it was perfectly reasonable to expect that Biden’s victory in South Carolina could have been around 29%. Indeed, Biden’s final margin of victory in South Carolina was 28.5%, which is within the prediction margin. Therefore, it seems it is unlikely Jim Clyburn’s endorsement boosted Biden’s victory in South Carolina.
  • Given the density function above, we can make some more interesting calculations:
  • P(Biden win > 5%) = 1 - \int_{-\infty}^{5} f(x) dx = 0.904 = 90.4%
  • P(Biden win > 10%) = 1 - \int_{-\infty}^{10} f(x) dx = 0.799 = 79.9%
  • P(Biden win > 15%) = 1 - \int_{-\infty}^{15} f(x) dx = 0.710 = 71.0%
  • P(Biden win > 20%) = 1 - \int_{-\infty}^{20} f(x) dx = 0.567 = 56.7%

What these calculations show is that the probability that Biden would have won by more than 5% before Clyburn’s endorsement was 90.4%. The probability that Biden would have won by more than 10% before Clyburn’s endorsement was 79.9%. The probability that Biden would have won by more than 20% before Clyburn’s endorsement was 56.7%, and so on.

Given these calculations, it actually seems unlikely that Clyburn’s endorsement made a huge impact on Biden’s win in South Carolina. This analysis shows that Biden would have likely won by more 15%-20% regardless.

The Probability of An Illegal Immigrant Committing a Crime In The United States

Trump has once again put The U.S. on the world stage this time at the expense of innocent children whose families are seeking asylum. The Trump administration’s justification is that:

 

“They want to have illegal immigrants pouring into our country, bringing with them crime, tremendous amounts of crime.”

 

I decided to try to analyze this statement quantitatively. Indeed, one can calculate the probability that an illegal immigrant will commit a crime within The United States as follows. Let us denote crime (or criminal) by C, while denoting illegal immigrant by ii. Then, by Bayes’ theorem, we have:

\boxed{P(C | ii) = \frac{P(ii | C) P(c)}{P(ii)}}

It is quite easy to find data associated with the various factors in this formula. For example, one finds that

  1. P(ii |c) = 0.21
  2. P(c) = 0.02
  3. P(ii) = 0.037

Putting all of this together, we find that:

P(C|ii) = 0.1135 = 11.35 \%

That is, the probability that an illegal immigrant will commit a crime (of any type) while in The United States is a very low 11.35%.

 

Therefore, Trump’s claim of “tremendous amounts of crime” being brought to The United States by illegal immigrants is incorrect.

 

Note that, the numerical factors used above were obtained from:

  1. https://www.justice.gov/opa/pr/departments-justice-and-homeland-security-release-data-incarcerated-aliens-94-percent-all
  2. https://www.washingtontimes.com/news/2017/aug/1/immigrants-22-percent-federal-prison-population/
  3. https://en.wikipedia.org/wiki/Incarceration_in_the_United_States

 

 

 

The Trump Rally, Really?

Today, The Dow Jones Industrial Average (DJIA) surpassed the 20,000 mark for the first time in history. At the time of the writing of this posting (12:31 PM on January 25), it is actually 20,058.29, so, I am not sure if it will close above 20,000 points, but, nevertheless, a lot of people are crediting this to Trump’s presidency, but I’m not so sure you can do that. First, the point must be made, that it is really the Obama economic policies that set the stage for this. On January 20, 2009, when Obama was sworn in, the Dow closed at 7949.089844 points. On November 8, 2016, when Trump won the election, the Dow closed at 18332.74023. So, during the Obama administration, the Dow increased by approximately 130.63%. I just wanted to make that point.

Now, the question that I wanted to investigate was would the Dow have closed past 20,000 points had Trump not been elected president. That is, assuming that the Obama administration policies and subsequent effects on the Dow were allowed to continue, would the Dow have surpassed 20,000 points.

For this, I looked at the DJIA data from January 20, 2009 (Obama’s first inauguration) to November 08, 2016 (Trump’s election). I specifically calculated the daily returns and discovered that they are approximately normally distributed using a kernel density method:

obamadowpdf

Importantly, one can calculate that the mean daily returns, \mu = 0.00045497596503813, while the volatility in daily returns, \sigma = 0.0100872666938282. Indeed, the volatility in daily returns for the DJIA was found to be relatively high during this period. Finally, the DJIA closed at 18332.74023 points on election night, November 08, 2016, which was 53 business days ago.

The daily dynamics of the DJIA can be modelled by the following stochastic differential equation:

S_{t} = S_{t-1} + \mu S_{t-1} dt + \sigma S_{t-1} dW,

where dW denotes a Wiener/Brownian motion process. Simulating this on computer, I ran 2,000,000 Monte Carlo simulations to simulate the DJIA closing price 53 business days from November 08, 2016, that is, January 25, 2017. The results of some of these simulations are shown below:

djiaclosingvaluesims

We concluded the following from our simulation. At the end of January 25, 2017, the DJIA was predicted to close at:

18778.51676 \pm 1380.42445

That is, the DJIA would be expected to close anywhere between 17398.0923062336 and 20158.94121. This range, albeit wide, is due to the high volatility of the daily returns in the DJIA, but, as you can see, it is perfectly feasible that the DJIA would have surpassed 20,000 points if Trump would not have been elected president.

Further, perhaps what is of more importance is the probability that the DJIA would surpass 20,000 points at any time during this 54-day period. We found the following:

probofexceeding

One sees that there is an almost 20% (more precisely, 18.53%) probability that the DJIA would close above 20,000 points on January 25, 2017 had Trump not been elected president. Since, by all accounts, the DJIA exceeding 20,000 points is considered to be an extremely rare/historic event, the fact that the probability is found to be almost 20% is actually quite significant, and shows, that it is quite likely that a Trump administration actually has little to do with the DJIA exceeding 20,000 points.

Although, this simulation was just for 53 working days from Nov 08, 2016, one can see that the probability of the DJIA exceeding 20,000 at closing day is monotonically increasing with every passing day. It is therefore quite feasible to conclude that Trump being president actually has little to do with the DJIA exceeding 20,000 points, rather, one can really attribute it to the day-to-day volatility of the DJIA!

The Relationship Between The Electoral College and Popular Vote

An interesting machine learning problem: Can one figure out the relationship between the popular vote margin, voter turnout, and the percentage of electoral college votes a candidate wins? Going back to the election of John Quincy Adams, the raw data looks like this:

Electoral College Party Popular vote  Margin (%)

Turnout

Percentage of EC

John Quincy Adams D.-R. -0.1044 0.27 0.3218
Andrew Jackson Dem. 0.1225 0.58 0.68
Andrew Jackson Dem. 0.1781 0.55 0.7657
Martin Van Buren Dem. 0.14 0.58 0.5782
William Henry Harrison Whig 0.0605 0.80 0.7959
James Polk Dem. 0.0145 0.79 0.6182
Zachary Taylor Whig 0.0479 0.73 0.5621
Franklin Pierce Dem. 0.0695 0.70 0.8581
James Buchanan Dem. 0.12 0.79 0.5878
Abraham Lincoln Rep. 0.1013 0.81 0.5941
Abraham Lincoln Rep. 0.1008 0.74 0.9099
Ulysses Grant Rep. 0.0532 0.78 0.7279
Ulysses Grant Rep. 0.12 0.71 0.8195
Rutherford Hayes Rep. -0.03 0.82 0.5014
James Garfield Rep. 0.0009 0.79 0.5799
Grover Cleveland Dem. 0.0057 0.78 0.5461
Benjamin Harrison Rep. -0.0083 0.79 0.58
Grover Cleveland Dem. 0.0301 0.75 0.6239
William McKinley Rep. 0.0431 0.79 0.6063
William McKinley Rep. 0.0612 0.73 0.6532
Theodore Roosevelt Rep. 0.1883 0.65 0.7059
William Taft Rep. 0.0853 0.65 0.6646
Woodrow Wilson Dem. 0.1444 0.59 0.8192
Woodrow Wilson Dem. 0.0312 0.62 0.5217
Warren Harding Rep. 0.2617 0.49 0.7608
Calvin Coolidge Rep. 0.2522 0.49 0.7194
Herbert Hoover Rep. 0.1741 0.57 0.8362
Franklin Roosevelt Dem. 0.1776 0.57 0.8889
Franklin Roosevelt Dem. 0.2426 0.61 0.9849
Franklin Roosevelt Dem. 0.0996 0.63 0.8456
Franklin Roosevelt Dem. 0.08 0.56 0.8136
Harry Truman Dem. 0.0448 0.53 0.5706
Dwight Eisenhower Rep. 0.1085 0.63 0.8324
Dwight Eisenhower Rep. 0.15 0.61 0.8606
John Kennedy Dem. 0.0017 0.6277 0.5642
Lyndon Johnson Dem. 0.2258 0.6192 0.9033
Richard Nixon Rep. 0.01 0.6084 0.5595
Richard Nixon Rep. 0.2315 0.5521 0.9665
Jimmy Carter Dem. 0.0206 0.5355 0.55
Ronald Reagan Rep. 0.0974 0.5256 0.9089
Ronald Reagan Rep. 0.1821 0.5311 0.9758
George H. W. Bush Rep. 0.0772 0.5015 0.7918
Bill Clinton Dem. 0.0556 0.5523 0.6877
Bill Clinton Dem. 0.0851 0.4908 0.7045
George W. Bush Rep. -0.0051 0.51 0.5037
George W. Bush Rep. 0.0246 0.5527 0.5316
Barack Obama Dem. 0.0727 0.5823 0.6784
Barack Obama Dem. 0.0386 0.5487 0.6171

Clearly, the percentage of electoral college votes a candidate depends nonlinearly on the voter turnout percentage and popular vote margin (%) as this non-parametric regression shows:

electoralmap.png

We therefore chose to perform a nonlinear regression using neural networks, for which our structure was:

nnetplot

As is turns out, this simple neural network structure with one hidden layer gave the lowest test error, which was 0.002496419 in this case.

Now, looking at the most recent national polls for the upcoming election, we see that Hillary Clinton has a 6.1% lead in the popular vote. Our neural network model then predicts the following:

Simulation Popular Vote Margin Percentage of Voter Turnout Predicted Percentage of Electoral College Votes (+/- 0.04996417)
1 0.061 0.30 0.6607371
2 0.061 0.35 0.6647464
3 0.061 0.40 0.6687115
4 0.061 0.45 0.6726314
5 0.061 0.50 0.6765048
6 0.061 0.55 0.6803307
7 0.061 0.60 0.6841083
8 0.061 0.65 0.6878366
9 0.061 0.70 0.6915149
10 0.061 0.75 0.6951424

One sees that even for an extremely low voter turnout (30%), at this point Hillary Clinton can expect to win the Electoral College by a margin of 61.078% to 71.07013%, or 328 to 382 electoral college votes. Therefore, what seems like a relatively small lead in the popular vote (6.1%) translates according to this neural network model into a large margin of victory in the electoral college.

One can see that the predicted percentage of electoral college votes really depends on popular vote margin and voter turnout. For example, if we reduce the popular vote margin to 1%, the results are less promising for the leading candidate:

Pop.Vote Margin Voter Turnout % E.C. % Win E.C% Win Best Case E.C.% Win Worst Case
0.01 0.30 0.5182854 0.4675000 0.5690708
0.01 0.35 0.5244157 0.4736303 0.5752011
0.01 0.40 0.5305820 0.4797967 0.5813674
0.01 0.45 0.5367790 0.4859937 0.5875644
0.01 0.50 0.5430013 0.4922160 0.5937867
0.01 0.55 0.5492434 0.4984580 0.6000287
0.01 0.60 0.5554995 0.5047141 0.6062849
0.01 0.65 0.5617642 0.5109788 0.6125496
0.01 0.70 0.5680317 0.5172463 0.6188171
0.01 0.75 0.5742963 0.5235109 0.6250817

One sees that if the popular vote margin is just 1% for the leading candidate, that candidate is not in the clear unless the popular vote exceeds 60%.

 

Optimal Strategies for the Clinton/Trump Debate

Consider modelling the Clinton/Trump debate via a static game in which each candidate can choose between two strategies: \{A,P\}, where A denotes predominantly “attacking” the other candidate, while P denotes predominantly discussing policy positions.

Further, let us consider the mixed strategies \sigma_1 = (p,1-p) for Clinton, and \sigma_2 = (q,1-q) for Trump. That is, Clinton predominantly attacks Trump with probability p, and Trump predominantly attacks Clinton with probability q.

Let us first deal with the general case of arbitrary payoffs, thus, generating the following payoff matrix:

\left( \begin{array}{cc} \{a,b\} & \{c,d\} \\ \{e,f\} & \{g,h\} \\ \end{array} \right)

That is, if Clinton attacks Trump and Trump attacks Clinton, the payoff to Clinton is a, while the payoff to Trump is b. If Clinton attacks Trump, and Trump ignores and discusses policy positions instead, the payoff to Clinton is c, while the payoff to trump is d. If Clinton discusses policy positions while Trump attacks, the payoff to Clinton is e, while the payoff to Trump is f, and if both candidates discuss policy positions instead of attacking each other, the payoff to them both will be g and h respectively.

With this information in hand, we can calculate the payoff to Clinton as:

\pi_c(\sigma_1, \sigma_2) = a p q+c p (1-q)+e (1-p) q+g (1-p) (1-q)

while the payoff to Trump is:

\pi_t(\sigma_1,\sigma_2) = b p q+d p (1-q)+f (1-p) q+h (1-p) (1-q)

With these payoff functions, we can compute each candidate’s best response to the other candidate by solving the following equations:

\hat{\sigma}_1 \in \text{argmax}_{\sigma_1} \pi_1(\sigma_1,\sigma_2)

\hat{\sigma}_{2} \in \text{argmax}_{\sigma_2} \pi_2(\sigma_1,\sigma_2)

where \hat{\sigma}_{1,2} indicates the best response strategy to a fixed strategy for the other player.

Solving these equations, we obtain the following:

If

latex-image-32
then,

Clinton’s best response is to choose p = 1/2.

If

latex-image-33

then,

Clinton’s best response is to choose  p = 1.

Otherwise, her best response is to choose p = 0.

 

While for Trump, the best responses are computed as follows:

If

latex-image-34

Trump’s best response is to choose q = 1/2.

If

latex-image-35

Trump’s best response is to choose q = 1.

Otherwise, Trump’s best response is to choose q = 0.

To demonstrate this, let us work out an example. Assume (for this example) that the payoffs for each candidate are to sway independent voters / voters that have not made up their minds. Further, let us assume that these voters are more interested in policy positions, and will take attacks negatively. Obviously, this is not necessarily true, and we have solved the general case above. We are just using the following payoff matrix for demonstration purposes:

\left( \begin{array}{cc} \{-1,-1\} & \{-1,1\} \\ \{1,-1\} & \{1,1\} \\ \end{array} \right)

 

Using the above equations, we see that if 0 \leq q \leq 1, Clinton’s best response is to choose p=0. While, if 0 \leq p \leq 1, Trump’s best response is to choose q =0. That is, no matter what Trump’s strategy is, it is always Clinton’s best response to discuss policy positions. No matter what Clinton’s strategy is, it is always Trump’s best response to discuss policy positions as well. The two candidates’ payoff functions take the following form:

payofffuncs

What this shows for example is that there is a Nash equilibrium of:

(\sigma_1^{*}, \sigma_{2}^{*}) = (0,0).

The expected payoffs for each candidate are evidently

\pi_c = \pi_t = 1.

Let us work out an another example. This time, assume that if Clinton attacks Trump, she receives a payoff of +1, while if Trump attacks Clinton, he receives a payoff of -1. While, if Clinton discusses policy, while being attacked by Trump, she receives a payoff of +1, while Trump receives a payoff of -1. On the other hand, if Trump discusses policy while being attacked by Clinton, he receives a payoff +1, while Clinton receives a payoff of -1. If Clinton discusses policy, while Trump discusses policy, she receives a payoff of +1, while Trump receives a payoff of -1. The payoff matrix is evidently:

\left( \begin{array}{cc} \{1,-1\} & \{1,-1\} \\ \{1,-1\} & \{1,-1\} \\ \end{array} \right)

In this case, if 0 \leq q \leq 1, then Clinton’s best response is to choose p = 1/2. While, if 0 \leq p \leq 1, then Trump’s best response is to choose q = 1/2. The Nash equilibrium is evidently

(\sigma_1^{*}, \sigma_{2}^{*}) = (1/2,1/2).

The expected payoffs for each candidate are evidently

\pi_c = 1, \pi_t = -1.

In this example,  even though it is the optimal strategy for each candidate to play a mixed strategy of 50% attack, 50% discuss policy, Clinton is expected to benefit, while Trump is expected to lose.

Let us also consider an example of where the audience is biased towards Trump. So, every time Trump attacks Clinton, he gains an additional point. Every time Trump discusses policy, while Clinton does the same he gains an additional point. While, if Clinton attacks while Trump discusses policy positions, she will lose a point, and he gains a point. Such a payoff matrix can be given by:

\left( \begin{array}{cc} \{1,2\} & \{-1,1\} \\ \{0,1\} & \{0,1\} \\ \end{array} \right)

Solving the equations above, we find that if q = 1/2, Clinton’s best response is to choose p =1/2. If 1/2 < q \leq 1, Clinton’s best response is to choose p = 1. Otherwise, her best response is to choose p = 0. On the other hand, if p = 0, Trump’s best response is to choose q = 1/2. While, if 0 < p \leq 1, Trump’s best response is to choose q = 1. Evidently, there is a single Nash equilibrium (as long as 1/2 < p \leq 1):

 (\sigma_1^{*}, \sigma_{2}^{*}) = (1,1).

Therefore, in this situation, it is each candidate’s best strategy to attack one another. It is interesting that even in an audience that is heavily biased towards Trump, Clinton’s best strategy is still to attack 100% of the time.

The interested reader is invited to experiment with different scenarios using the general results derived above.

2016 Real-Time Election Predictions

Further to my original post on using physics to predict the outcome of the 2016 US Presidential elections, I have now written a cloud-based app using the powerful Wolfram Cloud to pull the most recent polling data on the web from The HuffPost Pollster, which “tracks thousands of public polls to give you the latest data on elections, political opinions and more”.  This app works in real-time and applies my PDE-solver / machine learning based algorithm to predict the probability of a candidate winning a state assuming the election is held tomorrow.

The app can be accessed by clicking the image below: (Note: If you obtain some type of server error, it means Wolfram’s server is busy, a refresh usually works. Also, results are only computed for states for which there exists reliable polling data. )