07 Interpreting two-way tables

%%html
<iframe width="700" height="400" src="https://www.youtube.com/embed/MarqSlyz-lU/" frameborder="0" allowfullscreen></iframe>

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import plotly.graph_objects as go
import seaborn as sns

import findspark

findspark.init()
from pyspark.context import SparkContext
from pyspark.sql.session import SparkSession

spark = SparkSession.builder.appName("statistics").master("local").getOrCreate()

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/home/runner/work/statistics/spark-3.1.3-bin-hadoop3.2/jars/spark-unsafe_2.12-3.1.3.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

22/07/21 02:32:22 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

dataset = {
    "Candidate": ["Obama", "Romney", "Other"],
    "Men": [0.42, 0.52, 0.06],
    "Women": [0.52, 0.43, 0.05],
}

df = pd.DataFrame(dataset).set_index("Candidate")
df = df.append(df.sum().rename("Total"))
df

/tmp/ipykernel_4332/3566769890.py:2: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
  df = df.append(df.sum().rename("Total"))

	Men	Women
Candidate
Obama	0.42	0.52
Romney	0.52	0.43
Other	0.06	0.05
Total	1.00	1.00

sdf = spark.createDataFrame(zip(*dataset.values()), schema=list(dataset.keys()))
sdf.registerTempTable("sdf_table")
sdf.show()

[Stage 0:>                                                          (0 + 1) / 1]

+---------+----+-----+
|Candidate| Men|Women|
+---------+----+-----+
|    Obama|0.42| 0.52|
|   Romney|0.52| 0.43|
|    Other|0.06| 0.05|
+---------+----+-----+

df.plot(kind="bar")

<AxesSubplot:xlabel='Candidate'>

../_images/07 Interpreting two-way tables_9_1.png

plt.bar(df.index, df["Men"])
plt.bar(df.index, df["Women"], alpha=0.6)
plt.xlabel("Candidate")
plt.show()

../_images/07 Interpreting two-way tables_10_0.png

melt_df = df.reset_index().melt(
    id_vars="Candidate", var_name="Gender", value_name="Percentage"
)
melt_df

	Candidate	Gender	Percentage
0	Obama	Men	0.42
1	Romney	Men	0.52
2	Other	Men	0.06
3	Total	Men	1.00
4	Obama	Women	0.52
5	Romney	Women	0.43
6	Other	Women	0.05
7	Total	Women	1.00

sns.catplot(x="Candidate", y="Percentage", hue="Gender", data=melt_df, kind="bar")
plt.show()

../_images/07 Interpreting two-way tables_12_0.png

data = [
    go.Bar(x=df.index, y=df["Men"], marker=dict(color="cornflowerblue"), name="Men"),
    go.Bar(x=df.index, y=df["Women"], marker=dict(color="orange"), name="Women"),
]
layout = go.Layout(xaxis=dict(title="Candidate"), yaxis=dict(title="Percentage"))
fig = go.Figure(data=data)
fig.show()

Statstics with Python

07 Interpreting two-way tables#