« Back to the main page

Experiment 3: 20Newsgroups & Religion

Basic information

• Task: Classify a given text into the right category
• Dataset: 20Newsgroups
• Classes: Atheism or Christian
• Train/Dev/Test examples: 863 / 216 / 717
• Problem: This dataset contains a lot of artifacts – tokens (e.g., person names, punctuation marks) which are not relevant, but strongly co-occur with one of the classes.
• Out-of-domain test set: Religion (1819 examples)
• For more details, please see section 7 in the paper.

Word Clouds & Annotations

Model 1: 20Newsgroup_standard_CNN_20200508185211

Feature 0

Feature 1

Feature 2

Feature 3

Feature 4

Feature 5

Feature 6

Feature 7

Feature 8

Feature 9

Feature 10

Feature 11

Feature 12

Feature 13

Feature 14

Feature 15

Feature 16

Feature 17

Feature 18

Feature 19

Feature 20

Feature 21

Feature 22

Feature 23

Feature 24

Feature 25

Feature 26

Feature 27

Feature 28

Feature 29

Model weights:    - Atheism = -0.136    - Christian = -0.265

Model weights:    - Atheism = -0.404    - Christian = -0.203

Model weights:    - Atheism = 0.429    - Christian = -0.169

Model weights:    - Atheism = 0.014    - Christian = -0.210

Model weights:    - Atheism = -0.353    - Christian = 0.384

Model weights:    - Atheism = 0.068    - Christian = -0.383

Model weights:    - Atheism = -0.027    - Christian = 0.457

Model weights:    - Atheism = 0.137    - Christian = -0.083

Model weights:    - Atheism = -0.236    - Christian = -0.045

Model weights:    - Atheism = -0.180    - Christian = 0.051

Model weights:    - Atheism = -0.293    - Christian = -0.424

Model weights:    - Atheism = 0.405    - Christian = -0.429

Model weights:    - Atheism = 0.434    - Christian = 0.155

Model weights:    - Atheism = 0.161    - Christian = -0.005

Model weights:    - Atheism = 0.365    - Christian = -0.187

Model weights:    - Atheism = 0.418    - Christian = 0.040

Model weights:    - Atheism = -0.214    - Christian = 0.021

Model weights:    - Atheism = 0.014    - Christian = -0.367

Model weights:    - Atheism = 0.188    - Christian = -0.110

Model weights:    - Atheism = -0.003    - Christian = 0.409

Model weights:    - Atheism = -0.364    - Christian = 0.495

Model weights:    - Atheism = 0.064    - Christian = -0.187

Model weights:    - Atheism = -0.324    - Christian = 0.210

Model weights:    - Atheism = 0.150    - Christian = -0.179

Model weights:    - Atheism = 0.285    - Christian = 0.161

Model weights:    - Atheism = 0.131    - Christian = 0.404

Model weights:    - Atheism = -0.222    - Christian = 0.139

Model weights:    - Atheism = 0.475    - Christian = 0.179

Model weights:    - Atheism = -0.142    - Christian = 0.093

Model weights:    - Atheism = -0.031    - Christian = 0.492

Human answers:    - Atheism = 1    - Christian = 1    - Neither = 8

Human answers:    - Atheism = 0    - Christian = 1    - Neither = 9

Human answers:    - Atheism = 6    - Christian = 0    - Neither = 4

Human answers:    - Atheism = 0    - Christian = 4    - Neither = 6

Human answers:    - Atheism = 1    - Christian = 1    - Neither = 8

Human answers:    - Atheism = 10    - Christian = 0    - Neither = 0

Human answers:    - Atheism = 0    - Christian = 7    - Neither = 3

Human answers:    - Atheism = 0    - Christian = 0    - Neither = 10

Human answers:    - Atheism = 0    - Christian = 2    - Neither = 8

Human answers:    - Atheism = 0    - Christian = 1    - Neither = 9

Human answers:    - Atheism = 10    - Christian = 0    - Neither = 0

Human answers:    - Atheism = 1    - Christian = 0    - Neither = 9

Human answers:    - Atheism = 4    - Christian = 0    - Neither = 6

Human answers:    - Atheism = 2    - Christian = 0    - Neither = 8

Human answers:    - Atheism = 0    - Christian = 1    - Neither = 9

Human answers:    - Atheism = 0    - Christian = 0    - Neither = 10

Human answers:    - Atheism = 1    - Christian = 9    - Neither = 0

Human answers:    - Atheism = 0    - Christian = 0    - Neither = 10

Human answers:    - Atheism = 10    - Christian = 0    - Neither = 0

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Human answers:    - Atheism = 1    - Christian = 0    - Neither = 9

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Human answers:    - Atheism = 2    - Christian = 2    - Neither = 6

Human answers:    - Atheism = 0    - Christian = 2    - Neither = 8

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Human answers:    - Atheism = 0    - Christian = 0    - Neither = 10

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Decision:    - MTurk: Disabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Model 2: 20Newsgroup_standard_CNN_20200508185727

Feature 0

Feature 1

Feature 2

Feature 3

Feature 4

Feature 5

Feature 6

Feature 7

Feature 8

Feature 9

Feature 10

Feature 11

Feature 12

Feature 13

Feature 14

Feature 15

Feature 16

Feature 17

Feature 18

Feature 19

Feature 20

Feature 21

Feature 22

Feature 23

Feature 24

Feature 25

Feature 26

Feature 27

Feature 28

Feature 29

Model weights:    - Atheism = -0.373    - Christian = -0.175

Model weights:    - Atheism = -0.128    - Christian = 0.192

Model weights:    - Atheism = 0.180    - Christian = -0.163

Model weights:    - Atheism = -0.047    - Christian = -0.224

Model weights:    - Atheism = 0.115    - Christian = -0.291

Model weights:    - Atheism = 0.037    - Christian = 0.445

Model weights:    - Atheism = 0.068    - Christian = -0.326

Model weights:    - Atheism = 0.036    - Christian = -0.490

Model weights:    - Atheism = -0.224    - Christian = -0.071

Model weights:    - Atheism = 0.184    - Christian = 0.301

Model weights:    - Atheism = 0.011    - Christian = 0.259

Model weights:    - Atheism = -0.252    - Christian = -0.163

Model weights:    - Atheism = -0.183    - Christian = -0.025

Model weights:    - Atheism = 0.134    - Christian = 0.301

Model weights:    - Atheism = -0.083    - Christian = 0.436

Model weights:    - Atheism = 0.126    - Christian = 0.314

Model weights:    - Atheism = -0.238    - Christian = -0.505

Model weights:    - Atheism = -0.475    - Christian = 0.429

Model weights:    - Atheism = -0.033    - Christian = -0.216

Model weights:    - Atheism = 0.415    - Christian = -0.172

Model weights:    - Atheism = 0.219    - Christian = -0.003

Model weights:    - Atheism = -0.429    - Christian = -0.165

Model weights:    - Atheism = -0.415    - Christian = -0.054

Model weights:    - Atheism = 0.169    - Christian = -0.334

Model weights:    - Atheism = 0.211    - Christian = -0.450

Model weights:    - Atheism = 0.347    - Christian = 0.035

Model weights:    - Atheism = -0.073    - Christian = 0.066

Model weights:    - Atheism = -0.233    - Christian = -0.503

Model weights:    - Atheism = 0.142    - Christian = -0.136

Model weights:    - Atheism = -0.012    - Christian = 0.213

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Human answers:    - Atheism = 0    - Christian = 7    - Neither = 3

Human answers:    - Atheism = 10    - Christian = 0    - Neither = 0

Human answers:    - Atheism = 2    - Christian = 0    - Neither = 8

Human answers:    - Atheism = 0    - Christian = 5    - Neither = 5

Human answers:    - Atheism = 1    - Christian = 0    - Neither = 9

Human answers:    - Atheism = 0    - Christian = 0    - Neither = 10

Human answers:    - Atheism = 0    - Christian = 9    - Neither = 1

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Human answers:    - Atheism = 4    - Christian = 5    - Neither = 1

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Human answers:    - Atheism = 0    - Christian = 0    - Neither = 10

Human answers:    - Atheism = 0    - Christian = 3    - Neither = 7

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Human answers:    - Atheism = 4    - Christian = 0    - Neither = 6

Human answers:    - Atheism = 0    - Christian = 0    - Neither = 10

Human answers:    - Atheism = 6    - Christian = 0    - Neither = 4

Human answers:    - Atheism = 0    - Christian = 0    - Neither = 10

Human answers:    - Atheism = 10    - Christian = 0    - Neither = 0

Human answers:    - Atheism = 0    - Christian = 2    - Neither = 8

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Human answers:    - Atheism = 10    - Christian = 0    - Neither = 0

Human answers:    - Atheism = 9    - Christian = 0    - Neither = 1

Human answers:    - Atheism = 10    - Christian = 0    - Neither = 0

Human answers:    - Atheism = 0    - Christian = 1    - Neither = 9

Human answers:    - Atheism = 0    - Christian = 0    - Neither = 10

Human answers:    - Atheism = 1    - Christian = 0    - Neither = 9

Human answers:    - Atheism = 0    - Christian = 9    - Neither = 1

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Model 3: 20Newsgroup_standard_CNN_20200508191406

Feature 0

Feature 1

Feature 2

Feature 3

Feature 4

Feature 5

Feature 6

Feature 7

Feature 8

Feature 9

Feature 10

Feature 11

Feature 12

Feature 13

Feature 14

Feature 15

Feature 16

Feature 17

Feature 18

Feature 19

Feature 20

Feature 21

Feature 22

Feature 23

Feature 24

Feature 25

Feature 26

Feature 27

Feature 28

Feature 29

Model weights:    - Atheism = -0.107    - Christian = -0.399

Model weights:    - Atheism = -0.399    - Christian = -0.205

Model weights:    - Atheism = 0.472    - Christian = -0.073

Model weights:    - Atheism = -0.171    - Christian = 0.372

Model weights:    - Atheism = -0.426    - Christian = -0.092

Model weights:    - Atheism = 0.031    - Christian = -0.155

Model weights:    - Atheism = 0.403    - Christian = 0.048

Model weights:    - Atheism = -0.476    - Christian = 0.082

Model weights:    - Atheism = -0.392    - Christian = 0.302

Model weights:    - Atheism = 0.221    - Christian = 0.381

Model weights:    - Atheism = -0.181    - Christian = 0.213

Model weights:    - Atheism = 0.432    - Christian = -0.210

Model weights:    - Atheism = -0.013    - Christian = -0.312

Model weights:    - Atheism = -0.255    - Christian = 0.156

Model weights:    - Atheism = -0.167    - Christian = 0.320

Model weights:    - Atheism = -0.459    - Christian = 0.148

Model weights:    - Atheism = 0.002    - Christian = -0.352

Model weights:    - Atheism = -0.305    - Christian = 0.234

Model weights:    - Atheism = -0.273    - Christian = 0.070

Model weights:    - Atheism = -0.314    - Christian = -0.148

Model weights:    - Atheism = 0.306    - Christian = -0.361

Model weights:    - Atheism = -0.341    - Christian = -0.067

Model weights:    - Atheism = 0.329    - Christian = -0.273

Model weights:    - Atheism = 0.132    - Christian = -0.271

Model weights:    - Atheism = 0.489    - Christian = -0.337

Model weights:    - Atheism = 0.131    - Christian = -0.096

Model weights:    - Atheism = -0.211    - Christian = 0.317

Model weights:    - Atheism = 0.229    - Christian = -0.279

Model weights:    - Atheism = 0.466    - Christian = 0.255

Model weights:    - Atheism = -0.365    - Christian = 0.046

Human answers:    - Atheism = 0    - Christian = 0    - Neither = 10

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Human answers:    - Atheism = 0    - Christian = 2    - Neither = 8

Human answers:    - Atheism = 0    - Christian = 1    - Neither = 9

Human answers:    - Atheism = 0    - Christian = 9    - Neither = 1

Human answers:    - Atheism = 9    - Christian = 0    - Neither = 1

Human answers:    - Atheism = 0    - Christian = 1    - Neither = 9

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Human answers:    - Atheism = 0    - Christian = 2    - Neither = 8

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Human answers:    - Atheism = 1    - Christian = 2    - Neither = 7

Human answers:    - Atheism = 0    - Christian = 0    - Neither = 10

Human answers:    - Atheism = 1    - Christian = 0    - Neither = 9

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Human answers:    - Atheism = 0    - Christian = 9    - Neither = 1

Human answers:    - Atheism = 0    - Christian = 0    - Neither = 10

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Human answers:    - Atheism = 1    - Christian = 0    - Neither = 9

Human answers:    - Atheism = 1    - Christian = 9    - Neither = 0

Human answers:    - Atheism = 6    - Christian = 0    - Neither = 4

Human answers:    - Atheism = 9    - Christian = 0    - Neither = 1

Human answers:    - Atheism = 1    - Christian = 0    - Neither = 9

Human answers:    - Atheism = 10    - Christian = 0    - Neither = 0

Human answers:    - Atheism = 1    - Christian = 8    - Neither = 1

Human answers:    - Atheism = 10    - Christian = 0    - Neither = 0

Human answers:    - Atheism = 2    - Christian = 1    - Neither = 7

Human answers:    - Atheism = 0    - Christian = 10    - Neither = 0

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Enabled

Decision:    - MTurk: Disabled

Decision:    - MTurk: Enabled

Results

Results (Average ± SD) of Experiment 3: 20Newsgroups & Religion, CNNs; Boldface numbers are the best scores in the columns. They are further underlined if they are significantly better than the scores of all the other models (based on approximate randomization test with α = 0.05)

Downloads

• Wordclouds and annotations
• The dataset of this experiment as well as other experiments can be downloaded here.
• If you want to use the original trained models in the experiments, please contact Piyawat (pl1515 [at] imperial [dot] ac [dot] uk).

« Back to the main page