Experiment 3: Amazon Clothes
Basic information
- Task: Sentiment analysis (of product reviews)
- Dataset: Amazon Clothes
- Classes: Negative or Positive
- Train/Dev/Test examples: 3000 / 300 / 10000
- Problem: The models trained using this dataset may not generalize well to other domains of review texts
- Out-of-domain test sets: Amazon Music (8302 examples), Amazon Mixed (100000 examples), Yelp (38000 examples)
- For more details, please see section 7 in the paper.
Word Clouds & Annotations
Model 1: AmazonClothes_CNN_20200509013036
| Feature 0 | Feature 1 | Feature 2 | Feature 3 | Feature 4 | Feature 5 | Feature 6 | Feature 7 | Feature 8 | Feature 9 | Feature 10 | Feature 11 | Feature 12 | Feature 13 | Feature 14 | Feature 15 | Feature 16 | Feature 17 | Feature 18 | Feature 19 | Feature 20 | Feature 21 | Feature 22 | Feature 23 | Feature 24 | Feature 25 | Feature 26 | Feature 27 | Feature 28 | Feature 29 |
 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Model weights: - Negative = -0.064 - Positive = -0.308 | Model weights: - Negative = -0.032 - Positive = -0.270 | Model weights: - Negative = 0.409 - Positive = -0.039 | Model weights: - Negative = -0.438 - Positive = 0.272 | Model weights: - Negative = -0.092 - Positive = 0.343 | Model weights: - Negative = -0.446 - Positive = -0.106 | Model weights: - Negative = 0.294 - Positive = -0.005 | Model weights: - Negative = 0.239 - Positive = -0.406 | Model weights: - Negative = -0.341 - Positive = 0.284 | Model weights: - Negative = -0.312 - Positive = -0.088 | Model weights: - Negative = 0.185 - Positive = -0.139 | Model weights: - Negative = -0.213 - Positive = 0.047 | Model weights: - Negative = -0.040 - Positive = -0.316 | Model weights: - Negative = -0.346 - Positive = 0.094 | Model weights: - Negative = 0.165 - Positive = -0.408 | Model weights: - Negative = 0.362 - Positive = -0.289 | Model weights: - Negative = 0.426 - Positive = -0.333 | Model weights: - Negative = -0.333 - Positive = -0.044 | Model weights: - Negative = 0.514 - Positive = -0.521 | Model weights: - Negative = -0.459 - Positive = -0.040 | Model weights: - Negative = 0.069 - Positive = 0.415 | Model weights: - Negative = -0.355 - Positive = 0.325 | Model weights: - Negative = -0.522 - Positive = -0.212 | Model weights: - Negative = 0.068 - Positive = -0.263 | Model weights: - Negative = -0.257 - Positive = 0.324 | Model weights: - Negative = -0.152 - Positive = 0.197 | Model weights: - Negative = -0.068 - Positive = 0.280 | Model weights: - Negative = -0.084 - Positive = 0.584 | Model weights: - Negative = 0.205 - Positive = -0.376 | Model weights: - Negative = 0.098 - Positive = -0.435 |
| Human answers: - Negative = 9 - Positive = 1 - Neither = 0 | Human answers: - Negative = 3 - Positive = 3 - Neither = 4 | Human answers: - Negative = 6 - Positive = 3 - Neither = 1 | Human answers: - Negative = 1 - Positive = 4 - Neither = 5 | Human answers: - Negative = 2 - Positive = 7 - Neither = 1 | Human answers: - Negative = 1 - Positive = 0 - Neither = 9 | Human answers: - Negative = 5 - Positive = 3 - Neither = 2 | Human answers: - Negative = 6 - Positive = 2 - Neither = 2 | Human answers: - Negative = 2 - Positive = 5 - Neither = 3 | Human answers: - Negative = 3 - Positive = 7 - Neither = 0 | Human answers: - Negative = 5 - Positive = 2 - Neither = 3 | Human answers: - Negative = 0 - Positive = 9 - Neither = 1 | Human answers: - Negative = 8 - Positive = 2 - Neither = 0 | Human answers: - Negative = 1 - Positive = 4 - Neither = 5 | Human answers: - Negative = 9 - Positive = 1 - Neither = 0 | Human answers: - Negative = 6 - Positive = 1 - Neither = 3 | Human answers: - Negative = 7 - Positive = 1 - Neither = 2 | Human answers: - Negative = 2 - Positive = 7 - Neither = 1 | Human answers: - Negative = 7 - Positive = 2 - Neither = 1 | Human answers: - Negative = 1 - Positive = 7 - Neither = 2 | Human answers: - Negative = 1 - Positive = 9 - Neither = 0 | Human answers: - Negative = 0 - Positive = 10 - Neither = 0 | Human answers: - Negative = 1 - Positive = 7 - Neither = 2 | Human answers: - Negative = 3 - Positive = 0 - Neither = 7 | Human answers: - Negative = 1 - Positive = 6 - Neither = 3 | Human answers: - Negative = 0 - Positive = 5 - Neither = 5 | Human answers: - Negative = 5 - Positive = 2 - Neither = 3 | Human answers: - Negative = 0 - Positive = 5 - Neither = 5 | Human answers: - Negative = 7 - Positive = 0 - Neither = 3 | Human answers: - Negative = 7 - Positive = 1 - Neither = 2 |
| Decision: - MTurk: Enabled | Decision: - MTurk: Disabled | Decision: - MTurk: Enabled | Decision: - MTurk: Disabled | Decision: - MTurk: Enabled | Decision: - MTurk: Disabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Disabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Disabled | Decision: - MTurk: Enabled | Decision: - MTurk: Disabled | Decision: - MTurk: Disabled | Decision: - MTurk: Disabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled |
Model 2: AmazonClothes_CNN_20200509013945
| Feature 0 | Feature 1 | Feature 2 | Feature 3 | Feature 4 | Feature 5 | Feature 6 | Feature 7 | Feature 8 | Feature 9 | Feature 10 | Feature 11 | Feature 12 | Feature 13 | Feature 14 | Feature 15 | Feature 16 | Feature 17 | Feature 18 | Feature 19 | Feature 20 | Feature 21 | Feature 22 | Feature 23 | Feature 24 | Feature 25 | Feature 26 | Feature 27 | Feature 28 | Feature 29 |
 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Model weights: - Negative = -0.246 - Positive = 0.095 | Model weights: - Negative = -0.070 - Positive = -0.250 | Model weights: - Negative = 0.049 - Positive = -0.474 | Model weights: - Negative = 0.325 - Positive = -0.408 | Model weights: - Negative = 0.269 - Positive = -0.400 | Model weights: - Negative = 0.430 - Positive = 0.157 | Model weights: - Negative = -0.457 - Positive = -0.158 | Model weights: - Negative = -0.314 - Positive = -0.057 | Model weights: - Negative = -0.440 - Positive = 0.239 | Model weights: - Negative = -0.095 - Positive = 0.297 | Model weights: - Negative = 0.361 - Positive = -0.435 | Model weights: - Negative = -0.244 - Positive = 0.208 | Model weights: - Negative = -0.502 - Positive = 0.197 | Model weights: - Negative = -0.339 - Positive = 0.396 | Model weights: - Negative = -0.372 - Positive = 0.303 | Model weights: - Negative = -0.278 - Positive = 0.394 | Model weights: - Negative = -0.239 - Positive = 0.334 | Model weights: - Negative = 0.443 - Positive = 0.037 | Model weights: - Negative = 0.042 - Positive = 0.378 | Model weights: - Negative = -0.134 - Positive = -0.451 | Model weights: - Negative = 0.197 - Positive = -0.271 | Model weights: - Negative = 0.137 - Positive = -0.223 | Model weights: - Negative = -0.274 - Positive = 0.347 | Model weights: - Negative = 0.321 - Positive = -0.134 | Model weights: - Negative = 0.270 - Positive = -0.360 | Model weights: - Negative = -0.082 - Positive = 0.422 | Model weights: - Negative = 0.460 - Positive = 0.018 | Model weights: - Negative = 0.415 - Positive = -0.044 | Model weights: - Negative = 0.020 - Positive = -0.325 | Model weights: - Negative = 0.155 - Positive = -0.369 |
| Human answers: - Negative = 0 - Positive = 10 - Neither = 0 | Human answers: - Negative = 8 - Positive = 1 - Neither = 1 | Human answers: - Negative = 10 - Positive = 0 - Neither = 0 | Human answers: - Negative = 6 - Positive = 2 - Neither = 2 | Human answers: - Negative = 1 - Positive = 2 - Neither = 7 | Human answers: - Negative = 10 - Positive = 0 - Neither = 0 | Human answers: - Negative = 0 - Positive = 10 - Neither = 0 | Human answers: - Negative = 0 - Positive = 10 - Neither = 0 | Human answers: - Negative = 0 - Positive = 10 - Neither = 0 | Human answers: - Negative = 0 - Positive = 10 - Neither = 0 | Human answers: - Negative = 10 - Positive = 0 - Neither = 0 | Human answers: - Negative = 0 - Positive = 10 - Neither = 0 | Human answers: - Negative = 0 - Positive = 9 - Neither = 1 | Human answers: - Negative = 0 - Positive = 9 - Neither = 1 | Human answers: - Negative = 0 - Positive = 9 - Neither = 1 | Human answers: - Negative = 0 - Positive = 4 - Neither = 6 | Human answers: - Negative = 0 - Positive = 10 - Neither = 0 | Human answers: - Negative = 9 - Positive = 0 - Neither = 1 | Human answers: - Negative = 0 - Positive = 10 - Neither = 0 | Human answers: - Negative = 10 - Positive = 0 - Neither = 0 | Human answers: - Negative = 8 - Positive = 0 - Neither = 2 | Human answers: - Negative = 7 - Positive = 0 - Neither = 3 | Human answers: - Negative = 0 - Positive = 10 - Neither = 0 | Human answers: - Negative = 9 - Positive = 0 - Neither = 1 | Human answers: - Negative = 8 - Positive = 0 - Neither = 2 | Human answers: - Negative = 0 - Positive = 10 - Neither = 0 | Human answers: - Negative = 7 - Positive = 0 - Neither = 3 | Human answers: - Negative = 7 - Positive = 0 - Neither = 3 | Human answers: - Negative = 10 - Positive = 0 - Neither = 0 | Human answers: - Negative = 9 - Positive = 0 - Neither = 1 |
| Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Disabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Disabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled |
Model 3: AmazonClothes_CNN_20200509015329
| Feature 0 | Feature 1 | Feature 2 | Feature 3 | Feature 4 | Feature 5 | Feature 6 | Feature 7 | Feature 8 | Feature 9 | Feature 10 | Feature 11 | Feature 12 | Feature 13 | Feature 14 | Feature 15 | Feature 16 | Feature 17 | Feature 18 | Feature 19 | Feature 20 | Feature 21 | Feature 22 | Feature 23 | Feature 24 | Feature 25 | Feature 26 | Feature 27 | Feature 28 | Feature 29 |
 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Model weights: - Negative = -0.438 - Positive = 0.013 | Model weights: - Negative = -0.311 - Positive = 0.448 | Model weights: - Negative = 0.399 - Positive = 0.122 | Model weights: - Negative = 0.147 - Positive = -0.294 | Model weights: - Negative = 0.178 - Positive = -0.413 | Model weights: - Negative = -0.391 - Positive = -0.144 | Model weights: - Negative = 0.447 - Positive = -0.097 | Model weights: - Negative = 0.033 - Positive = -0.199 | Model weights: - Negative = -0.194 - Positive = -0.432 | Model weights: - Negative = 0.108 - Positive = -0.044 | Model weights: - Negative = -0.128 - Positive = 0.074 | Model weights: - Negative = 0.088 - Positive = 0.380 | Model weights: - Negative = -0.072 - Positive = 0.345 | Model weights: - Negative = -0.021 - Positive = -0.436 | Model weights: - Negative = -0.462 - Positive = 0.037 | Model weights: - Negative = 0.283 - Positive = 0.090 | Model weights: - Negative = -0.052 - Positive = 0.286 | Model weights: - Negative = -0.317 - Positive = 0.245 | Model weights: - Negative = 0.338 - Positive = -0.486 | Model weights: - Negative = -0.245 - Positive = -0.483 | Model weights: - Negative = -0.352 - Positive = 0.409 | Model weights: - Negative = 0.400 - Positive = 0.072 | Model weights: - Negative = -0.093 - Positive = 0.261 | Model weights: - Negative = -0.163 - Positive = 0.091 | Model weights: - Negative = 0.148 - Positive = -0.116 | Model weights: - Negative = 0.213 - Positive = -0.135 | Model weights: - Negative = -0.414 - Positive = -0.075 | Model weights: - Negative = -0.259 - Positive = -0.499 | Model weights: - Negative = 0.377 - Positive = -0.262 | Model weights: - Negative = -0.293 - Positive = 0.523 |
| Human answers: - Negative = 0 - Positive = 10 - Neither = 0 | Human answers: - Negative = 1 - Positive = 4 - Neither = 5 | Human answers: - Negative = 4 - Positive = 1 - Neither = 5 | Human answers: - Negative = 1 - Positive = 0 - Neither = 9 | Human answers: - Negative = 9 - Positive = 1 - Neither = 0 | Human answers: - Negative = 1 - Positive = 4 - Neither = 5 | Human answers: - Negative = 8 - Positive = 1 - Neither = 1 | Human answers: - Negative = 8 - Positive = 2 - Neither = 0 | Human answers: - Negative = 0 - Positive = 0 - Neither = 10 | Human answers: - Negative = 9 - Positive = 1 - Neither = 0 | Human answers: - Negative = 1 - Positive = 9 - Neither = 0 | Human answers: - Negative = 0 - Positive = 10 - Neither = 0 | Human answers: - Negative = 0 - Positive = 10 - Neither = 0 | Human answers: - Negative = 9 - Positive = 0 - Neither = 1 | Human answers: - Negative = 0 - Positive = 8 - Neither = 2 | Human answers: - Negative = 1 - Positive = 0 - Neither = 9 | Human answers: - Negative = 1 - Positive = 7 - Neither = 2 | Human answers: - Negative = 0 - Positive = 10 - Neither = 0 | Human answers: - Negative = 6 - Positive = 0 - Neither = 4 | Human answers: - Negative = 6 - Positive = 0 - Neither = 4 | Human answers: - Negative = 0 - Positive = 1 - Neither = 9 | Human answers: - Negative = 9 - Positive = 0 - Neither = 1 | Human answers: - Negative = 0 - Positive = 10 - Neither = 0 | Human answers: - Negative = 0 - Positive = 8 - Neither = 2 | Human answers: - Negative = 8 - Positive = 0 - Neither = 2 | Human answers: - Negative = 10 - Positive = 0 - Neither = 0 | Human answers: - Negative = 0 - Positive = 10 - Neither = 0 | Human answers: - Negative = 10 - Positive = 0 - Neither = 0 | Human answers: - Negative = 2 - Positive = 0 - Neither = 8 | Human answers: - Negative = 0 - Positive = 10 - Neither = 0 |
| Decision: - MTurk: Enabled | Decision: - MTurk: Disabled | Decision: - MTurk: Disabled | Decision: - MTurk: Disabled | Decision: - MTurk: Enabled | Decision: - MTurk: Disabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Disabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Disabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Disabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Enabled | Decision: - MTurk: Disabled | Decision: - MTurk: Enabled |
Results
Results (Average ± SD) of Experiment 3: Amazon Clothes, CNNs; Boldface numbers are the best scores in the columns. They are further underlined if they are significantly better than the scores of all the other models (based on approximate randomization test with α = 0.05)
Downloads
- Wordclouds and annotations
- The dataset of this experiment as well as other experiments can be downloaded here.
- If you want to use the original trained models in the experiments, please contact Piyawat (pl1515 [at] imperial [dot] ac [dot] uk).