Experiment 1: Yelp
Basic information
- Task: Sentiment analysis (of restaurant reviews)
- Dataset: Yelp
- Classes: Negative or Positive
- Train/Dev/Test examples: 500 / 100 / 38000
- Problem: The training data is very small.
- For more details, please see section 5 in the paper.
Word Clouds & Annotations
Model 1: YelpSmall500_CNN_20200515014923
| Feature 0 | Feature 1 | Feature 2 | Feature 3 | Feature 4 | Feature 5 | Feature 6 | Feature 7 | Feature 8 | Feature 9 | Feature 10 | Feature 11 | Feature 12 | Feature 13 | Feature 14 | Feature 15 | Feature 16 | Feature 17 | Feature 18 | Feature 19 | Feature 20 | Feature 21 | Feature 22 | Feature 23 | Feature 24 | Feature 25 | Feature 26 | Feature 27 | Feature 28 | Feature 29 |
 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Model weights: - negative = 0.084 - positive = -0.039 | Model weights: - negative = -0.443 - positive = -0.198 | Model weights: - negative = -0.135 - positive = 0.137 | Model weights: - negative = 0.026 - positive = 0.136 | Model weights: - negative = -0.244 - positive = -0.100 | Model weights: - negative = -0.073 - positive = 0.440 | Model weights: - negative = -0.292 - positive = 0.086 | Model weights: - negative = -0.067 - positive = -0.025 | Model weights: - negative = 0.278 - positive = -0.020 | Model weights: - negative = 0.385 - positive = 0.209 | Model weights: - negative = 0.427 - positive = 0.058 | Model weights: - negative = -0.318 - positive = 0.125 | Model weights: - negative = 0.059 - positive = -0.431 | Model weights: - negative = 0.438 - positive = -0.388 | Model weights: - negative = -0.171 - positive = 0.045 | Model weights: - negative = -0.371 - positive = 0.289 | Model weights: - negative = -0.121 - positive = -0.307 | Model weights: - negative = 0.436 - positive = -0.174 | Model weights: - negative = -0.014 - positive = -0.393 | Model weights: - negative = 0.061 - positive = -0.134 | Model weights: - negative = -0.277 - positive = 0.003 | Model weights: - negative = 0.416 - positive = -0.425 | Model weights: - negative = -0.423 - positive = -0.179 | Model weights: - negative = 0.141 - positive = -0.326 | Model weights: - negative = -0.219 - positive = 0.315 | Model weights: - negative = -0.064 - positive = -0.316 | Model weights: - negative = 0.492 - positive = -0.066 | Model weights: - negative = -0.323 - positive = 0.418 | Model weights: - negative = 0.277 - positive = -0.406 | Model weights: - negative = 0.229 - positive = -0.306 |
| Human answers: - mostly negative = 4 - partially negative = 3 - neither = 3 - partially positive = 0 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 0 - partially positive = 0 - mostly positive = 10 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 2 - partially positive = 4 - mostly positive = 4 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 1 - partially positive = 1 - mostly positive = 8 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 5 - partially positive = 2 - mostly positive = 3 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 5 - partially positive = 3 - mostly positive = 2 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 0 - partially positive = 1 - mostly positive = 9 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 7 - partially positive = 2 - mostly positive = 1 | Human answers: - mostly negative = 9 - partially negative = 0 - neither = 0 - partially positive = 0 - mostly positive = 1 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 6 - partially positive = 1 - mostly positive = 3 | Human answers: - mostly negative = 2 - partially negative = 3 - neither = 4 - partially positive = 1 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 0 - partially positive = 4 - mostly positive = 6 | Human answers: - mostly negative = 10 - partially negative = 0 - neither = 0 - partially positive = 0 - mostly positive = 0 | Human answers: - mostly negative = 1 - partially negative = 3 - neither = 6 - partially positive = 0 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 0 - partially positive = 1 - mostly positive = 9 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 0 - partially positive = 2 - mostly positive = 8 | Human answers: - mostly negative = 7 - partially negative = 2 - neither = 0 - partially positive = 0 - mostly positive = 1 | Human answers: - mostly negative = 0 - partially negative = 2 - neither = 5 - partially positive = 3 - mostly positive = 0 | Human answers: - mostly negative = 7 - partially negative = 2 - neither = 1 - partially positive = 0 - mostly positive = 0 | Human answers: - mostly negative = 7 - partially negative = 3 - neither = 0 - partially positive = 0 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 3 - partially positive = 4 - mostly positive = 3 | Human answers: - mostly negative = 3 - partially negative = 4 - neither = 2 - partially positive = 0 - mostly positive = 1 | Human answers: - mostly negative = 0 - partially negative = 1 - neither = 1 - partially positive = 3 - mostly positive = 5 | Human answers: - mostly negative = 2 - partially negative = 2 - neither = 4 - partially positive = 1 - mostly positive = 1 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 0 - partially positive = 2 - mostly positive = 8 | Human answers: - mostly negative = 10 - partially negative = 0 - neither = 0 - partially positive = 0 - mostly positive = 0 | Human answers: - mostly negative = 9 - partially negative = 0 - neither = 0 - partially positive = 0 - mostly positive = 1 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 1 - partially positive = 3 - mostly positive = 6 | Human answers: - mostly negative = 3 - partially negative = 5 - neither = 1 - partially positive = 1 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 2 - neither = 7 - partially positive = 1 - mostly positive = 0 |
| Summary: - Average score: 1.1 - Rank: B | Summary: - Average score: 2.0 - Rank: A | Summary: - Average score: 1.2 - Rank: B | Summary: - Average score: 1.7 - Rank: A | Summary: - Average score: 0.8 - Rank: C | Summary: - Average score: 0.7 - Rank: C | Summary: - Average score: 1.9 - Rank: A | Summary: - Average score: 0.4 - Rank: C | Summary: - Average score: 1.6 - Rank: A | Summary: - Average score: -0.7 - Rank: C | Summary: - Average score: 0.6 - Rank: C | Summary: - Average score: 1.6 - Rank: B | Summary: - Average score: 2.0 - Rank: A | Summary: - Average score: 0.5 - Rank: C | Summary: - Average score: 1.9 - Rank: A | Summary: - Average score: 1.8 - Rank: A | Summary: - Average score: 1.4 - Rank: B | Summary: - Average score: -0.1 - Rank: C | Summary: - Average score: 1.6 - Rank: B | Summary: - Average score: 1.7 - Rank: A | Summary: - Average score: 1.0 - Rank: B | Summary: - Average score: 0.8 - Rank: C | Summary: - Average score: 1.2 - Rank: B | Summary: - Average score: 0.3 - Rank: C | Summary: - Average score: 1.8 - Rank: A | Summary: - Average score: 2.0 - Rank: A | Summary: - Average score: 1.6 - Rank: B | Summary: - Average score: 1.5 - Rank: B | Summary: - Average score: 1.0 - Rank: B | Summary: - Average score: 0.1 - Rank: C |
Model 2: YelpSmall500_CNN_20200515024859
| Feature 0 | Feature 1 | Feature 2 | Feature 3 | Feature 4 | Feature 5 | Feature 6 | Feature 7 | Feature 8 | Feature 9 | Feature 10 | Feature 11 | Feature 12 | Feature 13 | Feature 14 | Feature 15 | Feature 16 | Feature 17 | Feature 18 | Feature 19 | Feature 20 | Feature 21 | Feature 22 | Feature 23 | Feature 24 | Feature 25 | Feature 26 | Feature 27 | Feature 28 | Feature 29 |
 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Model weights: - negative = -0.330 - positive = -0.053 | Model weights: - negative = 0.050 - positive = 0.270 | Model weights: - negative = 0.462 - positive = -0.433 | Model weights: - negative = -0.362 - positive = 0.439 | Model weights: - negative = 0.396 - positive = -0.229 | Model weights: - negative = -0.092 - positive = -0.434 | Model weights: - negative = -0.175 - positive = -0.014 | Model weights: - negative = -0.285 - positive = -0.199 | Model weights: - negative = -0.439 - positive = -0.127 | Model weights: - negative = -0.075 - positive = 0.063 | Model weights: - negative = 0.137 - positive = -0.015 | Model weights: - negative = 0.183 - positive = 0.035 | Model weights: - negative = -0.120 - positive = -0.403 | Model weights: - negative = 0.415 - positive = -0.016 | Model weights: - negative = 0.441 - positive = -0.015 | Model weights: - negative = 0.121 - positive = 0.438 | Model weights: - negative = -0.138 - positive = -0.375 | Model weights: - negative = 0.163 - positive = -0.248 | Model weights: - negative = -0.052 - positive = 0.112 | Model weights: - negative = 0.031 - positive = 0.194 | Model weights: - negative = -0.108 - positive = 0.031 | Model weights: - negative = -0.418 - positive = -0.071 | Model weights: - negative = -0.065 - positive = 0.046 | Model weights: - negative = 0.323 - positive = 0.478 | Model weights: - negative = 0.205 - positive = -0.279 | Model weights: - negative = 0.151 - positive = -0.085 | Model weights: - negative = -0.381 - positive = 0.137 | Model weights: - negative = 0.375 - positive = -0.242 | Model weights: - negative = -0.007 - positive = 0.308 | Model weights: - negative = 0.064 - positive = -0.452 |
| Human answers: - mostly negative = 0 - partially negative = 0 - neither = 5 - partially positive = 3 - mostly positive = 2 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 2 - partially positive = 6 - mostly positive = 2 | Human answers: - mostly negative = 7 - partially negative = 1 - neither = 2 - partially positive = 0 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 0 - partially positive = 2 - mostly positive = 8 | Human answers: - mostly negative = 8 - partially negative = 2 - neither = 0 - partially positive = 0 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 7 - neither = 1 - partially positive = 2 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 1 - partially positive = 5 - mostly positive = 4 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 1 - partially positive = 3 - mostly positive = 6 | Human answers: - mostly negative = 1 - partially negative = 0 - neither = 3 - partially positive = 1 - mostly positive = 5 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 3 - partially positive = 4 - mostly positive = 3 | Human answers: - mostly negative = 0 - partially negative = 1 - neither = 7 - partially positive = 0 - mostly positive = 2 | Human answers: - mostly negative = 3 - partially negative = 6 - neither = 1 - partially positive = 0 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 8 - partially positive = 2 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 3 - neither = 5 - partially positive = 0 - mostly positive = 2 | Human answers: - mostly negative = 6 - partially negative = 1 - neither = 2 - partially positive = 0 - mostly positive = 1 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 0 - partially positive = 0 - mostly positive = 10 | Human answers: - mostly negative = 5 - partially negative = 3 - neither = 2 - partially positive = 0 - mostly positive = 0 | Human answers: - mostly negative = 9 - partially negative = 0 - neither = 1 - partially positive = 0 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 0 - partially positive = 2 - mostly positive = 8 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 3 - partially positive = 6 - mostly positive = 1 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 0 - partially positive = 3 - mostly positive = 7 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 0 - partially positive = 2 - mostly positive = 8 | Human answers: - mostly negative = 0 - partially negative = 1 - neither = 0 - partially positive = 1 - mostly positive = 8 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 1 - partially positive = 4 - mostly positive = 5 | Human answers: - mostly negative = 0 - partially negative = 1 - neither = 9 - partially positive = 0 - mostly positive = 0 | Human answers: - mostly negative = 8 - partially negative = 0 - neither = 2 - partially positive = 0 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 0 - partially positive = 3 - mostly positive = 7 | Human answers: - mostly negative = 3 - partially negative = 3 - neither = 4 - partially positive = 0 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 1 - partially positive = 1 - mostly positive = 8 | Human answers: - mostly negative = 0 - partially negative = 1 - neither = 9 - partially positive = 0 - mostly positive = 0 |
| Summary: - Average score: 0.7 - Rank: C | Summary: - Average score: 1.0 - Rank: B | Summary: - Average score: 1.5 - Rank: B | Summary: - Average score: 1.8 - Rank: A | Summary: - Average score: 1.8 - Rank: A | Summary: - Average score: 0.5 - Rank: C | Summary: - Average score: 1.3 - Rank: B | Summary: - Average score: 1.5 - Rank: B | Summary: - Average score: 0.9 - Rank: C | Summary: - Average score: 1.0 - Rank: B | Summary: - Average score: -0.3 - Rank: C | Summary: - Average score: 1.2 - Rank: B | Summary: - Average score: -0.2 - Rank: C | Summary: - Average score: -0.1 - Rank: C | Summary: - Average score: 1.1 - Rank: B | Summary: - Average score: 2.0 - Rank: A | Summary: - Average score: 1.3 - Rank: B | Summary: - Average score: 1.8 - Rank: A | Summary: - Average score: 1.8 - Rank: A | Summary: - Average score: 0.8 - Rank: C | Summary: - Average score: 1.7 - Rank: A | Summary: - Average score: 1.8 - Rank: A | Summary: - Average score: 1.6 - Rank: A | Summary: - Average score: 1.4 - Rank: B | Summary: - Average score: 0.1 - Rank: C | Summary: - Average score: 1.6 - Rank: B | Summary: - Average score: 1.7 - Rank: A | Summary: - Average score: 0.9 - Rank: C | Summary: - Average score: 1.7 - Rank: A | Summary: - Average score: 0.1 - Rank: C |
Model 3: YelpSmall500_CNN_20200515025212
| Feature 0 | Feature 1 | Feature 2 | Feature 3 | Feature 4 | Feature 5 | Feature 6 | Feature 7 | Feature 8 | Feature 9 | Feature 10 | Feature 11 | Feature 12 | Feature 13 | Feature 14 | Feature 15 | Feature 16 | Feature 17 | Feature 18 | Feature 19 | Feature 20 | Feature 21 | Feature 22 | Feature 23 | Feature 24 | Feature 25 | Feature 26 | Feature 27 | Feature 28 | Feature 29 |
 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Model weights: - negative = -0.308 - positive = -0.163 | Model weights: - negative = -0.391 - positive = -0.325 | Model weights: - negative = 0.385 - positive = 0.295 | Model weights: - negative = -0.169 - positive = -0.152 | Model weights: - negative = 0.122 - positive = 0.324 | Model weights: - negative = 0.027 - positive = 0.344 | Model weights: - negative = -0.119 - positive = 0.153 | Model weights: - negative = 0.327 - positive = 0.422 | Model weights: - negative = 0.083 - positive = -0.277 | Model weights: - negative = -0.397 - positive = 0.183 | Model weights: - negative = -0.361 - positive = 0.285 | Model weights: - negative = -0.080 - positive = -0.343 | Model weights: - negative = -0.300 - positive = 0.260 | Model weights: - negative = 0.044 - positive = -0.227 | Model weights: - negative = -0.032 - positive = 0.081 | Model weights: - negative = 0.431 - positive = -0.130 | Model weights: - negative = 0.146 - positive = 0.266 | Model weights: - negative = 0.271 - positive = -0.058 | Model weights: - negative = -0.429 - positive = 0.046 | Model weights: - negative = 0.299 - positive = -0.237 | Model weights: - negative = -0.422 - positive = 0.367 | Model weights: - negative = -0.320 - positive = -0.432 | Model weights: - negative = -0.245 - positive = -0.168 | Model weights: - negative = 0.022 - positive = -0.387 | Model weights: - negative = -0.117 - positive = -0.152 | Model weights: - negative = 0.310 - positive = -0.222 | Model weights: - negative = -0.367 - positive = 0.127 | Model weights: - negative = 0.150 - positive = 0.332 | Model weights: - negative = 0.149 - positive = -0.380 | Model weights: - negative = 0.205 - positive = 0.154 |
| Human answers: - mostly negative = 0 - partially negative = 3 - neither = 3 - partially positive = 2 - mostly positive = 2 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 9 - partially positive = 1 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 7 - partially positive = 2 - mostly positive = 1 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 2 - partially positive = 3 - mostly positive = 5 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 8 - partially positive = 2 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 6 - partially positive = 3 - mostly positive = 1 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 0 - partially positive = 2 - mostly positive = 8 | Human answers: - mostly negative = 0 - partially negative = 1 - neither = 5 - partially positive = 4 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 8 - partially positive = 2 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 4 - neither = 5 - partially positive = 1 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 0 - partially positive = 4 - mostly positive = 6 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 9 - partially positive = 0 - mostly positive = 1 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 7 - partially positive = 2 - mostly positive = 1 | Human answers: - mostly negative = 7 - partially negative = 2 - neither = 1 - partially positive = 0 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 4 - partially positive = 5 - mostly positive = 1 | Human answers: - mostly negative = 6 - partially negative = 3 - neither = 1 - partially positive = 0 - mostly positive = 0 | Human answers: - mostly negative = 1 - partially negative = 0 - neither = 0 - partially positive = 5 - mostly positive = 4 | Human answers: - mostly negative = 0 - partially negative = 4 - neither = 5 - partially positive = 1 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 1 - neither = 2 - partially positive = 5 - mostly positive = 2 | Human answers: - mostly negative = 6 - partially negative = 2 - neither = 2 - partially positive = 0 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 8 - partially positive = 2 - mostly positive = 0 | Human answers: - mostly negative = 3 - partially negative = 1 - neither = 4 - partially positive = 2 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 1 - neither = 5 - partially positive = 3 - mostly positive = 1 | Human answers: - mostly negative = 1 - partially negative = 6 - neither = 1 - partially positive = 2 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 2 - neither = 5 - partially positive = 3 - mostly positive = 0 | Human answers: - mostly negative = 2 - partially negative = 4 - neither = 4 - partially positive = 0 - mostly positive = 0 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 1 - partially positive = 4 - mostly positive = 5 | Human answers: - mostly negative = 0 - partially negative = 0 - neither = 7 - partially positive = 2 - mostly positive = 1 | Human answers: - mostly negative = 0 - partially negative = 7 - neither = 2 - partially positive = 1 - mostly positive = 0 | Human answers: - mostly negative = 1 - partially negative = 4 - neither = 5 - partially positive = 0 - mostly positive = 0 |
| Summary: - Average score: 0.3 - Rank: B | Summary: - Average score: 0.1 - Rank: C | Summary: - Average score: -0.4 - Rank: C | Summary: - Average score: 1.3 - Rank: A | Summary: - Average score: 0.2 - Rank: C | Summary: - Average score: 0.5 - Rank: B | Summary: - Average score: 1.8 - Rank: A | Summary: - Average score: 0.3 - Rank: C | Summary: - Average score: -0.2 - Rank: C | Summary: - Average score: -0.3 - Rank: C | Summary: - Average score: 1.6 - Rank: A | Summary: - Average score: -0.2 - Rank: C | Summary: - Average score: 0.4 - Rank: B | Summary: - Average score: 1.6 - Rank: A | Summary: - Average score: 0.7 - Rank: B | Summary: - Average score: 1.5 - Rank: A | Summary: - Average score: 1.1 - Rank: A | Summary: - Average score: 0.3 - Rank: C | Summary: - Average score: 0.8 - Rank: A | Summary: - Average score: 1.4 - Rank: A | Summary: - Average score: 0.2 - Rank: C | Summary: - Average score: 0.5 - Rank: B | Summary: - Average score: 0.4 - Rank: B | Summary: - Average score: 0.6 - Rank: B | Summary: - Average score: -0.1 - Rank: C | Summary: - Average score: 0.8 - Rank: A | Summary: - Average score: 1.4 - Rank: A | Summary: - Average score: 0.4 - Rank: B | Summary: - Average score: 0.6 - Rank: B | Summary: - Average score: 0.6 - Rank: B |
Results
Results (Average ± SD) of Experiment 1: Yelp, CNNs; Boldface numbers are the best scores in the columns. They are further underlined if they are significantly better than the scores of all the other models (based on approximate randomization test with α = 0.05)
Downloads
- Wordclouds and annotations
- The dataset of this experiment as well as other experiments can be downloaded here.
- If you want to use the original trained models in the experiments, please contact Piyawat (pl1515 [at] imperial [dot] ac [dot] uk).