Cloudspotting

<div>
<style type="text/css">.xaringan-extra-logo {
width: 90px;
height: 128px;
z-index: 0;
background-image: url(../icons/cloud-qlvl-blue.png);
background-size: contain;
background-repeat: no-repeat;
position: absolute;
top:1em;right:1em;
}
</style>
<script>(function () {
  let tries = 0
  function addLogo () {
    if (typeof slideshow === 'undefined') {
      tries += 1
      if (tries < 10) {
        setTimeout(addLogo, 100)
      }
    } else {
      document.querySelectorAll('.remark-slide-content:not(.title-slide):not(.inverse):not(.hide_logo)')
        .forEach(function (slide) {
          const logo = document.createElement('a')
          logo.classList = 'xaringan-extra-logo'
          logo.href = 'https://www.arts.kuleuven.be/ling/qlvl/projects/current/nephological-semantics'
          slide.appendChild(logo)
        })
    }
  }
  document.addEventListener('DOMContentLoaded', addLogo)
})()</script>
</div>

.measure.mytitle[
# Cloudspotting

## Visual Analytics for Distributional Semantics

**Mariana Montes**

![kuleuven icon](../icons/kuleuven.png)

]

---
layout: true

---

# Outline

-  What for?

-  Distributional Semantics

-  Visual Analytics

-  Insights

---

# Outline

-  **What for?**

-  Distributional Semantics

-  Visual Analytics

-  Insights

---

# Introduction

.bg-lightest-blue.b--dark-blue.ba.bw2.br3.shadow-5.center[
Using distributional semantics for semasiological research
]
<br>
--

.fl.w-third.center.ph2.pv2[
![Photo of dictionary page, by Joshua Hoehne on Unsplash](img/dictionary.jpg)
]

.fl.w-third.center.ph2.pv2[
![Photo of someone writing, by Kat Stokes on Unsplash](img/writing.jpg)
]

.fl.w-third.center.ph2.pv2[

![Fragment of concordance](img/text.png)

]

???

Image credits.
Dictionary foto: Joshua Hoehne on Unsplash (https://unsplash.com/@mrthetrain)

Writing foto: Kat Stokes on Unsplash (https://unsplash.com/@katstokes_)

---
class: hide_logo

.fl.w-third.ph1[
![Fragment of concordance](img/text_001.png)
]
--
.fl.w-third.ph1[
![Fragment of concordance](img/text_002.png)
]
--
.fl.w-third.ph1[
![Fragment of concordance](img/text_003.png)
]

---

.fl.w-40.ph3[
![Fragments of concordance](img/pages.png)
]

.fl.w-10.p3[
<br>

<br>

![:img 100%, Right arrow.](img/arrow.png)
]

.fl.w-50[

![Plot of horde.](img/horde1.png)]

---

![:img 80%, Plot of horde with context.](img/highlight_nemen2.png)
]

---

# Outline

-  .gray[What for?]

-  **Distributional Semantics**

-  Visual Analytics

-  Insights

---

## Distributional Hypothesis

.bg-dark-blue.b--dark-blue.ba.bw2.br3.shadow-5.mh5.center.white[
Words with similar distribution are semantically similar.
]

<br>

**Similar distribution**: attested (with similar frequency) in similar circumstances.

Can we use frequency information to represent semantic relationships?

---

## What are vector space models?

<br>

`$$PMI_{(\mathrm{linguistics}, \mathrm{language})} = \log\frac{p(\mathrm{linguistics}, \mathrm{language})}{p(\mathrm{linguistics})p(\mathrm{language})}$$`

.footnote[
<svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;" xmlns="http://www.w3.org/2000/svg">  <path d="M256 8C119.043 8 8 119.083 8 256c0 136.997 111.043 248 248 248s248-111.003 248-248C504 119.083 392.957 8 256 8zm0 110c23.196 0 42 18.804 42 42s-18.804 42-42 42-42-18.804-42-42 18.804-42 42-42zm56 254c0 6.627-5.373 12-12 12h-88c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h12v-64h-12c-6.627 0-12-5.373-12-12v-24c0-6.627 5.373-12 12-12h64c6.627 0 12 5.373 12 12v100h12c6.627 0 12 5.373 12 12v24z"></path></svg>
Actual values from GloWbE (Global Web-based English corpus), with a window-span of 10 words to either side.

PMI: [Pointwise Mutual information](https://en.wikipedia.org/wiki/Pointwise_mutual_information)]

---
template: vsm-intro

<br>

`$$PMI_{(\mathrm{linguistics}, \mathrm{language})} = \log\frac{285/N}{\frac{20,740}{N}\frac{1.58 \times 10^6}{N}}$$`
---

<br>

`$$PMI_{(\mathrm{linguistics}, \mathrm{language})} = \log\frac{285}{20,740\times 1.58 \times 10^6}9.14 \times 10^8 = 4.37$$`
---
name: type-level

## Type level vectors

- Each row is the vector of a word
   - aggregating over all its occurrences

- Each column is a context feature

---
template: type-level

- Values are PMI

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> target </th>
   <th style="text-align:right;"> language/n </th>
   <th style="text-align:right;"> word/n </th>
   <th style="text-align:right;"> english/j </th>
   <th style="text-align:right;"> speak/v </th>
   <th style="text-align:right;"> flemish/j </th>
   <th style="text-align:right;"> eat/v </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> linguistics/n </td>
   <td style="text-align:right;"> 4.37 </td>
   <td style="text-align:right;"> 0.99 </td>
   <td style="text-align:right;"> 3.16 </td>
   <td style="text-align:right;"> 0.41 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> lexicography/n </td>
   <td style="text-align:right;"> 3.51 </td>
   <td style="text-align:right;"> 2.18 </td>
   <td style="text-align:right;"> 2.19 </td>
   <td style="text-align:right;"> 2.09 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> computational/j </td>
   <td style="text-align:right;"> 1.60 </td>
   <td style="text-align:right;"> 0.08 </td>
   <td style="text-align:right;"> -1.00 </td>
   <td style="text-align:right;"> -1.80 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> research/n </td>
   <td style="text-align:right;"> 0.20 </td>
   <td style="text-align:right;"> -0.84 </td>
   <td style="text-align:right;"> -0.50 </td>
   <td style="text-align:right;"> -0.38 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> -0.68 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> chocolate/n </td>
   <td style="text-align:right;"> -1.72 </td>
   <td style="text-align:right;"> -0.53 </td>
   <td style="text-align:right;"> -0.73 </td>
   <td style="text-align:right;"> -1.13 </td>
   <td style="text-align:right;"> 1.28 </td>
   <td style="text-align:right;"> 3.08 </td>
  </tr>
</tbody>
</table>

---
template: type-level

- Values are **P**PMI: negative values turned to 0.

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> target </th>
   <th style="text-align:right;"> language/n </th>
   <th style="text-align:right;"> word/n </th>
   <th style="text-align:right;"> english/j </th>
   <th style="text-align:right;"> speak/v </th>
   <th style="text-align:right;"> flemish/j </th>
   <th style="text-align:right;"> eat/v </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> linguistics/n </td>
   <td style="text-align:right;"> 4.37 </td>
   <td style="text-align:right;"> 0.99 </td>
   <td style="text-align:right;"> 3.16 </td>
   <td style="text-align:right;"> 0.41 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> lexicography/n </td>
   <td style="text-align:right;"> 3.51 </td>
   <td style="text-align:right;"> 2.18 </td>
   <td style="text-align:right;"> 2.19 </td>
   <td style="text-align:right;"> 2.09 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> computational/j </td>
   <td style="text-align:right;"> 1.60 </td>
   <td style="text-align:right;"> 0.08 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> research/n </td>
   <td style="text-align:right;"> 0.20 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.04 </td>
   <td style="text-align:right;"> 0.00 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> chocolate/n </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 1.28 </td>
   <td style="text-align:right;"> 3.08 </td>
  </tr>
</tbody>
</table>

---
name: study-tokens

## Token level vectors

### Tokens of *to study*

(1) Would you like to **study** linguistics?

(2) They **study** this in computational linguistics too.

(3) I eat chocolate while I **study**.

<hr>

---
template: study-tokens

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> context feature </th>
   <th style="text-align:right;"> language/n </th>
   <th style="text-align:right;"> word/n </th>
   <th style="text-align:right;"> english/j </th>
   <th style="text-align:right;"> speak/v </th>
   <th style="text-align:right;"> flemish/j </th>
   <th style="text-align:right;"> eat/v </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> linguistics/n </td>
   <td style="text-align:right;"> 4.37 </td>
   <td style="text-align:right;"> 0.99 </td>
   <td style="text-align:right;"> 3.16 </td>
   <td style="text-align:right;"> 0.41 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> computational/j </td>
   <td style="text-align:right;"> 1.60 </td>
   <td style="text-align:right;"> 0.08 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
</tbody>
</table>

---

### Merging vectors

--
<br>

.center[
<svg viewBox="0 0 320 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#0266a0;" xmlns="http://www.w3.org/2000/svg">  <path d="M143 256.3L7 120.3c-9.4-9.4-9.4-24.6 0-33.9l22.6-22.6c9.4-9.4 24.6-9.4 33.9 0l96.4 96.4 96.4-96.4c9.4-9.4 24.6-9.4 33.9 0L313 86.3c9.4 9.4 9.4 24.6 0 33.9l-136 136c-9.4 9.5-24.6 9.5-34 .1zm34 192l136-136c9.4-9.4 9.4-24.6 0-33.9l-22.6-22.6c-9.4-9.4-24.6-9.4-33.9 0L160 352.1l-96.4-96.4c-9.4-9.4-24.6-9.4-33.9 0L7 278.3c-9.4 9.4-9.4 24.6 0 33.9l136 136c9.4 9.5 24.6 9.5 34 .1z"></path></svg>
<svg viewBox="0 0 320 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#0266a0;" xmlns="http://www.w3.org/2000/svg">  <path d="M143 256.3L7 120.3c-9.4-9.4-9.4-24.6 0-33.9l22.6-22.6c9.4-9.4 24.6-9.4 33.9 0l96.4 96.4 96.4-96.4c9.4-9.4 24.6-9.4 33.9 0L313 86.3c9.4 9.4 9.4 24.6 0 33.9l-136 136c-9.4 9.5-24.6 9.5-34 .1zm34 192l136-136c9.4-9.4 9.4-24.6 0-33.9l-22.6-22.6c-9.4-9.4-24.6-9.4-33.9 0L160 352.1l-96.4-96.4c-9.4-9.4-24.6-9.4-33.9 0L7 278.3c-9.4 9.4-9.4 24.6 0 33.9l136 136c9.4 9.5 24.6 9.5 34 .1z"></path></svg>
<svg viewBox="0 0 320 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#0266a0;" xmlns="http://www.w3.org/2000/svg">  <path d="M143 256.3L7 120.3c-9.4-9.4-9.4-24.6 0-33.9l22.6-22.6c9.4-9.4 24.6-9.4 33.9 0l96.4 96.4 96.4-96.4c9.4-9.4 24.6-9.4 33.9 0L313 86.3c9.4 9.4 9.4 24.6 0 33.9l-136 136c-9.4 9.5-24.6 9.5-34 .1zm34 192l136-136c9.4-9.4 9.4-24.6 0-33.9l-22.6-22.6c-9.4-9.4-24.6-9.4-33.9 0L160 352.1l-96.4-96.4c-9.4-9.4-24.6-9.4-33.9 0L7 278.3c-9.4 9.4-9.4 24.6 0 33.9l136 136c9.4 9.5 24.6 9.5 34 .1z"></path></svg>
<svg viewBox="0 0 320 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#0266a0;" xmlns="http://www.w3.org/2000/svg">  <path d="M143 256.3L7 120.3c-9.4-9.4-9.4-24.6 0-33.9l22.6-22.6c9.4-9.4 24.6-9.4 33.9 0l96.4 96.4 96.4-96.4c9.4-9.4 24.6-9.4 33.9 0L313 86.3c9.4 9.4 9.4 24.6 0 33.9l-136 136c-9.4 9.5-24.6 9.5-34 .1zm34 192l136-136c9.4-9.4 9.4-24.6 0-33.9l-22.6-22.6c-9.4-9.4-24.6-9.4-33.9 0L160 352.1l-96.4-96.4c-9.4-9.4-24.6-9.4-33.9 0L7 278.3c-9.4 9.4-9.4 24.6 0 33.9l136 136c9.4 9.5 24.6 9.5 34 .1z"></path></svg>
<svg viewBox="0 0 320 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:#0266a0;" xmlns="http://www.w3.org/2000/svg">  <path d="M143 256.3L7 120.3c-9.4-9.4-9.4-24.6 0-33.9l22.6-22.6c9.4-9.4 24.6-9.4 33.9 0l96.4 96.4 96.4-96.4c9.4-9.4 24.6-9.4 33.9 0L313 86.3c9.4 9.4 9.4 24.6 0 33.9l-136 136c-9.4 9.5-24.6 9.5-34 .1zm34 192l136-136c9.4-9.4 9.4-24.6 0-33.9l-22.6-22.6c-9.4-9.4-24.6-9.4-33.9 0L160 352.1l-96.4-96.4c-9.4-9.4-24.6-9.4-33.9 0L7 278.3c-9.4 9.4-9.4 24.6 0 33.9l136 136c9.4 9.5 24.6 9.5 34 .1z"></path></svg>
]

<br>

---

### Token level vectors

<br>
.bb.b--light-blue.shadow-1.pv1.ph2[.light-blue.b[(2)] They **study** this in *computational linguistics* too.]

<br>
.bb.b--green.shadow-1.mt4.pv1.ph2[.green.b[(3)] I eat *chocolate* while I **study**.]

]

<br>

.center[
<table>
 <thead>
  <tr>
   <th style="text-align:left;"> target </th>
   <th style="text-align:right;"> language/n </th>
   <th style="text-align:right;"> word/n </th>
   <th style="text-align:right;"> english/j </th>
   <th style="text-align:right;"> speak/v </th>
   <th style="text-align:right;"> flemish/j </th>
   <th style="text-align:right;"> eat/v </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;font-weight: bold;color: #E69F00 !important;"> study<sub>1</sub> </td>
   <td style="text-align:right;"> 4.37 </td>
   <td style="text-align:right;"> 0.99 </td>
   <td style="text-align:right;"> 3.16 </td>
   <td style="text-align:right;"> 0.41 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
  </tr>
  <tr>
   <td style="text-align:left;font-weight: bold;color: #56B4E9 !important;"> study<sub>2</sub> </td>
   <td style="text-align:right;"> 5.97 </td>
   <td style="text-align:right;"> 1.07 </td>
   <td style="text-align:right;"> 3.16 </td>
   <td style="text-align:right;"> 0.41 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
  </tr>
  <tr>
   <td style="text-align:left;font-weight: bold;color: #009E73 !important;"> study<sub>3</sub> </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 1.28 </td>
   <td style="text-align:right;"> 3.08 </td>
  </tr>
</tbody>
</table>
]

---

#### Original text

.gold.b[(1)] Would you like to **study** *linguistics*?

.light-blue.b[(2)] They **study** this in *computational linguistics* too.

.green.b[(3)] I eat *chocolate* while I **study**.

]

<table class="table" style="font-size: 11px; margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;"> target </th>
   <th style="text-align:right;"> language/n </th>
   <th style="text-align:right;"> word/n </th>
   <th style="text-align:right;"> english/j </th>
   <th style="text-align:right;"> speak/v </th>
   <th style="text-align:right;"> flemish/j </th>
   <th style="text-align:right;"> eat/v </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;font-weight: bold;color: #E69F00 !important;"> study<sub>1</sub> </td>
   <td style="text-align:right;"> 4.37 </td>
   <td style="text-align:right;"> 0.99 </td>
   <td style="text-align:right;"> 3.16 </td>
   <td style="text-align:right;"> 0.41 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
  </tr>
  <tr>
   <td style="text-align:left;font-weight: bold;color: #56B4E9 !important;"> study<sub>2</sub> </td>
   <td style="text-align:right;"> 5.97 </td>
   <td style="text-align:right;"> 1.07 </td>
   <td style="text-align:right;"> 3.16 </td>
   <td style="text-align:right;"> 0.41 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
  </tr>
  <tr>
   <td style="text-align:left;font-weight: bold;color: #009E73 !important;"> study<sub>3</sub> </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 0.00 </td>
   <td style="text-align:right;"> 1.28 </td>
   <td style="text-align:right;"> 3.08 </td>
  </tr>
</tbody>
</table>

]

.pull-left.dist[

#### Token-token distance matrix
<table>
 <thead>
  <tr>
   <th style="text-align:left;"> target </th>
   <th style="text-align:left;"> study<sub>1</sub> </th>
   <th style="text-align:left;"> study<sub>2</sub> </th>
   <th style="text-align:left;"> study<sub>3</sub> </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;font-weight: bold;color: #E69F00 !important;"> study<sub>1</sub> </td>
   <td style="text-align:left;"> <span style="     color: grey !important;">0</span> </td>
   <td style="text-align:left;"> <span style="     color: black !important;">0.01</span> </td>
   <td style="text-align:left;"> <span style="     color: black !important;">1</span> </td>
  </tr>
  <tr>
   <td style="text-align:left;font-weight: bold;color: #56B4E9 !important;"> study<sub>2</sub> </td>
   <td style="text-align:left;"> <span style="     color: black !important;">0.01</span> </td>
   <td style="text-align:left;"> <span style="     color: grey !important;">0</span> </td>
   <td style="text-align:left;"> <span style="     color: black !important;">1</span> </td>
  </tr>
  <tr>
   <td style="text-align:left;font-weight: bold;color: #009E73 !important;"> study<sub>3</sub> </td>
   <td style="text-align:left;"> <span style="     color: black !important;">1</span> </td>
   <td style="text-align:left;"> <span style="     color: black !important;">1</span> </td>
   <td style="text-align:left;"> <span style="     color: grey !important;">0</span> </td>
  </tr>
</tbody>
</table>
]

--
.pull-right[
#### t-SNE visualization
![:img 60%, t-SNE simulation](../gifs/bubbles-cropped.gif)
]

---

# Outline

-  .gray[What for?]

-  .gray[Distributional Semantics]

-  **Visual Analytics**

-  Insights

---
.center[
![:img 80%, Plot of horde basic](img/horde1.png)]

---

![:img 90%, Plot of horde with context.](img/highlight_nemen2.png)

]

---

![:img 80%, Plot of horde with context words per cluster.](img/horde-hulls.png)

]

---

# Outline

-  .gray[What for?]

-  .gray[Distributional Semantics]

-  .gray[Visual Analytics]

-  **Insights**

---

# Insights

- Models do represent contextual patterns.

- BUT they do not necessarily correspond to semantic phenomena.

- Their shape and strength correspond to **frequency**, **distinctiveness** and **relatedness** of the context words.

<hr>

Can we use the models for automatic disambiguation?

> **No**, not reliably enough.
The correspondence between patterns and senses varies per word.

Should we then discard distributional models altogether?

> **No**, they can still be informative.
We can learn about the *structure* itself, about the patterns and their relationships.

> The visualization tools will help you!

---
layout: false
class: title-slide

# Thank you!

[mariana.montes@kuleuven.be](mailto:mariana.montes@kuleuven.be)