Model</th>	Herschel</th>	Lena</th></tr></thead>
Baseline</td>	19.8 ± 1.7</td>	28.8 ± 3.0</td></tr>
Baseline+Aug</td>	22.9 ± 3.0</td>	25.8 ± 10.2</td></tr>
FixMatchSeg</td>	23.4 ± 0.8</td>	32.4 ± 3.2</td></tr>
Adversarial</td>	26.6 ± 3.9</td>	25.1 ± 15.1</td></tr>
PixelDINO</td>	30.2 ± 2.7</td>	39.5 ± 6.5</td></tr> </tbody></table> Indeed, PixelDINO outperforms not only the supervised base methods, but also the two other semi-supervised segmentation methods we tested: FixMatchSeg</a> and Adversarial Semi-Segmentation</a>. In practice, this improved training method leads to less false positives and more faithful reconstruction of RTS shapes:</p> </iframe> </div> The developed PixelDINO method should prove useful not only for permafrost monitoring, but also for other usecases in remote sensing, where spatial variability poses a challenge. We hope that this work can inspire follow-up research for other applications.</p> COBRA: A Deep Active Contour Model for Delineating Glacier Calving Fronts 2023-07-31T00:00:00+00:00 </p> </figcaption> </figure> Existing approaches for calving front detection generally work by first performing a pixel-wise segmentation or edge detection, and then extract the actual calving front in a post-processing step. Our goal in this study is to build a model that only needs a single step, and directly outputs the calving front as a polyline.</p> </p> </figcaption> </figure> Following the idea of explicit contour prediction, we have developed a new method called “Charting Outlines by Recurrent Adaptation” (COBRA). It works by combining the idea of Active Contour models with deep learning. First, a 2D CNN backbone derives feature maps from the input imagery. Then, a 1D CNN (Snake Head) iteratively deforms an initial contour until to match the true contour.</p> Results</h1> These animations show how COBRA iteratively predicts glacier calving fronts:</p> </img> </img> </img> </img> </img> </img> </div> Paper</h1> Preprint available on arxiv</a>, published article available on IEEE Explore</a></p> Code</h1> If you would like to have a closer look at the implementation details, work with our method, or reproduce our results, you can find all of our code on Github</a>.</p> Inference Map</h1> </iframe> Seeing the Bigger Picture: Enabling Large Context Windows in Neural Networks by Combining Multiple Zoom Levels 2021-10-12T00:00:00+00:00 </iframe> </div> Read the full paper on IEEE Xplore</a>.</p> Self-supervised Audiovisual Representation Learning for Remote Sensing Data 2021-08-02T00:00:00+00:00 </p> Read the full paper at Science Direct</a>.</p> HED-UNet: Combined segmentation and edge detection for monitoring the Antarctic coastline 2021-03-23T00:00:00+00:00 </iframe> </div> Read the full paper on IEEE Xplore</a>.</p> Metaballs 2020-05-16T09:00:00+01:00 Metaballs</a> are a fun way of creating blobby looking shapes. The idea stems from a method for rendering molecules, where each atom contributes to the overall electron density 1</a></sup>.</p> In the easy case of hydrogen, this contribution is a radially symmetric shape centered around the atom’s center \(c\):</p> $$ f(v; c) = a \exp(- \\|v - c\\|_2^2) $$</p> To render a collection of such atoms, one then takes the sum over these contributions, and renders a level-set:</p> $$ \left\{\,v \,\middle\|\, \sum_{i=1}^n a_i\exp(- \\|v - c_i\\|_2^2) = L\,\right\} $$</p> Here’s a visualization in two dimensions with moving centers:</p> </iframe> Evolution 2019-12-29T00:00:00+00:00 Not much to do in this one… Simply watch the virtual creatures evolve to walk as fast as possible.</p> </iframe> Remote Sensing for Assessing Drought Insurance Claims in Central Europe 2019-11-14T00:00:00+00:00 Read the full paper on IEEE Xplore</a>.</p> Soap Bubbles 2019-06-06T09:00:00+01:00 Have you ever wondered about the rainbows on CDs, gasoline puddles or soap bubbles? All of these have the same cause: The interference of light on thin surfaces</a>. Today, we’ll try to render something that looks like a soap bubble.</p> Some Physics</h2> When a ray of light hits the surface of the soap bubble, it is either instantly reflected, or it enters the soap film and is refracted. When it is refracted, it can then be reflected off the other end of the soap, and then leave it again at a slightly different spot. There are countless other possiblities for the ray to bounce around, but these are the two that we will focus on here:</p> Because the width of the soap layer is comparable to the wavelength of visible light, the resulting color of the light rays will change considerably depending on the incidence angle.</p> Finished visualization</h2> Putting it all together, we get this nice little interactive visualization.</p> </iframe> Particle Life 2019-01-31T09:00:00+01:00 Particle Simulations are great. Life Simulations are fun. Meet Particle Life – the intersection of these two worlds.</p> Inspired by Biologist Lynn Margulis’</a> theory of endosymbiosis, Jeffrey Ventrella invented his Clusters</a>. The main idea is simulating microorganisms that have pretty basic interaction patterns. Some species are drawn towards certain others, while some will be repelled by others. Implementing a particle simulation with these rules leads to very interesting patterns.</p> Compared to physics-based particle simulations, this approach explicitly violates the Conservation of Momentum</a>. This allows for self-propelling structures, clusters that look like biological cells, and much more.</p> Porting this simulation to WebGL code was pretty straight-forward using CindyJS</a>. Play around with the simulation here:</p> </iframe> Reversing Hill Shading in Digital Elevation Models 2019-01-03T09:00:00+01:00 A while back on the Geographic Information Systems Subreddit</a> I stumbled upon this question</a>. A user wished to get height information from this elevation map they had found somewhere:</p> Unforunately it has hill-shading</a> applied on top of the color bar. While this shading technique makes it easier to understand the terrain structure at a glance, it also makes it harder to extract the raw height data from the map. So first we’ll have to understand how hill-shading works in order to find a way to get around it.</p> Hill-shading</h2> The point of hill-shading is make a height map appear 3D by shading slopes according to imaginary sunlight that hits the plane at some angle.</p> When creating a regular elevation map, one takes the raw elevation values and maps them to \((r, g, b)\) values according to a specified color scale. To achieve hill-shading, one additionally specifies a global light vector \(\vec l\) and a surface normal vector \(\vec n\) for each point. The shading factor can then be expressed as the scalar product \(s = \langle -\vec l, \vec n\rangle\).</p> The important point here is that the shading factor is applied uniformly to all three color channels. This means that hillshading only affects the brightness of a pixel, but not its hue. So if the color scale plays nicely with the hue, we will be able to extract the original height data.</p> Getting our hands dirty</h2> Time to whip out python and get going! Reading the image is pretty straight forward.</p> `import</span> numpy</span> as</span> np</span></span> import</span> matplotlib</span>.</span>pyplot</span> as</span> plt</span></span> import</span> colorsys</span></span> </span> img</span> =</span> plt</span>.</span>imread</span>(</span>'hillshade_rotated.png'</span>)</span></span></code></pre> We then take a look at the image and find a line that slices through the color bar. From this bar we’ll deduce the transform needed to convert hue to elevation</p>` plt</span>.</span>figure</span>(</span>figsize</span> =</span> (</span>10</span>,</span> 10</span>))</span></span> plt</span>.</span>grid</span>(</span>False</span>)</span></span> plt</span>.</span>imshow</span>(</span>img</span>)</span></span> plt</span>.</span>plot</span>([</span>97</span>,</span> 97</span>], [</span>95</span>,</span> 755</span>])</span>;</span></span></code></pre> We can now take the data from the color bar. To be sure, we plot the color bar data to verify that we actually extracted what we wanted.</p> colorbar</span> =</span> img</span>[</span>755</span>:</span>95</span>:</span>-</span>1</span>,</span> 97</span>:</span>98</span>, :]</span></span> plt</span>.</span>grid</span>(</span>False</span>)</span></span> plt</span>.</span>imshow</span>(</span>np</span>.</span>kron</span>(</span>colorbar</span>,</span> np</span>.</span>ones</span>((</span>1</span>,</span> 60</span>,</span> 1</span>))).</span>transpose</span>(</span>1</span>,</span> 0</span>,</span> 2</span>))</span>;</span></span></code></pre> The extracted color bar </p> </figcaption> </figure> Upon extracting the hue channel from the color bar data, we see that it is in fact monotonic and close to injective! This means we can reconstruct the original elevation data from just this channel.</p> def</span> convert</span>(</span>image</span>):</span></span> fun</span> =</span> np</span>.</span>vectorize</span>(</span>colorsys</span>.</span>rgb_to_hsv</span>)</span></span> return</span> np</span>.</span>stack</span>(</span>fun</span>(</span>image</span>[:,:,</span>0</span>],</span> image</span>[:,:,</span>1</span>],</span> image</span>[:,:,</span>2</span>]))</span></span> HUE</span>,</span> SATURATION</span>,</span> VALUE</span> =</span> convert</span>(</span>colorbar</span>)</span></span> </span> HIGH</span> = -</span>40</span></span> LOW</span> = -</span>78.08</span></span> def</span> correct</span>(</span>x</span>):</span></span> H</span> =</span> np</span>.</span>sort</span>(</span>HUE</span>.</span>reshape</span>(</span>-</span>1</span>))</span></span> return</span> HIGH</span> -</span> (</span>HIGH</span> -</span> LOW</span>)</span> *</span> H</span>.</span>searchsorted</span>(</span>x</span>)</span> /</span> len</span>(</span>H</span>)</span></span> </span></span> fig</span>, (</span>ax1</span>,</span> ax2</span>)</span> =</span> plt</span>.</span>subplots</span>(</span>1</span>,</span> 2</span>,</span> figsize</span>=</span>(</span>10</span>,</span> 5</span>))</span></span> ax1</span>.</span>plot</span>(</span>HUE</span>)</span></span> ax1</span>.</span>set_title</span>(</span>'Hue Progression'</span>)</span></span> ax2</span>.</span>plot</span>(</span>correct</span>(</span>HUE</span>))</span></span> ax2</span>.</span>set_title</span>(</span>'Corrected Elevation Progression'</span>)</span></span></code></pre> The left plot shows how the hue progresses over the color bar. The right plot shows how the color bar looks when corrected using np.searchsorted</code>:</p> Now, we can calculate the hue channel for the entire image and reconstruct the elevation values using the correction function we just wrote.</p> h</span>,</span> s</span>,</span> v</span> =</span> convert</span>(</span>img</span>)</span></span> corrected</span> =</span> correct</span>(</span>h</span>)</span></span> </span> plt</span>.</span>figure</span>(</span>figsize</span>=</span>(</span>10</span>,</span> 10</span>))</span></span> plt</span>.</span>grid</span>(</span>False</span>)</span></span> plt</span>.</span>imshow</span>(</span>corrected</span>,</span> cmap</span>=</span>'gray'</span>)</span></span> plt</span>.</span>imsave</span>(</span>'elevation_rotated.png'</span>,</span> corrected</span> ,</span> cmap</span>=</span>'gray'</span>)</span></span></code></pre> As a quick check, we’ll plot the output:</p> There’s a lot of ugly artifacts going on! Why is that? As you may have guessed, JPEG compression is to blame here. Pixels close to the border of the heightmap that should</em> be white get assigned some color that slightly differs from white. Because we only look at the hue channel, this is enough to distort these pixels completely.</p> This is where the saturation channel comes in handy. These almost-white pixels have close to no saturation, so it is enough to cut off the pixels at some saturation value</p> corrected</span>[</span>s</span> <</span> 0.1</span>]</span> =</span> None</span></span> </span> plt</span>.</span>figure</span>(</span>figsize</span>=</span>(</span>10</span>,</span> 10</span>))</span></span> plt</span>.</span>grid</span>(</span>False</span>)</span></span> plt</span>.</span>imshow</span>(</span>corrected</span>,</span> cmap</span>=</span>'gray'</span>)</span></span> plt</span>.</span>imsave</span>(</span>'elevation.png'</span>,</span> corrected</span> ,</span> cmap</span>=</span>'gray'</span>)</span></span></code></pre> This looks much better!</p> And because stuff is always more fun in 3D, we will end this post with a quick interactive surface plot of the extracted data in plotly:</p> import</span> plotly</span>.</span>graph_objs</span> as</span> go</span></span> from</span> plotly</span>.</span>offline</span> import</span> plot</span>,</span> init_notebook_mode</span></span> </span> init_notebook_mode</span>()</span></span> </span> Z</span> =</span> corrected</span>[:,</span> 240</span>:</span>610</span>]</span></span> Z</span>[</span>Z</span>==</span>1</span>]</span> =</span> 0</span></span> surf</span> =</span> go</span>.</span>Surface</span>(</span>z</span>=</span>Z</span>)</span></span> </span> layout</span> =</span> go</span>.</span>Layout</span>(</span></span> title</span>=</span>'Heightmap'</span>,</span></span> autosize</span>=</span>True</span>,</span></span> width</span>=</span>640</span>,</span></span> height</span>=</span>640</span>,</span></span> )</span></span> fig</span> =</span> go</span>.</span>Figure</span>(</span>data</span>=</span>[</span>surf</span>],</span> layout</span>=</span>layout</span>)</span></span> plot</span>(</span>fig</span>,</span> filename</span>=</span>'hillshade_elevation'</span>)</span></span></code></pre> </iframe> Pachelballs 2018-12-25T09:00:00+01:00 When I was a kid, I really enjoyed this game called Electroplankton</a>. Especially the game mode called Hanenbow</em>, where you launched small tadpoles onto leaves. On impact the leaves would emit glockenspiel sounds, and create a nice sounding melody. Here’s what it looked like:</p> </iframe> </div> I wondered, how hard could it be to implement something similar? If you just want to see the animation, click here</a>. If you’re interested in the background, read on!</p> Concept</h2> The basic idea: Cannon, Balls, Bouncers </p> </figcaption> </figure> First we have a cannon that shoots out balls at a certain frequency. These balls then fly through a field of bouncers, that act like small trampolines for the balls. Upon hitting a bouncer, a ball will emit a certain tone.</p> Physics</h2> We’re going to be using the wonderful CindyJS Framework</a>, that does a lot of the heavy lifting behind the scenes.</p> Ball flying physics is easy. A ball with position \(x\) and velocity \(v\) will be updated in two steps:</p> Updating the speed: \(v \gets v + dt \cdot g\), with a time step \(dt\) and the gravity vector \(g\)</li> Updating the position: \(x \gets x + dt \cdot v\)</li> </ul> The collisions with the bouncers are where it gets tricky. First of all, how do we detect a collision?</p> This is what we want to detect: \(x\) and \(\bar x\) are on opposite sides of the bouncer. </p> </figcaption> </figure> Say a ball is updated from position \(x\) to position \(\bar x\). We want to check if it collides with a bouncer during that update. A bouncer is essentially a line between two points \(b_1\) and \(b_2\).</p> One can see that we have a collision if and only if these two conditions are fulfilled:</p> \(x\) and \(\bar x\) lie on opposing sides of the line \(b_1b_2\).</li> \(b_1\) and \(b_2\) lie on opposing sides of the line \(x\bar x\).</li> </ul> There is a neat way of calculating which side of a line a point is on using determinants. The determinant</p> $$\det\begin{pmatrix}x_A&x_B&x_C\\ y_A&y_B&y_C\\ 1&1&1\end{pmatrix}$$</p> will have a positive sign if \(C\) lies left of the line \(AB\), and a negative one if it lies on the right:</p> </iframe>

PixelDINO: Semi-Supervised Semantic Segmentation for Detecting Permafrost Disturbances

2024-06-03T00:00:00+00:00

In sync with the changing climate, permafrost is undergoing rapid transformations. As temperatures rise, the frozen ground starts to thaw, which has various consequences. Not only does permafrost thawing pose risks to local infrastructure such as roads and buildings, but it also tightly connected to the global climate system by potentially releasing stored carbon into the atmosphere.</p>

Distribution of permafrost in the northern hemisphere. (Visualisation based on data from NSICD</a>) </p> </figcaption> </figure>

Covering more than 10% of the Earth’s land surface, permafrost areas are often remote and sparsely populated, making them difficult to monitor through traditional means. In-situ measurements are limited to specific locations and times, usually when expeditions visit these sites or when local sensors collect data. To overcome these limitations, remote sensing is a more efficient alternative for monitoring permafrost.</p>

Permafrost Remote Sensing</h2>

Coastal retrogressive thaw slump on the Bykovsky Peninsula in northern Siberia. © Ingmar Nitze (AWI)</a> </p> </figcaption> </figure>
With satellite imagery, we can easily get observations of the entire Arctic. While permafrost itself is mostly subsurface, we can monitor surface indicators closely linked to permafrost health or degradation. One such example are retrogressive thaw slumps (RTS), slow landslides resulting from the thawing of ice-rich permafrost. Despite their small size and scattered distribution, RTS can be detected in satellite images due to their distinct shape and spectral signature.</p>
But here’s the catch: While deep learning algorithms show promise in identifying RTS from satellite images, they need huge amounts of labeled training data. In fact, only a very small fraction of the Arctic has been labelled for RTS:</p>
</a>
</iframe>

COBRA: A Deep Active Contour Model for Delineating Glacier Calving Fronts

Code</h1>
If you would like to have a closer look at the implementation details, work with our method, or reproduce our results, you can find all of our code on Github</a>.</p>

Inference Map</h1>
</iframe>

Video</h1>
</iframe> </div>

Seeing the Bigger Picture: Enabling Large Context Windows in Neural Networks by Combining Multiple Zoom Levels

Self-supervised Audiovisual Representation Learning for Remote Sensing Data

HED-UNet: Combined segmentation and edge detection for monitoring the Antarctic coastline

Metaballs

Evolution

Remote Sensing for Assessing Drought Insurance Claims in Central Europe

Soap Bubbles

Finished visualization</h2>
Putting it all together, we get this nice little interactive visualization.</p>
</iframe>

Particle Life

Reversing Hill Shading in Digital Elevation Models

Pachelballs

Finished animation</h2>
</a> Putting it all together, we get this nice little interactive animation.</p>
</iframe>

konrad.earth

Japan Shader

PixelDINO: Semi-Supervised Semantic Segmentation for Detecting Permafrost Disturbances

konrad.earth

Japan Shader

The shader</h1> The core of this app is the WebGL shader. For each pixel on the screen, it marches along the corresponding ray in 3d space and checks whether the ray intersects with the terrain at any point. The ray marching loop is quite simple:</p>

PixelDINO: Semi-Supervised Semantic Segmentation for Detecting Permafrost Disturbances

COBRA: A Deep Active Contour Model for Delineating Glacier Calving Fronts

Results</h1> These animations show how COBRA iteratively predicts glacier calving fronts:</p>

Video</h1> </iframe> </div>

Seeing the Bigger Picture: Enabling Large Context Windows in Neural Networks by Combining Multiple Zoom Levels

Self-supervised Audiovisual Representation Learning for Remote Sensing Data

HED-UNet: Combined segmentation and edge detection for monitoring the Antarctic coastline

Metaballs

Evolution

Remote Sensing for Assessing Drought Insurance Claims in Central Europe

Soap Bubbles

Finished visualization</h2> Putting it all together, we get this nice little interactive visualization.</p> </iframe> </svg> </button> </div>

Particle Life

Reversing Hill Shading in Digital Elevation Models

Getting our hands dirty</h2> Time to whip out python and get going! Reading the image is pretty straight forward.</p>

Pachelballs

Video</h1>
</iframe> </div>

Finished visualization</h2>
Putting it all together, we get this nice little interactive visualization.</p>
</iframe>