Visit ArabTopics.com

Mannomatic smoothing: technical details

This post is rather technical, and it is intended mainly for the historical completeness. So unless you are very, very interested in the tiny technical details of the HS saga, you can safely skip this.

As most readers are aware, and stated in my post few hours after CG broke out, Mike’s Nature trick was first uncovered by UC here. He was able to replicate (visually perfectly) the smooths of MBH9x thereby showing that the smooths involved padding with the instrumental data. The filter used by UC was the zero-phase Butterworth filter (an IIR filter), which has been Mann’s favourite since at least 2003. However, there was something else that I felt was odd: UC’s emulation required a very long (100 samples or so) additional zero padding. So about two years ago, I decided to take an additional look at the topic with UC.

Indeed, after digitalizing Mann’s smooths we discovered that UC’s emulation was very, very good but not perfect. After a long research, and countless hours of experimenting (I won’t bore you with the details), we managed to figure out the “filter” used by Mann before Mann (2004)-era. Mann had made his own version of the Hamming filter (windowing method, an FIR filter)! Instead of using any kind of usual estimate for the filter order, which is usually estimated from the transition bandwidth (see, e.g., Mitra: Digital Signal Processing) and has typically the length of a few dozen coefficients at maximum, he used the filter length equal to the length of the signal to be filtered! As Mann’s PCA was apparently just a “modern” convention, this must be a “modern” filter design. Anyhow, no digital signal processing expert I consulted about the matter had ever seen anything like that.

In order to see how absurd the “filter design” is, consider filtering a signal of length 999 samples. According to Mann, you should design a Hamming filter of the same length. One should always disregard half of the filter length amount (i.e., 499 in our example) of filtered values from both ends, so in Mann’s case one would end up with a single smoothed value! In Mann’s implementation, however, one ends up with a filtered series of the same length as the original signal.

Another way to think of Mann’s “filter” is to consider it as a normal filter with a huge (half the signal length) zero padding to the both ends of the signal. This interpretation also gives hints why UC’s emulation was so successful. One can also speculate, if the similarity of the results between zero-phase Butterworth and Mann’s original filters is the reason Mann chose the Butterworth filter in the first place.

If someone wants to explore this topic further, I’ve place my Octave/Matlab implementation of Mann’s smoother here. The code includes references to the original Mann’s code I uncovered. Finally, the exact parameters of the trick in MBH9x were as follows. MBH98 has 50-year smoothing with padding of 1981-1995 instrumental. Additionally, the smoothing is cut back 25 samples (half of the “filter length”) from both ends. MBH99 used 40-year filtering with 1981-1997 (not 1998!) instrumental padding. The smooth is cut back 20 samples from the end but not from the beginning.

Source: 
Climate Audit

Dear friends of this aggregator

  • Yes, I intentionally removed Newsbud from the aggregator on Mar 22.
  • Newsbud did not block the aggregator, although their editor blocked me on twitter after a comment I made to her
  • As far as I know, the only site that blocks this aggregator is Global Research. I have no idea why!!
  • Please stop recommending Newsbud and Global Research to be added to the aggregator.

Support this site

News Sources

Source Items
WWI Hidden History 51
Grayzone Project 219
Pass Blue 240
Dilyana Gaytandzhieva 16
John Pilger 416
The Real News 367
Scrutinised Minds 29
Need To Know News 2739
FEE 4827
Marine Le Pen 383
Francois Asselineau 25
Opassande 53
HAX on 5July 220
Henrik Alexandersson 981
Mohamed Omar 383
Professors Blog 10
Arg Blatte Talar 40
Angry Foreigner 18
Fritte Fritzson 12
Teologiska rummet 32
Filosofiska rummet 115
Vetenskapsradion Historia 162
Snedtänkt (Kalle Lind) 225
Les Crises 2962
Richard Falk 176
Ian Sinclair 112
SpinWatch 61
Counter Currents 10027
Kafila 502
Gail Malone 42
Transnational Foundation 221
Rick Falkvinge 95
The Duran 9971
Vanessa Beeley 143
Nina Kouprianova 9
MintPress 5680
Paul Craig Roberts 1969
News Junkie Post 58
Nomi Prins 27
Kurt Nimmo 191
Strategic Culture 5056
Sir Ken Robinson 25
Stephan Kinsella 101
Liberty Blitzkrieg 859
Sami Bedouin 65
Consortium News 2685
21 Century Wire 3655
Burning Blogger 324
Stephen Gowans 92
David D. Friedman 154
Anarchist Standard 16
The BRICS Post 1519
Tom Dispatch 543
Levant Report 18
The Saker 4462
The Barnes Review 536
John Friend 493
Psyche Truth 160
Jonathan Cook 155
New Eastern Outlook 4195
School Sucks Project 1782
Giza Death Star 1975
Andrew Gavin Marshall 15
Red Ice Radio 623
GMWatch 2375
Robert Faurisson 150
Espionage History Archive 34
Jay's Analysis 1018
Le 4ème singe 90
Jacob Cohen 210
Agora Vox 16272
Cercle Des Volontaires 438
Panamza 2238
Fairewinds 117
Project Censored 987
Spy Culture 556
Conspiracy Archive 77
Crystal Clark 11
Timothy Kelly 583
PINAC 1482
The Conscious Resistance 863
Independent Science News 81
The Anti Media 6738
Positive News 820
Brandon Martinez 30
Steven Chovanec 61
Lionel 298
The Mind renewed 445
Natural Society 2619
Yanis Varoufakis 1028
Tragedy & Hope 122
Dr. Tim Ball 114
Web of Debt 148
Porkins Policy Review 434
Conspiracy Watch 174
Eva Bartlett 618
Libyan War Truth 339
DeadLine Live 1913
Kevin Ryan 64
BSNEWS 2088
Aaron Franz 245
Traces of Reality 166
Revelations Radio News 121
Dr. Bruce Levine 150
Peter B Collins 1620
Faux Capitalism 205
Dissident Voice 11030
Climate Audit 225
Donna Laframboise 453
Judith Curry 1142
Geneva Business Insider 40
Media Monarchy 2440
Syria Report 78
Human Rights Investigation 93
Intifada (Voice of Palestine) 1685
Down With Tyranny 12269
Laura Wells Solutions 44
Video Rebel's Blog 440
Revisionist Review 485
Aletho News 21012
ضد العولمة 27
Penny for your thoughts 3091
Northerntruthseeker 2463
كساريات 37
Color Revolutions and Geopolitics 27
Stop Nato 4724
AntiWar.com Blog 3151
AntiWar.com Original Content 7095
Corbett Report 2417
Stop Imperialism 491
Land Destroyer 1222
Webster Tarpley Website 1118

Compiled Feeds

Public Lists

Title Visibility
Funny Public