Estimize Global Rankings
- Posted by Leigh Drogen
- on July 24th, 2012
Today we released a hand full of exciting updates to the Estimize platform, you can read more about them here.
Right now though, I’d like to introduce the new Estimize Global Rankings. One of the great aspects of our community sharing structured data is that we have the ability to effectively rank and rate contributors. But exactly how to go about doing that has been a point of discussion for several months within our company.
I’m proud with what we’ve cooked up, and I’m excited for us to release a deeper feature set soon which includes full overall, sector, industry, and asset specific leaderboards. While the Estimize platform has undoubtedly succeeded at proving a diverse crowd of individuals can be collectively smarter than Wall Street (wisdom of the crowd theory), it’s time for us to turn our attention a bit towards providing tools to uncover the best individuals (finding the needles in the haystack).
Below you will find a detailed walk through of how the overall ranking system works. The ranking system was developed by our amazing developer and budding data scientist Jesse Youngmann. As he is straight out of academia and likes to be thorough with his writing, I’ll provide a brief overview.
- Your recent estimates count more towards your score, allowing new community members to quickly rise through the ranks if they prove to be very accurate, and incenting highly ranked community members to continue to estimate or have their scores deteriorate over time.
- The number of estimates you’ve made is accounted for in your score, community members are assumed to be “average” via diluting your score with an average of the community until you surpass a certain number of scored estimates at which point your score will more accurately reflect your precision.
See Your Overall Ranking….. or read on
While developing the global leaderboard of all active analysts on the Estimize platform, we were faced with the problem of clearly and usefully comparing the abilities of a large number of analysts with different estimating habits. We wanted to consider a user’s accuracy changing over time, the increased difficulty of estimates on a large, broad range of assets, and our confidence that a given number of scored estimates represents an analysts ‘true’ ability.
The algorithm we settled on is:
rank_score = ( time_weighted_sum + ( global_average_score * global_average_estimate_count * ad_hoc_constant ) ) /
( global_average_estimate_count * ad_hoc_constant + time_weighted_estimate_count )
where time_weighted_sum is the sum of each of your estimates’ score, weighted by the time since it was published.
We’re using a Bayesian Average where your estimates are weighted by time. Essentially, we find the average user score, and the average number of scored estimates per active user. Then we average your score with an ad-hoc constant times the average number of average score estimates. This weights all users towards the global average, but as you make more estimates, your weighted rank score will increasingly represent your actual score, while new users are assumed to be average until they’ve made enough estimates to prove otherwise.
When computing your own average, instead of using your total average throughout time, we weight your estimates by their age, so that your more recent estimates count more to your weighted rank score. Our algorithm for this is fully counting any estimate made in the last three months, and then counting estimates older than that at 0.90 ^ ( age_in_months – 3 months), so a 4 month old estimate is only worth 90% of a recent estimate, and a year old estimate is only worth about 40% of a recent estimate. This helps make sure your ranking represents your current abilities, and it ensures that inactive analysts can’t maintain their position forever.
This algorithm can be unfair to analysts who specialize in a small subset of stocks or industries, who may be very accurate but overall make a smaller number of estimates. Luckily for them, we’re currently developing Sector, Industry and Stock specific leaderboards that vary the relative weightings.
Full Disclosure: Nothing on this site should ever be considered to be advice, research or an invitation to buy or sell any securities, please see the Disclaimer page for a full disclaimer.
blog comments powered by Disqus
Leigh Drogen is the founder and chief investment officer of Surfview Capital, LLC, a New York based investment management firm employing an intermediate term long/short momentum strategy. More »
- SunTrust Joins The Estimize Platform, All 299 Other Sell Side Firms to Follow
- The Real Problem With SigFig’s $10/Month Portfolio For Everyone
- NHLPA Idiocy Needs To Stop
- The Promise, and Problems of BayesDB
- The Bad Days
- Two Years of Estimize
- The Smarter You Are, The More Roadblocks You’ll See
- Is Bitcoin a Commodity, Currency, or Technology?
- Why I Bought One Bitcoin
- On Bloomberg TV Talking Twitter and Tesla Estimates
- December 2013
- November 2013
- October 2013
- September 2013
- August 2013
- July 2013
- June 2013
- May 2013
- April 2013
- March 2013
- February 2013
- January 2013
- December 2012
- November 2012
- October 2012
- September 2012
- August 2012
- July 2012
- June 2012
- May 2012
- April 2012
- March 2012
- February 2012
- January 2012
- December 2011
- November 2011
- October 2011
- September 2011
- August 2011
- July 2011
- June 2011
- May 2011
- April 2011
- March 2011
- February 2011
- January 2011