Estimize Global Rankings

Today we released a hand full of exciting updates to the Estimize platform, you can read more about them here.

Right now though, I’d like to introduce the new Estimize Global Rankings. One of the great aspects of our community sharing structured data is that we have the ability to effectively rank and rate contributors. But exactly how to go about doing that has been a point of discussion for several months within our company.

I’m proud with what we’ve cooked up, and I’m excited for us to release a deeper feature set soon which includes full overall, sector, industry, and asset specific leaderboards. While the Estimize platform has undoubtedly succeeded at proving a diverse crowd of individuals can be collectively smarter than Wall Street (wisdom of the crowd theory), it’s time for us to turn our attention a bit towards providing tools to uncover the best individuals (finding the needles in the haystack).

Below you will find a detailed walk through of how the overall ranking system works. The ranking system was developed by our amazing developer and budding data scientist Jesse Youngmann. As he is straight out of academia and likes to be thorough with his writing, I’ll provide a brief overview.

  • Your recent estimates count more towards your score, allowing new community members to quickly rise through the ranks if they prove to be very accurate, and incenting highly ranked community members to continue to estimate or have their scores deteriorate over time.
  • The number of estimates you’ve made is accounted for in your score, community members are assumed to be “average” via diluting your score with an average of the community until you surpass a certain number of scored estimates at which point your score will more accurately reflect your precision.

See Your Overall Ranking….. or read on

While developing the global leaderboard of all active analysts on the Estimize platform, we were faced with the problem of clearly and usefully comparing the abilities of a large number of analysts with different estimating habits. We wanted to consider a user’s accuracy changing over time, the increased difficulty of estimates on a large, broad range of assets, and our confidence that a given number of scored estimates represents an analysts ‘true’ ability.

The algorithm we settled on is:

rank_score = ( time_weighted_sum + ( global_average_score * global_average_estimate_count * ad_hoc_constant ) ) /
( global_average_estimate_count * ad_hoc_constant + time_weighted_estimate_count )

where time_weighted_sum is the sum of each of your estimates’ score, weighted by the time since it was published.

We’re using a Bayesian Average where your estimates are weighted by time. Essentially, we find the average user score, and the average number of scored estimates per active user. Then we average your score with an ad-hoc constant times the average number of average score estimates. This weights all users towards the global average, but as you make more estimates, your weighted rank score will increasingly represent your actual score, while new users are assumed to be average until they’ve made enough estimates to prove otherwise.

When computing your own average, instead of using your total average throughout time, we weight your estimates by their age, so that your more recent estimates count more to your weighted rank score. Our algorithm for this is fully counting any estimate made in the last three months, and then counting estimates older than that at 0.90 ^ ( age_in_months – 3 months), so a 4 month old estimate is only worth 90% of a recent estimate, and a year old estimate is only worth about 40% of a recent estimate. This helps make sure your ranking represents your current abilities, and it ensures that inactive analysts can’t maintain their position forever.

This algorithm can be unfair to analysts who specialize in a small subset of stocks or industries, who may be very accurate but overall make a smaller number of estimates. Luckily for them, we’re currently developing Sector, Industry and Stock specific leaderboards that vary the relative weightings.

See Your Overall Ranking Now

Full Disclosure: Nothing on this site should ever be considered to be advice, research or an invitation to buy or sell any securities, please see the Disclaimer page for a full disclaimer.


blog comments powered by Disqus