Hey folks, this topic might get a little brainy, but I know my SSX 2012 competitors are a bright group of people, and I want your input. Guys like Xedec, Hero Complexity, and any one of the SSX 2012 folks can give their input. I'll try to explain the situation as best I can. But please, stick with this, because if there's one thing I know about my users, it's that we have some of the brightest people in the video game community - you don't shave one-hundredth of a second off of a score without some serious brain power.

A week ago, Pemdas82 brought up an issue with the rankings - he noticed that RockLee was ranked second in the Best Overall (not just Best SSX 2012 Overall or even Best Overall Survive) rankings, despite only owning 18 of 40 or so top scores in one category of one game. Obviously, as much as I respect RockLee, I agree with Pemdas in that this seems out of proportion to where RockLee should really be in the rankings, so I asked our developer to check it out and see what's going on.

But before I tell you what he said, I need to fill you all in on how the scoreboard works. It goes like this:

Showoff is figured with this equation:

catscore is your score on a particular track. cattopscore is the first place score on a particular track.

Racing is done using this one:


What both of these formulas do is judge your score on a particular track against the best score/time on that track, and then equate that performance into a number out of 10,000. First place will have 10,000/10,000, everyone else below that will have a number below 10,000.

Example of racing math:
Let's equate this to something on the scoreboard, like totalling a score on Best Overall SSX Racer. For example, let's say a person has four scores on five tracks in SSX Racing:
User's Time --- Track Best Time (all in seconds)
---------------------------------------
45------------------25
60------------------60
25------------------20
47------------------40
None---------------55 (User's time is filled in as 55 + 60 = 115)

What you see on that last entry is that the user has no score for a particular track - when this mod was first created (and I used the math above) we gave them a dummy score that is 60 seconds below last place. Nowadays, that figure is 1 second below last place, but I didn't want to redo the math so I left it as it is. For showoff I believe they get a score 100 points below last place.

With these figures, mean(catscore - cattopscore) is this:
((45 - 25) + (60 - 60) + (25 - 20) + (47 - 40) + (115 - 55)) / 5
(20 + 0+ 5 +7 + 65) / 5

97/5 = 19.4

10000 times 0.99 to the power of 19.4, correct?

10,000 times 0.822853982 = 8228.54 out of 10,000

Pretty elegant if you ask me.

Additional Methodology for Best Overall Categories
So each of these equations per track are worked out and then added up and averaged for each of the Best Game and Best Game Mode tracks (SSX 3, SSX 3 Showoff/Racing etc.) When we get to the Best Overall rankings it gets a little more complicated.

We run the same process as above, except we then weight the totals of each game according to the percentage of scores of each game. So for example, when we started SSX 3 had about 40% of scores, and SSX Tricky has about 26% of scores. Those two games were weighted the heaviest, followed by SSX On Tour, SSX and then SSX Blur. Likewise, these ratios will changes as new scores are added. If an old game suddenly gets a lot of new submissions, it's weight will increase in the formula. We also instituted a "minimum number of scores" requirement to make people at least try on certain games to avoid the penalty.

Now that you know how it works overall, we go back to the problem - why is RockLee is ranked second in Best Overall even though he only has 18 or so top scores out of more than 200 tracks in one game (SSX 2012)? Here is the quote from our programmer:


Now, before I go any further, you have to understand how the scoreboard works.

When the scoreboard rankings idea was first concieved, we had to tackle many, many different details in order to create as even a playing field as possible. You can read the workflow and methodology in this topic (warning, it's very math heavy.) One of the things we decided was that games that have more scores are "worth more" than games with few scores. For example, a first place on SSX Blur (which has 124 total scores) is worth much less in the overall rankings than a first place on SSX 3 (which has 753 scores total). This was a mathematical way of evaluating entries that benefits scores on those tracks that have more competition.

And this was a very elegant feature, and it worked well for many years. But now we're running into an issue, and it's that SSX 2012 is too popular. Let me quote the developer I worked with to explain:


So that's the issue why RockLee is second in rankings over people who have many more first place scores in many more games. Now, when we calculate the Best Overall Player across ALL SSX games, it should value a player who has scores in MORE games rather than just a lot of first place scores in one game. However, like I mentioned above, SSX 2012 is so popular that it's skewed the formula. So I need your input as to what you feel would be the best way to address.

I like SSX 2012, but the Best Overall category is no longer accurate, because SSX 2012 has tipped the balance so far to its side that it's throwing the whole thing out of balance, so we need to figure out how to even the scales again. Here are the specifics of the equation that determines the value of each score for a user in the Best Overall category:


So, as you can see, SSX 2012 has completely skewed the formula and the rankings page is no longer accurate. So in thinking about this solution, it's apparent to me that we need a good way to "devalue" SSX 2012's entries so that it is fairly represented in the overall scheme of the scoreboard, but doesn't skew the older games. While SSX 2012 has the most competition by far, there must be a way to even out the field on the Best Overall Player field. The question is, how to do that?

Give me your input, and I'm going to let my brain stew on this. In the meantime, ask any questions you all have and I fill you all in on the details of how the scoreboard works.

_________________

EDIT: So, I put this out before you edited it yourself. I have no idea what is going on with those numbers on the top of your post now, but what is stated underneath may not be easy as I believe.

SSX 2012 has nearly twice as many scores posted already than all the other games combined with more likely to come (2012=1762, others=1985). Considering not many more scores will be posted for the others and the rough equality between those two statistics, why not make two separate formulas. One for 2012, the other for the rest.

I don't think there is a concrete value to perfect it, but weighing 2012 0.66 and the others 0.33 that would help, I believe. Or to get more technical, I suppose weighing 2012 with 0.75, 3-Tricky-On Tour with 0.5, and SSX-Blur with 0.25.

I am not the best on code and might be wrong here, but wouldn't putting in concrete values in replacement of the formula work. If that is the case, couldn't a like 2 month assessment be made and change possible outliers as needed?

_________________
"You came to prepare for the future, I live for right now"
Xbox Gamercards
PSN Gamercards

Yeah, I realized that my topic made no sense without an overview of how the scoreboard works, so I went back and edited it in. My apologies if it was confusing, but it should be clearer now.



Not a bad idea. Care to expound on why? Would you say that when the next SSX game comes out, SSX 2012 will be pushed into roughly the same category as the others after a while? For the longest time, SSX 3 was the heavyweight, the game everyone wanted to play. It's possible SSX 2012 may go by the wayside the way the other games did when the next one comes out.


Can you give me a better idea of what you mean? The total of all the other values equals 1.0, because what we're totalling up here is the percentage of scores in the database, and that's how the scores are weighed when you calculate the total value of each score a person has across ALL the games.


The problem with a set formula is then I have to change it all the time. My optimal solution is to have something based on numbers of the scoreboard, a dynamica formula that would take into account the changing nature of the scoreboard and weight it that way.

_________________

If it was up to me I'd have Vix put a poll up that says "Which game(s) do you think should be weighted more when calculating the Scoreboard Overall Honors system?", leave it up for a good long while, then take the percentage calculated from those votes and put them in in place of that code up there.

(~how it would probably play out)

SSX - 8%

SSX Tricky - 19%

SSX 3 - 23%

SSX OT - 13%

SSX Blur - 6%

SSX 2012 - 31%

Not too mathy, but its efficient.

Admittedly, you'd have to update it from time to time, but doing it the month leading up to the Merq awards would work fine I think. Then that ratio would serve as the SSX Overall rankings for the year.

As much as that idea sounds good on paper, that's not the way the reality would play out. It would probably be more like this:

SSX 2012: 85%
All the others: 15%

Because most of the people here now are here for SSX 2012. When the next game comes out, most of them will be here for that -that's the way the system works.

So my brain is starting to come up with ideas. We need some other variable to weight the scores by - something that is not simply a percentage of total scores - because SSX 2012 is unfairly overbalanced due to how many tracks it has. Because it has over 100 tracks, it's skewing the results way in it's favor.

So that means we need to find a new formula to determine how much weight scores have. That means we can weight it based not only on percentage of scores, but also a few other factors like 1) amount of time the game's been out, longer = higher weight or 2) number of users that have submitted scores for a game or 3) some other variable that is hard coded into the scoreboard data and not based on qualitative data (read: people's opinions). What do you guys think of that idea?

_________________

Find some way to weight each game evenly by number of tracks?
Like SSX3 has 20 events, but SSX 2012 has what, 150? Each track in SSX3 needs to be worth 6 or 7x what each track in SSX2012 is worth...

I'm tired and can't do math right now but it sounds simple enough?

_________________

Hmm...interesting. Good idea as a basis! Let me think it through and see what the negatives might be in that decision.

_________________

Hmmmmm....

What's your objection to them all being a weighted value of 1? I'm not proposing it, but your answer could spark some ideas maybe.

If all the scores have the same weight, that wouldn't really solve much. That's the way it was in the beginning, and people who had lots of #1 scores on games that have very few scores (like SSX Blur) always were in the lead because their scores were way high since there was NO competition on those tracks. Read the details on the math and you'll see what I'm talking about. Likewise, if all games held the same value, we'd be back at square one - SSX 2012 would be dominant in the standings and users with top scores on that would run rampant on the Best Overall category.

_________________

So the current waiting system is:


n = number of scores for a game and
s = number of total scores.
You don't need the 100s in there.

The simplest fix would be to pull it back to the middle by expanding the calculation with an even weight between the games, something like this:

n = number of scores for a game and
s = number of total scores.
g = number of SSX games available to submit scores to.

How does this change the weights?

To even out more, weight could be added to the new part of the equation, except that means giving more free weighting to less popular games such as SSX and SSX Blur.

Another idea I thought about is suggest creating a function that stays true to the original design, but is slightly altered to create resistance to large amounts of submissions, seen below:

n = number of scores for a game and
s = number of total scores.

Below is the changes to the weights, NEW is the original formula I suggested and SQRT is the formula I suggested above.

I personally think this is a more elegant solution. Using square root makes it more difficult to gain weight if you already have a large amount of scores, in addition it isn't just giving games free weight. To make any serious decision, a future projection of score submissions would be needed to get an idea of how much weight is needed to keep the weighting in a desirable range.

The downside is both of these formulas I've proposed would give incentive to play less played games, since games such as SSX and SSX Blur are getting significant boosts in weight. I realize that the current weighting system was trying to avoid that, but it's likely any mathematical solution is going to have this problem.

========

This seems like a good place to bring up a second problem with the current formula. When Rocklee puts up a ridiculous record, the second place person will get significantly less than when a Xedec or a Hero puts up a ridiculous score on Race or even Trick.

Just based on observation, I suspect showoff and survive are calculated by the same method, so here are arbitrary examples, using the scoreboard calculation on one track:

Monster Survive:
1st: Rocklee 502,500.50 meters -- calculation 10,000.00
2nd: Ahlyis 102,465.43 meters -- calculation 2,039.11

Serenity Trick:
1st: Hero 210,189,872 pts -- calculation 10,000.00
2nd: Yako 161,501,504 pts -- calculation 7683.60

Off The Grid Race:
1st: Xedec 74.43 seconds -- calculation 10,000.00
2nd: KeKo 70.56 seconds -- calculation 9618.52

The problem that is illustrated is Rocklee can gain a greater net advantage by putting up a ridiculous score than Hero or Xedec can by putting a ridiculous score. In addition, a person can put up above average race times and climb up the board faster than if a person puts up above average trick or survive scores.

I assume the intention was to create incentive for putting up a ridiculous score, but the unintended result is that the best way to move up the scoreboard is to simply put up scores in racing categories where an average score is calculated as competitive and do your absolute best to put up a ridiculous scores in a areas that have wider gaps between scores.

At the moment, I don't have a solution and I'm not sure you really even want to address it.

_________________
[/url]
XBL GT: iTofu | Always looking for rivals.
Ozmo/WinonaNET | MerqCity Scoreboard

TL;DR

_________________
WinonaNET is a RiderNet experiment with a Cross-Platform Scoreboard: http://ssx.ozmonet.com

I just read everything going on here, and i cant say i fully understood it all. Some new things turned up on my score honors like overall #1 n best #2 hadnt ever noticed that. Is there a possibility to just take the titles from me? Or even if i have to delete my file to fix it, because i know im not the best at this game BUT i mean id be more than happy to do that. I apologize if my topic gets side tracked but why not just keep fixing the source than worryin about those numberss haha. U lost most of me after n + x = 0.776875 or whatever it was.

Rock, whatever is talked about in this thread, there is absolutely no problem with you. You are dominating the survival category in 2012 and that is awesome. In fact, we should be glad you did, because your dominance raised a lot of issues with our system that I didn't think of when SSX 2012 went live. Don't worry about the title stuff, because that's what we're aiming to fix and it's not a big deal if you have a few extra awards for a while.

Itofu - this is the sort of stuff I was looking for when I threw it out to the community. I'm not a math whiz by any stretch, though I am capable of wrapping my brain around the big issues. Both your suggestions and your spotting the problem with the Survival category having no actual ceiling are both greatly appreciated.

_________________

It's tough being this good.

_________________
WinonaNET is a RiderNet experiment with a Cross-Platform Scoreboard: http://ssx.ozmonet.com

It's a good idea. I'm going to type out your formula, because it definitely could be explained more clearly. lol

For Trick & Survive:
Sc = The player's individual score.
µSc = Mean score for track(s).
σ = Standard deviation of scores on the track(s).

For Race:
Sc = The player's individual score(s).
µSc = Mean score for track(s).
σ = Standard deviation of scores on the track(s).

For Totals:
Scores = the scores derived from the formulas above this one, for the tracks you are obtaining a total score for.
Tracks = total number of tracks in the set of tracks you are obtaining a total score for.


This addresses the problem I brought up in the second half of my post, which is the same problem you brought up.

The distribution is still going to be a little rough in areas without much competition and enormous outliers, like many of Rocklee's survival scores, but I don't think you really can fix that completely with any formula without minimizing the weight of the score and adding in weight for the placement of the score on the track.

_________________
[/url]
XBL GT: iTofu | Always looking for rivals.
Ozmo/WinonaNET | MerqCity Scoreboard

THAT'S WHAT I SAID!!

I've read this all and I get the general idea of how the system works, but I don't get why it needs to be so complex. Forgive me if I have missed something or if this is just absolutely ridiculous but why can't you just do the following,





There must be a reason why it can't be this simple otherwise it would have been done, but can someone explain why?

It doesn't really show the magnitude of the greatness. For example, Rocklee would simply get 1 more point than the second place person. The problem with addition/subtraction is that it treats every score as being equal, which in RockLee's case, is not true. If I scored a MILLION meters and over 2 standard deviations higher than mean, I'd want it to be worth more! Each race has a theoretical minimum, even if you start at 300m/s+ and fly in perfectly straight to the end (think jet plane), you still CANNOT complete Serenity any faster than 2 mins. (I didn't do the math, it's just an example). If someone squeaks out of second closer to that theoretical minimum, it should be worth however much that new score is AWAY from the group, not just "one more point".

The standard deviation calculation allows for the acknowledgement of someone's GREATNESS on a track by having a (subdued) multiplier. Therefore, the farther you are AWAY from the group, the better your score is PERCEIVED.

If I beat the number 2 person on 150 drops, by 0.01 meters (or 1 point, or 0.01 seconds), my score with addition and subtraction is 150 points greater, and in order to beat me, you'd have to beat me 150 times. That's a VERY hard feat for such a small margin to win. It's out of proportion. With the STDDEV, my score would be only MINIMALLY higher (probably about 0.1 points higher). So that if the number 2 person, beat me by a LOT on a handful of tracks, it would weigh heavier, and he'd be number 1 by a very small margin.

The person with the MOST #1s isn't the best, it's the person who whips the MOST ass.

Ah I see.

However I'd just like to add that for example, in Formula 1 racing, the person who finishes 1st place gets the same amount of points whether he is 0.01 seconds faster than 2nd or 10 minutes faster than 2nd.

1000 years ago we thought the earth was the center of the universe, 500 years later we thought the earth was flat, and there are still people to this day that believe both, but smarter people know better.

Formula 1 Racing believes the best way to find 1st place is to count the number of wins.

I added the Average, and Standard Deviations along with the multipliers to my scoreboard and user pages so you can see sample distributions. In this scoreboard, a person is centered around 0, where 0 is the average. Therefore, the higher number, the more awesomer. I've named it the "Awesome-Value"

Edit:
The more I look at it, the more I like it. It shows you HOW much more awesomer you are than the average.

I'm not sure how or why that is relevant but I understand why there has to be a special equation now.

Here's a few Examples using this calculation:

TickleMeOzmo - Average Awesomeness: 0.22 (Race: -0.49 | Trick: -0.89 | Survive: 1.32 ) (I was testing )
MarcusAnnex - Average Awesomeness: 2 (Race: 1.05 | Trick: 2 | Survive: 3.43 )
Xedec - Average Awesomeness: 2.04 (Race: 2.11 | Trick: Not Enough Tricks | Survive: Not Enough Survives )
Zockende_Socke - Average Awesomeness: 2.37 (Race: 2.03 | Trick: 3 | Survive: Not Enough Survives )
greatsinister - Average Awesomeness: 1.29 (Race: 1.5 | Trick: Not Enough Tricks | Survive: Not Enough Survives )
iTofu - Average Awesomeness: 1.19 (Race: 1.11 | Trick: 1.48 | Survive: 0.72 )
chromemaps - Average Awesomeness: 1.29 (Race: 1.17 | Trick: 1.14 | Survive: 1.99 )

I don't really have anything to contribute to this, but I think if you want to illustrate the magnitude of greatness, you gotta put some logarithms and shit here and there in hope it will curve the uber-high scores. No? I'm nowhere where tickleme is programming-wise and I'm not that kind of headguy. I usually do things trial-and-error and then fine-tune until I like it. And if it doesn't work, I create exceptions rather then revamp the whole system.

So yeah, the first thing that came into my mind is to write some kind of exception for Rocklee.

_________________

This should be your tagline on the forums.

Let's not be rude Ozmo.

I'm very impressed with your posts Ozmo. I haven't been posting much because it's been super busy, but let me say I'm thinking over stuff.

_________________