Markdown formatting is now available for admin notes. I added a Markdown extension to allow bare links to work as well.
So for those of you who want a clickable email address or URL in an admin note (and there are a few of you), just type it in now. Yay.
Saturday, January 17, 2009
Friday, January 9, 2009
memcached
I'd held off on caching for Swagbook so far. Finally decided to give it a shot for just a few pages now that the calculation framework is in and it's relatively easy to tell when a guild data has changed.
So far just the character list and run list are cached - they're the most often hit up that will benefit from caching. You may or may not notice a difference.
Yeah, I like to commit.
So far just the character list and run list are cached - they're the most often hit up that will benefit from caching. You may or may not notice a difference.
Yeah, I like to commit.
The new calculation engine, a.k.a. That Shouldn't Be So Difficult
I received quite a few requests for Zero Sum support - usually something along the lines of "could you show the sum of all the gear values for the run divided by how many people attended, so we can fill in the value ourselves?"
While this would be one way to implement Zero Sum, it lacked that certain special something. Changing one thing in the run resulted in a huge headache for all involved - and I'm not a fan of that.
So I decided the best approach to take would be to let people treat the attendance values as weights and add a Zero Sum DKP priority calculation. By treating the attendance as a weight, users can enter in any value they like - each attendee is allocated their share of the gear value. This allows very straightforward ways of implementing partial attendance (enter in what % of the raid they attended) and other fun things.
One nice result of this is that you can flip your pool over to Zero Sum without changing any of your data and it will Just Work.
I will warn you about something, though: the numbers you see may be slightly off from what you've seen if you're importing from another system. The reason for this is that Swagbook preserves the Zero Sum property on bonuses and penalties - that is that bonuses and penalties affect the whole pool. If you give a 5 point bonus to Adam, 5/n points are deducted from everyone, where n is the number of people in the pool. For the most part this isn't that noticeable unless you're doing a ton of bonuses and penalties.
Well, that's not precisely true - there is one situation where it's noticeable. Let's say that you attend a run and afterward your priority is 5. You do not attend the next run, but a new person does. Your DKP will then NOT be 5 like you expect, because now the "pool cost" of bonuses and penalties are being sliced up a little finer. This is one of the undesirable qualities of Zero Sum - your priority score can change without any action on your part.
(the rest of this is boring programmer talk, if you want the short version: Zero Sum is in. Please let me know about any bugs.)
Satisfied that I'd found a good way to approach the problem, the problem of implementing this came next. For those of you who don't know (ie you aren't me), previously every time a DKP value was changed, the run was marked "dirty". Just before closing the database connection, a query would run that would find all the dirty runs and (using one giant SQL query) calculate the sum attendance, gear, bonuses, and penalties for each run in that pool. As a result, getting priorities was largely a question of grabbing the precalculated values from the table so generated.
This was the reason for the short delay after saving changes, incidentally.
Easy enough, right?
Well, with Zero Sum the query would simply become untenable. No longer just a sum() of the column, Zero Sum is quite intricate - particularly if you consider bonuses and penalties.
So, I tried moving all of the calculations into Python, just to see if I could. Would have been easy to maintain, so it was worth a shot. After all was in place I fired up the benchmark, and...waited about 18 minutes for the front page to load.
Yeah, not going to work.
Next up, I tried making a table of stored calculations and triggers to remove dirty calculations. It was my hope that caching intermediate results might make recalculation faster (after all, rarely does the first run in a pool get changed). I also trimmed the fat in the code to stop re-calculating the same values. Render time went down to about 50 seconds, though truth to tell it was more the re-calculation culling than anything else.
So, my next question was: is it the calculation time or the marshalling time (transferring the values between the SQL server and python)? A quick experiment showed that run time was almost directly proportional to number of queries.
I decided to try out a stored procedure for calculating rather than using Python, and the change was remarkable. Think 5 seconds down to 0.02 seconds. Stored procedures were implemented for the rest of the calculations, and we were off to the races.
This is not to say that the stored procedures didn't go through a few changes. I started out being really stupid and having calculate_sum_attendance(xxx) only insert results into the calculations table. That's the fun at looking back at all this: I notice how much my thinking was constrained at each step by the method I'd attempted to use previously. Python to slow? Cache it. Still too slow? Build the cache with a stored procedure. Waaaait a second.
The caching is still there, but now calculate_sum_attendance(xxx) actually returns the value - as a result, only one query per calculation, rather than "refresh then load".
How much does the cache help? Honestly not 100% sure. The framework was all there, it cost me very little to use for the other procedures, and I know it's about 5x faster when the calculation already exists...but it could be that the repeated INSERTs are the majority of the cost. At the moment it all works and works well, so I don't think I'm going to mess with it.
So the new calculation methods are in. Zero Sum works like a charm now, and the way is paved for more intricate priority systems. After the next round of bugfixes, of course.
While this would be one way to implement Zero Sum, it lacked that certain special something. Changing one thing in the run resulted in a huge headache for all involved - and I'm not a fan of that.
So I decided the best approach to take would be to let people treat the attendance values as weights and add a Zero Sum DKP priority calculation. By treating the attendance as a weight, users can enter in any value they like - each attendee is allocated their share of the gear value. This allows very straightforward ways of implementing partial attendance (enter in what % of the raid they attended) and other fun things.
One nice result of this is that you can flip your pool over to Zero Sum without changing any of your data and it will Just Work.
I will warn you about something, though: the numbers you see may be slightly off from what you've seen if you're importing from another system. The reason for this is that Swagbook preserves the Zero Sum property on bonuses and penalties - that is that bonuses and penalties affect the whole pool. If you give a 5 point bonus to Adam, 5/n points are deducted from everyone, where n is the number of people in the pool. For the most part this isn't that noticeable unless you're doing a ton of bonuses and penalties.
Well, that's not precisely true - there is one situation where it's noticeable. Let's say that you attend a run and afterward your priority is 5. You do not attend the next run, but a new person does. Your DKP will then NOT be 5 like you expect, because now the "pool cost" of bonuses and penalties are being sliced up a little finer. This is one of the undesirable qualities of Zero Sum - your priority score can change without any action on your part.
(the rest of this is boring programmer talk, if you want the short version: Zero Sum is in. Please let me know about any bugs.)
Satisfied that I'd found a good way to approach the problem, the problem of implementing this came next. For those of you who don't know (ie you aren't me), previously every time a DKP value was changed, the run was marked "dirty". Just before closing the database connection, a query would run that would find all the dirty runs and (using one giant SQL query) calculate the sum attendance, gear, bonuses, and penalties for each run in that pool. As a result, getting priorities was largely a question of grabbing the precalculated values from the table so generated.
This was the reason for the short delay after saving changes, incidentally.
Easy enough, right?
Well, with Zero Sum the query would simply become untenable. No longer just a sum() of the column, Zero Sum is quite intricate - particularly if you consider bonuses and penalties.
So, I tried moving all of the calculations into Python, just to see if I could. Would have been easy to maintain, so it was worth a shot. After all was in place I fired up the benchmark, and...waited about 18 minutes for the front page to load.
Yeah, not going to work.
Next up, I tried making a table of stored calculations and triggers to remove dirty calculations. It was my hope that caching intermediate results might make recalculation faster (after all, rarely does the first run in a pool get changed). I also trimmed the fat in the code to stop re-calculating the same values. Render time went down to about 50 seconds, though truth to tell it was more the re-calculation culling than anything else.
So, my next question was: is it the calculation time or the marshalling time (transferring the values between the SQL server and python)? A quick experiment showed that run time was almost directly proportional to number of queries.
I decided to try out a stored procedure for calculating rather than using Python, and the change was remarkable. Think 5 seconds down to 0.02 seconds. Stored procedures were implemented for the rest of the calculations, and we were off to the races.
This is not to say that the stored procedures didn't go through a few changes. I started out being really stupid and having calculate_sum_attendance(xxx) only insert results into the calculations table. That's the fun at looking back at all this: I notice how much my thinking was constrained at each step by the method I'd attempted to use previously. Python to slow? Cache it. Still too slow? Build the cache with a stored procedure. Waaaait a second.
The caching is still there, but now calculate_sum_attendance(xxx) actually returns the value - as a result, only one query per calculation, rather than "refresh then load".
How much does the cache help? Honestly not 100% sure. The framework was all there, it cost me very little to use for the other procedures, and I know it's about 5x faster when the calculation already exists...but it could be that the repeated INSERTs are the majority of the cost. At the moment it all works and works well, so I don't think I'm going to mess with it.
So the new calculation methods are in. Zero Sum works like a charm now, and the way is paved for more intricate priority systems. After the next round of bugfixes, of course.
Wednesday, December 17, 2008
And that's a wrap for today
Okay, the final spots in the code that were not obeying guild preferences for date formatting are gone, gone, gone. Enjoy.
- New page for just viewing guild logs, for those who are into that sort of thing.
- No more spurious "admin message changed" logs for characters
- Fixed a javascript bug that kept some logs from showing
- Bulk gear changes now create log entries
Okay, one caveat:
If you were to, say, rename "Aegis of the Argent Crusade" to "Aegis of the Argent Crusa", you'd see log entries for each drop showing the drop had been changed. That's all well and good.
When the item scanner comes through and tries to find your item, it's going to change "Argent Crusa" to "Argent Crusade". So your previous change would be worthless. If this is the case, those log entries will be deleted. Whoosh. Gone.
Try not to panic. Everything is under control.
- New page for just viewing guild logs, for those who are into that sort of thing.
- No more spurious "admin message changed" logs for characters
- Fixed a javascript bug that kept some logs from showing
- Bulk gear changes now create log entries
Okay, one caveat:
If you were to, say, rename "Aegis of the Argent Crusade" to "Aegis of the Argent Crusa", you'd see log entries for each drop showing the drop had been changed. That's all well and good.
When the item scanner comes through and tries to find your item, it's going to change "Argent Crusa" to "Argent Crusade". So your previous change would be worthless. If this is the case, those log entries will be deleted. Whoosh. Gone.
Try not to panic. Everything is under control.
Dates and you
Okay, Aussies. You got your wish.
Time and date formats are now (somewhat) customizable. Set it up in the Guild Control Panel.
If there are formats you would like that are not in the list, let me know.
There are a couple of pages that have old date formatting. That'll be going away real soon now.
Time and date formats are now (somewhat) customizable. Set it up in the Guild Control Panel.
If there are formats you would like that are not in the list, let me know.
There are a couple of pages that have old date formatting. That'll be going away real soon now.
Small requested feature
The item scanner is far more flexible now. Means you should start seeing icons and info faster...since I'm not having to do as many manual changes anymore.
Also on the new list: if a character has been on a run in the last 30 days, they'll show up in the active list even if inactive. Much easier to keep track of people, says I.
Also on the new list: if a character has been on a run in the last 30 days, they'll show up in the active list even if inactive. Much easier to keep track of people, says I.
Sunday, December 14, 2008
A week of small changes
Only two tables left now in the entire system using the old dynamic tables. Rendering is now much faster - not seeing near as many multi-second page render times.
Coming up next: per-pool character listings. This should be the last feature needed before multi-priorities come in. Multi-priorities are where we might take into consideration, say, rank before DKP. Or attendance might be a factor.
Along with multi-priorities will come a rework of the pools and events page. Right now it's pretty confusing (as in there's next to no data whatsoever). I'm looking to create a page that will walk you through basic decisions about DKP so first-timers have a better time of it.
Oh, and give people the ability to actually delete events. That'd be nice.
Coming up next: per-pool character listings. This should be the last feature needed before multi-priorities come in. Multi-priorities are where we might take into consideration, say, rank before DKP. Or attendance might be a factor.
Along with multi-priorities will come a rework of the pools and events page. Right now it's pretty confusing (as in there's next to no data whatsoever). I'm looking to create a page that will walk you through basic decisions about DKP so first-timers have a better time of it.
Oh, and give people the ability to actually delete events. That'd be nice.
Subscribe to:
Posts (Atom)