RuneTrack Forums
It is currently Thu Aug 17, 2017 8:22 pm

All times are UTC [ DST ]




Reply to topic  [ 28 posts ]  Go to page 1, 2  Next
Author Message
PostPosted: Tue Jun 01, 2010 3:04 am 
Offline

Joined: Tue Jun 01, 2010 3:01 am
Posts: 4
The adventurer's log shows a players stats even under 30, therefore stats that are under 30 should be pulled from there, to complete the stats even though it doesnt show rank. It still shows the xp which is what many of us use to track, not by rank. I dont have any stats under 30 but when dung came out it woulda been kinda cool to see xp gain from the early levels.


Top
 Profile  
 
PostPosted: Wed Jun 02, 2010 6:21 am 
Offline
User avatar

Joined: Sun Jun 14, 2009 10:13 am
Posts: 26
Actually it doesn't because the Adventurer's Log runs off of the highscores. Even when I was under 30 Dungeoneering when it was released I still couldn't see my level. So either way you have to get it past 30 for it to be visible.

_________________
Image


Top
 Profile  
 
PostPosted: Wed Jun 09, 2010 8:23 pm 
Offline

Joined: Tue Jun 01, 2010 3:01 am
Posts: 4
No you dont. check this out. http://services.runescape.com/m=adventurers-log/display_player_profile.ws?searchName=dastrowner1
Hunter level is shown, and so is the xp. and its only level 20.


Top
 Profile  
 
PostPosted: Sun Jul 25, 2010 3:45 pm 
Offline
User avatar

Joined: Thu Mar 25, 2010 6:26 pm
Posts: 100
Adventurer's Log is hard to fetch data from because it can't show all information from all your skills at the same time. And there are a lot of other things like the div's and other html attributes that are making it difficult to process.

_________________
Image

Image

I <3 Runetrack


Top
 Profile  
 
PostPosted: Sun Jul 25, 2010 3:47 pm 
Offline
User avatar

Joined: Thu Mar 25, 2010 6:26 pm
Posts: 100
A little help for swordy:
Code:
http://services.runescape.com/m=adventurers-log/display_player_profile.ws?searchName=Playername&skill=SkillID
this will make it easier. If you want me to make a php script to fetch it just pm me ;)

_________________
Image

Image

I <3 Runetrack


Top
 Profile  
 
PostPosted: Tue Jul 27, 2010 12:26 am 
Offline
Site Admin

Joined: Sat Jun 06, 2009 12:38 am
Posts: 546
DutchStarz wrote:
Adventurer's Log is hard to fetch data from because it can't show all information from all your skills at the same time. And there are a lot of other things like the div's and other html attributes that are making it difficult to process.

Actually, it does display information from all skills in the "skills" variable around line 65 of the source on the Adventure Log profile page, regardless of what skill you've selected.

However, you're correct that the Adventure Log is much harder to process because of its bulk.

Take, for example, how short the source code is for the RS highscores:
http://hiscore.runescape.com/index_lite ... ord_Kill11

It only takes 0.5 seconds to fetch highscores data from the above link using file_get_contents(). However, the Adventure Log source code is of course much longer and can take many seconds to fetch:
http://services.runescape.com/m=adventu ... ord_Kill11

I've been working on this recently and the hard part isn't really extracting the data from the page once it's fetched - but the long time it takes to fetch the page in the first place. This update wouldn't be worth doing if the finished product didn't work fast enough.

Using cURL, I've been able to get the fetch time down to 1.7 seconds. Here's the current test code I'm using:
Code:
function fetch_log($url){
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
    $page = curl_exec($ch);
    curl_close($ch);
    return $page;
}
 

$url 
"http://services.runescape.com/m=adventurers-log/display_player_profile.ws?searchName=Sword_Kill11";
$inc 20;

$time = microtime(true);

for ($x = 0; $x < $inc; $x++) {
    $store[$x] = fetch_log($url);
}
echo (microtime(true)-$time)/$inc; // Echoes ~1.7    

If someone can find any way by altering the above code to fetch the source code of an Adventure Log using PHP so fast that the page echoes a time below 1.7 seconds, then I'll credit you in a news post on the front page and implement it for use. :)

_________________
ImageImage


Top
 Profile  
 
PostPosted: Tue Jul 27, 2010 12:55 pm 
Offline
User avatar

Joined: Thu Mar 25, 2010 6:26 pm
Posts: 100
Will take a look at it, but currently adventurer's log is down (maintenance).

_________________
Image

Image

I <3 Runetrack


Top
 Profile  
 
PostPosted: Wed Mar 02, 2011 11:07 am 
Offline

Joined: Tue Mar 01, 2011 1:32 am
Posts: 5
Support :)

_________________
Image


Top
 Profile  
 
PostPosted: Thu Mar 03, 2011 1:48 am 
Offline

Joined: Thu Mar 03, 2011 1:39 am
Posts: 3
Support

_________________
Image


Top
 Profile  
 
PostPosted: Fri May 27, 2011 9:25 am 
Offline
User avatar

Joined: Tue Jul 27, 2010 8:21 am
Posts: 161
Location: Australia
support 99%! bazinga!

Image

Image

Image

"You will not believe what i've been through tonight!"
:Duck: :Fighter: :OopsSign: :Yahoo: :Beer: :Sick:

_________________
Image


Top
 Profile  
 
PostPosted: Tue Jun 14, 2011 3:09 am 
Offline

Joined: Tue Jun 07, 2011 2:09 am
Posts: 48
Referring to you wanting a fast method for fetching stats: I don't understand why you had to run the fetch 20 times and if 20 runs is necessary per person, isn't diving by 20 to show a single fetch time giving an inaccurate answer for the amount of time necessary to run?

I took you up on that challenge and wrote a script to fetch my stats. My script organizes all of the stats into a 2 dimensional array: first dimension is skill, second dimension is the name, experience, level, and rank. Since I did not understand the dividing by 20, I left the 20 out and did my first time fetch before I did anything at all, and my last time fetch right after I did everything and then subtracted the first time from the second. I keep refreshing to see the time I get and the majority fall in the 1.4 to 1.6 second range.


Top
 Profile  
 
PostPosted: Tue Jun 14, 2011 5:49 am 
Offline
Site Admin

Joined: Sat Jun 06, 2009 12:38 am
Posts: 546
Dsctatom wrote:
Referring to you wanting a fast method for fetching stats: I don't understand why you had to run the fetch 20 times and if 20 runs is necessary per person, isn't diving by 20 to show a single fetch time giving an inaccurate answer for the amount of time necessary to run?

Running it 20 times was just for benchmarking purposes. I ran the script 20 times than divided the result by 20 in order to get a reliable average speed, as just taking a speed test for a single fetch would be unreliable.

Dsctatom wrote:
I took you up on that challenge and wrote a script to fetch my stats. My script organizes all of the stats into a 2 dimensional array: first dimension is skill, second dimension is the name, experience, level, and rank. Since I did not understand the dividing by 20, I left the 20 out and did my first time fetch before I did anything at all, and my last time fetch right after I did everything and then subtracted the first time from the second. I keep refreshing to see the time I get and the majority fall in the 1.4 to 1.6 second range.

Ah, that's interesting - what function are you using to fetch the stats? file_get_contents() and cURL both were running at around 1.7 seconds. However, looking at the time stamp of my post above - this was back when RuneTrack was hosted on a slower server.

Interestingly enough, I re-tested the exact same script above on the new server RuneTrack is currently running on which is much faster and got an average run time of 1.2 seconds.

Additionally, using multi-threaded cURL (http://php.net/manual/en/function.curl-multi-exec.php) this can be expanded to run up to 50 simultaneous threads (on the RuneTrack server at least), taking the time down to 0.024 (1.2/50) seconds (absolute best case scenario - cURL doesn't have a high success rate like file_get_contents() so some requests may have to be made multiple times which adds some additional time to the equation). I didn't really go into this aspect in my previous post, but I'm pretty confident that curl_multi_exec() is the best bulk fetching function available in PHP - I was just curious to see whether someone knew of anything better than cURL as I'm always looking to speed up the system updates. :)

_________________
ImageImage


Top
 Profile  
 
PostPosted: Tue Jun 14, 2011 6:07 am 
Offline

Joined: Tue Jun 07, 2011 2:09 am
Posts: 48
Sword Kill11 wrote:
Additionally, using multi-threaded cURL (http://php.net/manual/en/function.curl-multi-exec.php) this can be expanded to run up to 50 simultaneous threads (on the RuneTrack server at least), taking the time down to 0.024 (1.2/50) seconds (absolute best case scenario - cURL doesn't have a high success rate like file_get_contents() so some requests may have to be made multiple times which adds some additional time to the equation). I didn't really go into this aspect in my previous post, but I'm pretty confident that curl_multi_exec() is the best bulk fetching function available in PHP - I was just curious to see whether someone knew of anything better than cURL as I'm always looking to speed up the system updates. :)


50 simultaneous threads with different input, I take it? Is the fetch for our stats being run every X <insert unit of time measurement> or is it when we look at our stats that it updates (apart from the server update when it would have to fetch an update for everyone)? If it's the former: that seems like quite a server load; if it's the latter: I wouldn't expect 50 simultaneous runs to be occurring at the same moment anyway.

The method I used was file_get_contents() inside of a preg_match in order to narrow it down to only the skills, then I formatted it into a php array (which did not seem to have any noticeable change to the load duration). Assuming you only grabbed the skills array, this piece should organize the data for you:

Code:
$levels = rtrim(ltrim($alllevels[0],"var skills=[new skill\("),"\')];");

$levels_split= preg_split('/\),new skill\(/', $levels);

for ($x = 0; $x < 26; $x++) {
$levels_array[$x] = preg_split('/([^0-9],|,[^0-9])/',$levels_split[$x]);
$levels_array[$x] = str_replace("'","",$levels_array[$x]); 
} 


I had to strip it of JavaScript formatting and Jagex decided to be inconsistent on their use of quotations, which added some more regex to it, but the final product allows for: $levels_array[0][1] to tell you total experience and so forth.


Top
 Profile  
 
PostPosted: Tue Jun 14, 2011 6:45 am 
Offline
Site Admin

Joined: Sat Jun 06, 2009 12:38 am
Posts: 546
Dsctatom wrote:
Sword Kill11 wrote:
Additionally, using multi-threaded cURL (http://php.net/manual/en/function.curl-multi-exec.php) this can be expanded to run up to 50 simultaneous threads (on the RuneTrack server at least), taking the time down to 0.024 (1.2/50) seconds (absolute best case scenario - cURL doesn't have a high success rate like file_get_contents() so some requests may have to be made multiple times which adds some additional time to the equation). I didn't really go into this aspect in my previous post, but I'm pretty confident that curl_multi_exec() is the best bulk fetching function available in PHP - I was just curious to see whether someone knew of anything better than cURL as I'm always looking to speed up the system updates. :)


50 simultaneous threads with different input, I take it? Is the fetch for our stats being run every X <insert unit of time measurement> or is it when we look at our stats that it updates (apart from the server update when it would have to fetch an update for everyone)? If it's the former: that seems like quite a server load; if it's the latter: I wouldn't expect 50 simultaneous runs to be occurring at the same moment anyway.

Yes, 50 simultaneous threads to capture 50 different Adventure Log or stat pages.

Essentially, there's two update systems:
- Every time you view your Profile Page, your stats update via a file_get_contents() call.
- The system update which runs at 2am Eastern daily uses curl_multi_exec() to update every user's stats before the official capture for that day is taken.

Dsctatom wrote:
The method I used was file_get_contents() inside of a preg_match in order to narrow it down to only the skills, then I formatted it into a php array (which did not seem to have any noticeable change to the load duration). Assuming you only grabbed the skills array, this piece should organize the data for you:

I'm not exactly sure what you meant by this - but putting file_get_contents() inside a preg_match won't speed up the process. You're not only fetching a certain portion of the page in that scenario - you're fetching the entire page, then just parsing out all of the unneeded content (not entirely sure if this is what you meant though, so I apologize if I'm assuming anything incorrectly here).

The only way to speed up file_get_contents() is to explicitly limit its fetching length, using the fifth parameter:
Code:
// Grab the first 100 characters 
$page = file_get_contents("http://google.com", null, null, 0, 100); 

As you only need about the first 5,000 characters to get the stats array on the Adventure Log from the JavaScript, you can do the entire fetching and parsing using just this:
Code:
$url = "http://services.runescape.com/m=adventurers-log/display_player_profile.ws?searchName=Dsctatom";
$page = file_get_contents($url, null, null, 0, 5000);
$start = strpos($page, 'new skill');
$end = strpos($page, '];', $start);
$array = str_replace(",", "", substr($page, $start, ($end-$start)));
preg_match_all("/new skill\(\'(.*?)\'\'(.*?)\'(.*?)\'(.*?)\'\)/", $array, $stats, PREG_SET_ORDER); 

This actually runs in about 0.5 seconds, even though it's using file_get_contents(), and produces a $stats array which contains an element for each skill name, level, rank, and xp (for example, in the above code, $stats[0][3] holds the total level).

Thanks for all your efforts on this again - always cool to revisit things like this and work through new solutions. :)

_________________
ImageImage


Top
 Profile  
 
PostPosted: Tue Jun 14, 2011 6:54 am 
Offline

Joined: Tue Jun 07, 2011 2:09 am
Posts: 48
Sword Kill11 wrote:
I'm not exactly sure what you meant by this - but putting file_get_contents() inside a preg_match won't speed up the process. You're not only fetching a certain portion of the page in that scenario - you're fetching the entire page, then just parsing out all of the unneeded content (not entirely sure if this is what you meant though, so I apologize if I'm assuming anything incorrectly here).


I was just stating my method, not trying to say that preg_match sped anything up.

Sword Kill11 wrote:
This actually runs in about 0.5 seconds, even though it's using file_get_contents(), and produces a $stats array which contains an element for each skill name, level, rank, and xp (for example, in the above code, $stats[0][3] holds the total level).


Considering 0.5 not being that slow, what if you only used the adventure log as a secondary resource for users that didn't have all levels 30+? This would result in roughly twice the time for users with lower skills, but as far as I can see, no significant change to other users since it'd probably be handled by a simple if statement.


Top
 Profile  
 
PostPosted: Sun Jun 19, 2011 10:14 pm 
Offline

Joined: Thu Jan 27, 2011 7:53 pm
Posts: 152
Ok

_________________
Image


Top
 Profile  
 
PostPosted: Sun Sep 11, 2011 12:39 pm 
Offline
User avatar

Joined: Thu Mar 25, 2010 6:26 pm
Posts: 100
Just giving my opinion on this, but since the lite scores actually show level 30 or over i think it would be not of much use anymore to use the adventurer's log for tracking also since it's members only it would be unable to track those who are F2p and people who really care about tracking are most certainly over level 30 i think. These first 13k xp would not matter to such persons. I think its a less then 1% case when people tracking have not yet got a level over 30 so for this group to has this extra feature would be a waste of resources for the runetrack servers which could be used better for features that more users can benefit from.

Greetings.

_________________
Image

Image

I <3 Runetrack


Top
 Profile  
 
PostPosted: Sun Sep 11, 2011 9:43 pm 
Offline

Joined: Tue Jun 07, 2011 2:09 am
Posts: 48
Although it isn't common, it does happen. Since most people that are tracked are over level 30, then it wouldn't be much of a waste of resources since it would rarely be pulling from the adventure log.


Top
 Profile  
 
PostPosted: Mon Sep 12, 2011 9:16 pm 
Offline
User avatar

Joined: Thu Mar 25, 2010 6:26 pm
Posts: 100
Just has to pull from database which are under 30 yes but just not that handy when having to make a whole extra check and fetch script.

_________________
Image

Image

I <3 Runetrack


Top
 Profile  
 
PostPosted: Tue Oct 04, 2011 5:20 am 
Offline
User avatar

Joined: Tue Jul 27, 2010 8:21 am
Posts: 161
Location: Australia
Yeah

_________________
Image


Top
 Profile  
 
Display posts from previous:  Sort by  
Reply to topic  [ 28 posts ]  Go to page 1, 2  Next

All times are UTC [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group