Dsctatom wrote:
Sword Kill11 wrote:
Additionally, using multi-threaded cURL (
http://php.net/manual/en/function.curl-multi-exec.php) this can be expanded to run up to 50 simultaneous threads (on the RuneTrack server at least), taking the time down to 0.024 (1.2/50) seconds (absolute best case scenario - cURL doesn't have a high success rate like file_get_contents() so some requests may have to be made multiple times which adds some additional time to the equation). I didn't really go into this aspect in my previous post, but I'm pretty confident that curl_multi_exec() is the best bulk fetching function available in PHP - I was just curious to see whether someone knew of anything better than cURL as I'm always looking to speed up the system updates.
50 simultaneous threads with different input, I take it? Is the fetch for our stats being run every X <insert unit of time measurement> or is it when we look at our stats that it updates (apart from the server update when it would have to fetch an update for everyone)? If it's the former: that seems like quite a server load; if it's the latter: I wouldn't expect 50 simultaneous runs to be occurring at the same moment anyway.
Yes, 50 simultaneous threads to capture 50 different Adventure Log or stat pages.
Essentially, there's two update systems:
- Every time you view your Profile Page, your stats update via a file_get_contents() call.
- The system update which runs at 2am Eastern daily uses curl_multi_exec() to update every user's stats before the official capture for that day is taken.
Dsctatom wrote:
The method I used was file_get_contents() inside of a preg_match in order to narrow it down to only the skills, then I formatted it into a php array (which did not seem to have any noticeable change to the load duration). Assuming you only grabbed the skills array, this piece should organize the data for you:
I'm not exactly sure what you meant by this - but putting file_get_contents() inside a preg_match won't speed up the process. You're not only fetching a certain portion of the page in that scenario - you're fetching the entire page, then just parsing out all of the unneeded content (not entirely sure if this is what you meant though, so I apologize if I'm assuming anything incorrectly here).
The only way to speed up file_get_contents() is to explicitly limit its fetching length, using the fifth parameter:
Code:
$page = file_get_contents("http://google.com", null, null, 0, 100);
As you only need about the first 5,000 characters to get the stats array on the Adventure Log from the JavaScript, you can do the entire fetching and parsing using just this:
Code:
$url = "http://services.runescape.com/m=adventurers-log/display_player_profile.ws?searchName=Dsctatom";
$page = file_get_contents($url, null, null, 0, 5000);
$start = strpos($page, 'new skill');
$end = strpos($page, '];', $start);
$array = str_replace(",", "", substr($page, $start, ($end-$start)));
preg_match_all("/new skill\(\'(.*?)\'\'(.*?)\'(.*?)\'(.*?)\'\)/", $array, $stats, PREG_SET_ORDER);
This actually runs in about 0.5 seconds, even though it's using file_get_contents(), and produces a $stats array which contains an element for each skill name, level, rank, and xp (for example, in the above code, $stats[0][3] holds the total level).
Thanks for all your efforts on this again - always cool to revisit things like this and work through new solutions.
Login
Register
Go Back To RuneTrack.com
FAQ
Search
Members
