I got around the yaml issue by using some custom string hacking - and PHP's string functions are pretty quick. By using split() and strpos() I could slice up my big yaml file into more digestible sizes (see previous post).
The propel issue, however, was always going to be more problematic. I considered removing all my ORM calls and either using delayed inserts or LOAD DATA INFILE statements. Both of which I have used successfully in the past, but implementing it in this case meant a complete rewrite of over 2000 lines of tested code.
So, I decided to use a similar approach to the ORM. Like YAML, Propel 1.2 is very good at what it does inside small to medium processes. It just isn't very efficient when used in big jobs - like inside large loops. So instead of trying to process my massive array that I managed to construct from the original yaml, I sliced it up and sent smaller yaml files back into the queue.
What this means is that instead of having a single item of 13,000 recipients sitting in the queue I have 130 items of 100 instead. My queue processing task can now batch 100 emails at a time and then terminate, thereby freeing up memory for the next batch. While there is some extra overhead in writing files to disk and duplicating some of the yaml, the benefits are well worth it. The end result is a scalable, fast and robust system that can handle almost any request by simply slicing it up into manageable chunks. TIme and money saved, client happy :)
| [ 0 trackbacks ] | permalink | related link |
( 3.1 / 179 )This is a problem I often come up against - sending email to thousands of recipients. The way I usually handle it is to do as little as possible in the master loop.
The best technique I have found is to defer database writes by writing to a temporary textfile in memory (using php://memory) and then using LOAD DATA INFILE to write contents to the database. That puts all the email data into a database queue, allowing a cron task to send the emails - say 50 per minute.
Using this technique I can send 50,000+ emails quite easily. The queue function takes under a minute to run and the cron job hacks away at the queue for about 24 hours or so.
The problem I have with this technique is that it evolved via a process of whittling away at the various framework layers used in my "enterprise applications". Framework layers supposedly designed to provide scalability.
Now you can probably tell from this blog that I use Symfony extensively (although I have dabbled with other frameworks, I always come back to Symfony). Perhaps it is due to the fact that my largest projects (over 1,000,000 lines of code) are locked into Symfony 1.0 and - more importantly - Propel 1.2. Arguably, what I should be doing is writing a Symfony plugin that performs large data processing using the techniques I've described, but it seems a shame to have a detailed, documented and unit-tested ORM that can only be used for serving small jobs like "web pages", and must be bypassed for any data intensive stuff. Isn't that a core requirement of "enterprise level" scalability?
Anyway, I'll try to end the rant with some code. Last night I wrote a custom yaml parser because the sfYaml::load() method in Symfony 1.0 (which uses spyc) was taking over 20 minutes to load a 1MB yaml file. I managed to load it in about 5 seconds. Now, I'll admit it's not really a yaml parser but more of a key-pair extractor but because it doesn't use any regex it is waay faster than the old sfYaml.
(NB: I couldn't install syck on my CentOS4 server and the newer sfYaml component from 1.2 failed to load yaml collections in the format shown below).
The yaml (multiply this by 10,000):
Recipients:
- title: 'Mr'
firstname: 'Bob'
lastname: 'Dobbs'
company: 'Subgenius Network'
email: 'bob@subgenius.com'
- title: 'Mrs'
firstname: 'Jane'
lastname: 'Dobbs'
company: 'Subgenius Network'
email: 'jane@subgenius.com'
The recipient extractor method:
public static function extractRecipients($yaml)
{
//separate recipients from yaml
$recipients_string = "Recipients:";
$recipients_start = strpos($yaml,$recipients_string);
$recipients_end = strpos($yaml,"bcc:");
$recipients_yaml = substr($yaml,$recipients_start,
($recipients_end-$recipients_start));
$yaml_start = substr($yaml,0,$recipients_start);
$yaml_end = substr($yaml,$recipients_end);
$yaml = $yaml_start . $yaml_end;
//load yaml
$yaml_array = sfYaml::load($yaml);
//get recipients array
$recipients_yaml = str_replace($recipients_string,"",
$recipients_yaml);
$recipients_array = split("-",$recipients_yaml);
//loop over recipients to get getails as array
$recipients_list = array();
foreach ($recipients_array as $r1) {
//loop over each line item
$r1_array = split("\r\n",$r1);
$my_recipient = array();
foreach ($r1_array as $r2) {
if (strpos($r2,":")!==false){
//extract key pair from line item
$r2_array = split(":",$r2);
if (sizeof($r2_array) == 2){
$key = str_replace("''","'",
trim(trim($r2_array[0]),"'"));
$value = str_replace("''","'",
trim(trim($r2_array[1]),"'"));
$my_recipient[$key] = $value;
}
}
}
if (sizeof($my_recipient) > 1){
//add this recipient to list
$recipients_list[] = $my_recipient;
}
}
//add recipients to yaml data
$yaml_array[trim($recipients_string,":")] = $recipients_list;
//return yaml data
return $yaml_array;
}
| [ 0 trackbacks ] | permalink | related link |
( 3 / 174 )I have enabled comments on the blog after a few requests for it... play nice!
| [ 0 trackbacks ] | permalink | related link |
( 3 / 162 )I recently had to integrate a forum into a Symfony 1.0 app and, having selected PhpBB3 for the job, went about writing an authentication module. Here's how...
1. Create the auth module
Lets say I want to call the module 'symfony'. I create a file called auth_symfony.php and drop it in the phpbb /includes/auth folder.
2. Write the autologin method
Because I want my forums to be on the same domain, I have saved the phpbb files in web/forum. This way I have access to the session variables stored in my symfony app. So when a user goes into the forums, the autologin method can be used to interrogate my smyfony session.
In a nutshell, the autologin method needs to verify the user's session, grab their details and log them into phpbb. So as to avoid having to manage forum users from my symfony app, I also create new phpbb users here if one doesn't already exist for the current user.
Note that phpbb needs its own session, so we need to switch between sessions here as well.
/**
* Autologin function
*/
function autologin_symfony()
{
include_once('includes/functions_user.php');
global $db, $config, $user;
$sess = session_name();
session_name('symfony');
session_start();
$sfSession = $_SESSION['symfony/user/sfUser/attributes']['userData'];
@session_name($sess);
@session_start();
if (isset($_REQUEST['admin'])){
$_SESSION['admin'] = $_REQUEST['admin'];
}
if (isset($_SESSION['data'])){
return $_SESSION['data'];
}elseif (!isset($sfSession['username']) &&
!isset($_SESSION['admin'])){
header("Location: /");
exit;
}elseif (isset($sfSession['username'])){
$user_row = array(
'username' => $sfSession['username'],
'user_password' => phpbb_hash($sfSession['username']),
'user_email' => $sfSession['email'],
'user_type' => USER_NORMAL,
'group_id' => 2
);
$sql ='SELECT *
FROM ' . USERS_TABLE . "
WHERE user_email = '"
. $db->sql_escape(utf8_clean_string($sfSession['email']))
."'";
$result = $db->sql_query($sql);
$row = $db->sql_fetchrow($result);
$db->sql_freeresult($result);
if ($row){
// Successful login...
$data = array_merge($row,array(
'status' => LOGIN_SUCCESS,
'error_msg' => false,
'autologin' => 1,
'user_row' => $row
));
$_SESSION['data'] = $data;
return $data;
}else{
//check for existing name
$sql ='SELECT *
FROM ' . USERS_TABLE . "
WHERE username_clean = '"
. $db->sql_escape(utf8_clean_string($sfSession['username']))
."'";
$result = $db->sql_query($sql);
$row = $db->sql_fetchrow($result);
$db->sql_freeresult($result);
if ($row){
//randomise username if duplicate found
$user_row['username'] = $user_row['username']
. " - " . rand(1000,9999);
}
//create new user
$user_id = user_add($user_row);
$user_row['user_id'] = $user_id;
$data = array_merge($user_row,array(
'status' => LOGIN_SUCCESS_CREATE_PROFILE,
'error_msg' => false,
'autologin' => 1,
'user_row' => $user_row,
'user_type' => USER_NORMAL,
'group_id' => 2
));
$_SESSION['data'] = $data;
return $data;
}
}
}
Note also that the $_SESSION['data'] array is used to avoid hitting the database everytime the user loads a new page in phpbb.
3. Create admin login
You might have noticed that the above code looks for a session var called 'admin'. I use this to track whether a user has come from the backend. This is used to display the phpbb login form. Otherwise the user is simply sent back to the symfony app login.
/**
* Login function
*/
function login_symfony($username, $password)
{
global $db, $config, $user;
$sql ='SELECT * FROM ' . USERS_TABLE
. " WHERE username_clean = '"
.$db->sql_escape(utf8_clean_string($username))
."' and user_password = '".md5($password)."'";
$result = $db->sql_query($sql);
$row = $db->sql_fetchrow($result);
$db->sql_freeresult($result);
if ($row){
$data = array_merge($row,array(
'status' => LOGIN_SUCCESS,
'error_msg' => false,
'user_row' => $row
));
$_SESSION['data'] = $data;
return $data;
}else{
return array(
'status' => LOGIN_ERROR_PASSWORD,
'error_msg' => 'LOGIN_ERROR_PASSWORD',
'user_row' => array('user_id' => ANONYMOUS),
);
}
}
4. Create logout method
This simply wipes both sessions and redirects the user to the symfony app homepage.
/**
* Logout function
*/
function logout_symfony($user_row)
{
$sess = session_name();
session_name('symfony');
session_start();
session_destroy();
@session_name($sess);
@session_start();
@session_destroy();
header("Location: /");
exit;
}
5. Create validate session method
Last but not least this method is used to check that the current user session is valid. Not 100% required, but a good precaution.
/**
* Validate session function
*/
function validate_session_symfony()
{
$sess = session_name();
session_name('symfony');
session_start();
$auth = $_SESSION['symfony/user/sfUser/authenticated'];
@session_start($sess);
@session_start();
//always redirect to home
$admin = isset($_SESSION['admin']);
if ($admin || $auth){
return true;
}
return false;
}
6. Configure
You'll need to log into the phpbb ACP and set the authentication module to your custom module. Then you need to bypass symfony in the .htaccess for the /forum URL
RewriteRule ^forum/.*$ - [PT]
RewriteCond %{REQUEST_URI} !^/forum
And that's basically it :)
| [ 0 trackbacks ] | permalink | related link |
( 3 / 197 )A lot of people moan about EFI-X so I thought I'd tell a good story. I got my EFI-X chip about 6 months ago. After a botched OS X 10.5.7 update the chip fried. I sent it back. In the interim I toyed with every OSx86 kernel out there without any success. Then today my chip arrived back from Taiwan (desplite the "made in Holland" insignia). I popped it back into the PC and installed OSX again from a retail DVD first time no trouble at all. To be honest, I doubted I'd ever see that chip again, but now I'm feeling better...
| [ 0 trackbacks ] | permalink | related link |
( 3 / 169 )Shadowbox is a great way to play FLV videos in a lightbox, but it is a little shortsighted in terms of the parameters it expects you'll want to pass onto JWPlayer. So here's a little workaround I came up with.
If you want to send a playlist position and repeat variable you need to modify the shadowbox-flv.js file as follows:
var D=["file="+this.obj.content,"height="+F,"width="+I,
"autostart="+C,"displayheight="+K,"showicons="+H,
"backcolor=0x000000","frontcolor=0xCCCCCC","lightcolor=0x557722",
'repeat='+(this.obj.repeat?this.obj.repeat:'false'),
'playlist='+(this.obj.playlist?this.obj.playlist:'none')]
Then you can send the params from your JS code:
Shadowbox.open({
player: 'flv',
title: 'FLV',
content: 'playlist.xml',
height: 400,
width: 550,
autostart: true,
repeat: 'list',
playlist: 'right'
});| [ 0 trackbacks ] | permalink | related link |
( 3 / 165 )I look forward to the day IE6 is wiped from the face of the internet like doomsday cults await their rapturous glory. I have read rumours - and I tend to agree - that IE6 is costing the world economy millions of dollars - I would like to see the figure properly researched, it could be billions!
Slowly we see big sites like YouTube phasing out IE6 support, slowly. Alas, I have client sites with 20% of visitors using IE6 so the end is not yet in sight...
The Tech Crunch story
What Ajaxian had to say about it
Join the list of sites not supporting IE6
| [ 0 trackbacks ] | permalink | related link |
( 3 / 137 )Using Symfony's observe_field() on radio buttons will work only the first time the value is changed. If a user changes the values more than once the onchange event stops firing. To get around this you need to use the onclick event instead (as observer_field() uses onchange - and to support keyboard events you need to add an onkeyup handler as well).
A solution was posted at Symfony nerds in the comments suggesting a hardcoded onclick Ajax call in the radiobutton() method. I suggest using the remote_function() instead to simplify the code, avoid duplication and allow the helper to do the work. After all, that's what helpers are for...
<?php
echo radiobutton_tag(
"tt_product",
1,
true,
'onclick=toolPref(1)'
);
?>
<?php
echo javascript_tag("
function toolPref(val){
".remote_function(array(
'url' => 'user/tallypref',
'with' => "'tally_product='+val"))."
}"
);
?>
| [ 0 trackbacks ] | permalink | related link |
( 3 / 128 )Thought this was worth posting as it could save someone a lot of grief. I had a slave MySQL server stop updating without warning. A SELECT SLAVE STATUS showed there was an error due to an existing record. This would have occurred when a SELECT DATA INFILE populated 50,000 records and things went a little haywaire. The solution was quite simple:
1. STOP SLAVE;
2. TRUNCATE TABLE table_name;
3. START SLAVE;
This wipes out all data in the table with errors and allows the master to rebuild it from scratch. If you have foreign keys you might need to turn them off first...
| [ 0 trackbacks ] | permalink | related link |
( 3 / 113 )Want your web app to send email from your gmail or google apps hosted email account? This way your messages are stored in your sent box. Here's how to do it with a failover to localhost (in case there is a problem with the gmail server or authentication)...
try{
$connection = new Swift_Connection_SMTP(
sfConfig::get("app_gmail_host"),
Swift_Connection_SMTP::PORT_SECURE,
Swift_Connection_SMTP::ENC_TLS
);
$connection->setUsername(sfConfig::get("app_gmail_username"));
$connection->setPassword(sfConfig::get("app_gmail_password"));
$connection->attachAuthenticator(
new Swift_Authenticator_PLAIN()
);
$mailer = new Swift($connection);
$mailer->disconnect();
echo "Using gmail..." . PHP_EOL;
unset($mailer);
} catch (Exception $e) {
echo "Using localhost..." . PHP_EOL;
$connection = new Swift_Connection_NativeMail();
}
//use our working mailer
$mailer = new Swift($connection);
Note you need to disconnect, destroy and recreate your SMTP mailer or else gmail will accuse you of trying to change identity!
| [ 0 trackbacks ] | permalink | related link |
( 3 / 102 )Back Next
Avatar
