\\ELEVEN33\\ | Here is my box of tricks!

Rails.cache + Typhoeus + Hydra

Posted on October 1, 2011 by xrs

I needed to make multiple calls to an api to get music album information. The api only allows for single search term querying. So I could only search for Artist: “Coheed and Cambria” rather than being able to search for multiple artists at one time. The API returned a huge amount of data with the supplied artist and other artists who were similar. So I needed to use the power of Typhoeus+Hydra to do parallel requests so i could search for multiple artists, albums, and tracks on a single page load within Rails. However the responses would sometimes take up to 8 seconds for a return. This was not a good user experience in my opinion. So I wanted to cache the search criteria with ActiveSupport::Cache (Rails.cache) to speed up the return of home page results. Here is how I did it:
#Method to handle response def handle_response(response) if response.is_a? Hash response else if response.code == 200 JSON.parse(response.body) elsif response.code == 404 nil else raise response.body end end end

#Make multiple requests and cache results def many(requests, hash = false) #Create temporary return Array sendback = [] #Instantiate Hydra hydra = Typhoeus::Hydra.new #Set behavior of Hydra Cache Getter hydra.cache_getter do |request| Rails.cache.read(request.cache_key) rescue nil end #Set behavior of Hydra Cache Setter hydra.cache_setter do |request| Rails.cache.write(request.cache_key,self.handle_response(request.response),expires_in: request.cache_timeout) end #Iterate through requests requests.each do |r| #Sets the cache key to the api query #Create the request that will use the cached response if Rails.cache exists #otherwise will query the api request = Typhoeus::Request.new(r,cache_timeout: 1.day) request.on_complete do |response| sendback << self.handle_response(response) end hydra.queue request end hydra.run sendback end
I hope this helps just one other person that is looking for a solution using Typhoeus::Hydra with Rails.cache.

Posted in Programming | Leave a comment

Ruby on rails install with MAMP on OS X

Posted on October 25, 2010 by xrs

Install Readline

First Download and Install Readline
cd $HOME/Downloads $ curl -O ftp://ftp.cwru.edu/pub/bash/readline-6.0.tar.gz $ tar xzvf readline-6.0.tar.gz $ cd readline-6.0 $ ./configure --prefix=/usr/local $ make $ sudo make install

Next get rid of the default Ruby & Rubygems installed on OS X. This may be overkill, but I cant stand when some os x library screws me for a couple of hours.
$ sudo mv /usr/bin/ruby /usr/bin/old_ruby $ sudo mv /usr/bin/gems /usr/bin/old_gems $ sudo mv /usr/bin/rake /usr/bin/old_rake sudo mv /System/Library/Frameworks/Ruby.framework/Versions/Current /System/Library/Frameworks/Ruby.framework/Versions/Current_old

Install Ruby

$ cd $HOME/Downloads $ curl -O ftp://ftp.ruby-lang.org//pub/ruby/1.9/ruby-1.9.2-p0.tar.gz $ tar xzvf ruby-1.9.2-p0.tar.gz $ cd ruby-1.9.2-p0 $ ./configure --prefix=/usr/local --enable-pthread --enable-shared $ make $ sudo make install

Rubygems comes installed with Ruby 1.9.2 so need to run a separate install.

Install Rails

$ sudo gem install rails

Install Mysql for MAMP with libraries

This part is derived from Boondocks.net

Download and Install “MAMP” from the Web.
Start MAMP
Download the “MAMP 1.7.2” src code.
Unzip the source code file.
Open the terminal and go into the source code directory ex:

$ cd $HOME/Downloads/MAMP_1.7.2_src

Untar the mysql file and go into the untarred directory:

$ tar -xzvf mysql-5.0.41.tar.gz $ cd mysql-5.0.41

Compile Mysql libraries for MAMP extensions.

$ ./configure --with-unix-socket-path=/Applications/MAMP/tmp/mysql/mysql.sock --without-server --prefix=/Applications/MAMP/Library $ make -j2

Copy the compiled libraries into MAMP:

$ cp libmysql/.libs/*.dylib /Applications/MAMP/Library/lib/mysql

Copy the MySQL headers into MAMP

$ mkdir /Applications/MAMP/Library/include $ cp -R include /Applications/MAMP/Library/include/mysql

Install the Mysql Ruby GEM.

$ sudo env ARCHFLAGS="-arch i386" gem install mysql -- --with-mysql-config=/Applications/MAMP/Library/bin/mysql_config

I just hope someone finds this useful.

Posted in Programming | 3 Comments

How to set up a Mysql Cluster in Amazon EC2

Posted on February 9, 2010 by xrs

I didn’t write this. I am just syndicating from a site that keeps trying to redirect me through a javascript hack.

This is a quick Howto for setting up MySQL cluster using Amazon EC2 instances. I have used 3 small instances (1 mgmt + 2 data nodes) from RightScaleâ€™s CentOS 5 public image to test things. Iâ€™ve a working cluster using these steps but thereâ€™s no guarantee that these steps works for you as well.

Management server for cluster:
Although we can set up management server for cluster in one of the data node but we should have a third node as if one of the two MySQL cluster nodes fails, and the management server is not running, then the data on the two cluster nodes will become inconsistent and if the same node fails which have management server, then we will not be able to create another node or configure cluster.

Beginning with MySQL 5.1.6, it is possible to store the non-indexed columns of NDB tables on disk, rather than in RAM as with previous versions of MySQL Cluster. Here Iâ€™m using MySQL 5.1.26-rc as its the latest version while writing this article.

Letâ€™s dive in practical:
1. Fireup 3 instances and note down their public and private IPs, dedicate roles(mgmt, node) for them and login in management node. Install ncurses library for MySQL compilation requirement:
# yum install ncurses ncurses-devel

2. Download mysql source from its source download page:
# cd /usr/src
# groupadd mysql
# useradd -g mysql mysql
# wget http://dev.mysql.com/get/Downloads/MySQL-5.1/mysql-5.1.26-rc.tar.gz/from/ftp://mirror.services.wisc.edu/mirrors/mysql/
# tar xzf mysql-5.1.26-rc.tar.gz
# cd mysql-5.1.26-rc

3. Compile and install:
# ./configure –enable-community-features –with-mysqld-user=mysql –with-mysqlmanager –with-plugins=ndbcluster
# make && make install

4. Create required directories:
# mkdir -p /var/lib/mysql/mysql-cluster/backup
# chown -R mysql:mysql /var/lib/mysql
# cd /var/lib/mysql/mysql-cluster
# vim config.ini
â€”
[NDBD DEFAULT]
NoOfReplicas=2
DataMemory=80M # How much memory to allocate for data storage
IndexMemory=18M # How much memory to allocate for index storage
# For DataMemory and IndexMemory, we have used the
# default values. Since the “world” database takes up
# only about 500KB, this should be more than enough for
# this example Cluster setup.
[MYSQLD DEFAULT]
[NDB_MGMD DEFAULT]
[TCP DEFAULT]
# Section for the cluster management node
[NDB_MGMD]
# IP address of the management node (this system)
HostName=10.252.169.x
# Section for the storage nodes
[NDBD]
# IP address of the first storage node
HostName=10.254.237.x
DataDir=/var/lib/mysql/mysql-cluster
BackupDataDir=/var/lib/mysql/mysql-cluster/backup
DataMemory=2048M
[NDBD]
# IP address of the second storage node
HostName=10.251.74.x
DataDir=/var/lib/mysql/mysql-cluster
BackupDataDir=/var/lib/mysql/mysql-cluster/backup
DataMemory=2048M

# one [MYSQLD] per storage node
[MYSQLD]
[MYSQLD]
â€”
Save and close the file.

5. Start management server. You can also copy ndb_mgmd in system path for easy access.
# /usr/src/mysql-5.1.26-rc/storage/ndb/src/mgmsrv/ndb_mgmd -f /var/lib/mysql/mysql-cluster/config.ini

***
Setup of Node1 and Node2:

1. These settings are identical for Node 1 and Node 2. Letâ€™s start by adding mysql user and group:
# groupadd mysql
# useradd -g mysql mysql

2. Install required ncurses library for MySQL compilation:
# yum install ncurses ncurses-devel

3. Download, compile and install MySQL:
# cd /usr/src
# wget http://dev.mysql.com/get/Downloads/MySQL-5.1/mysql-5.1.26-rc.tar.gz/from/ftp://mirror.services.wisc.edu/mirrors/mysql/
# tar xzf mysql-5.1.26-rc.tar.gz
# cd mysql-5.1.26-rc
# ./configure –enable-community-features –with-mysqld-user=mysql –with-mysqlmanager –with-plugins=ndbcluster
# make && make install

4. Install initial mysql db:
# /usr/src/mysql-5.1.26-rc/scripts/mysql_install_db –user=mysql

5. Create directories:
# mkdir -p /var/lib/mysql/mysql-cluster/backup
# chown -R mysql:mysql /var/lib/mysql

6. Copy init script from mysql source to preferred location:
# cp /usr/src/mysql-5.1.26-rc/support-files/mysql.server /etc/init.d/
# chmod 755 /etc/init.d/mysql.server

7. Create config file for MySQL:
# vim /etc/my.cnf
â€”
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1
ndbcluster
# IP address of the cluster management node
ndb-connectstring=10.253.191.210
default-storage-engine=NDBCLUSTER
#Those are for future tuning
#max_connections=341
#query_cache_size=16M
#thread_concurrency = 4
[client]
port=3306
socket=/var/lib/mysql/mysql.sock
[mysql.server]
user=mysql
basedir=/usr/local
[mysql_cluster]
# IP address of the cluster management node
ndb-connectstring=10.253.191.x
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
â€”
Save and close the file.

8. You can also include MySQL directories in your path (Optional).
# PATH=$PATH:/usr/local/mysql/bin:/usr/local/mysql/libexec
# export PATH

9. Start MySQL Server now:
# /etc/init.d/mysql.server start

10. Repeat these steps in node 2 and start MySQL on both nodes. MySQL should start without any complain. You should also update the password for root user.

Test the cluster:

1. Focus again on Management node to have a look of Cluster connectivity. Open the ndb cluster management console. You should add this directory in PATH or can copy files to your system bin. (Iâ€™m a bit lazy
# /usr/src/mysql-5.1.26-rc/storage/ndb/src/mgmclient/ndb_mgm
– NDB Cluster — Management Client –

2. Issue command to view status of cluster:
ndb_mgm> show;
Connected to Management Server at: localhost:1186
Cluster Configuration
———————
[ndbd(NDB)] 2 node(s)
id=2 @10.254.237.x (Version: 5.1.26, Nodegroup: 0, Master)
id=3 @10.251.74.x (Version: 5.1.26, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @10.252.169.x (Version: 5.1.26)

[mysqld(API)] 2 node(s)
id=4 @10.254.237.x (Version: 5.1.26)
id=5 @10.251.74.x (Version: 5.1.26)
ndb_mgm> quit

Everything should run fine.

3. Jump on the node 1 (or 2, doesnt matter in a cluster and try to create db, table and issue commands. BE SURE to use ENGINE=NDBCLUSTER while table creation else cluster will not work, means data will not be replicated throughtout cluster. Insert some records and check that on other node.

Conclusion:
Itâ€™s a basic test cluster and I would not reccommend it for any production use. In fact, as per one of the MySQL experts, â€œI would advise you to consider if MySQL Cluster is really a right choice for your project, itâ€™s often not, probably in 99.9% of the cases I encountered, and does not give expected resultsâ€ and when asking the alternate, â€œusually some MySQL/InnoDB in Master/Master, Master/Slave setup can work well enough to handle very high rate of transactions.â€

Of course, ther are several considerations you have to take care, like network connectivity between instanes, backup to S3 and automatic promotion of second master in case first one failed. Iâ€™m looking on these and as well as on to setup Master/Master replication with S3 backup. Iâ€™ll post that here, once done.

Posted in Linux, Programming | 2 Comments

Ruby + Thin + Camping : Serving static files with Thin vs. Camping’s work around.

Posted on March 31, 2009 by xrs

I have been working with camping for a while and I didnt like how it required me to write methods to handle my static files. Stumbling across some things I read about using “Thin” and “Rack” I was able to serve static files through thin server versus using the camping framework. The file I am using has a .ru extension rather than a .rb extension because it is a rack config file.

%w[ rubygems camping active_record camping/db camping/session reststop json active_support net/http rack ].each { |x| require x}

Camping.goes :Operator

#So after the Camping Instantiation I use this little piece of code to let thin handle the static files.
use Rack::Static, :urls => ["/static", "/images", "/css", "/javascripts"], :root => File.expand_path(File.dirname(__FILE__))

#All my Operator Code goes in here#

#Here us the end of my file. To start the Thin + Camping application.

Operator::Models::Base.establish_connection :adapter => ‘sqlite3′, :database => ‘operator.db’
Operator::Models::Base.logger = Logger.new(‘operator.log’)
run Rack::Adapter::Camping.new(Operator)

now type: thin start -r ./myfile.ru

I hope this helps all who like to go camping but take the “Thin” path to get to where they are going.

Posted in Programming | Leave a comment

Ruby + Soap4r : Complex headers

Posted on March 31, 2009 by xrs

I was was using Soap4r to talk to a Enterprise Messaging Bus and was able to come up with this to pass complex headers that were required by the Bus. So I am happy with my outcome. However this begs the question. “Why are we still using SOAP?”

require ‘soap/header/simplehandler’

n = XSD::NS.new
n.assign(“soap”, “”)

class HeaderHandler < SOAP::Header::SimpleHandler
attr_accessor :value

def initialize(ns,tag)
super(XSD::QName.new(ns, tag))
@value = value
end

def on_simple_outbound
@value
end
End

h =
HeaderHandler.new(“http://service.asurion.com/schemas/MessageHeader/1.0.0“,
“MessageHeader”)
driver.headerhandler << h
h.value = {XSD::QName.new(“http://example.com/schemas/MessageHeader/1.0.0“,
“Context”) => {XSD::QName.new(“http://example.com/schemas/
exampleCanonicalModel/1.0.0″, “Culture”) => “en-US” ,
XSD::QName.new(“http://example.com/schemas/ exampleCanonicalModel/1.0.0″,
“TimeStamp”) => “2009-01-09T17:44:32.6441748Z”, XSD::QName.new(“http://
example.com/schemas/exampleCanonicalModel/1.0.0″, “Principal”) =>
{XSD::QName.new(“http://example.com/schemas/exampleCanonicalModel/1.0.0“,”Sy
stemUsername”) => “clientname”,
XSD::QName.new(“http://example.com/schemas/exampleCanonicalModel/1.0.0“,”Sys
temPassword”) => 55555555},
XSD::QName.new(“http://example.com/schemas/exampleCanonicalModel/1.0.0“,
“LineOfBusiness”) => “Handset”,
XSD::QName.new(“http://example.com/schemas/exampleCanonicalModel/1.0.0“,
“Application”) => “Aggregator”,
XSD::QName.new(“http://example.com/schemas/exampleCanonicalModel/1.0.0“,
“Client”) => “Google”,
XSD::QName.new(“http://example.com/schemas/exampleCanonicalModel/1.0.0“,
“Channel”) => “InStore”}}

###################################### Output of header from wiredump

<soap:Header>
<n1:MessageHeader
xmlns:n1=”http://example.com/schemas/MessageHeader/1.0.0”
soap:mustUnderstand=”0″>
<n1:Context
xmlns:n2=”http://example.com/schemas/exampleCanonicalModel/1.0.0“>
<n2:LineOfBusiness>Handset</n2:LineOfBusiness>
<n2:Client>Google</n2:Client>
<n2:Channel>InStore</n2:Channel>
<n2:Application>Aggregator</n2:Application>
<n2:Principal>
<n2:SystemPassword>55555555</n2:SystemPassword>
<n2:SystemUsername>clientname</n2:SystemUsername>
</n2:Principal>
<n2:TimeStamp>2009-01-09T17:44:32.6441748Z</n2:TimeStamp>
<n2:Culture>en-US</n2:Culture>
</n1:Context>
</n1:MessageHeader>
</soap:Header>

Posted in Programming | Leave a comment

Python + SOAPpy module

Posted on July 14, 2008 by xrs

I have been tinkering around with Python lately and have had a little trouble here and there. I first had a problem with installing the psycopg2 adapter for Postgres integration. come to find out that installing packages from macports is not always the best thing. So a simple “sudo port uninstall psycopg2” and then compile from binary fixed the problem.

Now I am wresteling with the SOAPpy install. It keeps giving me the error when building ”

Traceback (most recent call last):
File “setup.py”, line 8, in <module>
from SOAPpy.version import __version__
File “/Users/xrs/Downloads/SOAPpy-0.12.0/SOAPpy/__init__.py”, line 5, in <module>
from ClientÂ Â Â Â Â import *
File “/Users/xrs/Downloads/SOAPpy-0.12.0/SOAPpy/Client.py”, line 46
from __future__ import nested_scopes
SyntaxError: from __future__ imports must occur at the beginning of the file

“. So after some reading and banging my head on the keyboard i found this handy article.

http://www.python.org/dev/peps/pep-0236/

Upon reading I found that in Python 2.5 wants all nested_scopes to be called at the beginning of the file and that only certain statements can be ahead of the import statement. So “yeah”, this is something a experienced Python programmer would have figured out in a couple of minutes. Oh well, just like with any other language you have to do some walking around in circles before you realize the end to the loop is to read the manual.

Posted in Programming | 1 Comment

Flex Builder + Ubuntu

Posted on June 24, 2008 by xrs

So recently Adobe released its Flex Builder for Linux. After learning this the first thing that popped in my head is awesome I can figure out how to unsuccessfully install something after 10 tries again. So after fiddleing with Flex Builder for about 30 minutes I got it installed. Heres what I had to do.

First check your system for the necessities needed forÂ the Flex Builder Eclipse Plugin to work

Linux Header (linux-headers-2.6.24-19-generic)

sudo apt-get install linux-headers-2.6.24-16-generic

Eclipse Classic 3.3.2

wget http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/R-3.3.2-200802211800/eclipse-SDK-3.3.2-linux-gtk.tar.gz

tar xzvf eclipse-SDK-3.3.2-linux-gtk.tar.gz

this should give you a directory called eclipse. I moved mine to my home directory.

mv ./eclipse /home/<username>/

(this will come in handy during the installation process.)

Java JRE 1.5

I had to use 1.5 otherwise Flex Builder would not install

–minimum install

sudo apt-get install sun-java5-jre

– what I did

sudo apt-get install sun-java5-jre sun-java5-plugin sun-java5-bin sun-java5-fonts

after installing java you have to tell Ubuntu what version of Java you want to use

sudo update-alternatives –config java

There are 4 alternatives which provide `java’.

SelectionÂ Â Â Alternative
———————————————–
1Â Â Â /usr/bin/gij-4.2
+Â Â Â Â Â Â Â 2Â Â Â /usr/lib/jvm/java-gcj/jre/bin/java
3Â Â Â /usr/lib/jvm/java-6-sun/jre/bin/java
*Â Â Â Â Â Â Â Â 4Â Â Â /usr/lib/jvm/java-1.5.0-sun/jre/bin/java

Press enter to keep the default[*], or type selection number:

select java-1.5.0, in my case #4

Next Download the flex builder.

wget http://download.macromedia.com/pub/labs/flex/flexbuilder_linux/flexbuilder_linux_install_a3_033108.bin

The INSTALL file will tell you to make the bin file executable (chmod +x) but I didn’t feel like it. So I just did the following

sh ./flexbuilder_linux_install_a3_033108.bin

and so the adobe installer begins.

The installer will ask you where you want to put “Adobe_Flex_Builder_Linux” directory

/home/<username>/Adobe_Flex_Builder_Linux

ehhhh why not.

Next the installer will ask you to select you current eclipse directory. (remember from earlier?)

/home/<username>/eclipse

after hitting next it should install the flex builder without any errors. If you do get errors I would sugget “sudo su -” then running the .bin file again. Pay attention to the paths of the Adobe_Flex_Builder_Linux directory creation.

Well I hope this helps those out there who love Ubuntu, but sometimes get exhausted with always having to configure 5 other things before they can use the program they want.

Posted in Linux, Programming | 3 Comments