Scott Ru gives us some insights on the process Amazon uses to handle user reviews on their site.
You start with some philosophical rules, and you try to make them stick. Providing guidelines is the only way to start.
Archive for the 'tech' Category
Via Kvardek-du through Planet Lisp
CNet reports that Tim O’Reilly keynoted a message to the Web 2.0 community that they should stop building SuperPoke and try to challenge more real problems.
The CNet article jabs at O’Reilly Media itself helped to spawn some of these SuperPoke applications with all of these tech conferences that they have held but overall O’Reilly’s main message in the keynote resonates with me:
And you have to conclude, if you look at the focus of a lot of what you call ‘Web 2.0,’ the relentless focus on advertising-based consumer models, lightweight applications, we may be living in somewhat of a bubble, and I’m not talking about an investment bubble. (It’s) a reality bubble
So recently I saw this when doing a svn checkout of a project and ran into the following:
$ svn co http://svn.somewhere.com/svn/projects
svn: Unrecognized format for the relative external URL ”.
Wonderful. This indicated to me there was a problem with the svn externals somewhere. After noodling a little I decided to Google around and found this. Basically, duplicate listings in your svn:externals is a bad thing.
Seems that some cloud vendors (sheesh I really only knew about Google Ape… err App Engine, Amazon’s EC2 service, and GoGrid) have been having some issues watching customer data go up in a poof. oops.
Datacenter Knowledge mentions Flexiscale having issues
The problems for FlexiScale began when one of the main storage volumes was accidentally deleted by an employee during a system upgrade earlier this week.
as well as another company called LinkUp (although I guess it’s down now) having issues too:
a cloud storage previously known as Media Max, which shut down Aug. 8 after losing “an unspecified amount of customer data
Ouch. As the industry fashionistas and the hordes of zombies (and I guess everyone else pulled in from the vortex created) moves towards trying to fulfill what Nicholas Carr dubs the Big Switch I can only surmise that more cases of cloud computing companies will continue to go belly up due to dumb mistakes, poor execution, etc, name your classic dumb mistake here. This will leave behind the vendors who have spent time at multiple levels (besides just technology) to ensure that many of the typical mistakes that will hit any IT business can be recovered from instead of being the critical wound that kils.
Read more!
There are many multiple file rename utilities that you can dig up.
One that I am used to on Debian/Ubuntu-based distros is called rename which is one that is derived from the Perl Cookbook. However, note that this rename script does not seem to exist on other UNIX variants in a packaged format. It definitely does not seem to exist on a Fedora Core 9 box I use nor does it exist on a FreeBSD machine.
While it would not be too hard to swipe the perl script and copy it in place onto a system. I hate having to remember moving that script around and around with me from place to place so I looked around for other equivalents and found one called mmv. One nice thing is that there are packages for at least FreeBSD, FC9, and Debian/Ubuntu variants so it should be quite lazy to install compared to rename.
Below I give an example of using it to do a multiple renaming where I want to append the value 1 to the end of a bunch of filenames.
$ ls cat_100 cat_15 cat_200 cat_50 cat_500
$ mmv cat"*" cat#1_1 $ ls cat_100_1 cat_15_1 cat_200_1 cat_500_1 cat_50_1
Doug Hellman’s svnbackup script tool is a really handy tool for setting up automated backups for a subversion repository.
However, the non-fun time comes when one wants to restore a subversion repository that has way too many dumpfiles parts. The instructions for restoration are basically ‘roll your own’ if you want to try to automate the restore procedure. What would be nicer is if there was the converse tool that made it easier to not have to figure out how to re-order the backup files in the proper manner in order to perform the restore.
I spent a few minutes thinking about it and wrote a small Ruby script to help with this that I call svnbackup-restore.rb. Here is the (hastily written) source code. (Download here)
#!/usr/bin/ruby
# Program Name: Restore the restore
# Purpose: Take all the svn dumpfiles generated from svnbackup
# sort them and try loading them via svnadmin
# Usage: 1. Create the new repo path with svnadmin create
# 2. Set repo_name to the repo backup file names
# 3. Set restore_path to the new path to restore to
# 4. Run it ./svnbackup-restore.rb
# Assumptions: svnrestore-backup.rb is in the same dir as the svn dump files
# CHANGE PARAMETERS HERE
repo_name = 'myrepo'
restore_path = '/path/to/myrepo'
# Print out debugging?
DEBUG = true
# DON'T CHANGE BELOW
# Filename format is 'dumpfile---.bzip2'
svn_dumpfiles = Dir["dumpfile-#{repo_name}*.bzip2"]
sorted_files = svn_dumpfiles.sort do |a, b|
left_rev = a.split(’-')[2].to_i
right_rev = b.split(’-')[2].to_i
left_rev right_rev
end
sorted_files.each do |dump_part|
results = `bzcat #{dump_part} | svnadmin load #{restore_path}; echo $?`
puts results if DEBUG
res = results.split.last.to_i
if res != 0
puts “Error on trying to load up #{dump_part}!”
exit 1
end
end
Lightning talks are one of the more interesting parts of a conference in my opinion since 5 minutes really forces the speaker to get to the point and it becomes painfully obvious if the presentation has no focus or if there has been real work to get across the main message in the shortest amount of time.
So here are some highlights
Kuwata on Java to Ruby
- No, not the book Java to Ruby
- Going from Java to Ruby requires a change in mindset. Not a change of code
- Some historical example: COBOL in Java (ed. That sounds frightening) is like Java in Ruby
- Don’t hold back experts and too many things spend too much time on beginners
- Do not keep beginners as beginners (teach them!)
- Bragging about the largeness of a project size is the wrong type of bragging (suggested large code + many devs == lack of ability
dRuby & Security by Nishiyama
- druby is not built by default to handle the wild Internet
- There is a feature called insecure method list that will help prevent certain methods from being invocated remotely
- Use $SAFE however it won’t save against a DoS. Also don’t forget about rlimit
Ruby + ODE by Sasaki
- Showed a very nice 3D demo walkthrough world controlled with a Wii-mote and looked like the Pitagora Switch world
- Can summon objects pressing one of the buttons
- Can stop time
- The project should be on Code Repos
- (ed. Seeing this in action was far more interesting than reading these notes)
- Conclusion first: Sorry they don’t
- Showed some very nice examples of using the Enumerator library for many bad cases (each_slice)
- Beginning Ruby programmers really love each to the point of going overboard
Folk programming with Ruby by mootoh
- Folk Programming was introduced at YAPC Asia 2008 (See my notes)
- Show examples outside of building web applications such as Rich UI exploration
- Showed examples of Plugins to other programs since it’s easier than writing a big app (although with Ruby the apps should be nice and small)
- Some examples: Safari + Hatena bookmark, Quicksilver + Twitter (looks dangerous), Quartz Composer + Gainer, Vim + Refe
- Ruby is a very good glue
Again as a Rubyist… toRuby by Ikezawa
- (ed. I liked this talk a lot since it wasn’t by some uber-Ruby hacker)
- Didn’t use Ruby until 2000
- Background was as a consultant since 1984. Helped found a Wapro Kissaten!
- Moving to Ruby hit the OOP barrier. Was not used to OOP methodologies at all
- Stopped however after a long time in June 2007, found a local Ruby guru to help him out and that got the ball rolling again
- Invites others to join in
Ruby 1.9 with Rails 2.1 by matsuda
- Went through the history of web programming (or his version of it)
- Ancient History - PHP
- Recent History - DB Framework with ORM (in Java… lots of these frameworks)
- Current - Rails
- However DB access is a Rails weakness
- Introduced named_scope feature (Is this a Rails 2.1 specific feature? Need to review this myself
- More info on this at blog.dio.jp
Read code with Testing by Endoh
- Rookie Ruby Committed
- His suggestion on learning is by reading real code
- Use TBCR (Test Based Code Reading)
- Run make test-all
- Find code paths that are never executed
- Add tests (requires more code reading)
- Using this method have increased Ruby’s test coverage to 85%
- In contrast Python is 80%, Perl 63%, PHP is 51% (plans to target PHP next)
Okay this is my notes from Matz’s Keynote. I was 5 minutes late since finding lunch took a really long time to find anything around the area unfortunately. Unfortunately, most of the visiting Rubyists decided to do KFC however Charles Nutter decided to stick it through and we finally found a nice Yakiniku restaurant to eat at but that ended up being the reason I was 5 minutes late… okay anyways here are my notes from in the middle of the talk…
Matz was talking about sanctuaries and how some technologies have built their sanctuaries over time. Here are some sanctuaries he mentioned and some of their defining characteristics:
- UNIX
- The filter (wahoo!)
- “Worse is better” aka the New Jersey School of Design (I always ask myself just how much worse though..)
- Convenience over perfection
- Smalltalk
- OOP, deep OOP
- Targeted at children (funny how only greybeards really use this language now discounting eToys)
- Bytecode VM but not the first to have one but one of the more prominent ones
- A dynamic language implementation with decades of hard-earned experience, wisdom and knowledge around it
- java
- Well Java seems to fulfill business and ‘enterprise needs’ (for now)
- Java was originally slow however time has changed that perception. Java’s speed complaints have mostly faded away (But I still complain about a 30 second startup time for the VM)
- Absorbed many other ideas from other languages (VM, Garbage Collection, Exception Handling)
- Java has one of the fastest Garbage Collected VMs now (with the amount of engineering effort thrown at it I’d hope so)
- Hackers gathered
- New technology is born
- The world changed (because of the technology? Missed this…)
- For Rubyists, feeling matters. Focus on the human and the joys programming
- It inherited many things from the past. Lisp metaprogramming, Smalltalk OOP, UNIX text processing
- Productivity matters. Machines are faster but people’s time is much more expensive
- Agility matters. Environments change and embracing the changes is the right thing
Matz then describes that Sanctuaries tend to go through the following phases
Don’t forget about centripetal forces during these stages. The community matters quite a bit. According to Matz, 50% of the types of OSes out there in this world are some type of UNIX.
Finally, he goes on to describe the Ruby Sanctuary and its defining characteristics
At this point a person from Rakuten comes up on stage and talks about some projects that Matz has been collaborating with.
One is called ROMA which is some sort of distributed memory data storage (think memcached with some more ability for data integrity in case of failures)
The other one is called fairy which is supposed to be a lightweight distributed programming framework.
Unfortunately it seems that both projects are rather delayed and it doesn’t seem apparent if these projects will be Open Sourced or not. That is rather disappointing to me but can’t have everything for free now can we?
Evan Phoenix on Rubinius
Evan started off with a quick intro on Rubinius and proceeded to talk about some of the big pieces of Rubinius from a 10,000 feet high view.
- The kernel of the system
- The C Compatibility Layer
- It’s a big project and Evan wants to have a conversation with anyone who wants to have one on it
For the VM nerds here are some feature highlights…
- Accurate generational Garbage Collector
- Bytecode based VM
- Capable of bootstrapping itself
Evan suggested we should think of Rubinius sort of like an OS.
Then he started diving right into a tour of some of the cool internals of Rubinius and explaining some of the internal objects that really are the heart of Rubinius (above the VM layer)
BTW what is the <<? Evan coins it the left chevron.
With MethodContext and CompiledMethod it’s possible to implement eval in Ruby and follow the principle of Code as Data. Taking advantage of Ryan Davis’s ParseTree (included in Rubinius) one can take something like
"1 + 1".to_sexp
Which will take 1 + 1 and change it into an s-expression, convert it into an AST, and a User Visitor Pattern implementation can walk through this and spit out bytecode. (I might have recollected this wrong.. if so, sorry.. Not a VM implementor)
Some other big features that Evan mentioned were
- Extensions
- Getting close to running Ruby C-extensions
- However they are still 2nd-class citizens (they will take a performance hit but working is better than fast and broken)
- MRI in this case is still faster
- Multi-VM implementation
- stdin and stdout are implemented as pipes
- Functional but still highly experimental (can try calculating Pi across a multi-core machine)
- args = ["-e", "puts", "hello"]
vm = Rubinius::Vm.spawn args
puts vm.stdout.gets - Channels
- One of the main internal communication channels used in Rubinius
- I/O events use channels for example
Then Evan goes on to showing some cool demo of implementing Binding.of_caller in front of the audience. He also showed one neat thing about rbx. If you run a command it will load irb automatically assuming you want a REPL (cool!)
Q & A
- Q: What are your final goals with Rubinius?
- A: Have an image like SmallTalk? The VM only knows about byte codes not Ruby. Could even write an implementation of SmallTalk on the Rubinius VM however Evan said he has no plans to
- Q: Anything on shared benchmark testing?
- A: (Matz) There has been movements to start collecting code as of last week. Trying to make sure that it is more ‘real world’ code benchmarks. There should be an Infoq article on this.
(Evan) Started on this. Check out GitHub for more info - Q: How’s the performance?
- A: No really hard numbers on Rubinius’ performance. Some things have gotten faster over time such as the generational part. Awhile ago Evan wrote a bunch of small benchmarks to test things out but seems to have misplaced them somewhere
Here’s a continuation of my scribbled notes from the RubyKaigi. This one from the JRuby implementors. Some of my own thoughts and comments are inserted in parentheses.
Charles Nutter on JRuby
- Quick intro and the impressive live demos of JRuby in action (or as the IRC channel said Nice Live Coding (NLC))
- This summer, work on Ruby 1.9 features (seems there’s an option flag to turn on 1.9 compatibility mode already)
- Current 1.1.3 Progress Report
- A new interpreter that is up to 30% faster
- Compile-time performance enhancements
- Many Rails Bottlenecks fixed
- No new release of JRuby this year but…
- Wanted to show some uses of JRuby
- IDEs (NetBeans, Eclipse, INtelliJ use JRuby in their IDEs
- Swing GUI development
- Swing development can be very complex but Ruby helps to simplify it
- Benefits from the ‘Write once, Run everywhere’
- There are a plethora of options to choose from (why does this seem to happen in the Java world all the time?)
- Cheri
- Profligacy (by the infamous Rails Ghetto-man Zed Shaw)
- MonkeyBars (Proceeded to show an impressive demo)
- MonkeyBars leverages existing GUI utils (that’s a major plus)
- MVC-like structure (hey looks like Rails…)
- Graphics (Showed some really sweet demos of ruby-processing in action)
- Processing had a way to interface to audio input devices
- Remember to check out “A Face for Stephen Hawking”. Wow that code looked really clean…
- Rails (the benchmark app that all implementors need to measure implementation readiness by)
- There has been a shift in Java Web frameworks lately (but still happening at a glacial pace)
- Deployment problem is “solved” (I keep hearing this…)
- mongrel is “old school” (at least for JRuby-based deployments)
- Made mention of Phusion/Passenger but it still had the issue of rails process spawning and how to manage that (Seems Jruby doesn’t have these issues)
- Some war deployment demoes
- A warbler demo (A tool to help create war files)
- Will package in Jruby automagically into the war (That’s very nice)
- Just run warble inside the rails application directory (Hopefully it doesn’t package up a gigantic development.log)
- As simple as GLASSFISH_HOME/bin/asadmin deploy *.war
- Has Rack supoprt (can handle Merb and anything else with Rack support
- Can be configurable
- From his demo, it seems that warble will make a war of anything (aka be careful what directory you run this command)
- Showed some really nice benchmarks of rails and JRuby (I need to put this through the ringer when I get more time)
- Showed another deployment tool called the glassfish gem
- 3MB gem that had everything needed for JRuby on Rails
- Tries to match the agile process
- JRuby Users (in production): CSI’s Disease Surveillance, Oracle’s Mix website, Sun’s Media Cast, Thoughtworks Mingle
Q & A
- Q: The Parser Implementation seems hard. Any war stories
- A: I hate parsers and refuse to write one. Someone wrote something that used J. I don’t know much about it but Tom’s pains dealing with it seem very hard. No plans to change parsers anytime in the near future.
- Q: Object Space & Binding
- A: On Jruby, this will incur an automatic performance penalty. binding & caller will be required each time which incurs a heavy cost
- Q: Non-local break performance?
- A: Implemented as a Java Exception. By doing this, the JVM can optimize that as a Jump. JRuby tries to use this as much as possible behind the scenes for performance
Ruby Kaigi 2008 is happening. I’m sure there other posts on the Ruby Kaigi happening however here are some of my own hasty scribbles for anyone that cares…
Introduction Speech
- This is the 3rd year of the conference
- One theme of this year is multiple implementatios
- More people coming into the Ruby community and we need to greet the (Dave Thomas’ speech from last year)
- Two tracks this year (I mostly followed the main track)
The first set of presentations from the main track were from the Ruby implementors.
Koichi SASADA - Ruby VM Development
- 1.9.0-2 released as of 6/20/2008 (mainly bug fixes)
- A little discussion of 1.9.1 Roadmap (Something about 1.9.1 being the more stable release?)
- A plethora of interpreters available now (lists all of them)
- Sasada-san felt a little sheepish over Matz mentioning taking only about technology is ‘boring’ since his talk focused on mainly technology
- University of Tokyo has become a haven for since Sasada-san now has a laboratory (wahoo!) which means…
- Ruby related research-projects
- Student Research Projects on Ruby
- Parallel Thread Execution
- Better multi-core support
- Memory resource usage (or was that resource contention?) can be a problem
- Multiple-VMs
- Run Multiple VMs in same Ruby process
- Jruby + Nakada-san are the primary drivers
- The JRuby guys are ready to implement it (waiting on API) and Rubinius seems to have one already
- API done
- bootstrap done
- Creation of interpreter mostly done (seems to have some small issues?)
- Still needs documentation (What project doesn’t?)
- Ruby implementations needs FP optimization for HPC
- Ruby 1.9.x quite fast compared to other implementations (according to the Benchmarks Sasada-san showed)
- Create an Optimal interpreter for “you”
- Make it easier for Ruby to be used in embedded environments
- Make it easier for Ruby customized for specific environments (iPhone anyone?)
- Make it possible to plug in and out the core pieces
- Byte code serialization and embedded
- Make the GC customizeable
Before there was Rails, there existed other Web Frameworks for Ruby. One of the promising ones is Nitro however it fell into realm of ignorance. On some random surfing I ran across this blog post from one of the main authors of the Nitro Web Framework.
It’s an interesting read since the author laments how Nitro could have been a contender but it never took off. The comments are enlightening since it seems there were project (mis)management issues that prevented a community from really forming around Nitro. Anyways, by now, Rails has the lion’s share of the attention so any hopes now would rest on it being so much better that people will switch. My understanding is that it has some things that are better but not such a big enough jump to make people interested, unfortunately.
I looked over mod_rails today and realized how nice and easy looking it was to install.
The general install process is:
1. Install rubygems
2. Run gem install passenger
3. passenger-install-apache2-module
4. Follow the instructions and install any missing dependencies (it actually is smart enough to detect which ones you need before splatting itself into the filesystem)
5. Add some lines into the apache conf somewhere, declare a virtual host and point DocumentRoot to the public folder of the desired Rails app
By piggypacking onto the gems installation infrastructure they made it really simple to install even without support from native package managers (although my hope is that it will be as easy as apt-get install modrails one day).
However, always the skeptic I decided to give it a quick and dirty test run and lo’ and behold it really is as easy as advertised and looks like it’s working okay.
Impressive stuff. Now I just need to read a little more on how it actually works (seems to hand off to some sort of backend process that is intelligent enough to spawn a set number of backend rails processes as needed and kills any that are idle off or continue serving if things are busy). So for those of you looking to deploy rails within an apache setup, this might be a nice and simple way (assuming you have access to manage the apache server process) to integrate rails within Apache.
I’m still unclear how many rails apps can be multiplexed on a realistic basis with one Apache installation with this method but a little bit of trial and error should give some ideas.
18
Jun
Happy half-a-decade blog
Yeesh, I just looked at the date of my first post and it’s been a whole 5 years since I started blogging here.
What have I learned? People who blog multiple times a day must either have some kick-ass tools to do it quickly or way too much time. I seem to have neither so this blog gets infrequently updated.
Recent Comments