Tuesday, September 14, 2010

Parsing HTML with Cthulhu

You can't parse [X]HTML with regex. Because HTML can't be parsed by regex. Regex is not a tool that can be used to correctly parse HTML. As I have answered in HTML-and-regex questions here so many times before, the use of regex will not allow you to consume HTML. Regular expressions are a tool that is insufficiently sophisticated to understand the constructs employed by HTML. HTML is not a regular language and hence cannot be parsed by regular expressions. Regex queries are not equipped to break down HTML into its meaningful parts. so many times but it is not getting to me. Even enhanced irregular regular expressions as used by Perl are not up to the task of parsing HTML. You will never make me crack. HTML is a language of sufficient complexity that it cannot be parsed by regular expressions. Even Jon Skeet cannot parse HTML using regular expressions. Every time you attempt to parse HTML with regular expressions, the unholy child weeps the blood of virgins, and Russian hackers pwn your webapp. Parsing HTML with regex summons tainted souls into the realm of the living. HTML and regex go together like love, marriage, and ritual infanticide. The <center> cannot hold it is too late. The force of regex and HTML together in the same conceptual space will destroy your mind like so much watery putty. If you parse HTML with regex you are giving in to Them and their blasphemous ways which doom us all to inhuman toil for the One whose Name cannot be expressed in the Basic Multilingual Plane, he comes. HTML-plus-regexp will liquify the n​erves of the sentient whilst you observe, your psyche withering in the onslaught of horror. Rege̿̔̉x-based HTML parsers are the cancer that is killing StackOverflow it is too late it is too late we cannot be saved the trangession of a chi͡ld ensures regex will consume all living tissue (except for HTML which it cannot, as previously prophesied)dear lord help us how can anyone survive this scourge using regex to parse HTML has doomed humanity to an eternity of dread torture and security holes using regex as a tool to process HTML establishes a breach between this world and the dread realm of c͒ͪo͛ͫrrupt entities (like SGML entities, butmore corrupt) a mere glimpse of the world of reg​ex parsers for HTML will ins​tantly transport a programmer's consciousness into a world of ceaseless screaming, he comes, the pestilent slithy regex-infection wil​l devour your HT​ML parser, application and existence for all time like Visual Basic only worse he comes he comes do not fi​ght he com̡e̶s, ̕h̵i​s un̨ho͞ly radiańcé destro҉ying all enli̍̈́̂̈́ghtenment, HTML tags lea͠ki̧n͘g fr̶ǫm ̡yo​͟ur eye͢s̸ ̛l̕ik͏e liq​uid pain, the song of re̸gular exp​ression parsing will exti​nguish the voices of mor​tal man from the sp​here I can see it can you see ̲͚̖͔̙î̩́t̲͎̩̱͔́̋̀ it is beautiful t​he final snuffing of the lie​s of Man ALL IS LOŚ͖̩͇̗̪̏̈́T ALL I​S LOST the pon̷y he comes he c̶̮omes he comes theich​or permeates all MY FACE MY FACE ᵒh god no NO NOO̼O​O NΘ stop the an​*̶͑̾̾​̅ͫ͏̙̤g͇̫͛͆̾ͫ̑͆l͖͉̗̩̳̟̍ͫͥͨe̠̅s ͎a̧͈͖r̽̾̈́͒͑e n​ot rè̑ͧ̌aͨl̘̝̙̃ͤ͂̾̆ ZA̡͊͠͝LGΌ ISͮ̂҉̯͈͕̹̘̱ TO͇̹̺ͅƝ̴ȳ̳ TH̘Ë͖́̉ ͠P̯͍̭O̚​N̐Y̡ H̸̡̪̯ͨ͊̽̅̾̎Ȩ̬̩̾͛ͪ̈́̀́͘ ̶̧̨̱̹̭̯ͧ̾ͬC̷̙̲̝͖ͭ̏ͥͮ͟Oͮ͏̮̪̝͍M̲̖͊̒ͪͩͬ̚̚͜Ȇ̴̟̟͙̞ͩ͌͝S̨̥̫͎̭ͯ̿̔̀ͅ

Want your own psychopathic crazy text? http://www.textozor.com/ serves all your smudged and blotted needs.

Thursday, August 26, 2010

Malware and Virus Scans

So, a lot of my associates are Windows users out of necessity.

Got an email this morning:
"Our pc's fans started goin off again on full blast. We already did the virus detection and surprisngly it found viruses and deleted them. But today it started on full blast again. Any ideas?"

... Stop letting people use Internet Explorer? ;)

There's some .PDF, Flash, and JAVA vulnerabilities going around now; make sure you patch up your Adobe Reader and Flash Player to current, Update your JAVA to 6u22 or higher, run TFC.exe and then run a malware bytes scan.

After MBAM finishes cleaning up; go into the Windows Scheduled Tasks in control panel and remove any suspicious jobs (Lot of the nasty ones are untitled and look like a GUID like

Then fire up process explorer, Options Menu -> Verify Image Signatures, View -> Select Columns and make sure "Verified Signer" is checkboxed, View -> Show Process Tree, click the - by wininit.exe to hide system services and kill off anything that doesn't have an entry in the Company Name field. (Kill *Any* Unsigned EXEs, even stuff from Logitech mouse drivers and Realtek audio controls)

In fact, you can usually prune every user process off except for the root Explorer.exe process.
If you're careful, you can even kill off most services other than svchost.exe and
anything with "(Verified) Microsoft Windows" in the Verified Signer column.

Then fire up autoruns.exe and remove any suspicious startup entries. Heck, if you can, remove everything but the nvidia/ATI driver autoruns.

Now run TFC.exe once more to make sure all the tempdirs are emptied.

Alternatively, you can run an offline scan with a bootcd like AVIRA's Rescue CD -- the ISOs are updated weeklyish.


AVAST's also a good option.

Your best choice is to get ChromeChromium, or Firefox and load them with the Adblock and PDF Download extension that prompts you to download PDFs -- just click cancel if the prompt comes up unless you intentionally clicked a PDF link yourself.

Myself, I use Chromium and Chrome Adblock. Chromium has an internal PDF renderer that doesn't rely on Adobe, and it's sandboxed by default. You can enable it from chrome://plugins/ which also will warn you if other plugins are not up to date.

Still, it pays to be careful.
If you'd like to see how sophisticated some of these attacks are, check out this series of articles:

Drupal on nginx

Just got an email from someone asking for my drupal config template, Ask and ye shall receive:

# Search and replace subdomain.fqdn.com with your domain info.
server {
listen 80;
server_name subdomain.fqdn.com;
access_log /var/www/sites/subdomain.fqdn.com/logs/access.log;
error_log /var/www/sites/subdomain.fqdn.com/logs/error.log;

#location / {
# root /var/www;
# index index.php;
# error_page 404 = @drupal;
#location @drupal {
# rewrite ^(.*)$ /index.php?q=$1 last;

# Set site_root
root /var/www/sites/subdomain.fqdn.com/public/;
index index.php index.html;

# Set doc_root (Drupal CleanURLs)
location / {
if (!-e $request_filename) {
rewrite ^/(.*)$ /index.php?q=$1 last;

# hide protected files
location ~* \.(engine|inc|info|install|module|profile|po|sh|.*sql|theme|tpl(\.php)?|xtmpl)$|^(code-style\.pl|Entries.*|Repository|Root|Tag|Template)$ {
deny all;

# serve static files directly
location ~* ^.+\.(jpg|jpeg|gif|css|png|js|ico)$ {
rewrite ^/favicon.ico$ /sites/default/themes/mytheme/favicon.ico break;
access_log off;
expires 30d;

# Install Imagecache module, update the location, enable this directive
location ^~ /sites/default/files/imagecache/ {
index index.php index.html;
# assume a clean URL is requested, and rewrite to index.php
if (!-e $request_filename) {
rewrite ^/(.*)$ /index.php?q=$1 last;

location ^~ /sites/default/files/downloads/ {
index index.php index.html;
autoindex on;

# FastCGI Via Socket
location ~ \.php$ {
fastcgi_pass unix:/tmp/php-fastcgi.socket; #;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME /var/www/sites/subdomain.fqdn.com/public$fastcgi_script_name;
include fastcgi_params;

# Proxy Via Socket or TCP
#location ~ \.php$ {
# proxy_pass unix:/tmp/php-fastcgi.socket; #;
# proxy_connect_timeout 15;
# proxy_redirect default;
# proxy_set_header Host $host;
# proxy_set_header X-Real-IP $remote_addr;
# proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# #proxy_set_header X-Forwarded-Proto https;
# }

### Redirect www to root domain?
#server {
# listen 80;
# server_name www.subdomain.fqdn.com;
# rewrite ^/(.*) http://subdomain.fqdn.com/$1 permanent;
# }

### Redirect root domain to www?
#server {
# listen 80;
# server_name subdomain.fqdn.com;
# rewrite ^/(.*) http://www.subdomain.fqdn.com/$1 permanent;
# }

### HTTPS server
#server {
# listen 443;
# server_name subdomain.fqdn.com;
# ssl on;
# ssl_certificate /var/www/sites/subdomain.fqdn.com/private/cert.pem;
# ssl_certificate_key /var/www/sites/subdomain.fqdn.com/private/cert.key;
# ssl_session_timeout 5m;
# ssl_protocols SSLv2 SSLv3 TLSv1;
# ssl_ciphers ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
# ssl_prefer_server_ciphers on;
# access_log /var/www/sites/subdomain.fqdn.com/logs/secure.access.log;
# error_log /var/www/sites/subdomain.fqdn.com/logs/secure.error.log;
# location / {
# root /var/www/sites/subdomain.fqdn.com/public/;
# index index.php index.html;
# }
# location ~ \.php$ {
# fastcgi_pass unix:/tmp/php-fastcgi.socket; #;
# fastcgi_index index.php;
# fastcgi_param SCRIPT_FILENAME /var/www/sites/subdomain.fqdn.com/public$fastcgi_script_name;
# include fastcgi_params;
# }
# }

Wednesday, August 25, 2010

Sometimes you must make the best of what you have available...

Super Armatrons rock in so many ways.

Every robot that you ever heard of normally employs some sort of individual servo mechanism for powering each joint of the robot. Not so with Armatron; There are no electronics inside at all, in fact the toy is entirely powered by a single motor. All of the toy's functions are controlled by a complex array of mechanical transmissions and gear trains. It's powered by 2 D batteries, directly connected to a power switch and a dinky brushed DC motor that looks to run around 300 RPM; it's very slow. Some clever folks have replaced the motor with a steam engine.

The Box, and the Manual (For posterity)

Yes, you are supposed to keep score yourself. :)

This toy has so many moving parts it almost defies verbal explanation. Shown above is an overview of the base mechanisms. Each of the two joysticks can be moved up/down, left/right and clockwise/counter-clockwise for a total of six movements of the arm. Each of the six movements is bi-directional. Four of the six movements are bi-directional with two speeds in each direction. The joystick linkages (orange parts) control six transmissions (black and white parts). All the transmissions are powered by a single motor on the right. The gearing above the motor is for a timing mechanism described below. The gearing and shafting behind the transmissions connect the transmissions to a series of ring gears described below.

There are three sections to the transmission system: (1) a row of six black forward gears, (2) a row of six black reverse gears and (3) a row of six white indexing drums. The picture above shows the lower row of forward gears without the other two sections of the transmission. The picture also shows how the forward gears drive the output shafting.

The picture above shows how the upper row of six reverse gears mesh with the lower row of six forward gears.

The picture above shows three of the six indexing drums. The red arrow indicates an inner gear fixed to the main drive shaft. One of the outer black gears is for high-speed operation. The other outer black gear is for low-speed operation. Friction between the gears causes the entire drum to rotate about the shaft unless the drum's position is held by one of five indexing tabs. Green arrows indicate four of the five tabs distributed around the drum. The tabs determine which of the two gears on the drum engages the forward or reverse gear. The combinations are slow-forward, fast-forward, slow-reverse and fast-reverse. The fifth index is neutral where no gears are engaged. Note how the left drum is thinner and has only one outer gear. This corresponds to the claw movements, which have only one speed in forward and reverse.

The images above show how the up/down movement of the right joystick controls the indexing of the rightmost drum. This corresponds to the up/down motion of Armatron's shoulder joint.

This picture shows the gear covers back in place. The arm assembly fits into the round ring gear socket. Armatron can continuously rotate through a full 360 degrees. This is made possible by the stack of ring gears. Each movement is transmitted from the base to the arm via a ring gear. The orange and back barrel is a timer mechanism that cuts power to the arm after the allotted time has expired.

Here are all the ring gears layered back in the socket.

This shot shows the underside of the arm assembly that fits into the socket of ring gears.

Here's some drawings of the mechanism ripped from the patents.

Thursday, August 19, 2010

Catching up with the configuration curve

Okay, so, I discovered some new stuff.

First: nginx can be convinced to do close-to-automatic deployments.

sudo tee /etc/nginx/conf.d/autoconfig.conf <<-\EOA
    client_max_body_size 5M;
    client_body_buffer_size 128k;
    proxy_connect_timeout 90;
    proxy_send_timeout 180;
    proxy_read_timeout 180;
    proxy_buffer_size 16k;
    proxy_buffers 8 16k;
    proxy_busy_buffers_size 32k;
    proxy_intercept_errors on;
    fastcgi_connect_timeout 60;
    fastcgi_send_timeout 180;
    fastcgi_read_timeout 180;
    fastcgi_buffer_size 128k;
    fastcgi_buffers 4 256k;
    fastcgi_busy_buffers_size 256k;
    fastcgi_temp_file_write_size 256k;
    fastcgi_intercept_errors on;
    include /var/www/sites/*/config/*;

Now when you drop a new site into /var/www/sites your configuration will be picked up after an 'reload nginx'! Just rename the config folder to config-disabled and reload 'nginx' to disable a site.

Second: PostgreSQL!

I'm sick of mySQL, Sun, and Oracle. Screw 'em all, time to jump ship to Postgres.
Yay for Drizzle and MariaDB; keep up the good work and don't let The Man (Oracle) keep you down. Maybe I'll come back from the pg-side, maybe not.

apt-get install postgresql php5-pgsql phppgadmin

This is a little more involved to setup than mysql, due to the lack of dbconfig-common support requiring manual setup.

nano /usr/share/phppgadmin/conf/config.inc.php

Now find and change the following line
$conf['extra_login_security'] = true;
$conf['extra_login_security'] = false;

and now all userids can login.

Now, postgresql itself comes in a locked down config out of the box.

sudo su postgres -c psql template1

ALTER USER postgres WITH PASSWORD 'securePassword!';

sudo passwd -d postgres
sudo su postgres -c passwd

nano /etc/postgresql/8.4/main/postgresql.conf

Change the line:
#listen_addresses = 'localhost'
listen_addresses = '*'
and also change the line:
#password_encryption = on
password_encryption = on

nano /etc/postgresql/8.4/main/pg_hba.conf

Set up something like this: (IPv6 is used by default for some reason.)

# IPv6 local connections:
host    all         redmine_user         ::1/128               trust
host    all         all         ::1/128               md5

Easy way to do this is to highlight the ipv6 line, hit Ctrl-K, the up arrow, Ctrl-U, then change the 'all' username to 'redmine_user'.

service postgresql-8.4 restart

And you should be set.

mkdir -p /var/www/sites/phppgadmin/config

sudo tee /var/www/sites/phppgadmin/config/phppgadmin.conf <<-\EOA
server {
        listen 80 default;
        server_name localhost;
        access_log /var/www/sites/phppgadmin/logs/phppgadmin.access.log;
        error_log /var/www/sites/phppgadmin/logs/phppgadmin.error.log;

        location / {
            root /usr/share/phppgadmin;
            index index.php;

        location ~ \.php$ {
            include /etc/nginx/fastcgi_params;
            fastcgi_pass unix:/tmp/php-fastcgi.socket;
            fastcgi_index index.php;
            fastcgi_param SCRIPT_FILENAME /usr/share/phppgadmin$fastcgi_script_name;

rm /etc/nginx/sites-enabled/default
reload nginx

And viola, postgresql!

Wednesday, August 4, 2010

Weaving Redmine, RVM, Ruby, Rails, Rack, Thin, and nginx as a reverse proxy

Yesterday, it came down from On High.

Bossman say 'We need a trouble ticket system.' and Kami say "yes massa!"

A quick trawl of wikipedia's list later, and Redmine's the winner.
Time Tracking, Revision control integration, and full RBAC + lockdown, and all written in my favorite language, Ruby.

So, we'll be starting from where my previous post in the series left off.

You should have Lucid + nginx + php5-fastcgi + phpmyadmin set up already. (Honestly, you can skip the PHP stuff if you're not interested; I keep it around for Drupal and phpmyadmin.)

First: Get some required packages.

apt-get install curl patch git-core subversion autoconf build-essential bison libreadline6 libreadline-dev zlib1g zlib1g-dev libssl-dev libsqlite3-0 libsqlite3-dev sqlite3 libxml2-dev libmysqlclient-dev libpq-dev

Second, install rvm.
bash < <( curl -L http://bit.ly/rvm-install-system-wide )
tee -a ~/.bashrc <<-\EOA
[[ -s '/usr/local/lib/rvm' ]] && source '/usr/local/lib/rvm'
source '/usr/local/lib/rvm'

Disable ruby's documentation generation (Don't need it on rackspace; skip this if you're on a dev box.)
echo "gem: --no-rdoc --no-ri" >> /etc/gemrc

Install ruby 1.8.7.
rvm install 1.8.7

Now set it as the default.
rvm use --default 1.8.7

Get the sqlite3 gem compiled
gem install sqlite3-ruby
IF this fails, you're probably missing libsqlite3-dev

Get the mysql gem compiled
gem install mysql
IF this fails, you're probably missing libmysqlclient-dev

Get the pg gem compiled
gem install pg
IF this fails, you're probably missing libpq-dev

Install rack 1.0.1

gem install rack -v=1.0.1

Install rails 2.3.5

gem install rails -v=2.3.5

If you want the optional gantt chart generation, you'll need rmagick.
apt-get install libmagick9-dev imagemagick
That will pull down a bunch of -dev packages; now build the native extensions.
gem install rmagick

Now we need a better appserver than webrick. Let's go get thin.

gem install thin
Decide on where you'd like the redmine app to live; I like the private directory under the hostname.
mkdir -p /var/www/redmine.domain.com/private/ && cd /var/www/redmine.domain.com/private/

Now, checkout version 1.0's latest revision.

svn co http://redmine.rubyforge.org/svn/branches/1.0-stable redmine

Okay, we've got redmine; now to configure it. First, let's setup the database. Run this SQL in phpmyadmin:

create database redmine_core character set utf8;
create user 'redmine_user'@'localhost' identified by 'my_password';
grant all privileges on redmine_core.* to 'redmine_user'@'localhost';
Or phppgadmin: (Paste each line separately.)

CREATE DATABASE redmine_core WITH ENCODING='UTF8' OWNER=redmine_user;

Edit the config/database.yml and configure the database settings.
cd redmine/config/ && cp database.yml.example database.yml
  adapter: mysql
  database: redmine_core
  host: localhost
  username: redmine_user
  password: my_password
  encoding: utf8

Or for Postgresql:

  adapter: postgresql
  database: redmine_core
  host: localhost
  username: redmine_user
  password: my_password
  encoding: utf8

Now return to the redmine root (cd ..) and generate the session store.
rake generate_session_store

Now, we'll have redmine populate the database.
RAILS_ENV=production rake db:migrate

You'll be prompted to choose a language here; just hit enter for english.
RAILS_ENV=production rake redmine:load_default_data

Wooo! Let's test it.

ruby script/server thin -e production
Now visit port 3000. You should get a nice blank redmine instance.

Now, for configuring nginx for a reverse proxy test:
server {
        listen                  80;
        server_name             redmine.domain.com;
        access_log              /var/www/redmine.domain.com/log/access.log;
        error_log               /var/www/redmine.domain.com/log/error.log;
        root                    /var/www/redmine.domain.com/public/;

        location / {
                try_files       $uri @fallback;

        location @fallback {
                proxy_pass    ;
                proxy_connect_timeout   15;
                proxy_redirect          default;
                proxy_set_header        Host    $host;
                proxy_set_header        X-Real-IP       $remote_addr;
                proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
                #proxy_set_header       X-Forwarded-Proto       https;
Okay, this works!

Next step: Setting up thin clusters on boot.
thin install

Set up the icky old init-scripts:
update-rc.d -f thin defaults

Now we can add YAML config files in /etc/thin/ !
pid: tmp/pids/thin-redmine.pid
group: rvm
wait: 30
timeout: 30
log: log/thin.log
max_conns: 1024
require: []

environment: production
max_persistent_conns: 512
servers: 2
daemonize: true
user: www-data
socket: /tmp/thin-redmine.sock
chdir: /var/www/redmine.domain.com/private/redmine 

Okay, let's start thin.
service thin start

And fix up our nginx proxy config:
upstream thin_redmine {
   server   unix:/tmp/thin-redmine.0.sock;
   server   unix:/tmp/thin-redmine.1.sock;

server {
        listen                  80;
        server_name             redmine.domain.com;
        access_log              /var/www/redmine.domain.com/log/access.log;
        error_log               /var/www/redmine.domain.com/log/error.log;
        root                    /var/www/redmine.domain.com/public/;

        location / {
                try_files       $uri @fallback;

        location @fallback {
                proxy_pass              http://thin_redmine;
                proxy_connect_timeout   15;
                proxy_redirect          default;
                proxy_set_header        Host    $host;
                proxy_set_header        X-Real-IP       $remote_addr;
                proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
                #proxy_set_header       X-Forwarded-Proto       https;

Enable your configuration and restart nginx.
service nginx reload


Here's some links:

Monday, July 19, 2010

gPXE and the HTTP server that could

eBox 1.4 has support for serving a bootfile over TFTP.

Only it's broken. Oops.

Here's my quick fix:
sudo nano /etc/inetd.conf

tftp           dgram   udp     wait    root  /usr/sbin/in.tftpd /usr/sbin/in.tftpd -s /var/lib/tftpboot

That /var/lib/tftpboot is where tftp will serve files from.
eBox expects it to be serving a file such as:

tftp           dgram   udp     wait    root  /usr/sbin/in.tftpd /usr/sbin/in.tftpd -s /var/lib/ebox/conf/dhcp/eth0

Now when you 'upload' a new boot file to eBox's dashboard,
gets replaced with whatever you've uploaded.

But eBox won't set the dhcp-option 'filename' to serve it.

sudo nano /usr/share/ebox/stubs/dhcp/subnet.mas

Look for
% if(defined($info{'nextServer'})) {
                next-server <% $info{'nextServer'} %>;
% }
% if(defined($info{'filename'})) {
                filename "<% $info{'filename'} %>";
% }

And change it to

% if(defined($info{'nextServer'})) {
                next-server <% $info{'nextServer'} %>;
                # Filename entry added by Kamilion (dec 01 2009)
                filename firmware;
% }
% if(defined($info{'filename'})) {
                filename "<% $info{'filename'} %>";
% }

Now we need something to boot.

Go pick up the latest gPXE from here:

Click Customize.

Change the following Settings:



And paste in the following Embedded Script: (Good base, but edit if you wish)
echo "Greetings! Hit Ctrl-C to bail out."
sleep 5
echo "Going to DHCP on primary network adapter"
ifopen net0
dhcp net0
echo "Going to try http://netboot/default.gpxe"
chain http://netboot/default.gpxe
echo "Didn't work, we're still here. Falling back to http://boot/default.gpxe"
chain http://boot/default.gpxe
echo "Didn't work, we're still here. Falling back to BKO"
set 209:string pxelinux.cfg/default
set 210:string http://boot.kernel.org/bko/
echo "Here we go, off to boot.kernel.org!"
chain http://boot.kernel.org/bko/pxelinux.0
echo "Didn't work, we're still here. No Internet connection? Falling back to next BIOS Boot device"

You should get a single .pxe file back after clicking Get Image.

Go to DHCP -> Interface -> Advanced Options -> Thin Client.
Next server: eBox
File Name [browse]
File path in next server:

Click "Change" to complete the settings, then Save Changes.

Place this file in the root of your HTTP server, named default.gpxe, and create a DNS alias to that machine named 'netboot'.

chain http://netboot/boot/menu.gpxe

Here's an example you can use to load Parted Magic:

kernel -n img http://bigblock/boot/pmagic/4.5/bzImage load_ramdisk=1 prompt_ramdisk=0 keymap=us loglevel=0 rw sleep=4
initrd http://bigblock/boot/pmagic/4.5/initramfs
boot img

Here's an example you can use to boot from iSCSI.

#dhcp net0
set keep-san 1
sanboot iscsi:

More examples here: http://boot.sllabs.com/boot/

Friday, July 16, 2010

How to use u3-tool in Lucid

As my few blog visitors may be aware, I've been using the U3 customizer for windows for a long time on my 4GB cruzers to make them bootable.


I came across the u3-tool sometime ago when I bought my Clarion MiND, but I've only actually used it recently.

The U3 Customizer tools were released somewhere in 2005 and won't work in anything but windows XP 32bit, and won't recognize U3s over 8GB.

Gonzor discovered TwinMOS's application version supported Larger drives, Vista, and was released in 2007. However, the copy from TwinMOS didn't work for me -- and yet Gonzor's copy from mediafire did, both are version

I just bought an open-box 16GB Contour EXtreme with AES from newegg for $40, and it came yesterday.

So I started searching on how to hack a 16GB U3 drive.

"Oh, right, u3-tool... Almost forgot about that!"

In searching for 'how to use u3-tool', I ran across ubuntu launchpad bug report #534070 and played around a little.

I had some issues trying to get the windows version to work, I could resize the CD domain, but 'burning' the ISO failed consistently at 4-5% with a scsi error.

In Ubuntu Lucid, you can just 'sudo apt-get install u3-tool' but:
I was *NOT* able to get /dev/sg* or /dev/sr* to work -- I had to address the disk device itself as /dev/sdf to get it to work.

Hope this helps others out, as I think the major problem people are having is trying to use one of the /dev/sg like the u3-tool help text mentions.

Here's the log of the CD domain resize and burn I ran.

kamilion@SonyRA840G:~$ sudo u3-tool -i /dev/sdf
Total device size:   14.95 GB (16051601408 bytes)
CD size:             7.69 MB (8060928 bytes)
Data partition size: 14.94 GB (16043474944 bytes)
kamilion@SonyRA840G:~$ sudo u3-tool -l UbuntuLucid3264.iso /dev/sdf
CD image(1874288640 byte) is to big for current cd partition(8060928 byte), please repartition device.
kamilion@SonyRA840G:~$ sudo u3-tool -p 1874288640 /dev/sdf

WARNING: Loading a new cd image causes the whole device to be whiped. This INCLUDES
 the data partition.

Are you sure you want to continue? [yn] y
kamilion@SonyRA840G:~$ sudo u3-tool -l UbuntuLucid3264.iso /dev/sdf
|**************************************************| 100%

kamilion@SonyRA840G:~$ sudo u3-tool -i /dev/sdf
Total device size:   14.95 GB (16051601408 bytes)
CD size:             1.75 GB (1874329600 bytes)
Data partition size: 13.20 GB (14177271808 bytes)

After I changed the ISO, I also had to open Disk Utility (palimpsest), "Format the Disk"  to create a new MBR geometry, and then create a new NTFS partition.
(Which I've subsequently copied Windows 7's bootmgr to the root of the flash partition and ran "bootsect /nt60 U:", then copied the contents of the 7 install cd to the flash)
Both sections of the device are now bootable, the CD boots ubuntu lucid 32/64 TORAM=Yes, and the flash boots the 7 preinstallation environment.

(Yeah, I know the picture's screwed up -- I'll edit the blogspot CSS later.)

So that all worked just fine for me. Hope it does for you, too!

(And for the person who emailed me to ask about the background... It's from SSDD.)

Wednesday, June 9, 2010

Fun with nginx, upstart, and lucid

Howdy boys and girls!

Today we'll be mangling us some web.

I've started off by installing a fresh copy of Ubuntu Lucid Server in Virtualbox 3.2.4. Mind, if you're using a Linux host, turn *ON* host caching in the "SATA Controller" if your host's using EXT4 as the filesystem your VDIs are stored on, otherwise it will try to use AIO and corrupt your VDIs. The host kernel needs a patch to fix this, but neither karmic or lucid have it, apparently. Anyway.

The Core System

For my first act, I've shuttled over my SSH public keys so I can use my agent to login.

ssh kamilion@laptop # Say no or ^C to make empty ~/.ssh dir with proper perms!
scp kamilion@laptop:/home/kamilion/.ssh/authorized_keys ~/.ssh/authorized_keys
tee -a /etc/ssh/sshd_config <<-\EOA

# No passwords! Get bent, crackers!
PasswordAuthentication no

For my second act, I've tossed a couple of the more useful utilities on. (python-software-properties gives you "add-apt-repository ppa:freenx-team" shortcuts)

sudo apt-get install python-software-properties dnsutils openssh-server denyhosts screen htop rsync nethogs sqlite3

Make *sure* you add your REAL origin IPs to /etc/hosts.allow

tee -a /etc/hosts.allow <<-\EOA

AND edit /etc/denyhosts.conf to enable blocklist sync.

nano /etc/denyhosts

/etc/init.d/denyhosts restart

This will sync the ssh blocklists from the denyhosts server.
Uncomment SYNC_UPLOAD too, please contribute your stats!

(Yes, I know this is pointless after shutting off password auth, but BETTER SAFE THAN SORRY.)

Virtual Machines

If you're using a VM, this might be handy:

sudo apt-get install build-essential dkms
sudo rm /etc/init/tty[2-6].conf  #Disable other TTYs because this is a VM.

The virtualbox guest additions will use dkms to build the guest kernel modules automatically.

You can select the Install Guest Additions from the vbox menu to insert the ISO into the cdrom.


The Webserver

Now we'll install nginx.

sudo apt-get install nginx

And set it up to startup on boot with upstart.
The prestart tests the config; handy!

sudo tee /etc/init/nginx.conf <<-\EOA
# /etc/init/nginx.conf
# nginx - starts the nginx webserver

description "nginx"

start on (net-device-up and local-filesystems)
stop on runlevel [016]

pre-start exec /usr/sbin/nginx -t

expect fork
exec /usr/sbin/nginx

Now you should be able to

start nginx

and check port 80 for the default "Welcome to nginx!" response.

Right -- so this is served out of /var/www/ by default and controlled from /etc/nginx/ in such a manner that you dump config stanzas into a file in /etc/nginx/sites-available/ and link them as such:

sudo ln -s /etc/nginx/sites-available/fqdn.com /etc/nginx/sites-enabled/fqdn.com && sudo service nginx reload

you should get a working vhost with a proper config.

So, we're now capable of serving static pages, what else can we do?

Let's check which modules and defaults our nginx was compiled with:

nginx -V

nginx version: nginx/0.7.65
TLS SNI support enabled
configure arguments: --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/body --http-proxy-temp-path=/var/lib/nginx/proxy --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-debug --with-http_stub_status_module --with-http_flv_module --with-http_ssl_module --with-http_dav_module --with-http_gzip_static_module --with-http_realip_module --with-mail --with-mail_ssl_module --with-ipv6 --add-module=/build/buildd/nginx-0.7.65/modules/nginx-upstream-fair

Looks like we've got fastcgi, DAV, FLV streaming, gzip and mail-proxy by default.

Installing PHP5

Now for PHP5.

sudo apt-get install php5-cgi php5-mysql php5-pgsql php5-sqlite php5-suhosin php5-imap php5-mcrypt php5-gd # php5-gd is usually needed

# security disclosure risk: shut php up -- hide version!
sudo sed -i '/expose_php/s/\;exp/exp/;/expose_php/s/=.*/= 0/' /etc/php5/cgi/php.ini
grep 'expose_php' /etc/php5/cgi/php.ini

sudo tee /etc/init/php-fastcgi.conf <<-\EOA
# /etc/init/php-fastcgi.conf
# php-fastcgi - starts php-cgi as an external FASTCGI process

description "php-fastcgi - respawning UNIX Socket"

start on (net-device-up and local-filesystems)
stop on runlevel [!2345]

expect fork
exec /usr/bin/sudo -u www-data PHP_FCGI_CHILDREN=5 PHP_FCGI_MAX_REQUESTS=125 /usr/bin/php-cgi -q -b /tmp/php-fastcgi.socket

Okay, now to start it.

start php-fastcgi

PHP5 FastCGI for nginx

Now you'll need an nginx config.

Here's one of my templates:

# Enable with
# sudo ln -s /etc/nginx/sites-available/fqdn.com /etc/nginx/sites-enabled/fqdn.com && sudo service nginx reload

server {
        listen                  80;
        server_name             fqdn.com www.fqdn.com;
        access_log              /var/www/fqdn.com/log/access.log;
        error_log               /var/www/fqdn.com/log/error.log;

        location / {
                root            /var/www/fqdn.com/public/;
                index           index.php index.html;

        location ~ \.php$ {
                fastcgi_pass    unix:/tmp/php-fastcgi.socket;
                fastcgi_index   index.php;
                fastcgi_param   SCRIPT_FILENAME /var/www/fqdn.com/public$fastcgi_script_name;
                include         fastcgi_params;

### Redirect www to root domain?
#server {
#       listen                  80;
#       server_name             www.fqdn.com;
#       rewrite                 ^/(.*) http://fqdn.com/$1 permanent;
#       }

### Redirect root domain to www?
#server {
#       listen                  80;
#       server_name             fqdn.com;
#       rewrite                 ^/(.*) http://www.fqdn.com/$1 permanent;
#       }

### HTTPS server
#server {
#       listen                  443;
#       server_name             fqdn.com;
#       ssl                     on;
#       ssl_certificate         /var/www/fqdn.com/private/cert.pem;
#       ssl_certificate_key     /var/www/fqdn.com/private/cert.key;
#       ssl_session_timeout     5m;
#       ssl_protocols           SSLv2 SSLv3 TLSv1;
#       ssl_ciphers             ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
#       ssl_prefer_server_ciphers       on;
#       access_log              /var/www/fqdn.com/log/secure.access.log;
#       error_log               /var/www/fqdn.com/log/secure.error.log;
#       location / {
#               root            /var/www/fqdn.com/public/;
#               index           index.php index.html;
#               }
#       location ~ \.php$ {
#               fastcgi_pass    unix:/tmp/php-fastcgi.socket;
#               fastcgi_index   index.php;
#               fastcgi_param   SCRIPT_FILENAME /var/www/fqdn.com/public$fastcgi_script_name;
#               include         fastcgi_params;
#               }
#       }

The Database

Okay, let's move on and install a real database. Doesn't really matter which you choose. I'm gonna go with mysql for now so we can use phpmyadmin later.

sudo apt-get install mysql-server

Enter your new mysql root password twice.

Now for phpmyadmin.

sudo apt-get install phpmyadmin

It'll ask which server you want to configure it for -- leave both apache2 and lighttpd disabled and continue. Yes, we want to use dbconfig-common. Enter your mysql root password for the first one; then just hit enter to have phpmyadmin generate it's own account and random password.

sudo tee /etc/nginx/sites-available/phpmyadmin <<-\EOA
# Enable with
# sudo ln -s /etc/nginx/sites-available/phpmyadmin /etc/nginx/sites-enabled/phpmyadmin && sudo service nginx reload
server {
        listen 80 default;
        server_name localhost;
        access_log /var/www/apps/logs/phpmyadmin.access.log;
        error_log /var/www/apps/logs/phpmyadmin.error.log;

        location / {
            root /usr/share/phpmyadmin;
            index index.php;

        location ~ \.php$ {
            include /etc/nginx/fastcgi_params;
            fastcgi_pass unix:/tmp/php-fastcgi.socket; #;
            fastcgi_index index.php;
            fastcgi_param SCRIPT_FILENAME /usr/share/phpmyadmin$fastcgi_script_name;
        location /nginx_status {
            # copied from http://blog.kovyrin.net/2006/04/29/monitoring-nginx-with-rrdtool/
            stub_status on;
            access_log   off;
            allow; # Home
            allow; # Work
            deny all;


Make some empty logdirs or nginx will go splat when it can't access them.

mkdir -p /var/www/apps/logs/
chown -R www-data.www-data /var/www/apps/logs/

Kill the pesky default site now that we have phpmyadmin. (yes, "reload nginx" is a command. Nifty, huh? The magic of Upstart!)

ln -s /etc/nginx/sites-available/phpmyadmin /etc/nginx/sites-enabled/phpmyadmin
rm  /etc/nginx/sites-enabled/default
reload nginx

You should now be able to access phpmyadmin and login with root/mysqlpass.
phpmyadmin also can synchronize two mysql instances now, as well.

Additional Information and Tricks

SSH tunnels are your friend! Sync DBs over ssh with phpmyadmin!
ssh -vg -L 8080:localhost:80 -R 33306:mysql-server.home:3306 user@fqdn.com

firefox http://localhost:8080/server_synchronize.php

replace mysql-server.home with whatever hostname/ip has the mysqld instance you wish to sync with. You can pick any port instead of 33306, just make sure you tell phpmyadmin to look at "localhost" on that port.

You can clone whole servers too!
rsync -azvv -e ssh /var/www/ root@fqdn.com:/var/www/
rsync -azvv -e ssh /etc/nginx/ root@fqdn.com:/etc/nginx/

rsync -azvv -e ssh root@fqdn.com:/etc/nginx/ /etc/nginx/
rsync -azvv -e ssh root@fqdn.com:/var/www/ /var/www/

Want a remote mini-desktop? This'll take a couple minutes.
add-apt-repository ppa:freenx-team
apt-get install freenx-server firefox gnome-core synaptic

Boom, grab the nxclient.

Now, go forth and serve pages!