Lots of stuff and my random thoughts

Welcome to Andy Brown's personal website - thebmwz3.co.uk
Sunday, May 19 2013

Welcome

GeneralWelcome everyone,
I've finally got round to writing up a blog, so I'm going to generally post information and snippets on here, mainly because my memory is so bad, that I need a reference site for things I've thought of, etc.

If you'd like to contribute, or comment, then please do, I'd love to hear from you, especially if you've found any information of any use, or applied/tested anything and the results.


As per normal, everything here is (C) me, and also I'm not liable for any problems trying anything here causes, information here is not the thoughts or opinions of my employer, any groups I'm affiliated with or anybody else in my family, its the random gibberings of an idiot, so please take it as that and check anything twice, at least! tmp

Linksys WRT54GL bricked

General

For years I've used the Linksys WRT54 range of wifi routers, I like these units as they have a decent form-factor, generally just keep on running for years without any attention and two wifi antennas that have screw-terminals on them so can easily add or change the aerial. I've always flashed these routers with different firmware, generally I use DDWRT as this has a new GUI, easy setup and quite a few advanced functions to switch WIFI on and off at certain times, captive portals, firewall rules, etc.

Recently I was going to upgrade my firmware, so did the usual, downloaded from DDWRT and uploaded it, then waited for the reboot, wifi didn't come back. Let it do its longer reboot, still nothing and it just wouldn't come back. I checked on the unit and the power light was constantly flashing, this is a bad sign, this generally means the FLASH/firmware isn't working correctly and its stuck in a boot loop. The recovery options on the Linksys/Broadcom hardware are to try their failsafe IP, this is 192.168.1.1 and initialise during their CFE bootloader. The trick here is to see if you can get a ping from 192.168.1.1 during power on. Unfortunately, this also didn't work for me, so it looked increasingly likely that I'd lost the unit completely, so it would be down to using a JTAG re-flash of the system to recover. This isn't easy! So popped the cover off and located the JTAG port. JTAG is a 12-pin (6x6 pins) port that you can access the Broadcom chipset directly to re-flash and debug it. In my case I was trying to use my FTDI cable FT232RL. I now realise this was STUPID of me! This is simply a low-voltage (3v) TTL to serial/usb converter cable, not a JTAG. So I started to give up hope. Until I realised, on the Linksys board was a SERIAL pin header. I soldered up some pins and connected up at 115200-8-N-1 and powered on. Sure enough I saw the bootloader!

This was a relief, it showed me that the bootloader wasn't damaged, but it just hun as it tried to load the O/S (As expected). It also confirmed that it should have been listening on 192.168.1.1 but the time it waited was tiny, almost 1-2 seconds! But it gave me hope as I reckon I could try to recover it from this point. Here is the screen output I had:

 

CFE version 1.0.37 for BCM947XX (32bit,SP,LE)
Build Date: Tue Jun 20 16:22:41 CST 2006 (root@localhost.localdomain)
Copyright (C) 2000,2001,2002,2003 Broadcom Corporation.
 
Initializing Arena
Initializing Devices.
 
No DPN
et0: Broadcom BCM47xx 10/100 Mbps Ethernet Controller 3.90.37.0
CPU type 0x29008: 200MHz
Total memory: 16384 KBytes
 
Total memory used by CFE:  0x80300000 - 0x803A39C0 (670144)
Initialized Data:          0x803398D0 - 0x8033BFE0 (10000)
BSS Area:                  0x8033BFE0 - 0x8033D9C0 (6624)
Local Heap:                0x8033D9C0 - 0x803A19C0 (409600)
Stack Area:                0x803A19C0 - 0x803A39C0 (8192)
Text (code) segment:       0x80300000 - 0x803398D0 (235728)
Boot area (physical):      0x003A4000 - 0x003E4000
Relocation Factor:         I:00000000 - D:00000000
 
Boot version: v3.7
The boot is CFE
 
mac_init(): Find mac [00:1C:10:36:41:9E] in location 0
Nothing...
 
eou_key_init(): Find key pair in location 0
The eou device id is same
The eou public key is same
The eou private key is same
Device eth0:  hwaddr 00-1C-10-36-41-9E, ipaddr 192.168.1.1, mask 255.255.255.0
        gateway not set, nameserver not set
Loader:raw Filesys:raw Dev:flash0.os File: Options:(null)
Loading: .. 3916 bytes read
Entry at 0x80001000
Closing network.
Starting program at 0x80001000
 
You can see it was giving me its IP and trying to boot. So now to try and interrupt it and get a ping reply. It also occurred to me at this point, previously trying to connect/talk to it I had plugged the cable directly from the router into my laptop. This was problematic as linux was trying to auto-negotiate and DHCP to the interface, which seemed to also introduce a delay, the delays were small but when I'd see the bootloader via serial I realised how time-sensitive the whole approach was. So using a mini-hub I connected the two together and started the ping again. Again I rebooted and found it didn't reply to pings.
The CFE bootloader can be interrupted by a CTRL-C to go interactive (like grub, lilo, etc) so the next trick was to try and interrupt it. Again this was time-sensitive and I found I had to keep hammering away at CTRL-C whilst plugging power into the router. A very tricky thing to do, but keep trying. Probably 10-20 attempts. Finally I got in and it responded:
 
mac_init(): Find mac [00:1C:10:36:41:9E] in location 0
Nothing...

eou_key_init(): Find key pair in location 0
The eou device id is same
The eou public key is same
The eou private key is same
Device eth0:  hwaddr 00-1C-10-36-41-9E, ipaddr 192.168.1.1, mask 255.255.255.0
        gateway not set, nameserver not set
Automatic startup canceled via Ctrl-C
CFE> ^C
CFE> ^C
CFE> help
Available commands:

rndis               Broadcom USB RNDIS utility.
et                  Broadcom Ethernet utility.
modify              Modify flash data.
nvram               NVRAM utility.
reboot              Reboot.
flash               Update a flash memory device
memtest             Test memory.
f                   Fill contents of memory.
e                   Modify contents of memory.
d                   Dump memory.
u                   Disassemble instructions.
autoboot            Automatic system bootstrap.
batch               Load a batch file into memory and execute it
go                  Verify and boot OS image.
boot                Load an executable file into memory and execute it
load                Load an executable file into memory without executing it
save                Save a region of memory to a remote file via TFTP
ping                Ping a remote IP host.
arp                 Display or modify the ARP Table
ifconfig            Configure the Ethernet interface
show devices        Display information about the installed devices.
unsetenv            Delete an environment variable.
printenv            Display the environment variables
setenv              Set an environment variable.
 
As you can see, we can now get to the bootloader commands, whilst this appeared my pings to 192.168.1.1 worked, so I now had a chance to get into it! I immediately tried sending a new firmware via TFTP to the Linksys. Trying to send the firmware over tftp was failing, so I was a little confused, until I read about CFE and how it works, and realised, by interrupting its boot process, it stopped its ability to receive firmware. So I now started to read about the commands, as I wanted to do it all in one go now, no more reboots as who knows if I'd get back into this mode again! Checking the commands I saw the erase and flash options. Checking again at DDWRT it showed that I could erase the existing/faulty flash 'nvram erase' which confirmed back with a 0 (for ok). I then found parameters to the flash command, and this accepted flashing from a TFTP source, so by doing:
flash -ctheader : flash1.trx
I could start the flash, at the same time sending the file via tftp it started to upload:
 
CFE> nvram erase
*** command status = 0
CFE> flash -ctheader : flash1.trx
Reading :: CODE Pattern is CORRECT!
upgrade_ver[v4.20.6] upgrade_ver[42006] 4712_ver[15000]
Done. 3032064 bytes read
fname=flash1.trx 
CODE Pattern is correct! (W54G)
Programming...done. 3032032 bytes written
*** command status = 0
 
At that point I typed 'go' which told it to boot, and sure enough after a few seconds DDWRT kernel boot messages appeared and eventually got the DDWRT root prompt, so I'm back in.
 
Hopefully the above will be handy to anyone in the same situation and thinking their router is toast. Persistance and keep on reading for solutions, good luck!
 

 

Arduino and Sky box innards

General

 Well as its Christmas I've had a bit of time on my hands so thought I'd start on a project a friend of mine, Adam thought of. He's key developer on TVHeadend and liked the idea of replacing the Sky+ box. That's been the drive behind TVHeadend, XBMC, etc. So why not go a step further? Take that old Sky plus hardware/case and re-use it for the XBMC PC with fully functioning front-panel.

I've had a few Sky+ boxes lying in my garage for a while now (Faulty capacitors, or just old hardware so they just sit there), so I thought I'd rip one open and see whats going on. Firstly we know the insides of the Sky+ box,

its 1 main motherboard, a PSU board and hard-drive. Then a ribbon cable feeds the front panel. So HDD comes out, and I pull apart carefully the front panel. Luckily they're made to disassemble, so a couple of screws at the side and the front panel then can unclip with gently prising the locking tabs all round it.

This revealed the front panel innards, and a 2-part circuit board with a lot of circuitry on it.

   

Not what I was expecting, I was thinking more along the lines of either I2C or a simple LED cluster that fed back to the motherboard. I powered up and started to probe the pins to see what I was getting. Unfortunately I don't have an oscilloscope so I couldn't see any data/signalling, so just went on voltage initially. Unfortunately all I found was ground, power and standby-power. Pretty disappointing. I then tried to find out what the LEDs that light up the Sky playback circle (The sequence of LEDs that go round in a circle when playing back video, etc). Unfortunately as I don't have an active SKY card in, even the demo won't display this sequence, so again I've had to give up on this. So I've decided to pull the circuit board out, and do my own thing! Fit a cluster of LEDs in the back and just feed those back to my Arduino board.

The idea is that I'll plug the arduino into the XBMC/Openelec box using USB. I can then write a serial communication language that will send status to the arduino that will then display on the LEDs. On top of that I'm going to cut out part of the front panel of the Sky box and put in a backlit LCD screen, that will show current channel, now/next, something like that (Not fully decided). Last night I got as far as mounting the LEDs behind the facia and started to check they were aligned (tricky!). Next will be to start wiring that to the arduino to see if I can get the 'animation' of the LEDs looking anywhere near OK.

If anyone has done any more interfacing with the Sky+ front panels or figured out the control lines, etc, please let me know! I'd love to get the original circuit board running somehow.

 

 

Borri UPS for home use

General

Around a week ago my APC Smart UPS SC420 decided its battery needed replacing (again!). I only seem to get around 2-3 years out of mine which started me thinking, why on earth do I get such short life from something 'relatively' expensive. These units come in at around £120 and batteries around half that. For 420va that doesn't seem too great, as all I do is power 2 machines (desktops with a few add-ons) and a network switch/wifi. Mainly to protect the computers from nasty spikes and unexpected shutdowns more than running without power (since the internet router is in the house, not connected to a UPS).

However, I wanted to replace the UPS to keep the protection up, mainly as the power in my area seems to spike and brownout quite a bit and I've had a lot of things go pop as a result of this (X10 kit hates this kind of choppy power, hence most of my X10 home control stuff going bang). So I duly went to APC and looked up the replacement battery. It came out at around £50 after postage, etc. So I then looked at the APC upgrade wizard, see what they would offer me for a trade-in, as I particularly like the idea of trading in old hardware to get new for a bit of discount and I feel I've done the environment some good with them having batteries inside, etc. For trade-in I was disappointed. I could get a replacement unit for around £100, thats with the same 420va limitation, which since I run that at around 80-90% load isn't great (and I suspect why I go through batteries so often). So looking at the next up capacity 450va, this was £151 for really very little difference, or the 550va model for around the same price. Not very impressive.

So I started doing my usual answer to everything, google for recommendations or alternative brands, Borri kept coming up as they appear to do all sorts of UPS and generator equipment from SOHO right up to datacentres, etc, and looking quite promising I looked at the B60 range. These started at 800va and went up to 2000va, and shopping around for prices. I was pleasantly surprised, the B60-800va version would be £71 delivered, etc. So thats approximately double the rating for a nice £71.

The next issue before ordering, was it linux compatible? For years I've used APC units and used the ever popular opensource APCUPSD to shut down servers when batteries neared depletion, etc. Firstly Borri provided a download of linux software. An excellent start. Checking the software it wasn't what I'd hoped, it was a java-based server/applet that basically talked to the ups over USB and presented output via a flash-based web-ui on a custom java-tomcat url on your machine. OK, not the end of the world. I did more digging and thought of NUT for linux. This looked to support it, so worth giving it a go.

UPS arrived next day and it's quite a nice looking unit, really clear permanently lit LCD on the front telling you input volts, output volts and current status (Battery level in bars, current load level). I plugged it into my laptop to play around with and get an idea of the java tomcat web interface. It actually was quite neat, just a pity it was flash and didn't seem to have a raw data port/method of extracting data from it otherwise I'd have stuck with it.

Installing NUT and setting it up for the Borri wasn't too bad, it's partially supported according to NUT using the blazer_usb driver, but this doesn't give you battery level/capacity and time to depletion. No real issue, I can survive!

I've yet to add my load of machines to it and see how it copes, but so far its looking pretty good. If I figure out the battery level/capacity part of it I'll be very happy with this solution.

 

TVHeadend

 Quite a while since I did a post, and things in the TVHeadend and XBMC world have moved on quite a bit. TVHeadend is now being actively developed, primarily resolving the EPG issues that were around in the past, but then also moving onto loads more improvements. Therefore things have moved to a development environment, so if you want to try out the latest you can on github.com/tvheadend. Or via the original www.lonelycoder.com site where you can get debian packages and official releases. Hope you like all the updates!

A few people have also asked me about the dev stuff I've done, I've got a few testing areas for TVHeadend at my own github (http://github.com/andyb2000/) so take a look, and drop me a note if you want to try anything out as I'm always looking for new testers, or join the TVHeadend lot on IRC freenode.net #hts

 

XBMC, TVHeadend and watching live TV

 As some of you may have followed in earlier posts I've been replacing our old Sky satellite setup with my own setup here at home. Well over the past month or so its improved considerably so I thought I'd write up where I'm up to with the whole setup and how the family are getting along with it.

Firstly the problem, Sky is expensive and to be honest the majority are repeats and most of the channels we frequently watch are FTA (Free-to-air/satellite) in the UK, so dropping Sky was logical. So I then had the Sky dish and the problem of how to feed it into the house setup. Firstly the front-end. I've used XBMC since the early days on the original XBOX thanks to Bunnie Huang and his amazing books, etc. (Google it!). So I'm happy with XBMC, so setting that up on a small footprint PC with quiet fans and HDMI out. I bought a cheap Nvidia card with HDMI out (Cost around £35), as these cards can do hardware decoding, this is important as it takes the load off the PC's CPU and very important when your machine isn't the most powefull on the planet. So with that in the PC (It's a slim-line Dell so fitting the Nvidia was a pain, the case doesn't quite close properly, but its hidden out of sight so not a problem). The HDD in the XBMC machine isn't too important, big enough to hold the XBMC install is enough as you won't really be storing anything on it. Save your disks for the 2nd bit of hardware! Networking is also important, make sure its cabled up right and if possible use Gigabit network cards+switches as it'll speed the whole system up (Although for HD tv you can get away with 100Mbps as HD tv streams are 35Mbps (at worst) dropping down to 6Mbps.

Install XBMC: Simply download their Live CD XBMCUbuntu, burn that and run it. There is the option to install, choose that and wait whilst it flattens your HDD, partitions, installs, etc. Now a word of caution. Make sure your graphics card, hardware, etc, are all plugged in at install, as the installer does a really good job of installing the right components if it finds the right hardare. It'll save you a world of pain later! So now you should be booting XBMC from HDD and getting things looking reasonable. You can tweak this as you need. Insert a MS media centre remote for remote control/IR functions (Pretty much any will do, I found the older silver ones work better than the newer branded editions (HP badged, etc, I found didnt work reliably). There should be little else you need to do.

To get TV functions into XBMC you need the latest packages, Pulse-Eight provide these in an easy script, SSH to your XBMC machine, then run these commands:

       cd /tmp; wget http://packages.pulse-eight.net/ubuntu/install-xbmc.sh; chmod +x install-xbmc.sh; ./install-xbmc.sh

Thats all one one line to make copy/paste easier! What this does is grab the install script from Pulse-Eight, set it to executable and run it. This adds a repository into Ubuntu with the relevant updates, upgrades and does it all for you. When its done, just reboot and you should have XBMC again but this time it will have additional TV functions hiding. This is all we do with XBMC for now.

(NOTE: XBMC has a few problems with standby, resuming usb devices at the moment, this appears to be an Ubuntu issue and not XBMC, so if you set your settings > power savings to standby when power pressed and after resuming you find it won't make the IR remote work you've got the bug. There are loads of forum posts about it. I've tried them all and with no success as yet unfortunately).

If by chance your IR works after resume then you'll want to be able to wake the XBMC machine from its sleep by the remote (So a simple power button press on the remote will wake it). If thats the case you want to look at adding the following to /etc/rc.local:

 

echo "USB0" > /proc/acpi/wakeup
echo "USB1" > /proc/acpi/wakeup
echo "USB2" > /proc/acpi/wakeup
echo "USB3" > /proc/acpi/wakeup
echo "USB4" > /proc/acpi/wakeup
echo enabled > /sys/bus/usb/devices/7-2/power/wakeup
 
That last line you need to find the correct numeric value (Instead of 7-2). Do it just by typing lsusb, look for your IR device and the numbers you need are the BUS and DEVICE. So in mine above I had:
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 002 Device 002: ID 0644:0200 TEAC Corp. All-In-One Multi-Card Reader CA200/B/S
Bus 002 Device 003: ID 046d:08cc Logitech, Inc. Mic (PTZ)
Bus 007 Device 002: ID 045e:006d Microsoft Corp. eHome Remote Control Keyboard keys
 
So the last entry is my IR remote, take the 2 numbers 007 002, remove the 0's and you have the 7-2 needed for the wake command. This will allow your USB devices to wake the machine when its in sleep.
 
 
 
Now onto the Satellite part of things. TVHeadend. This is best ran on a separate machine. A few people run it on the same machine as XBMC but this appears problematic and you will also need to leave the machine on, or do some trickery to wake it at the right time to handle recordings, so we're skipping that and using a dedicated machine. The other advantage is you can stick this out of the way (In my case, my modified Garage) so its on all the time. This means you don't need to use quiet fans, etc. So purchase some Satellite/Terrestrial cards. I use 2 satellite cards (DVB-S DVB-S2 are Satellite), and one terrestrial (DVB-T). If you're buying satellite cards then I'd strongly recommend to get DVB-S2 (And not the S1 cards), because some channels in the UK now are only available on the more advanced S2 streams, so to future-proof I'd go DVB-S2. I'd also recommend against combination cards (2 receivers on one card, one sat one terrestrial, etc), as in some cases Linux won't be able to use the different elements separately and you'll end up restricting what to do.
So I installed one DVB-S card, one DVB-S2 card and one DVB-T card (All PCI as USB is also problematic in some cases). I did a normal/basic Ubuntu server installation (No graphical interface needed, but if you're a novice it won't harm). Get it all up to date and then get ready for the other installations.
 
Next I created a storage area for recordings. On my network I have a NAS (Network Attached Storage) so I used this, however for these instructions I'll show you how to make your TVHeadend server into a basic NAS for TV use. The reason I did this was that it allowed all machines on the network (Regardless of if they ran XBMC or TVHeadend integrated) to view and playback recorded material. On my network at home I used NFS as the majority of my network is Linux-based. So you need to install the nfs server to your TVHeadend machine:
apt-get install nfs-kernel-server
After install you need to create your storage area, so do something like:
mkdir /home/recordedtv
We'll use that to save all our TV to. So you need to export that from NFS:
echo "/home/recordedtv *(rw,sync,no_subtree_check,no_root_squash)" >> /etc/exports
This adds the folder to the export for NFS with a few parameters, WARNING: we are letting anyone that connects full read/write access without restrictions. Only do this if you trust your network, otherwise read up on safer NFS settings.
 
Now we need to install TVHeadend. At the moment there is some active development which is providing amazing changes to how EPG and the main TVH system works, this work is done by Adam Sutton who has made some great updates to TVH. I've been very lucky in being able to beta-test a lot of code and provide input to its development, so I'm very grateful to Adam who has humoured me all along! So at the moment this would be the branch I would recommend over the main tree or the debian packages. So it's a little more involved compiling but its worth it! So here are a few simple instructions to get you started:
apt-get update
apt-get install build-essential git 
cd /usr/src
mkdir tvheadend
cd tvheadend
git clone https://github.com/adamsutton/tvheadend.git
git pull origin initialscan
You should now have a full copy of the latest dev code. Go into that folder (cd tvheadend) and you can then start compiling. I can't remember if there were additional dependencies for TVH so if configure fails you will need to add the dependencies. If you're unsure pop a comment/tweet over to me and I'll help out and update this doc.
 
cd tvheadend
./configure --prefix=/usr --release
make
 
That'll give you a working TVH binary to use. Only a couple more steps to go now! 
Create a user for TVHeadend to use:
adduser tvheadend
You can then use that to run TVH, etc. So change to that user and do a simple setup task. This is because at present the work is in development. So carry out:
mkdir ~/.hts
mkdir ~/.hts/tvheadend
cp -Rp /usr/src/tvheadend/tvheadend/data/epggrab ~/.hts/tvheadend
All this does is creates the TVHeadend config folders, and also copies the libraries needed for EPG grabbing to work.
I tend to run TVHeadend it in a screen process so you can see if any errors occur, so you can do this:
screen (Press enter on the information screen that appears)
cd /usr/src/tvheadend/tvheadend
./build.linux/tvheadend -s -d
 
That'll fire up TVHeadend. Hopefully it'll run and got exit with an error. If so you can now swap to the webui and start to configure it.
The rest of the config is fairly standard, in the new EPG grabber tab make sure under the OTA section you enable OpenTV: Sky and EIT grabbing. The channel name/number/icon is experimental so you may wish to leave this off for now. Continue on to the TV Adapaters section and setup the rest of your TVH setup as normal. This is documented elsewhere and I've documented under my old XBMC section so take a look there is you're totally new to TVH.
 
Good luck, and as always please do get in touch if you'd like a bit of extra help with your setup.
 

 

UK Cookie Law

 UK Cookie Law Compliance.

Well the ICO have required all EU websites that use cookies to apply a compliance method on the 26th May 2012. What my views are on this? Well if you want to accept cookies, your browser will accept them. If you don't you set your browser to NOT accept cookies and you're done. But apparently that isn't the 'right' way according to the ICO's techs, so all website authors, operators, etc, have to do something a bit different. In most cases you have to block the cookies until you get confirmation from the user. However these have been updated and now gives us the option of implied consent which gives us a little simpler option to take here. We'll implement this on the website here, basically by visiting the site you are accepting that we will use cookies to assist in the website experience, however if you do not login or provide account details then this information will simply be used for statistical analysis and production of website usability.

The above is basically our cookie policy for this website, the website is a not-for-proft, hobbyist site and so the use of cookies are simply for user statistics, or where a user registers for easing their use of the site/login/administration.

 

(PS: You'll also notice at the bottom-right of the website my cookiecontrol compliance, this is a nifty little free add-on that helps you with the compliance, its from  http://civicuk.com/cookie-law/index if you want one!)

Arduino - Cheap wifi connectivity

General
Building a cheap ethernet connection for my Arduino. I needed to relay data from my Arduino back to my home server. The location
makes wired prohibitive, but I also don't have CAT5 cabled to the location I had the Arduino in. I then looked at the various
shields available, the Ethernet shield wasn't too expensive at around £20, but this would mean wiring, and the location in the kitchen
was particularly difficult to get wiring to.
 
I then looked at wifi shields, these came in at around £85 and way too expensive for just a little hobby project, the in-between options
of using Zigbee interfaces, which is a slightly cheaper option, but I was concerned about the distance and complexity of setting
this up with my server.
 
I therefore started posting to mailing lists and trying to find a way of doing a wifi-bridge cheap, or USB over wifi. All I needed was
to get the usb/serial data out from my Arduino to my server so that I could graph it, monitor it, etc.
A kind chap on an ISP mailing list brought forward the TP-Link TL-WR703N suggesting that this device can be flashed with OpenWRT
and then act as a wifi bridge. This sounded interesting, and you can get them on eBay for less than £15 inc delivery. This was sounding
good.
 
The TP-Link TL-WR703N has 3 physical ports (RJ45, USB host and power (via a micro-usb port)). It also has integrated WIFI. So my
idea was to connect the wifi up to my house wifi, then use the USB port to monitor the serial output from Arduino. Ideal!
 
The TL-WR703N arrived, all in its chinese writing and unfortunately its chinese web admin interface. Luckily for me, the OpenWRT
site for this device was very detailed and told me exactly what to click to upload OpenWRT.
Then OpenWRT site at http://wiki.openwrt.org/toh/tp-link/tl-wr703n Had all the details to flash it, very simple and quick.
Once flashed with OpenWRT a little more tweaking was needed. I've generally not done much with OpenWRT in the past, I stick to DDWRT
which has a nice web-gui, so OpenWRT involved SSH'ing to the unit and setting the rules correctly for what I wanted.
 
I did a bit of reading on the OpenWRT recipies site on what I needed which was to be a client on a mixed WPA2-Personal AES 
encrypted wifi with MAC protection.
The basic method is follow the outline of a MASQUERADE CLIENT on OpenWRT and adjust accordingly. So my /etc/config/network looked like:
 
config interface 'loopback'
option ifname 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
 
config interface 'lan'
option ifname 'eth0'
option type 'bridge'
option proto 'static'
option ipaddr '192.168.1.1'
option netmask '255.255.255.0'
 
config interface 'wan'
option ifname 'wlan0'
option proto 'dhcp'
 
(Note that I left LAN on the 192.168.1.1 setup, this meant that if the WAN or dhcp failed I can plug into it and still admin the thing)
 
/etc/config/wireless was a bit more tricky:
 
config wifi-device 'radio0'
option type 'mac80211'
option channel '1'
option macaddr '8c:21:0a:d8:7b:57'
option hwmode '11ng'
option htmode 'HT20'
list ht_capab 'SHORT-GI-20'
list ht_capab 'SHORT-GI-40'
list ht_capab 'RX-STBC1'
list ht_capab 'DSSS_CCK-40'
 
config wifi-iface
option device 'radio0'
option mode 'sta'
option ssid 'MYSSID'
option encryption 'psk2+ccmp'
option key 'MYPSKPASSWORD'
 
To find these settings you issue 'iwlist scan' and look for your SSID and the other settings. So in my case I set the channel of 1.
The encryption of psk2+ccmp (This refers to wpa2 with AES encryption). After doing that you can then do 'ifup wan' and 'wifi' which
should then get things working. Except in my case it didn't. I checked, I had the MAC address in my wifi router, all looked fine but it
wouldn't join the WIFI. The settings in OpenWRT are a little hazy so I assumed this was the problem and spent a while here, but in the end
I found turning MAC protection off temporarily on my DDWRT allowed it to connect, enabled it, saved and it all works each time now,
so unsure if I hit a bug in my DDWRT on my wifi access point or something, but for future, debugging with MAC protection turned off
is a lot easier!
 
Now I had it connected I needed to get info from the Arduino over the USB port. I plugged it in hoping for the /dev/ttyACM0 to be
created, but no luck. I got the USB insert message:
[   10.020000] usb 1-1: new full-speed USB device number 2 using ehci-platform
So it looks like the kernel driver doesn't exist.
Luckily the OpenWRT group have a packaging system that includes a lot of drivers and add-ons, so using the package management to install:
 
opkg update
opkg install kmod-usb-acm
 
Reboot, and when I plugin the arduino, success, I see:
[   10.020000] usb 1-1: new full-speed USB device number 2 using ehci-platform
[   10.160000] usbcore: registered new interface driver cdc_acm
[   10.160000] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
[   10.210000] cdc_acm 1-1:1.0: ttyACM0: USB ACM device
 
And the /dev/ttyACM0 is there. Last trick is how to get a couple of lines of data from the serial connection and SSH/SCP it up
to my server. This proved a little tricky, as first of all getting just a couple of lines of data from a constantly flowing serial
terminal, then the scp function itself.
 
To get just a couple of lines of serial I firstly tried the stty trick "stty -F /dev/ttyACM0 9600 -echo -onlcr time 50 min 0" and then
cat /dev/ttyACM0 which will read data for a specific amount of time. This was very unreliable and often hung the /dev/ACM0 interface.
So I looked to alternatives. I then tried a constant/background "cat /dev/ttyACM0 > /tmp/serial.log", periodically from cron reading
it and zero'ing it. Problem here was the filesize growing and causing the device to hang due to the limited ram on the device. Also when
/dev/null'ing the /tmp/serial.log the cat then failed and a kill/hup to the cat caused the ACM0 to hang again! So no good there.
Finally I tried "head -n 12 /dev/ttyACM0 > /tmp/arduino.log" So this output the last 12 lines from the serial console to a file.
(Overwriting the file each time keeps it low filesize). Then I can parse this for the info I need, and then scp it over. Except scp
doesn't exist as the ssh is just dropbear! So again to opkg and install the ssh client tools so I get the scp binary. Generate an
id_rsa key and then I can scp my output over: "scp -i /etc/dropbear/id_rsa /tmp/arduino.log root@192.168.xx.xx:/tmp"
Success! I can now parse this with my graphing tools.
 
Enjoy, and thank you to the TP-Link TL-WR703N for a cheap and nifty solution! (PS: Got any direct questions? See me on twitter: www.twitter.com/andyb2000 )

Arduino garden helper

General

 Over the recent bank holiday weekend we decided to plant a few vegetables in the garden. Nothing too radical, a few Onions, Peppers, Chilli and re-plant our little strawberry plant. We invested in one of those small polythene tunnels too, so help give them a bit of a boost start and see if we can get the plants happy despite the miserable cold weather we've been having. To this end I finally thought up a use for my Arduino I've had kicking around. So I started to think about a soil moisture and poly-tunnel temperature sensor type array that would feed back to the Arduino and tell us when it wanted watering, and also what the highs and lows are for temperature.

So I set about making a soil moisture sensor. My basic background in electronics helped and I knew the simplest method is to use a resistance type technique, so provide +DC to a probe, and them measure the resistance between that and a second probe. This should be easy for the Arudino as it has a few analogue inputs that I can use. The second input was temperature, so I used a temperature sensor I'd previously bought from oomlout.co.uk (TMP36) which has a nice range of -40oC up to 125oC so perfect for this. It's supplied in a standard semiconductor package so first I had to make it a little more weatherproof. Soldering up the 3 wires to a long length of cable (burglar alarm cable, which is my fave cable for general purpose as its 6-core and pretty reasonable in terms of weathering and loss of signal). I then coated each of the exposed wires with a bit of heatgun glue, followed by electricians tape round each leg, so each were electrically insulated and also hopefully sufficiently weather proofed. (Not sure how well that would survive totally exposed to the elements, but as this was going in the poly-tunnel I thought it would be fine).

Next was to build the soil moisture sensor. There are a few different ones on the net, so I went with one that seemed a common way of doing it. Set two probes (in my case 2 brass screws) in a piece of electrically non-conducting packing foam. So this lets the screws push down into the earth whilst keeping the rest of the screws insulated. At the top I then wire wrapped the alarm wire around the screws and added a few blobs of solder for good measure. A bit more electricians tape finished them off.

Here is the sensor that I came up with. You can see the TMP36 semiconductor sticking out the top, and the screw probes out the bottom.

 

After the probe was sorted, I ran it round outside into the poly-tunnel:

 

So the final part was the circuit into the Arduino, and also the code to make it function. The breadboard I had already connected to my Arduino had the LCD attached so I left that to give constant output on the Temperature and soil moisture, so all I had to add was the connections out to the remote probes. I found an interesting issue, at first the soil moisture values varied wildly, even when the probe wasn't in the soil, until I realised that this was because nothing was 'pulling' the input to ground, adding a quick resistor into the setup so that it pulled the input to 0 when it didn't have any other value sorted that out and I got values depending on the resistance being presented. At first I'd connected the soil moisture probe to +5ve and the other probe to my analogue input. One problem I'd found reported quite frequently online was about electrolysis of the metal probes (where electricity passing through them caused corrosion to take place). To counter this I moved the moisture probe to an output pin, and so I'll only switch on current to the probe when I wish to measure it, hopefully reducing the amount of electrolysis taking place.

A few iterations of code were required to get the reading showing anything sensible, and de-bouncing the inputs was essential. I found 4 iterations of the sensor read loop followed by an average calculation stopped a lot of the large fluctuations taking place and gave me what appear to be stable readings, and when compared with the temperature read-out from the car, the outdoor temperature looked around the right values. Soil moisture though is still a bit of an issue as I need to self-calibrate the sensor by checking the soil and deciding when it gets too dry to set that as the lower threshold.

I'll post the final code up here shortly as I'm still fine-tuning it, and also hopefully connecting my Arduino up so I can access the readings remotely and graph them using my rrd graphing system I use for other things around the house (heating, electricity, etc).

Here are a few images of the Arduino in its plastic (indoor) enclosure in operation. The Green LED lights up when the soil moisture drops below the triggered limit, and the Red LED lights when the readings are being taken. Debugging is also output via the serial port.

  

 

Aruino code + schematics to follow!

 

 

Update 10/May/2012 - Here is the current Arduino code I've been using, sorry about the messy code!

<pre>

// AndyB 7-may-2012 Outdoor temperature and soil moisture circuit

// ---------------------------------------------------------------

// include the library code:

#include <LiquidCrystal.h>

#include <EEPROM.h>

#include <Time.h>

#include <TimeAlarms.h>

 

// PINS

//   2 3 4 5 11 12 - LCD

//  9 = watering LED

// 10 = soil power pin

// 13 = Data collection LED

//  8 = LCD display light

//  6 = LCD display button light up

 

// initialize the library with the numbers of the interface pins

LiquidCrystal lcd(12, 11, 5, 4, 3, 2);

int sensorPin = A0;       // Temperature sensor

int soilMoisturePin = A1; // soil moisture

int sensorValue1 = 0;

int addr = 0;

int SoilValue1 = 0;

float temperatureC = 0;

const int buttonPin = 7;     // the number of the pushbutton pin

int buttonState = 0;         // variable for reading the pushbutton status

const int ledPin =  13;      // the number of the LED pin

const int waterPin = 9;      // the LED for watering

const int soilPowerPin = 10;  // Pin to power up the soil moisture sensor

const int lcdPowerPin = 8;    // Pin to light up the LCD

const int lcdButtonPin = 6;  // Switch to light up the LCD for a few loops

 

int lcdlightloop=0;

 

int logloop = 0;

 

int debug = 0;  // Debug setting

 

void setup() {

  Serial.begin(9600);

  if (debug == 1) {Serial.println("Program initialised");};

  pinMode(ledPin, OUTPUT);  

  pinMode(buttonPin, INPUT);

  pinMode(waterPin, OUTPUT);

  pinMode(soilPowerPin, OUTPUT);

  pinMode(lcdPowerPin, OUTPUT);

  pinMode(lcdButtonPin, INPUT);

  // set up the LCD's number of columns and rows: 

  lcd.begin(16, 2);

  digitalWrite(lcdPowerPin, HIGH);

  // Print a message to the LCD.

  // lcd.print("       oC");

  // write a 0 to all 512 bytes of the EEPROM

//  for (int i = 0; i < 512; i++) {

//    EEPROM.write(i, 0);

//  };

  // do not reset the EEprom, find the last value and set the pointer, avoids power resets loosing old data

  if (debug == 1) {Serial.println("reading EEPROM");};

  for (int i = 0; i < 512; i++) {

    byte tmp_val = EEPROM.read(i);

    Serial.print("EEPROM ");

    Serial.print(i);

    Serial.print(": ");

    Serial.println(tmp_val, DEC);

    int tmp_val_int=tmp_val;

    if (tmp_val_int == 0) {

        addr=i;

        if (debug == 1) {Serial.print("EEPROM empty at ");};

        if (debug == 1) {Serial.println(i);};

        break;

    };

  };

  digitalWrite(soilPowerPin, LOW);

//  sensorValue1 = analogRead(sensorPin);

//  float voltage = sensorValue1 * 5.0;

//  voltage /= 1024.0;

//  temperatureC = (voltage - 0.5) * 100;

//  SoilValue1 = analogRead(soilMoisturePin);

//  Serial.println("Back from reading sensors");

//  logdatatoeeprom();

  if (debug == 1) {Serial.println("Setup completed");};

  // Alarm.timerRepeat(9000, logdatatoeeprom); // every 15 minutes

  // Alarm.timerRepeat(15, logdatatoeeprom);

  digitalWrite(lcdPowerPin, LOW);

}

 

void loop() {

  if (debug == 1) {Serial.println("Starting main loop");};

  // check the button status

  buttonState = digitalRead(buttonPin);

  if (buttonState == HIGH) {     

    // Button pressed - Reset all of the EEPROM values

    if (debug == 1) {Serial.println("Button pressed - resetting EEPROM");};

    digitalWrite(ledPin, HIGH);  

    // write a 0 to all 512 bytes of the EEPROM

    for (int i = 0; i < 512; i++) {

      EEPROM.write(i, 0);

    };

    Serial.println("EEPROM ERASED");

    addr=0;

    logdatatoeeprom();

    digitalWrite(ledPin, LOW);

  };

  // check for LCD display button

  if (digitalRead(lcdButtonPin) == HIGH) {

      // LCD button was pressed

      lcdlightloop=1;

  };

  if (lcdlightloop > 0) {

      lcdlightloop += 1;

      digitalWrite(lcdPowerPin, HIGH);

  };

  if (lcdlightloop > 20) {

    lcdlightloop=0;

    digitalWrite(lcdPowerPin, LOW);

  };

  if (debug == 1) {Serial.println("Button checks complete");};

  // set the cursor to column 0, line 1

  // (note: line 1 is the second row, since counting begins with 0):

  digitalWrite(ledPin, HIGH);

  if (debug == 1) {Serial.println("Read temp sensor");};

  

  // READ SENSORS HERE

  temperatureC = readinput_temp();

  SoilValue1 = readinput_soil();

  

  if (SoilValue1 < 40) {

    digitalWrite(waterPin, HIGH);

  } else {

    digitalWrite(waterPin, LOW);

  };

 if (debug == 1) {Serial.println("Display to LCD");};

 Serial.println("Temp: ");

 Serial.println(temperatureC);

 Serial.println("Moisture: ");

 Serial.println(SoilValue1);

  lcd.clear();

  lcd.setCursor(0,0);

  lcd.print("temp: ");

  lcd.print(temperatureC,1);

  lcd.print(" oC");

  lcd.setCursor(0,1);

  lcd.print("soil: ");

  lcd.print(SoilValue1);

if (debug == 1) {Serial.println("Done writing LCD");};

    // we write to the eeprom every 10 iterations

    logloop=logloop+1;

    if (debug == 1) {Serial.print("Testing for logloop ");};

    if (debug == 1) {Serial.println(logloop);};

      if (logloop > 9) {

        logdatatoeeprom();

        logloop = 0;

      };

  digitalWrite(ledPin, LOW);

  if (debug == 1) {Serial.println("LED off");};

  Alarm.delay(10000); // wait ten second between clock display

  if (debug == 1) {Serial.println("Delay set");};

}

 

void logdatatoeeprom(){

  digitalWrite(ledPin, HIGH);

  lcd.setCursor(15,1);

  lcd.print("E");

  if (debug == 1) {Serial.println("logdatatoeeprom started");};

  if (debug == 1) {Serial.print("writing to eeprom ");};

  if (debug == 1) {Serial.print(addr);};

  if (debug == 1) {Serial.print(" : ");};

  if (debug == 1) {Serial.println(temperatureC);};

  EEPROM.write(addr, temperatureC);

  if (debug == 1) {Serial.println("back from writing to eeprom");};

  digitalWrite(ledPin, LOW);

  digitalWrite(ledPin, HIGH);

  digitalWrite(ledPin, LOW);

  // advance to the next address.  there are 512 bytes in 

  // the EEPROM, so go back to 0 when we hit 512.

  addr = addr + 1;

  if (addr == 512) {

    addr = 0;

  };

  if (debug == 1) {Serial.println("end logdatatoeeprom");};

}

 

float readinput_temp() {

  // Read inputs and de-bounce, so read 3 times and take an average

  if (debug == 1) {Serial.println("Readinputs (Temperature) function started");};

  float temp1 = 0;

  float temp2 = 0;

  int soil1 = 0;

  int soil2 = 0;

  int lpcount=0;

  

  while (lpcount<6) {

    sensorValue1 = analogRead(sensorPin);

    float voltage = sensorValue1 * 5.0;

    voltage /= 1024.0;

    temp2 = (voltage - 0.5) * 100;

    temp1 += temp2;

    if (debug == 1) {Serial.print("Temp loop :");};

    if (debug == 1) {Serial.print(lpcount);};

    if (debug == 1) {Serial.print(" : ");};

    if (debug == 1) {Serial.println(temp2);};

    lpcount=lpcount+1;

  };

  temperatureC = temp1/6;

  if (debug == 1) {Serial.print("Temperature average: ");};

  if (debug == 1) {Serial.println(temperatureC);};

  if (debug == 1) {Serial.println("Exit Readinputs (Temperature)");};

return temperatureC;

};

 

int readinput_soil() {

  if (debug == 1) {Serial.println("Readinputs (Soil) function started");};

  // Switch power pin on for the soil sensor

  digitalWrite(soilPowerPin, HIGH);

  float temp1 = 0;

  float temp2 = 0;

  int soil1 = 0;

  int soil2 = 0;

  int lpcount=0;

 

  if (debug == 1) {Serial.println("Read soil moisture sensor");};

 

  lpcount=0;

  while (lpcount<4) {

    soil2 = analogRead(soilMoisturePin);

    soil2 = (soil2 / 2);

    soil1 += soil2;

    if (debug == 1) {Serial.print("Soil loop :");};

    if (debug == 1) {Serial.print(lpcount);};

    if (debug == 1) {Serial.print(" : ");};

    if (debug == 1) {Serial.println(soil2);};

    lpcount=lpcount+1;

  };

  SoilValue1 = soil1/4;

  if (debug == 1) {Serial.print("Soil average: ");};

  if (debug == 1) {Serial.println(SoilValue1);};

  digitalWrite(soilPowerPin, LOW);

  if (debug == 1) {Serial.println("Exit Readinputs (Soil)");};

return SoilValue1;

}

 

</pre>

 ----------------------------------------------------------------------------------

Update: Tuesday 15th May 2012

I'm now graphing the output as I've hooked the Arduino up to a laptop via USB, below are the graphs (updated every hour):

    

 

XBMC, TVHeadend, MythTV, streaming to your TV

 I thought it was about time I wrote a bit more about what I've been up to with our TV setup at home. Firstly a bit of background, take a look at the previous posts on my XBMC setup, but that is now a bit out of date as I've found this isn't a set and forget sort of setup, so I can see why a lot of people won't bother with this day-to-day. But I'm persistant and wanted to get my setup as good as possible so we've got the best of several options.

Firstly FreeSat with the superior HD channels, then Freeview (via Aerial) as this still has some channels not on FreeSat, then PVR functionality for pause/rewind of live tv, finally playback from the network media drive. In addition I've also got a want to stream iPlayer, ITV, 4od and 5 via their online stream facilities.

So originally I had TVHeadend running (via their git repository) with a custom patch for adapter priorities. That patch allows me to 'weight' which adapter I want to be used, so the rule was, use the Freesat adapters, unless the channel wasn't available (or signal was too low). This worked great, but there are problems with the integration to XBMC (The frontend). The biggest problem was that the PVR functionality of pause/rewind didn't work, and the author of TVHeadend/XBMC PVR plug-in really doesn't have any interest in developing this as he has no need for it. That causes me a bit of a headache, as that component lets the system down quite a bit. There are also problems with TVHeadend in that it relies on XMLTV to grab TV listings, as the EIT grabber (Over the air/satellite) barely works at all, and again nobody can see to improve that at the moment. So these two items let it down quite a bit as it means we get dodgy entries in our TV Guide, so its not reliable, and we can't pause. Another huge bug, and I believe this is an XBMC problem, is that if a channel doesn't have an EPG entry, or the entry is deemed as 'invalid' by XBMC it actually hides the channel. Now this causes channels to disappear and re-appear due to bad EPG data, this is something I've asked numerous times on the forums on how to correct myself in the code and re-compile but nobody seemed to know the definitive answer, or didn't want to help. This also frustrated me!

So I thought, time to move to MythTV using XBMC as the front-end. Myth does have pause/rewind functions, has EIT working fully and a few other nice things so this seemed a good option to switch to. Installing Myth-backend is a pain as although they have debian/ubuntu packages, you have to guess which ones you need to run the backend (mythtv-backend, mythtv-web and a few others). Once installed onto my headless server that has my 2 FreeSat and 1 Freeview cards in it I needed to set it up. Here was the first problem, its a GUI that you run within an Xserver (a desktop). This is a headless server so I can't do that, so have to run it over an X forwarder to my laptop, so its a bit laggy but works. The mythtv-setup is HORRIBLE! It's clunky, and looks like its been designed to be used from a TV remote, but even that it would be horrible to use, so I didn't have a good experience there. The logic of channels, channel inputs, hardware adapters and channels is also a little confusing. So you first setup your capture card, thats pretty straight forward, the system finds my 3, you have to setup Diseqc and tell it just to use an LNB (even for the Freeview) which is weird as it seems to be a requirement rather than an option. Once they're all setup you have to create a video source. This is straight forward as I want to use EIT, so select that, and thats done. Now the bit that I've still not fully got my head around, Input Connections. You set these to match your channels, so on mine I've created 2, one for FreeSat and one for Freeview (I have no idea if this is right, as it doesn't seem 100% correct!), so I went in, scanned for FreeSat DVB-S first (this takes about 10 minutes), then for DVB-S2 (another 10 minutes). Then create another Input Connection for Freeview, and scan for channels on there (About 10 minutes again). Wow that takes a long time! Now you can go into channel listings. This is mostly pointless, as the interface is so unusable you have about 200 entries in a list and it expects you to click into them one-by-one and disable the ones you don't want, correct incorrect text, etc. I would be on forever with that, so opted to leave that as it is for now and just have a stupid number of channels visible. At this point you can save and exit.

Now you need to compile cmyth with XBMC. cmyth is a fork of XBMC as the developer has re-written some parts of XBMC to handle the Myth setup correctly. So I grabbed his sources and started compiling, hitting many incompatability problems along the way as debian packages were missing, I was running an old version of Ubuntu, so had to upgrade the distribution first. That took longer than I hoped, and if you do a custom compile, don't forget to put your ./configure command options in to make all your add-ons work. My line ended up as './configure --prefix=/usr --enable-vdpau --enable-static --disable-shared --enable-sdl --enable-rtmp' Doing this, a make and install got us up and running and I could see the EPG loading. Going in however I had about 10 BBC1's, 10 BBC2, etc. Basically what it had done is found each copy of the channel on the FreeSat devices, Freeview channels and just put each one into the listing, it didn't merge any! So that meant back to the MythTV listing setup (Also each time you go into myth-setup you have to shutdown the daemon, which disconnects all clients stopping TV from working again!). I tried again but found the channel editor impossible. Luckily I spotted the MythWeb add-on, went into that and found you can edit the channels through the web interface, so I went through removing channels I didnt want. But what to do about the duplicate BBC1? A bit more reading revealed that you should give these the same channel number. So find all the BBC1's in your listing and set them to a channel number (e.g. I used 101 to mirror the Sky numbering system). I've done that and sure enough it looks much better.

The main problems I have now (and not solved):

  • EPG loading time in XBMC is very slow. It can be up to 5 minutes and its not a good idea so start watching TV till this is finished as if you do that it seems to corrupt a lot of EPG entries. This is a pain as we all have to sit watching a loading bar before we watch TV now!
  • After selecting a channel, it takes almost a minute before the channel engages and starts streaming
  • Pressing anything like EPG, pause, etc, also freezes the playback for a few seconds as the box 'thinks'
  • Channels without proper EPG still don't appear in the EPG line-up, so again this is a big problem!

So after all of these, I'm unsure if I should revert back to TVHeadend, as there are a few too many bugs and problems that are frustrating, and I'm not sure if the gains (Pause TV) is enough to warrant it. I may re-do all of it over the weekend back to TVHeadend and see if I can correct the few bugs remaining there myself!