A Little Logger

It goes like this: … “it’s too bright to work, I need the window shades down” … and then a few days later … “there’s a great view but we don’t see it because the shades are always down”

The windows my wife and I could look out of as we work are covered a lot of the time. Despite the shades being electrically operated they are not “smart” and when you shut them they simply stay shut. I’ve been pondering for a while whether it might be viable to open and close them automatically somehow but it’s only just reached the point of any thought being applied.  The first idea that came to mind was to use the time of day and date to determine whether they should be closed or open – but the sunshine is pretty variable in Edinburgh and that probably would leave them down a lot of times they could be up. So I’ve decided to look into using brightness sensors to control them.

The next question that came to mind was “how bright is too bright” and “what does that mean in light-sensor-units that I can measure”.  So I felt the need for a little logger (data logger that is) to gather some data on brightness over a few days.

Connected Dev Boards Galore

Over the years I’ve made a lot of systems for collecting data – I think all (three of) the companies I’ve worked for/founded have concentrated on data collection – at least initially. But most data logging systems are pretty clunky and I just want something that I can get working in a matter of minutes and tinker with indefinitely – it shouldn’t be too hard – should it?

There are a lot of great little connected development boards around at the moment – the Electric Imp, Particle Photon, RedBear Duo, NodeMCU and the one I quickly chose the WiPy. Life is too short to mess around with C++ with something trivial like this so I ruled out the boards that don’t come ready loaded with a higher level language than C++, and I’m no fan of Lua so Damien George‘s MicroPython won the day.

IMG_0997RedBearDuo_thumbnail

I started late – but made quick progress!

It was already 10:30pm when I started and I wanted to get to bed in a reasonable time so I cut a few corners. I pulled out the WiPy board that I’d already connected to the home WiFi. Connecting to WiFi isn’t too hard to do (as explained at the bottom of this post) but the board comes in Access Point mode (as many devices do today) and you have to connect to it’s own WiFi network first which is a little bit of a faff.

Anyhow, I didn’t need to do that so I was quickly typing things like the following (copied directly from the great documentation) at the REPL prompt:

from machine import ADC
adc = ADC()
sensorLdr = adc.channel(pin='GP3')
sensorLdr()

Quick and DirtyI connected a light dependent resistor (LDR) between GP3 (a pin on the WiPy) and ground and a resistor (10K) up to 3.3V using a breadboard (I actually connected a few different sensors to compare them but have spared details).

Press enter and you have a reading – quite close to the max value of 4095 in this case as it was dark!

So, continuing in the quick and dirty mode, I taped the whole thing to the window with insulation tape and got onto the next bit.

Logging the data

I confess that I cheated a little on this part too – I already have some code to log HTTP GETs to a flat-file. It is really trivial code but it has proven extremely useful for situations like this. It uses nodejs and express which makes it really easy to set up on anything – including a Raspberry Pi – and it saves lots of time. The code is at the bottom of this page.

But basically that allowed me to type (again at the REPL prompt):

import socket
addr = socket.getaddrinfo('fractal', 5089)[0][-1]
s = socket.socket()
s.connect(addr)
s.send(b'GET /log/lightvalues/' + str(sensorLdr()) + ' HTTP/1.1\r\n \r\n\r\n')

And I could confirm that I received a new GET on my computer – called “fractal” port 5089 with url /log/lightvalues/1234

The final code

All that remained was to put the send in a loop, move the code to a file called main.py – the WiPy runs this immediately after boot.py – and store it in the root of the WiPy using FTP (as described below):

from machine import ADC
adc = ADC()
sensorLdr = adc.channel(pin='GP3')

import socket
addr = socket.getaddrinfo('fractal', 5089)[0][-1]
s = socket.socket()
s.connect(addr)

while(True):
 s.send(b'GET /log/lightvals/' + str(sensorLdr()) + ' HTTP/1.1\r\nHost: fractal\r\n\r\n')
 data = s.recv(1000)
 time.sleep(10)

That was it – done in 90 minutes and off to bed!

Next morning results

The flat file is perfect for directly loading into Excel – the bit of nonsense data at the start comes from the testing close to midnight – but otherwise some useful data …

20160715 Excel capture of light sensor tests

The lines are actually as follows:

  • blue line is the LDR (with a 10K resistor to 3.3V)
  • orange line is a visible-light phototransistor from Sharp (with a 10K resistor pulling down – I later changed this to a 1K as it was saturating)
  • grey line is an Adafruit ALS-PT19 light sensor module – but that is also close to saturation – maybe need a filter to use that for this purpose

Addendum1: Quick and dirty Nodejs/express flat-file logger

// Generated by CoffeeScript 1.7.1
var Express, app, formatDuration, formatHMS, formatYMD, http, moment, path, request, sortBy;
request = require('request');
http = require("http");
Express = require("express");
path = require("path");
moment = require("moment");

console.log("Starting Flat File Logger - Rob Dobson 2016");

app = Express();
app.set("port", process.env.PORT || 5089);
app.set("views", path.join(__dirname, "views"));
app.set("view engine", "jade");

app.use(function(req, res, next) {
 res.header("Access-Control-Allow-Origin", "*");
 res.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
 return next();
});

app.get("/", function(req, res) {
 res.send("<html><body><h1>Flat File Logger</h1></body></html>");
 console.log("Get home page");
});

app.get("/log/:logfile?/:logmsg?", function(req, res) {
 var fs, logfile, logmsg, timeStr;
 logfile = req.params.logfile !== void 0 ? req.params.logfile.toLowerCase() : "";
 logmsg = req.params.logmsg !== void 0 ? req.params.logmsg.toLowerCase() : "";
 timeStr = moment.utc().format('YYYY-DD-MM HH:mm:ss') + " ";
 console.log(timeStr + "Got log " + logfile + " " + logmsg);
 fs = require('fs');
 fs.appendFile("logs/" + logfile + ".txt", timeStr + "\t" + logmsg + "\n", function(error) {
 if (error) {
 return console.error("Error writing file", error);
 }
 });
 return res.send("OK");
});

app.get("/view/:logfile?", function(req, res) {
 var logfile, timeStr;
 logfile = req.params.logfile !== void 0 ? req.params.logfile.toLowerCase() : "";
 timeStr = moment.utc().format('YYYY-DD-MM HH:mm:ss') + " ";
 console.log(timeStr + "View log " + logfile);
 return res.sendFile(path.join(__dirname, "logs/" + logfile + ".txt"));
});

http.createServer(app).listen(app.get("port"), function() {
 console.log("Flat File Logger listening on port " + app.get("port"));
});

Addendum2: Setting up a WiPy on your own network

The process for getting a new WiPy onto your own WiFi network is like this:

  1. Power up the WiPy – ideally by plugging it into a WiPy Expansion Board and then connecting to a regular micro-USB
  2. Use a laptop to connect to the WiFi network that the WiPy creates – the SSID starts with wipy-wlan and the key is www.wipy.io
  3. The WiPy is now accessible at IP address 192.168.1.1 on it’s own network
  4. Open an FTP client and connect to ftp://192.168.1.1 as the user micro with password python
  5. Create a new file on the laptop with the content shown below …
  6. Change the ‘192.168.0.241’ line to the fixed IP address you want the WiPy to have (make sure it doesn’t conflict with the range of addresses dynamically allocated by your router’s DHCP function)
  7. Change the ‘mywifissid’ and ‘mywifipassword’ to your own WiFi’s SSID and password
  8. Save the file somewhere and call it boot.py
  9. Copy the boot.py file over FTP to the WiPy root folder – overwriting the one that is there by default

When the WiPy is restarted (there’s a button for that) it should appear on your WiFi network at the address you requested

Once you have done all this you can:

  1. FTP into the WiPy at the fixed IP address you chose (192.168.0.241 in my case)
  2. Telnet into the WiPy at the same address (again using the user: micro and password: python) – this gets you to a REPL (read, evaluate, print – loop) – just like a regular computer does when you type Python at the shell/command prompt
  3. Connect via serial on the USB port provided by the expansion board – this also gets to the REPL prompt (it is an echo)
import machine

from network import WLAN
from machine import UART
import os

# Uart access to REPL
uart = UART(0, 115200)
os.dupterm(uart)

# WiFi on startup
wlan = WLAN() # get current object, without changing the mode
if machine.reset_cause() != machine.SOFT_RESET:
wlan.init(WLAN.STA)
# configuration below MUST match your home router settings!!
wlan.ifconfig(config=('192.168.0.241', '255.255.255.0', '192.168.0.1', '192.168.0.1'))

if not wlan.isconnected():
# change the line below to match your network ssid, security and password
wlan.connect('mywifissid', auth=(WLAN.WPA2, 'mywifipassword'), timeout=5000)
while not wlan.isconnected():
machine.idle() # save power while waiting