• Tag Archives Arduino
  • Using date ranges in MongoDB and PHP.

    I have also written similar posts relating to date ranges in Python. You can find my Question here from when I was getting frustrated and The answer to my problems here.

    You seriously wouldn’t believe the trouble I’ve had in the past two weeks trying to make some progress on my Arduino, Raspberry pi, Python, PHP and MongoDB project.

    Work has been very busy so the only time I’ve had to work on this is on the bus on the way to and from Dublin and from time to time very late at night.

    Right, so here are a few of the problems I came up against:
    The first hing I wanted to do was limit the size of my table in MongoDB. I’m collecting quite a lot of sensor data from the Arduino but I don’t need to retain this data for any more than around 2 weeks.

    MongoDB allows you to create a TTL index which will delete data that was created more than a certain number of seconds ago. This is a really handy feature however, it didn’t really work for me. I think you need to store the date in BSON format and I had stored my date in ISO format because I think it will make it easier to retrieve and write the sensor entries.

    Regardless, here is the code I used:

    db.envirocheck.sensors.ensureIndex( { “Date”: 1 }, { expireAfterSeconds: 604800 } )

    You can learn more about TTL indexes using the Fantastic MongoDB documentation that covers TTL Indexes

    As I said, this didn’t work for me at all so a suggestion on Twitter that I received weeks before made me think of capped collections. These are similar to TTL indexes in that they delete old data but instead of the TTL index, this works by deleting entries that are old however it does so when the collection reaches a certain value. By writing data once a second, I find that with 500Bytes I can store just over two hours of data. I obviously need to figure out how many bytes I need for storing two weeks worth but that’s something to do when I’m feeling more awake.

    The code to create a capped collection is here:

    db.createCollection( “sensors”, { capped: true, size: 500000 } )

    Again, look here for the MongoDB documentation for capped collections.

    Next, I of course needed to set an index on my date field as I’m going to be using this to select specific temperature values for date and time ranges. That was quite straight forward.

    db.sensors.ensureIndex( { “Date”: 1 } )

    Next, I needed to find a way of selecting between two dates in MongoDB and PHP. You might think this is easy, but no! It’s far from it! I stupidly tried to get ahead of myself by making this really complicated. I looked at Doctrine but trust me on this, the documentation for this project is absolutely crap! Now, maybe it’s me. Maybe I’m not experienced enough to figure this out but for god sake, this documentation might seem great from a high level but unless you read it from start to finish like a book, it’s useless! there’s no context to any of their examples and huge chunks of code are missing without any pointers to the parts of the documentation that might reference them. I wasted a week reading that documentation. There’s also different variations and different versions so the whole thing is really frustrating. All I wanted to do was find data between a date or time range. I liked the simplicity of the query builder and I can really see the power of this library but the documentation really turned me off.

    Finally, I came to my senses last night at about 11:30PM when I really should have been a sleep. Come to think of it, I should really be a sleep now as well but I want to get all this out of my head and on paper so to speak before I forget it. I came across This post on the MongoDB blog which made things very very clear. I had of course tried something very similar to that before I started looking for alternatives but really, it was so simple! All I was missing was converting the date into strToTime before I tried to convert it into MongoDate format. I did a lot of searching on Google but although I could find shed lodes of documentation on converting from MongoDate into PHP, I couldn’t find anything on the other way around. I obviously wasn’t looking in the right place because as soon as I saw those few letters strToTime, it all clicked.

    Here’s the example from the MongoDB blog:

    $start = new MongoDate(strtotime(‘1971-01-01 00:00:00’));
    $end = new MongoDate(strtotime(‘1999-12-31 23:59:59’));
    $collection->find(array(“create_date” => array(‘$gt’ => $start, ‘$lte’ => $end)));

    This actually converts the date and time into a number like this:

    1393545599

    Armed with this information, I set about dynamically setting the date and time. This code will get the sensor values saved to MongoDB over the past day:

    $start = new MongoDate(strtotime(date(“Y-m-d H:i:s”,”-1 days”)));
    $end = new MongoDate(strtotime(date(“Y-m-d H:i:s”)));

    See how easy that is? Isn’t that frustrating! I’ve spent about ten hours reading about this. Such a waste in a lot of ways but I suppose I probably learned plenty on my travels to finding out more about Mongo and the way it handles dates. Funny, in the collection, the date is stored in ISO format. For example: 2014-27-02 23:46:05. It must do some very interesting conversion back into a standard format. When I tried to check using the format that the date is stored in within the collection using (Y-m-d H:m:s) it failed to pull back any records. Maybe because MongoDate is trying to parse that from the expected strToTime number. That’s weird though because that wasn’t even working when I wasn’t using MongoDate. It’s a question I must ask on the forums when I eventually get around to creating an account.

    As you can see, I’m still learning and in a lot of ways this is really frustrating. I could probably do with reading a few books on these subjects but where’s the fun in that? I rather learn as I go along.


  • Progressing slowly with the arduino and the Raspberry Pi

    Work on the Arduino and the Raspberry Pi is ongoing. So far, I’ve made Led’s flash, used a light meter to determine when the LED’s are on and off, taken the temperature of the room and moved a camera using a Servo. On the Pi side, I’ve set up Email alerts when motion has been detected by the phone and I even found an application that supports push on the IOS platform so I may even be able to get the Pi to send alerts directly to my iPhone.

    I’ve encountered a few challenges of course. Almost all of the tutorials for the Arduino use a delay function to pause when the servo is running or when the motor is spinning but that’s no good when you need the loop to continue processing while all of this happens. So, I looked into a few alternatives. Using a counter to count the milliseconds since the device was turned on was fine but this would need to reset after 34 days which would cause a problem with the timing of the loop. I then tried a library called ElapsedMilis. This works fine but I had a lot of problems figuring out the logic. I got there in the end though but unfortunately my approach wasn’t completely sound.

    After asking a question on the Arduino forum I was pointed in the direction of another library called Delay Timer. I don’t think it’s released because when I included it first in one of my projects I had a bit of debugging to do to get it working. I must subscribe to Git Hub to suggest my changes. Unlike my previous approach, with this library, I can use as many timers as I want. The last time, I was using one hardware timer and a number of intervals to try to mark when functions should be executed. This should have worked fine in theory but the problem is that the loop is processed so quickly that it can run through the process too quickly and miss an important event. With this new approach, I can have different timers running concurrently and I can check each one. When it is time to execute that function, the timer resets again and its place in the loop is never lost.

    Of course, it’s not all programming. Some of this work is also about putting the components together and making it sit properly. One problem I was having is that the servo was generating too much vibration. This would cause the camera to constantly think it was detecting motion. I came up with the great idea of using a bit of Velcro to help mount the servo to the side of the Raspberry Pi and then Velcro the camera board to the moving part of the servo. Now, the vibration is absorbed by the servo. It’s funny the solutions that show themselves when you’re stuck.


  • A blind person can use the Arduino. Just about.

    Continuing on with my Raspberry Pi and Arduino experimentation, I’ve been trying to get a few minutes here and there over the past two days. Christmas is always a very relaxing but very busy time for me. I’m spending time with family and friends and I try to do very little with technology considering I spend so much time in front of a computer during the rest of the year. This year is very different. It’s still as busy as ever but when I get a free moment, I can’t wait to jump back into the RP. RP from here on in will mean Raspberry Pi in case you’re not sure.

    Firstly, I had a few questions sent by Email after the last post. All of these questions are asked on the Raspberry Vi website but I’ll briefly give you a few answers here.

    1. Is Speakup supported?
      Technically, Speakup is supported by Raspbian but issues with the sound card integrated with the GPU on the device have caused a lot of problems in relation to software synthesis from ESpeak and other synthesizers. It may be possible to connect something like an Apollo2 to the RP via serial to get that talking. If someone wants to send me over a cable that will connect to the pins on the RP I’d be happy to try this.
    2. Is there any accessibility in the graphical environment?
      Raspbian uses the LXDE Window Display Manager. Orca works with Gnome and the KDE screen reader has a long way to go. Theoretically if Orca is running in LXDE and you start an application written using GDK Orca will work quite happily. However, it is worth remembering that the RP is a low power device. Even with a really good memory card at 96MB read and 92MB write speed the performance will not be nice to work with. My card is 45 write 48 read. Or something close to that.
    3. Is there sound output?
      Yes. Through the line out port and through HDMI if your television supports this.

    I think it’s important to understand that the Raspberry Pi is not meant to be a high power device. It is primarily a tool that should promote learning by children. For someone like me, it’s a low cost device that is really useful for experimentation. If I break it, it’s not the end of the world. If I cause the system not to boot it won’t take long to fix.

    The great thing for me is I have kind of fallen away from using Linux all that actively at home. I use it on servers a lot but I rarely use it for playing with Python or trying out new packages or configs.

    I bought an Arduino a year ago with the best intentions of trying out a few things that I had heard about with the added aim of getting involved with the local maker space. Unfortunately to my annoyance, the Arduino IDE wasn’t accessible to me as a screen reader user. This meant that the kit that I bought has gone on unused for far too long. Thanks to a really brilliant little tool that I found while casually looking around last week, I now have access to compile and upload to my Arduino.

    This tool is called Ino. It is exceptionally easy to use. The basic commands are

    Ino init Creates a new project.
    Ino build Compiles and creates the make file for your project.
    Ino upload Uploads the compiled project.
    Ino serial Displays the serial output from the Arduino.

    You need the Arduino package installed first as well as Python easy_install. There are also a few dependencies listed in the requirements.txt file that will need to be installed before the tool will work.

    The great thing is that it allows complete control of the Arduino without ever using the IDE. So, it can all be done using the console via SSH.

    All I’ve done so far is use a few sensors and lights to read from the serial port just to get started. I’ve used a few loops, called functions, used include files and set up a few checks using if statements. Nothing too complicated but it’s giving me a starting point. Thanks of course to Emma who is helping me with the very visual side of the Arduino. It’s not possible for me to wire up the board so I need sighted assistance to add new components.

    Of course, you would probably get some nice debugging tools using the IDE and the output from ino build isn’t great when it encounters a problem but so far, I’ve been able to debug it manually.

    The only terrible thing about the RP and the Arduino is it’s quite addictive when you get stuck into a project. I need to remember that I’ve an 11 week old baby fighting for my attention as well.