• Tag Archives Linux
  • Using date ranges in MongoDB and PHP.

    I have also written similar posts relating to date ranges in Python. You can find my Question here from when I was getting frustrated and The answer to my problems here.

    You seriously wouldn’t believe the trouble I’ve had in the past two weeks trying to make some progress on my Arduino, Raspberry pi, Python, PHP and MongoDB project.

    Work has been very busy so the only time I’ve had to work on this is on the bus on the way to and from Dublin and from time to time very late at night.

    Right, so here are a few of the problems I came up against:
    The first hing I wanted to do was limit the size of my table in MongoDB. I’m collecting quite a lot of sensor data from the Arduino but I don’t need to retain this data for any more than around 2 weeks.

    MongoDB allows you to create a TTL index which will delete data that was created more than a certain number of seconds ago. This is a really handy feature however, it didn’t really work for me. I think you need to store the date in BSON format and I had stored my date in ISO format because I think it will make it easier to retrieve and write the sensor entries.

    Regardless, here is the code I used:

    db.envirocheck.sensors.ensureIndex( { “Date”: 1 }, { expireAfterSeconds: 604800 } )

    You can learn more about TTL indexes using the Fantastic MongoDB documentation that covers TTL Indexes

    As I said, this didn’t work for me at all so a suggestion on Twitter that I received weeks before made me think of capped collections. These are similar to TTL indexes in that they delete old data but instead of the TTL index, this works by deleting entries that are old however it does so when the collection reaches a certain value. By writing data once a second, I find that with 500Bytes I can store just over two hours of data. I obviously need to figure out how many bytes I need for storing two weeks worth but that’s something to do when I’m feeling more awake.

    The code to create a capped collection is here:

    db.createCollection( “sensors”, { capped: true, size: 500000 } )

    Again, look here for the MongoDB documentation for capped collections.

    Next, I of course needed to set an index on my date field as I’m going to be using this to select specific temperature values for date and time ranges. That was quite straight forward.

    db.sensors.ensureIndex( { “Date”: 1 } )

    Next, I needed to find a way of selecting between two dates in MongoDB and PHP. You might think this is easy, but no! It’s far from it! I stupidly tried to get ahead of myself by making this really complicated. I looked at Doctrine but trust me on this, the documentation for this project is absolutely crap! Now, maybe it’s me. Maybe I’m not experienced enough to figure this out but for god sake, this documentation might seem great from a high level but unless you read it from start to finish like a book, it’s useless! there’s no context to any of their examples and huge chunks of code are missing without any pointers to the parts of the documentation that might reference them. I wasted a week reading that documentation. There’s also different variations and different versions so the whole thing is really frustrating. All I wanted to do was find data between a date or time range. I liked the simplicity of the query builder and I can really see the power of this library but the documentation really turned me off.

    Finally, I came to my senses last night at about 11:30PM when I really should have been a sleep. Come to think of it, I should really be a sleep now as well but I want to get all this out of my head and on paper so to speak before I forget it. I came across This post on the MongoDB blog which made things very very clear. I had of course tried something very similar to that before I started looking for alternatives but really, it was so simple! All I was missing was converting the date into strToTime before I tried to convert it into MongoDate format. I did a lot of searching on Google but although I could find shed lodes of documentation on converting from MongoDate into PHP, I couldn’t find anything on the other way around. I obviously wasn’t looking in the right place because as soon as I saw those few letters strToTime, it all clicked.

    Here’s the example from the MongoDB blog:

    $start = new MongoDate(strtotime(‘1971-01-01 00:00:00’));
    $end = new MongoDate(strtotime(‘1999-12-31 23:59:59’));
    $collection->find(array(“create_date” => array(‘$gt’ => $start, ‘$lte’ => $end)));

    This actually converts the date and time into a number like this:

    1393545599

    Armed with this information, I set about dynamically setting the date and time. This code will get the sensor values saved to MongoDB over the past day:

    $start = new MongoDate(strtotime(date(“Y-m-d H:i:s”,”-1 days”)));
    $end = new MongoDate(strtotime(date(“Y-m-d H:i:s”)));

    See how easy that is? Isn’t that frustrating! I’ve spent about ten hours reading about this. Such a waste in a lot of ways but I suppose I probably learned plenty on my travels to finding out more about Mongo and the way it handles dates. Funny, in the collection, the date is stored in ISO format. For example: 2014-27-02 23:46:05. It must do some very interesting conversion back into a standard format. When I tried to check using the format that the date is stored in within the collection using (Y-m-d H:m:s) it failed to pull back any records. Maybe because MongoDate is trying to parse that from the expected strToTime number. That’s weird though because that wasn’t even working when I wasn’t using MongoDate. It’s a question I must ask on the forums when I eventually get around to creating an account.

    As you can see, I’m still learning and in a lot of ways this is really frustrating. I could probably do with reading a few books on these subjects but where’s the fun in that? I rather learn as I go along.


  • Using the Tilda terminal in Linux with full accessibility for Orca users.

    This post was origionally written on friday the 29th of February 2008 however over the past few years it got lost due to blog upgrades. Because I’ve noticed a few people looking for this information I thought it would be a good idea to post it again.

    Yesterday, I decided to play around with a package called Tilda.  Tilda is a graphical console for the Gnome desktop.  It runs on KDE as well but its GTK based.  The main advantage it gives is more bells and whistles for people who like visual effects.  No, I’m not in to visual effects for obvious reasons however I was curious and I like the speed that it launches at


    After installing it yesterday, I was very happy to see that Orca worked with it right away.  When I ran Tilda for the first time, I was given a configuration wizard screen.  Orca spoke all of the focusable objects as if they were made for each other.  In the terminal it’s self, flat review could be used to read the console as you would expect with any accessible application.  Only problem was that Orca didn’t automatically speak new text as it was written to the screen.


    To try to rectify the situation, armed only with my Windows screen reader knowledge and my curiosity, I renamed the gnome-terminal.py file to tilda.py.  That didn’t do anything for me.  However, thinking back, I wonder if it didn’t do anything for me because I didn’t restart Orca first before trying tilda again.  My thinking behind this attempt was that Windows screen readers such as Jaws versions before 7 and Window Eyes used a script or macro type function that was more or less tied to the executable of the application.  For example, if notepad.exe was run, Jaws / Window eyes would run the settings / scripts for that file if it found a file named notepad.jsb or notepad.001.   This has changed in later versions of Jaws and Window eyes however I assumed that it was possible that the logic was similar in Orca.    That didn’t work though so I sent a brief email to the Orca discussion list asking for their suggestions. 


    Rich Burridge, an Orca developer, took some time out of his busy day to help me.  With some research, he determined that Tilda actually used VTE (Virtual Terminal Emulator) This is also used by Gnome-Terminal and has a lot of accessibility support already.  This meant that it was probably fine to use the Gnome-terminal script as it would most likely behave the same.  Only one small change was required.  He suggested that I add a few short lines to my orca-customizations.py file.  Look at the end of this post for the specific code.
    I want to take this opportunity to describe to you how Gnome accessibility differs from that provided by windows screen readers as in Windows, just copying this script from one application to another expecting it to behave the same would be completely unheard of.  Windows screen readers provide accessibility in windows. In Linux”, it’s gnome that provides its own accessibility.  Orca takes advantage of this and provides output customized to ensure that users receive the information they need in a way they can understand.  That’s the short version.  Now for some description. 


    In windows, if you are using a screen reader like Jaws and an instant messaging program like MSN for example, Jaws needs to monitor very high level behavior.  I.e, it needs to track changes to the interface, read text from the status bar, monitor the entire conversation history area and a lot more.  It does this to ensure you hear status updates, incoming messages, Contact information and of course, at times, it needs to keep track of your own actions so it can tell you where you are in any given window.  Most of this information is obtained by analyzing the interface.  Only a very small percentage of what Jaws gets from windows is obtained from information that the application or operating system gives it.  In other words, MSN does not communicate with Jaws to tell it that a new message has arrived.  Jaws determines this by watching for changes on the screen.


    Gnome on the other hand is completely different.  It provides assistive software such as the Orca screen reader with information so that it can relay this to the user.  In the gnome messaging client, pidgin, Orca is informed when a new message is sent to the message history window.   It then has events determined by scripts to tell it what to do with this information.  So, it doesn’t matter how you have pidgin configured, it will still send this information to Orca which in turn will relay it to the user.  So, bringing it back to the terminal, it doesn’t matter that Gnome Terminal is completely different to Tilda.  Tilda uses different colors, different positioning and a lot of eye candy.  It really doesn’t matter though as it utilizes this VTE that provides the required accessibility information to Orca! 


    I should also say here that although my description of the differences between how Windows and Gnome behaves should be accurate, I can’t say it with full certainty.  I’m not a developer and if you are really interested in the low level workings of the Gnome window manager and how it provides accessibility, I’d suggest you look into subscribing to the orca mailing list.


    That’s all the background and descriptions out of the way.  If you’re interested in getting up and running with tilda and Orca, use the following instructions:




    1. Go into a terminal.

      1. Press alt and F2 when in the Gnome desktop.
      2. Type gnome-terminal
      3. Press enter.

    2. Install the tilda terminal.

      1. Type apt-get install tilda
      2. Press the enter key.  When prompted to confirm the package download and installation, type the letter y and again, press enter.
      3. Exit the terminal window.

    3. Instruct orca to run the gnome-terminal.py script when you run tilda.

      1. Press alt f2 to start the run dialogue box.
      2. Type gedit then press enter.
      3. Paste the below code into the editor.

        import re
        import orca.settings
        from orca.orca_i18n import _
        orca.settings.setScriptMapping(re.compile(_(’tilda’)), “gnome-terminal”)

      4. Save the document by pressing control and s.
      5. Exit gedit by pressing alt and f4.

    4. Run the tilda terminal.

      1. Press alt f2 to start the run dialogue box.
      2. Type tilda and press enter.


    You’re done.  You are now in the tilda configuration screen.  Configure the package to your own preferences then use the ok button to save your changes and start the tilda terminal.  This wizard will not be shown automatically again when you run tilda.  To bring up the wizard, type tilda –C in the launch application dialogue box accessible with Alt F2.


    I think that should be clear enough.  Any problems or questions feel free to leave a comment and I’ll try to get to them.


    My thanks to Rich Burridge who so generously helped with this.  Without his help I’d probably be working at this still.


  • Installing QNotifier under Debian Linux.

    QNotifier seems to be a cool application that can report changes that may impact the performance of your Linux server right to your iPhone. If for example, the apache daemon goes down, QNotifier can be configured to notify you of this.

    Installing QNotifier is a little tricky as the installation instructions are very brief.

    the following commands will get QNotifier running on your Debian system. I assume this will most likely also run on Ubuntu.

    Install Ruby.

    apt-get update
    apt-get install ruby ruby1.8-dev libzlib-ruby rdoc irb libopenssl-ruby rubygens

    Install the update to Rubygens.

    gem install rubygems-update
    cd /var/lib/gems/1.8/bin
    ./update_rubygems

    Update ruby dependencies for Qnotifier.

    gem install hoe rexical racc rubyforge nokogiri archive-tar-minitar sup rails –include-dependencies

    Install the QNotifier package.

    gem install qnotifier

    Run the package to set it up. Get the registration key from the iPhone.

    /var/lib/gems/1.8/bin/qnotifier

    There are configuration files that can be tweeked to make the package report different information and these config packages are refereced in the short installation guide available from the QNotifier package after you install it to your iPhone.


  • using RSA or DSA for authentication to a Linux server via SSH or SCP.

    Following on from my post yesterday about backups, I thought I’d give a further explination as to how to copy down the archives that I created in the script.

    For this, I’m using SCP. However, if using SCP, you ordinarily need to log on.

    If your prompted for a username and password every time your script runs an scp command, it’s kind of pointless having cron run the script at all.

    So, to get around the requirement to log in, while at the same time keeping the set up secure, we use an RSA or DSA key.

    for the rest of this post, I’m going to call the machines backup and server. The backup is the machine I am copying the backup files to.

    On the backup machine, type the following commands to generate the files and copy the public file across to the server. I suggest you use a very restricted account on the backup and server for this.

    ssh-keygen -t rsa
    hit enter for the first question to agree to save the key to /home/YourUserName/.ssh/id_rsa
    Hit enter without typing anything for the second and third questions as we don’t want a password for this particular key. Note, this is usually not recommended but it should be ok for this type of situation.
    It will tell you that a public and private key has been created and it will give you the finger print of the newly created key as well.

    Next, you will want to copy the public key across to your server. Note, the server is the machine that hosts your backup scripts.
    scp .ssh/id_rsa.pub YourUserName@ServerName:.ssh/

    If this is the first time you’ve used a public key then use the following command as it will make things easier for you.
    scp .ssh/id_rsa.pub YourUserName@ServerName:.ssh/authorized_keys

    If however you have used other keys, do the following:
    ssh YourUserName@ServerAddress

    Type your username and password to log in.

    Now, type the following to append the id_rsa.pub to the authorized_keys file.
    echo .ssh/id_rsa.pub >> .ssh/authorized_keys

    Now, leave the ssh session by typing exit.

    From the backup machine, you can now log in via ssh without providing a password.

    Note!!!

    You might want to secure your public key. If it goes missing, this could go very very baddly for you as this key does not require a password.

    Log into the server by typing:
    ssh YourUserName:ServerAddress

    Now, change the permissions of the file so that this restricted user account is the only one with read and write access to the public key
    chmod 600 .ssh/authorized_keys

    Now, get out of the ssh session by typing exit.

    The next step will be running scp to download your backups and verify that their readable. If their not, we’ll want to log the failure.


  • GTK v QT.

    This is a response to my Blog post about Linux accessibility. I wrote it in response to another comment. It turned into a bit of a long one though and the information is applicable to a lot of things. So, have a read of this:

    KDE is based on QT and Gnome is based on GTK. It’s important to recognise the differences in these environments from the start. QT doesn’t support AT-SPI. It was planned to support this but it never happened. This is down to a decision by the QT developers.
    Here’s an article that discusses how Gnome communicates with Orca. http://accessibility.kde.org/developer/atk.php

    There’s information about the accessibility of QT packages at http://doc.trolltech.com/4.5/qt4-accessibility.html

    GTK is not the only standard for the graphical desktop manager in Linux. QT is just as much of a player. That’s where the problem comes in. Any application that is written for Gnome and follows the Gnome development guidelines will communicate the necessary information to Orca. The problem arises when you launch a QT based application such as Acrobat reader as Orca doesn’t get the required information from it.

    I understand your point about the limited progress in some areas due to developers not making enough progress with their compliants with the accessibility guidelines but tell me one platform where that isn’t an issue. It’s nothing to do with communication. It’s a problem with bad coding. Simple.

    In relation to the problems you have encountered accessing applications that run as root in Ubuntu, there are a number of well documented fixes to this. Basically, Orca, like every other Linux application can not obtain information from a process running as a more privlidged user. This is the reason why Linux is such a secure platform. However, there are work arounds to this as I said earlier. In fact, Listen to one of the recordings in the Linux section of www.lalrecordings.com and you will hear one of the ways of getting around this. In Vinux, the environment is preconfigured for optomal accessibility. Perhaps you should try this out? You might find that with the environment configured to provide the most accessibility, most of your issues are resolved. If nothing else, it will illustrate to you that on most distributions of Linux running Gnome, problems can be ironed out if your willing to spend some time on configuration.

    I’d also suggest you read http://live.gnome.org/Orca as I’m almost certain that you’ll find an alternative and accessible PDF reader. As for skype, use the plug in for Pidgin. I would bet that you will not find an application that doesn’t have a gTK counterpart in LInux. There are very few things in Linux that you will not be able to access if you keep at it. Just like the same can be said in windows or even OSX if your that way inclined.