FluxPlant- Cloud Based Water and Consumption Monitoring System
Introduction:
My project goal was to design a system that can track your home water usage and carbon footprint, and display the data for the average consumer in a useful way. We in California are experiencing a nasty drought, and when users want to make a change in their consumption habits, they find that their monthly water bill only displays an aggregated total of the month’s total water usage, which does not help in identifying the specific problem and area in which water is being consumed excessively. To help the alleviate this issue, I set out to create a system that can really help you with your water consumption.
After 6 painful weeks, I ended up with a system that contains four main components: a Bluetooth enabled sensor, a home unit for collected sensor data, a server, and a web app. I was able to successfully send sensor data to my server and display various analytics from the collected information. The server was designed with a modular mindset, meaning that the data it collects wont be tied to only “Milliliters Used” or “Temperature”. Users will be able to add any type of data, and this system would be able to be used with any sensor.
After BSE, I will be continuing development of the project, looking non invasive methods of measuring the water flow in a pipe, and linking that to my server and software. This will be beneficial for the average consumer as it can be easily installed and can give them comparative data on their usage.
Creating this project was definitely an experience to be remembered. In these six weeks, I learned an enormous amount from Arduino development, to web server design and data modeling. Cycling through the many obstacles that I faced gave me a lot of perspective on the development process, and made me even more interested in pursuing a career in engineering later on. Read ahead to see more details about my project and some of the challenges I faced during this process.
Final Project: Connecting everything and building User Interface
My final milestone was to bring all of the components of my project together and design a full functioning user interface for consumers to be able to view their data and pin point exactly what areas of their house need improvement. At this point I also completed my project with a final system that I am happy with. Here are all of the components of my final project and how each of them work. Click here to a link to my repository containing all of the source code for the items below.
Arduino Device
In the final stretch of my project, I changed a lot of elements with my Arduino module. Initially I had it step up with a WiFi shield to be able to send data directly over the web to whatever server I chose to use. However this ended up causing problems for me. For one the Arduino WiFi Shield was not capable of making SSL connections to my server, which proved problematic as most of my server commands needed to be SSL based. Also the WiFi shield began to exhibit explainable issues; random connection issues, occasional resets of the Arduino. Finally I realized that as a consumer product, this would anyways be impractical as connecting multiple devices to a user’s WiFi would at times go over the user’s router’s limit, and in general be cluttered and chaotic. To solve this issue I decided to move the submitting of data to a chrome toolkit that I would design, and just have the Arduino device record data and send it to the toolkit in CSV format for processing. Below is the final schematic for the Arduino Module:
The code basically will wait for a command to start the measurement process and then stop it just for testing purposes. Once started, the device will then start dumping the data over the Bluetooth connection.
Chrome Toolkit
Now that the sensor is sending data over bluetooth, I need to be able to receive that data somehow and be able to parse it into a JSON format and send it over the web. To do this I needed to create a desktop application that can both send web request and can have Bluetooth capabilities. Fortunately, chrome has the capability of writing packaged desktop apps that can do both actions and can even be written in HTML5, CSS, and Javascript, languages that I already know how to use. Using the Chrome API, I was able to quickly create a desktop application that scans bluetooth devices, connects to them, is able to send commands to these devices, and more importantly is able to receive data from them. Once the user selects a device, the Chrome API will connect and show a status code as shown here:
After connecting, for this specific Arduino module, you can send it a command to start, which will then give it the green light to start sending back data:
The data comes in an Array Buffer form which is converted to a String and concatenated to a running response String:
After sending the command to stop, and pressing the send to server button, the tool-kit uses a library called Papa Parse to convert the data to a JSON object, and then send it to the server. The id of the sensor is already pre-entered in the program, along with data type and other variable required for the API call:
Once you see a Success! message, the data has been sent to the server under the Sensor’s Session entry.
Server
The Server is the base of the application and the system that manages all of the User and Sensor data, and how exactly each information should be displayed. Between the last Milestone and now, I have added functionality to be able to store location and names of sensors, and track User Token Usage in my database.
On the handler side however, I added routes for Users to request the various different partials and pages that the webpage might require. The request sensor method allows the user to get a log of all the sessions a particular sensor has. This could be combined with the list of each sensor a User to get a sum of all of the User’s data.
The API handlers allow for basic CRUD operations on sensors which allows for testing and data submission via any external application.
The separate device path allows for non-SSL based devices to perform the same CRUD operations.
User Interface
The user interface is a web application designed for users to be able to view all of their sensor data. It is built on HTML,LESS/ CSS, and jQuery/Javascript which communicates with the server with AJAX commands. As a base for the design, I used the Admin LTE dashboard template to expedite the design process and use predesigned UI elements.
When you go here, you arrive at the login page of the website where currently you can only sign in with google plus.
The web app contains two main pages, the Dashboard and the Sensors page. The Dashboard page displays to the users their daily usage statistics. It includes graphics that shows them for example how many bottles of water they have used in a day.
Milestone 2: Create Server and Database Structure along with API Functions
My second milestone goal was to create a back end for my water monitoring system. During this time, I learned how to use the Google App Engine Platform, a managed platform that allows developers to create applications without dealing with lower level aspects of web development such as load balancing, scaling, and instance creation. Here are the components of my server so far:
Data Model
With Google App Engine, I am using their NDB API for interacting with the Google Datastore, a NoSQL schema less database. The main structure of the database is an Entity, Entity Kind, and Property model. I have three Entity Kinds: Users, Sensors, and Sessions. Sessions have Sensors as ancestors, and Users stores an array of Sensors. Each user contains various general profile information along with the Sensor Array. Each Sensor contains various sensor information such as IP address, MAC address, Sensor type, Name, and Location. Its most important property however is the Consumer Id, and ID which is randomly generated on creation which is used by users to submit data to it. The Sessions entity kind contains the actual sensor data that is logged for a given Consumer Id. Here you can store the mL used, the average temperature of the water, and the time the session was submitted.
Logic
The first part of my project is the logic of the server. Here I created functions for all of the type of CRUD operations that would be needed for my app. For sensors, I created a function that reads REST Web Requests and either takes aggregated data or a raw data set that gets aggregated by the server.
This is designed specifically for the sensor I’m working with, but I will later be just collecting attributes and storing them via the expando db model so the data won’t be restricted to just what is identified by the server. My other main CRUD operation is the ability to create sensors in the database and read out their data. The logic takes REST web requests with options such as IP address, MAC address, and sensor type, and stores in the datastore. The server also creates a random consumer id based on the entity key so that consumers can use that id as token to submit data to the server, a sort of “email address” for the sensor. When a user wants to view all the data for a sensor, he or she can make a web request with the sensor id that will return a JSON data-set of all the sessions and the data recorded in that session.
Authentication
My application will also contain users that can have sensors in their profile. The user data model is already created, however now I have to also be able authenticate users. To do this I used a utility called SimpleAuth (Link here) which allows me to manage sessions and authentication tokens. When a request is made to view your profile, or even user data, the logic gets the authentication token and verifies it with the database before serving the data. The authentication process is designed so that any third party OAuth provider can be linked to the user; this grants me a larger user base. In the future this can be expanded to take usernames and passwords to create my own native users by implementing a salting and hashing scheme such as PBKDF2.
API
Finally my server also includes a set of API handlers that can access all of the CRUD operations above. This allows other applications and devices to submit and manipulate data, but its primary function is to allow my website to be able to send AJAX requests to get webpages and various other data.
Issues
- Google App Engine Java Platform
- The first main issue I encountered during this milestone was learning the platform itself. Google app engine supports four main languages: Python, Java, Go, and PHP. Since I was primarily comfortable with Java, I started with Google App Engine’s Java framework. This however proved to be extremely complicated as it dealt with a lot of lower level functionality that needed to be configured just to handle simple requests. Due to all of the frameworks and libraries that I had to learn in Java, I decided to switch to python in hope for an easier environment. This was extremely beneficial as Google App Engine’s Python Framework is very easy to use and implement and it let me focus more on the brains of the app instead of the structure.
- SSL Capabilities
- Another major issue I ran into is that most applications and clients, including the Arduino, do not have SSL capabilities. Google’s cloud endpoint service provides API functions, however they can only use SSL connections. This was a huge issue as it inhibited my Arduino from sending data to my server. To solve this issue I started using a framework called WebApp2 on Google App Engine which allows me to create custom paths that do not require SSL.
- Data Modeling
- Another issue that I ran into was structuring my database for this project. I wanted to create a database that can store large data sets and describe relationships between data sets, sensors, and Users. My first option was to use a service called Google BigQuery, which allows for the storage of large datasets which you can easily query in just seconds. This service, however cost money. So instead I modeled my data with Sessions, Sensors, and Users, for which Sensors were parents of Sessions, and Users had a “has a” relationship with the Sensors. Also instead storing large amounts of raw data, I created a function that sifts through the POSTed data and aggregates it before storing in the database.
Milestone 1: Create Arduino module that can record data and send over the web.
For my first Milestone I aimed to write and test Arduino code to record and store water usage and temperature data from my Flow Rate Sensor and Temperature sensor. After the measurement cycle is complete, I wanted to send that data over the web using an HTTP POST. I employed the use of a test server that I found online called Henry’s HTTP Post Test Server, a server which you can post data to and get back a link that will display your data online.
Here is my schematic:
Program:
The program begins by initializing all of the components present on the board, from the temperature sensor, SD card, Flow Rate Sensor, and even the Wifi Shield itself. After initialization, the Wifi Shield searches the given SSID and then joins it with the given password. After joining the network, the Arduino requests the DHCP to obtain the devices IP Address, and also obtains the IP address of the server that we will be using to dump data to.
After initialization the program waits for a button (An extra button soldered on to the prototyping space on the Wifi Shield) to be pressed to start the measurements process. Once it begins, the Arduino logs the data in a JSON format (A format suited for web applications to read data) on the SD card. When the button is pressed once again, the measurement process stops, and an HTTP request is generated with all of the necessary headers along with the post body which is the data we logged with the sensors. After the request is built and stored in a file on the SD card, the WiFi Shield then opens a socket to the IP address of the server that we will be dumping our data to (Henry’s Server). Once the socket is open, the Arduino writes the HTTP request character by character to the server, and once that is complete, listens for a response from Henry’s Server. Once successful, the server will return a link on his website that will contain all of the data that we had recorded from the sensors on the Arduino.
Click here to view my code.
Issues:
- Program too large for Arduino Uno
- This was one of the biggest issues I had faced. After bringing in the libraries for the two different sensors and the shield, I was left with very little space on the Uno for all of the sketch logic. I struggled with this for a while, trying to shave off features and simplify, however the project was becoming something that I didn’t want. So I decided to move to the Arduino Mega. This gave me an immense amount of space to store as big of a sketch as I wanted to, and I was able to keep all of the features I wanted.
- HTTP request not having enough headers
- During this process, I had my first encounter with HTTP requests. When requests are sent with data, the requests are required to have headers that summarize the data and give instructions for the server on how to process the data. While dumping to Henry’s server, my Arduino would return a 500 server error. I realized that I needed to add a header describing how long the content is. While only one line of code was needed to implement this, it took me four days to debug this issue.
- JSON String being too big for program memory
- As the Arduino collects sensor data, it stores the data in JSON format, a format that is used when sending web request. However as the sketch parses the data, the program memory fills up and past a certain point cannot handle the size. With large measurement sessions this was an issue as they would always require large strings to be stored in the program data. Instead of storing in the programming memory, I enabled SD card capability so I can write the JSON object onto an SD .txt file, and when ready to dump the data to server, I can iterate a pointer over the file and read out each character to the server.
Starter Project: ‘Simon Says’
For my starter project, I chose to build the simon says game module. The game has a “repeat after me” format complete with lights and sound. The board includes an IC micro-controller, a buzzer, LEDs, a couple of capacitors and resistors. The resistors regulate the voltage from the batteries which powers the IC. The IC is loaded with a firmware that turns the LEDs on and off while sounding the buzzer. This is possible due to the capacitors which store charge and release it to allow the flashing of the LEDs and buzzing noises.