Introduction
Here in South Texas, outside water usage is dictated by the water level in the Edwards Aquifer, "a unique groundwater system and one of the most prolific artesian aquifers in the world" (https://en.wikipedia.org/wiki/Edwards_Aquifer). Home owners can only water their grass on certain days and times depending on the water restriction stage. The water restriction stage is determined by the level in the Edwards Aquifer. Instead of visiting the water level website everyday, I wanted to play around with the Twitter API to scrape the aquifer water level data, then tweet it out from the @edwardsaquabot account!
Logging
I used this project to get familiar with the Python logging
module that is used to print to the console as well as a
.log file.
# Setup logging
logFormatter = logging.Formatter("%(asctime)s [%(levelname)s] %(message)s")
rootLogger = logging.getLogger()
if __name__ == "__main__":
rootLogger.setLevel(20) # INFO
# Log file handling
fileHandler = logging.FileHandler(__file__.strip(".py") + "_" + log_timestamp() + "_PID-" + str(os.getpid()) + ".log")
fileHandler.setFormatter(logFormatter)
rootLogger.addHandler(fileHandler)
# Console logging
consoleHandler = logging.StreamHandler()
consoleHandler.setFormatter(logFormatter)
rootLogger.addHandler(consoleHandler)
Fetching Water Levels
The water levels can be found here, http://data.edwardsaquifer.org/j-17.php,
so that page is scraped using BeautifulSoup, a HTML parser. The fetch_level
function retrieves the relevant
information:
def fetch_level(self):
# Use beautiful soup to grab the levels...works, maybe not the best though
rootLogger.info("[*] Fetching water levels...")
page = urllib2.urlopen(self.url)
soup = BeautifulSoup(page.read())
table = soup.find_all('table')[1]
# Today's Reading
column = table.find_all('td')[0]
todayWaterLevel = column.find('span').contents[0].strip()
# Yesterday's Reading
column = table.find_all('td')[2]
yesterdayWaterLevel = column.find('span').contents[0].strip()
# 10 Day Average Reading
column = table.find_all('td')[4]
tenDayAverage = column.find('span').contents[0].strip()
return todayWaterLevel, yesterdayWaterLevel, tenDayAverage
The TwitterAPI Python module loads the account details, authenticates, and
posts the results. The first step is to parse the credentials in the TwitterAPI_creds.ini file using ConfigParser.
Note that the TwitterAPI_creds.ini
in the code must be populated with your specific API credentials.
def parse_twitter_creds(self):
parser = SafeConfigParser()
parser.read('TwitterAPI_creds.ini')
self.consumerKey = parser.get('creds', 'consumerKey')
self.consumerSecret = parser.get('creds', 'consumerSecret')
self.accessToken = parser.get('creds', 'accessToken')
self.accessTokenSecret = parser.get('creds', 'accessTokenSecret')
Post the information to Twitter.
def post_tweet(self, message):
twitter = TwitterAPI(self.consumerKey, self.consumerSecret, self.accessToken, self.accessTokenSecret)
request = twitter.request('statuses/update', {'status': message})
statusCode = request.status_code
if statusCode == 200:
rootLogger.info("Successfully tweeted: {0}".format(message))
else:
rootLogger.error("HTTP status code: {0} -- unsuccessfully tweeted: {1}".format(statusCode, message))
Conclusion
All of the code can be found on the Opsdisk Github repository here: https://github.com/opsdisk/aquabot. Comments, suggestions, and improvements are always welcome. Be sure to follow @opsdisk on Twitter for the latest updates.