I have a confession … I am not an “ideas” person
This post is a mix of me blathering on, bits of dotnet core C# code, bits of swift code and a final conclusion… if you just want to skip to the end for the tl-dr click here.
A lot of people’s views of software developers are potential Mark Zuckerbergs sitting at home on a myriad of amazing ideas just waiting to make their millions. I know a few guys who are just full of ideas, some of them are great too. I have NEVER had any of those. Almost any ideas I have are either much too expensive to get off the ground “just for fun” and the rest have usually been done before, and better than I thought they would be… so I usually just sit there pulling my hair out trying to think of something to do.
So recently I really decided that I wanted to upgrade my 27″ 1080p monitor to a 34″ 1440p curved widescreen… I was looking about eBay and Amazon trying to find something that wouldn’t break the bank, but would still be good quality. It’s not for games, it’s for development, some minor photo/video editing and general usage. There are a load of great options out there, but I really liked Dell’s offering. Currently Dell have 3 ultrawides in their Ultrasharp Range (U3415W, U3417W and the newest U3419W) I read a load of reviews, and the cheapest one, the U3415W, got great write ups, and was only a few years old. Bingo.
But at £675 that was a bit steep… I looked into some cheaper options but I would have ended up with something sub par and decided just to leave it…. until one day I glanced at the product and it was £313! … better than half price! AMAZING! I thought about it for a few minutes, chatted to my wife and decided to go for it… by the time I got back upstairs, it was back to £675. The amazing deal must have been too good to be true, or I missed it.
(I am getting to the point of this post soon, stay with me)
This happened a few times over the next few days, it dropped in price, and I had literally a few minutes before it shot back up again, and I always missed it. I started to get a bit annoyed at this and decided to do a quick search for a tool that monitors the prices of items on amazon… I came up with only one real option, camelcamelcamel a site which didn’t look great, and I couldn’t see any easy way to get notifications immediately (I believe they do email/tweets, but that’s not something that I check immediately and wasn’t really what I was after) so instead of looking for something else, or manually checking, I decided that I was going to have a bit of fun and write something to do the hard work for me.
I dusted off some fairly old code from a web scraper that used xpath to grab information of web pages (that hopefully don’t change much) and modified it to work with the price element of an amazon page. The basic code for scraping the markup is at the end of this post. Give it a URL, it gets the web page, rips through the markup, gets the price and returns it. Really really simple stuff, hard coded to the current page structure, but it works.
At least it did until today, where you need to change the xpath to this as the item is out of stock and only available via resellers.
var nodes = document.DocumentNode.SelectNodes( "/html/body/div[@id='a-page'][1]" +"/div[@id='dp'][1]" +"/div[@id='dp-container'][1]" +"/div[@id='centerCol'][1]" +"/div[@id='olp_feature_div'][1]" +"/div[1]" +"/span[2]" +"/span[1]");
I wrapped this up a bit in a console app, stuck it on a 30 minute timer and left it running.
Worked a treat, I just had to pop my head into the room and I could see what price it was currently at, and what it had been (to see if I could predict when it would next get cheap) … spoiler alert, there is no pattern.
I wanted to be able to dig into the data after the fact, so I added in some basic POCO objects for an item, an item price, did a one liner to use entity framework core with SQLite and bang, my little console app had a database tracking and recording when item prices changed.
NB: Before this works, you need to run a quick script to create the database…
#Run these to create database dotnet ef migrations add InitialCreate dotnet ef database update
My little ‘I want to see when that monitor is cheap’ application was now recording item prices if and when they changed. Great stuff. But that’s not great if I’m not near the computer when the price changes.
I have an iPhone, all this is happening on a Mac, why not write a shell application that can receive push notifications so that when the monitor changes price, it will pop up on my phone within 30 minutes and from there I can decide if I want to buy it or not through the amazon app.
A quick google turns up a dotnet core push notification library called PushSharpCore ….. a quick little
dotnet add package PushSharp.Core --version 1.0.0
in my project folder and we are good to go. I almost literally copy pasted their sample code, this doesn’t have to be fancy, I need to send (up to) one notification every 30 mins, it doesn’t have to be perfect.
I knocked up the world’s simplest (and I mean simple… no icon, one white screen, nothing else) iOS app that can receive notifications and deployed it to my phone directly using xcode.
Make your AppDelegate.swift a UNUserNotificationCenterDelegate and add these overloads and you are good to go (once you jump through Apple’s hoops with certificates and provisioning profiles for push notifications)
and that’s about it, we were live and monitoring!
The price jumped up and down by £5 a few times a day for a few days… then suddenly..
Bazinga! We have a winner! I jumped onto the official iPhone app to confirm the price, and a new seller had dumped a load of cheap ones on, I grabbed one at £318 and was one happy developer. Within 2 hours (I know because I got another notification) the price had gone back up and hasn’t dropped as low since.
In conclusion
I really enjoyed playing with this little project, it didn’t take too long and it served a very real purpose for me. My real takeaway from this is that good ideas don’t have to be complicated or expensive, and it doesn’t even matter if others are doing something similar. Good software doesn’t have to be hosted in the cloud running in microservices, adhering to patterns and practices with dependency injection and unit tests… it just has to serve a purpose, and do that to its best ability.
I think I will eventually flesh this out a bit, it’s written in dotnet core, so maybe I will wrap it up a bit more robustly, add tests, and leave it running on a raspberry pi keeping an eye on bits and bobs I am interested in.
If I find it working well and being useful, I will come up with a better way of grabbing the prices off amazon using their API or a more reliable way of parsing the DOM. I will probably flesh out the iPhone app so I can copy item URLs from Amazon and add them for monitoring via my phone, keeping a list of thier current, cheapest and most expensive prices. Hell if I really go crazy I will push the server app up into the cloud, I’ll release a more robust version of the app on the app store and let other people play with and maybe make a few quid from generating amazon referral links and small ads….
But regardless of where I go with this, my real takeaway is that software doesn’t have to start off massive, and complete, it can just be a console app that solves a problem for you, and if it does that, then in my mind, it’s good software.
Sample web scraper
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| using System; | |
| using System.IO; | |
| using System.Linq; | |
| using System.Threading.Tasks; | |
| using HtmlAgilityPack; | |
| namespace AmazonPriceCheck | |
| { | |
| public class AmazonUkStoreParser : IHtmlParser | |
| { | |
| public string GetItemCodeFromUrl(string url) | |
| { | |
| var itemCode = url.Substring(url.IndexOf("/dp/") + 4); | |
| itemCode = itemCode.Substring(0, itemCode.IndexOf("/")); | |
| return itemCode; | |
| } | |
| public decimal GetPriceForUrl(string url) | |
| { | |
| decimal thePrice = -1; | |
| if (string.IsNullOrEmpty(url)) | |
| { | |
| return -1; | |
| } | |
| if (!url.Contains(".co.uk")) | |
| { | |
| return -1; | |
| } | |
| if (!url.Contains("amazon.co.uk")) | |
| { | |
| return -1; | |
| } | |
| var pageString = HttpUtils.GetHtmlFromUrl(url); | |
| thePrice = ParseDocumentForPrice(pageString); | |
| return thePrice; | |
| } | |
| private decimal ParseDocumentForPrice(string pageString) | |
| { | |
| var document = new HtmlDocument(); | |
| document.LoadHtml(pageString); | |
| var nodes = document.DocumentNode.SelectNodes( | |
| "/html/body/div[@id='a-page'][1]" | |
| + "/div[@id='dp'][1]" | |
| + "/div[@id='dp-container'][1]" | |
| + "/div[@id='centerCol'][1]" | |
| + "/div[@id='desktop_unifiedPrice'][1]" | |
| + "/div[@id='unifiedPrice_feature_div'][1]" | |
| + "/div[@id='price'][1]" | |
| + "/table" | |
| + "/tr[@id='priceblock_ourprice_row'][1]" | |
| + "/td[2]" | |
| + "/span[@id='priceblock_ourprice']"); | |
| var formattedNode = nodes.First().InnerHtml; | |
| formattedNode = formattedNode.Replace(" ", ""); | |
| formattedNode = formattedNode.Replace("\n", ""); | |
| formattedNode = formattedNode.Replace("£", ""); | |
| decimal price = -1; | |
| Decimal.TryParse(formattedNode, out price); | |
| return price; | |
| } | |
| } | |
| } |
Mark McGookin 
Hey Mark. Interesting thoughts – recognise a lot in this! Love cobbling together a quick & dirty solution that just solves something specific. Doesn’t need to be shiny. I made one a couple of years ago that scans my webserver for hacked files, if it finds anything it updates an RSS feed, then I use the IFTTT app to monitor the RSS and give me a notification if the feed is updated. So I’m alerted instantly if my server’s compromised. Still works! Out of interest, why only 30 min updates, why not every 1 or 2 mins? Fear of being blocked?
LikeLike
Yeah the original application scraped App Stores for information on new apps and developers. It hit them pretty hard and got a lot of 403s. I could route this through tor and sip a new route every time I got a 403 but I just decided to have it check every now and then, so that if I add more products there still wouldn’t be a risk of amazon flagging it.
LikeLiked by 1 person