Python, selenium, and Windows...yes Windows.

I’ve spent much of the day hacking out some Python code to automate some functions…some of the functions rely on javascript that is only executed by actually pulling up a web page and clicking a few buttons.

I could (and might still) have decided to rewrite the javascript into Python so that I could just write a full on Python script to handle everything that was going on, and then just schedule a cron job to run the script every day…and be done with it.

However, the data structure behind the scenes for this particular process is fairly complex (multi-dimensional array)…and the javascript that get’s applied is also fairly complex (it basically applies various formulas depending on the various states of the data and what users click)…in a lot of ways you can think of the entire process like a poor mans, web-based excel…customized to solve some specific business problems.

So while I *could* rewrite it all into pure Python…it was going to be about the equiv. of rewriting it all from scratch…and was going to take me much much longer than I had time to do today.

So I needed to figure out another solution…and when I stepped back to think about the real problem…all I needed was for a computer to do exactly what the user has been doing up until now…open a browser and click some buttons.

Enter Selenium. Selenium is designed to help automate testing and Q/A tasks, but with a little hacking and tweaking you can really make it do just about anything.

In fact in about 50 lines of code, I had a perfectly working system on my local machine (awesome). Now the only challenge was to move this to a server and set it to run automatically each day.

Well the challenge there turns out to be that my code wanted to control a web browser…and to do that it needed to have a GUI environment with a web browser actually installed.

Now there are options out there that allow you to mimic a GUI with programs installed…but I am not *that* familiar with them and again didn’t want to spend too much of my time figuring that part out today either.

So I opt'ed to go with setting up a Windows Server on EC2 (they are dirt cheap these days after-all).

This of course came with a handful of it’s own challenges…first I hadn’t set up a Windows server in probably over two years now…and of course in that time, everything has changed.

Stack Overflow once again got me through all the road bumps (how to set a Path variable, how to install pip on Windows, how to get python listed in the registry, etc.).

BTW Stack has saved me countless hours over the years - such an awesome knowledge base.

Anyway - after a few hours of searching, clicking, and hacking (along with many heavy sighs of annoyance)…the system is up and running like a charm for us.

Yes, it’s a pure hack and I’ll probably replace it down the road…but it solved our immediate problem and should hold up just fine until I have more time/interest/need to replace it.

…and that’s what a day in my ‘hacker shoes’ is all about…

This post has received 46 loves.


ARCHIVE OF POSTS



This is the personal blog of Kevin Marshall (a.k.a Falicon) where he often digs into side projects he's working on for digdownlabs.com and other random thoughts he's got on his mind.

Kevin has a day job as CTO of Veritonic and is spending nights & weekends hacking on Share Game Tape. You can also check out some of his open source code on GitHub or connect with him on Twitter @falicon or via email at kevin at falicon.com.

If you have comments, thoughts, or want to respond to something you see here I would encourage you to respond via a post on your own blog (and then let me know about the link via one of the routes mentioned above).