Using Selenium & Beautiful Soup to Scrape AJAX Sites

I don’t really work in Python very often but I worked on a recent project for a dynamic scraper for a hefty list of domains. Besides the normal challenges of using an unfamiliar programming language, I ran into a series of problems I don’t usually have with JavaScript. These problems were with Ajax. I would use Chrome do my inspections and all of my jQuery would work. I’d check it all out and it worked but none the data from those css selectors were available in Beautiful Soup. After some searching around I found Selenium which was a life saver.

Selenium web driver opens up a browser with a timeout long enough for the Ajax to load and then saves the HTML. From there I save the HTLM doc as a BeautifulSoup object and CSS select my life away.

In this example I just wanted to grab the first image from the carousel in this site:

image



Code: 

Dealing with setbacks


I am a cyclist. Not in the competing in the Tour de France sense, but I’ll spend 10-15 hours a week riding my bike and I’m concerned with improving and becoming a better cyclist everyday. I plan my diet and sleep around it because I want to become better. It something I think about often, talk about often and spend a good amount of my free time doing. Over the winter I spent a great deal of time outside, despite the cold weather and bad air because I wanted to improve and do more challenging races come Spring and Summer.

image



This Spring was an exciting time for me as it would be warm again and I was excited to do some of the rides in some of Salk Lake Valley canyons. I wanted to push myself to be faster, to recover better and to just be happier with where I was on the saddle. Early in the season I had some of the best riding of my life. I was faster, better at climbing, had better endurance and it looked like I was well on my way to being better. I felt even better about it because of my pace of miles— I was well on my way to doubling the milage I had accumulated the previous year. At some point in late April, I started to experience cramping on most of my rides. Cramps were a problem for me the previous year but cramping was related to pushing myself too hard and no being prepared. If I went out on a long climb and didn’t have enough food or drink I would get cramps at the end. It was uncommon for me to ride less than 50 miles and get one. This Spring I started getting cramps after 10 miles into a ride. I found this strange because I bought expensive supplements and drinks, I was in good shape, I stretched and I was doing what I needed to.

image



It got to a point where I was unsure I could even finish a 25 mile ride. I started to fear getting caught cramped out in the middle of nowhere and slowly working my way back on a cramped out leg. I was so frustrated by this I took issue to my doctor at the time. He did some blood work and told me I should try some potassium/magnesium infusion techniques. At one point I was eating 3 bananas a day and taking supplements. None of this seemed to work, and I ended up getting tired of bananas. The cramps kept coming and no progress was being made.

image



As my frustration continued to mount I took to Google, searching for any edge cases for why people cramp. I noticed a number of people taking Tums for cramp relief, citing the calcium in it providing them instant relief. I felt wholly uncomfortable taking a medication for heart burn casually, even if it was over the counter but I thought there might be something to me having calcium deficiency. I have been eating a mostly Paleo diet for about 8 months now and calling it low in calcium would be an understatement. I went to GNC and got some calcium supplements and started to take them. I was hoping for a miracle turn around but early on it seemed like they made no difference. About 2 weeks into taking them I completed a 55 mile ride with a substantial climb and did not cramp out. I started to get more daring with mixed results. I decided I should mix leg weight lifting into the mix to try to build endurance as well. Slowly with the combination of riding, weight lifting and supplements I’ve gotten to a point where I feel good. I still get cramps and I’m not sure what the solution is but I’m excited about thinking through the problem and fixing it.

I’ve had many frustrated days and nights working on and thinking about my legs. I’ve eased off riding a bit and I want to talk to another doctor about it soon. What I learned most from this experience is how much I love cycling and my resolve to continue to try surprised me. Not everything is going to be easy, but the lessons learned from hardships will help carry me through my next challenge.

Next Career Steps

Ever since I first entered BYU campus in January 2009 I’ve had a strange relationship with Utah. I love the outdoors, I love my family and I love how affordable it is to live here. I love waking up on a 70 degree morning and riding my bike into the canyons, clearing my mind and challenging my body. I love paying $9 to golf 9 holes and I love living in a modern apartment at an affordable price. I have an equally long list of things I don’t like about Utah and I’ve wanted to see a different part of the country and live in a new place with new challenges for some time now. These are the thoughts that bounced around my mind as I searched for a new job this summer. Not only did I want a place that offered challenging work in a fun environment but in a place I enjoyed living.

My job search was brutal when I’m completely honest with myself. Every time I had an offer(s), I ended up with pro and cons lists that were infinitely long and ended up talking myself in and out of every scenario. This was all compounded by personal and family events taking place, making the decision even tougher. Ultimately, one place/position made sense and so I’ll be working at Adobe as an Analyst on the Digital Index team starting next week. I’m excited to utilize my skills in development and statistics and to work with great data set and tools. This also means I get to stay in Utah, continue to ride my bike and continue to play golf poorly.



I have a few other things going on that are career oriented, but I’ll save those for another post. Thanks for all those who helped in this journey.

My Instagram

©2010–2014 Jowanza's World