This quickly led me to receive the following htmlĪs you were browsing, something about your browser made us

However, the first task in my iterative approach was to just get a listing page for a StreetEasy search.

It had been a while since I had scraped sites and I wanted to do it all (mostly) on my own. However, I thought it was simple enough I’d prefer to do it myself. At first glance, there appeared to be a few different StreetEasy scraping scripts on Github. So first let me quickly describe the reason I wasn’t able to scrape StreetEasy. Spoiler alert I wasn’t able to do this for a reason I will explain in more detail, but this led me to use a lot of old tools I hadn’t used in a while and come up with a script for scraping through Tor and switching IP’s between requests. So I had a thought, I used to web scrape sites using Python, why not try it on StreetEasy and filter the apartments myself. After a quick search I realized that people have been requesting this feature for years (since Oct 2015 to be exact), and it was nowhere in sight and there didn’t seem to be any services out there that did it either. The other day I was starting the search for a new aparment in New York City, which I have done a couple of times now, and was frustrated that StreetEasy doesn’t allow you to filter apartments that are available after a certain date. Another thing to note is that some sites are able to automatically block IP’s that are Tor exit nodes, so this may not work for some sites that go to these measures. This was more an informational exercise and I wanted to share it with others. I do not condone the use of this information for creating illegal web crawlers.

How to use tor super safely