Merge pull request #8 from opsdisk/make-http-429-optional

Added yagooglesearch_manages_http_429s option

Author: opsdisk

README.md

@@ -77,16 +77,16 @@ for url in urls:
 ## Google is blocking me!
 
 Low and slow is the strategy when executing Google searches using `yagooglesearch`.  If you start getting HTTP 429
-responses, Google has rightfully detected you as a bot and will block your IP for a set period of time.
-`yagooglesearch` is not able to bypass CAPTCHA, but you can do this manually by performing a Google search from a
-browser and proving you are a human.
+responses, Google has rightfully detected you as a bot and will block your IP for a set period of time. `yagooglesearch`
+is not able to bypass CAPTCHA, but you can do this manually by performing a Google search from a browser and proving you
+are a human.
 
 The criteria and thresholds to getting blocked is unknown, but in general, randomizing the user agent, waiting enough
-time between paged search results (7-17 seconds), and waiting enough time between different Google searches
-(30-60 seconds) should suffice.  Your mileage will definitely vary though.  Using this library with Tor will likely get
-you blocked quickly.
+time between paged search results (7-17 seconds), and waiting enough time between different Google searches (30-60
+seconds) should suffice.  Your mileage will definitely vary though.  Using this library with Tor will likely get you
+blocked quickly.
 
-## HTTP 429 detection and recovery
+## HTTP 429 detection and recovery (optional)
 
 If `yagooglesearch` detects an HTTP 429 response from Google, it will sleep for `http_429_cool_off_time_in_minutes`
 minutes and then try again.  Each time an HTTP 429 is detected, it increases the wait time by a factor of
@@ -95,6 +95,43 @@ minutes and then try again.  Each time an HTTP 429 is detected, it increases the
 The goal is to have `yagooglesearch` worry about HTTP 429 detection and recovery and not put the burden on the script
 using it.
 
+If you do not want `yagooglesearch` to handle HTTP 429s and would rather handle it yourself, pass
+`yagooglesearch_manages_http_429s=False` when instantiating the yagooglesearch object.  If an HTTP 429 is detected, the
+string "HTTP_429_DETECTED" is added to a list object that will be returned, and it's up to you on what the next step
+should be.  The list object will contain any URLs found before the HTTP 429 was detected.
+
+```python
+import yagooglesearch
+
+query = "site:twitter.com"
+
+client = yagooglesearch.SearchClient(
+    query,
+    tbs="li:1",
+    verbosity=4,
+    num=10,
+    max_search_result_urls_to_return=1000,
+    minimum_delay_between_paged_results_in_seconds=1,
+    yagooglesearch_manages_http_429s=False,  # Add to manage HTTP 429s.
+)
+client.assign_random_user_agent()
+
+urls = client.search()
+
+if "HTTP_429_DETECTED" in urls:
+    print("HTTP 429 detected...it's up to you to modify your search.")
+
+    # Remove HTTP_429_DETECTED from list.
+    urls.remove("HTTP_429_DETECTED")
+
+    print("URLs found before HTTP 429 detected...")
+
+    for url in urls:
+        print(url)
+```
+
+![http429_detection_string_in_returned_list.png](img/http429_detection_string_in_returned_list.png)
+
 ## HTTP and SOCKS5 proxy support
 
 `yagooglesearch` supports the use of a proxy.  The provided proxy is used for the entire life cycle of the search to
@@ -119,10 +156,10 @@ Supported proxy schemes are based off those supported in the Python `requests` l
 
 * `http`
 * `https`
-* `socks5` - "causes the DNS resolution to happen on the client, rather than on the proxy server."  You likely
-  **do not** want this since all DNS lookups would source from where `yagooglesearch` is being run instead of the proxy.
-* `socks5h` - "If you want to resolve the domains on the proxy server, use socks5h as the scheme."  This is the
-  **best**  option if you are using SOCKS because the DNS lookup and Google search is sourced from the proxy IP address.
+* `socks5` - "causes the DNS resolution to happen on the client, rather than on the proxy server."  You likely **do
+  not** want this since all DNS lookups would source from where `yagooglesearch` is being run instead of the proxy.
+* `socks5h` - "If you want to resolve the domains on the proxy server, use socks5h as the scheme."  This is the **best**
+  option if you are using SOCKS because the DNS lookup and Google search is sourced from the proxy IP address.
 
 ## HTTPS proxies and SSL/TLS certificates
 
@@ -232,8 +269,8 @@ The `&tbs=` parameter is used to specify either verbatim or time-based filters.
 ## Limitations
 
 Currently, the `.filter_search_result_urls()` function will remove any url with the word "google" in it.  This is to
-prevent the returned search URLs from being polluted with Google URLs.  Note this if you are trying to explicitly
-search for results that may have "google" in the URL, such as `site:google.com computer`
+prevent the returned search URLs from being polluted with Google URLs.  Note this if you are trying to explicitly search
+for results that may have "google" in the URL, such as `site:google.com computer`
 
 ## License
 
@@ -248,4 +285,4 @@ Project Link: [https://github.com/opsdisk/yagooglesearch](https://github.com/ops
 ## Acknowledgements
 
 * [Mario Vilas](https://github.com/MarioVilas) for his amazing work on the original
-[googlesearch](https://github.com/MarioVilas/googlesearch) library.
+  [googlesearch](https://github.com/MarioVilas/googlesearch) library.

setup.py

@@ -5,7 +5,7 @@
 
 setuptools.setup(
     name="yagooglesearch",
-    version="1.3.0",
+    version="1.4.0",
     author="Brennon Thomas",
     author_email="info@opsdisk.com",
     description="A Python library for executing intelligent, realistic-looking, and tunable Google searches.",

yagooglesearch/__init__.py

@@ -12,7 +12,7 @@
 
 # Custom Python libraries.
 
-__version__ = "1.2.0"
+__version__ = "1.4.0"
 
 # Logging
 ROOT_LOGGER = logging.getLogger("yagooglesearch")
@@ -77,8 +77,9 @@ def __init__(
         country="",
         extra_params=None,
         max_search_result_urls_to_return=100,
-        delay_between_paged_results_in_seconds=list(range(7, 18)),
+        minimum_delay_between_paged_results_in_seconds=7,
         user_agent=None,
+        yagooglesearch_manages_http_429s=True,
         http_429_cool_off_time_in_minutes=60,
         http_429_cool_off_factor=1.1,
         proxy="",
@@ -88,28 +89,31 @@ def __init__(
 
         """
         SearchClient
-        :param str query: Query string. Must NOT be url-encoded.
+        :param str query: Query string.  Must NOT be url-encoded.
         :param str tld: Top level domain.
         :param str lang: Language.
         :param str tbs: Verbatim search or time limits (e.g., "qdr:h" => last hour, "qdr:d" => last 24 hours, "qdr:m"
             => last month).
         :param str safe: Safe search.
         :param int start: First page of results to retrieve.
         :param int num: Max number of results to pull back per page.  Capped at 100 by Google.
-        :param str country: Country or region to focus the search on. Similar to changing the TLD, but does not yield
+        :param str country: Country or region to focus the search on.  Similar to changing the TLD, but does not yield
             exactly the same results.  Only Google knows why...
-        :param dict extra_params: A dictionary of extra HTTP GET parameters, which must be URL encoded. For example if
+        :param dict extra_params: A dictionary of extra HTTP GET parameters, which must be URL encoded.  For example if
             you don't want Google to filter similar results you can set the extra_params to {'filter': '0'} which will
             append '&filter=0' to every query.
         :param int max_search_result_urls_to_return: Max URLs to return for the entire Google search.
-        :param int delay_between_paged_results_in_seconds: Time to wait between HTTP requests for consecutive pages for
-            the same search query.
+        :param int minimum_delay_between_paged_results_in_seconds: Minimum time to wait between HTTP requests for
+            consecutive pages for the same search query.  The actual time will be a random value between this minimum
+            value and value + 11 to make it look more human.
         :param str user_agent: Hard-coded user agent for the HTTP requests.
+        :param bool yagooglesearch_manages_http_429s: Determines if yagooglesearch will handle HTTP 429 cool off and
+           retries.  Disable if you want to manage HTTP 429 responses.
         :param int http_429_cool_off_time_in_minutes: Minutes to sleep if an HTTP 429 is detected.
         :param float http_429_cool_off_factor: Factor to multiply by http_429_cool_off_time_in_minutes for each HTTP 429
             detected.
         :param str proxy: HTTP(S) or SOCKS5 proxy to use.
-        :param bool verify_ssl: Verify the SSL certificate to prevent traffic interception attacks. Defaults to True.
+        :param bool verify_ssl: Verify the SSL certificate to prevent traffic interception attacks.  Defaults to True.
             This may need to be disabled in some HTTPS proxy instances.
         :param int verbosity: Logging and console output verbosity.
 
@@ -127,8 +131,9 @@ def __init__(
         self.country = country
         self.extra_params = extra_params
         self.max_search_result_urls_to_return = max_search_result_urls_to_return
-        self.delay_between_paged_results_in_seconds = delay_between_paged_results_in_seconds
+        self.minimum_delay_between_paged_results_in_seconds = minimum_delay_between_paged_results_in_seconds
         self.user_agent = user_agent
+        self.yagooglesearch_manages_http_429s = yagooglesearch_manages_http_429s
         self.http_429_cool_off_time_in_minutes = http_429_cool_off_time_in_minutes
         self.http_429_cool_off_factor = http_429_cool_off_factor
         self.proxy = proxy
@@ -362,14 +367,24 @@ def get_page(self, url):
 
         if http_response_code == 200:
             html = response.text
+
         elif http_response_code == 429:
+
             ROOT_LOGGER.warning("Google is blocking your IP for making too many requests in a specific time period.")
+
+            # Calling script does not want yagooglesearch to handle HTTP 429 cool off and retry.  Just return a
+            # notification string.
+            if not self.yagooglesearch_manages_http_429s:
+                ROOT_LOGGER.info("Since yagooglesearch_manages_http_429s=False, yagooglesearch is done.")
+                return "HTTP_429_DETECTED"
+
             ROOT_LOGGER.info(f"Sleeping for {self.http_429_cool_off_time_in_minutes} minutes...")
             time.sleep(self.http_429_cool_off_time_in_minutes * 60)
             self.http_429_detected()
 
             # Try making the request again.
             html = self.get_page(url)
+
         else:
             ROOT_LOGGER.warning(f"HTML response code: {http_response_code}")
 
@@ -432,6 +447,13 @@ def search(self):
             # Request Google search results.
             html = self.get_page(url)
 
+            # HTTP 429 message returned from get_page() function, add "HTTP_429_DETECTED" to the set and return to the
+            # calling script.
+            if html == "HTTP_429_DETECTED":
+                unique_urls_set.add("HTTP_429_DETECTED")
+                self.unique_urls_list = list(unique_urls_set)
+                return self.unique_urls_list
+
             # Create the BeautifulSoup object.
             soup = BeautifulSoup(html, "html.parser")
 
@@ -509,6 +531,11 @@ def search(self):
                 url = self.url_next_page_num
 
             # Randomize sleep time between paged requests to make it look more human.
-            random_sleep_time = random.choice(self.delay_between_paged_results_in_seconds)
+            random_sleep_time = random.choice(
+                range(
+                    self.minimum_delay_between_paged_results_in_seconds,
+                    self.minimum_delay_between_paged_results_in_seconds + 11,
+                )
+            )
             ROOT_LOGGER.info(f"Sleeping {random_sleep_time} seconds until retrieving the next page of results...")
             time.sleep(random_sleep_time)