The scraper runs but a failure shows that no zipcode returns (which may be caused as I have not upload the zipcode file to the pub/sub topic), and another exception appears as the chrome fails to start.
Reproduction:
Run the playbook.
When the step goes to the scraper running, an error shows as below.
fatal: [docker_container]: FAILED! => {"changed": true, "cmd": ["python3", "/root/pricer/fp_merge.py"], "delta": "0:00:04.051777", "end": "2023-06-16 20:34:24.221526", "msg": "non-zero return code", "rc": 1, "start": "2023-06-16 20:34:20.169749", "stderr": "Traceback (most recent call last):\n File \"/root/pricer/fp_merge.py\", line 1, in <module>\n from fp2_14 import get_product_prices\n File \"/root/pricer/fp2_14.py\", line 248, in <module>\n df_new = get_product_prices(\"fp-20230425\", \"fp-subscription-NFL-zipcodes\", \"https://www.sayweee.com/en/product/Want-Want-Senbei-Rice-Crackers/17119\")\n File \"/root/pricer/fp2_14.py\", line 30, in get_product_prices\n driver = webdriver.Chrome(options=options)\n File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/chrome/webdriver.py\", line 49, in __init__\n super().__init__(\n File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/chromium/webdriver.py\", line 54, in __init__\n super().__init__(\n File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py\", line 206, in __init__\n self.start_session(capabilities)\n File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py\", line 291, in start_session\n response = self.execute(Command.NEW_SESSION, caps)[\"value\"]\n File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py\", line 346, in execute\n self.error_handler.check_response(response)\n File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/errorhandler.py\", line 245, in check_response\n raise exception_class(message, screen, stacktrace)\nselenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally.\n (unknown error: DevToolsActivePort file doesn't exist)\n (The process started from chrome location /snap/chromium/2497/usr/lib/chromium-browser/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)\nStacktrace:\n#0 0x55d090bec023 <unknown>\n#1 0x55d090931076 <unknown>\n#2 0x55d090959ac6 <unknown>\n#3 0x55d090955ea0 <unknown>\n#4 0x55d090993d8d <unknown>\n#5 0x55d09099356f <unknown>\n#6 0x55d09098af93 <unknown>\n#7 0x55d090960a14 <unknown>\n#8 0x55d090961a7e <unknown>\n#9 0x55d090baf32d <unknown>\n#10 0x55d090bb3d25 <unknown>\n#11 0x55d090bbd21e <unknown>\n#12 0x55d090bb47ac <unknown>\n#13 0x55d090b85462 <unknown>\n#14 0x55d090bd5d88 <unknown>\n#15 0x55d090bd5f2c <unknown>\n#16 0x55d090be54d5 <unknown>\n#17 0x7f6f57ce4609 start_thread", "stderr_lines": ["Traceback (most recent call last):", " File \"/root/pricer/fp_merge.py\", line 1, in <module>", " from fp2_14 import get_product_prices", " File \"/root/pricer/fp2_14.py\", line 248, in <module>", " df_new = get_product_prices(\"fp-20230425\", \"fp-subscription-NFL-zipcodes\", \"https://www.sayweee.com/en/product/Want-Want-Senbei-Rice-Crackers/17119\")", " File \"/root/pricer/fp2_14.py\", line 30, in get_product_prices", " driver = webdriver.Chrome(options=options)", " File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/chrome/webdriver.py\", line 49, in __init__", " super().__init__(", " File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/chromium/webdriver.py\", line 54, in __init__", " super().__init__(", " File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py\", line 206, in __init__", " self.start_session(capabilities)", " File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py\", line 291, in start_session", " response = self.execute(Command.NEW_SESSION, caps)[\"value\"]", " File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py\", line 346, in execute", " self.error_handler.check_response(response)", " File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/errorhandler.py\", line 245, in check_response", " raise exception_class(message, screen, stacktrace)", "selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally.", " (unknown error: DevToolsActivePort file doesn't exist)", " (The process started from chrome location /snap/chromium/2497/usr/lib/chromium-browser/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)", "Stacktrace:", "#0 0x55d090bec023 <unknown>", "#1 0x55d090931076 <unknown>", "#2 0x55d090959ac6 <unknown>", "#3 0x55d090955ea0 <unknown>", "#4 0x55d090993d8d <unknown>", "#5 0x55d09099356f <unknown>", "#6 0x55d09098af93 <unknown>", "#7 0x55d090960a14 <unknown>", "#8 0x55d090961a7e <unknown>", "#9 0x55d090baf32d <unknown>", "#10 0x55d090bb3d25 <unknown>", "#11 0x55d090bbd21e <unknown>", "#12 0x55d090bb47ac <unknown>", "#13 0x55d090b85462 <unknown>", "#14 0x55d090bd5d88 <unknown>", "#15 0x55d090bd5f2c <unknown>", "#16 0x55d090be54d5 <unknown>", "#17 0x7f6f57ce4609 start_thread"], "stdout": "Created a dataframe: Empty DataFrame\nColumns: [Product ID, Zip Code, Shown Price, Minimum Quantity, Total Price, Unit Price, Sales Count, Sold Status]\nIndex: []", "stdout_lines": ["Created a dataframe: Empty DataFrame", "Columns: [Product ID, Zip Code, Shown Price, Minimum Quantity, Total Price, Unit Price, Sales Count, Sold Status]", "Index: []"]}
Expectation:
The scraper should give us a dataframe and the hdf5 file listing the pricing information of those 10 items listed in the url_list.
Research:
I run fp2_10.py initially to ensure that zipcodes have been published to the topic, and then I run the playbook. Unfortunately, this error still exists. I think there should be some issues regarding the fp_merge.py itself.
**Issue Summary:**
The scraper runs but a failure shows that no zipcode returns (which may be caused as I have not upload the zipcode file to the pub/sub topic), and another exception appears as the chrome fails to start.
**Reproduction:**
1. Run the playbook.
2. When the step goes to the scraper running, an error shows as below.
```
fatal: [docker_container]: FAILED! => {"changed": true, "cmd": ["python3", "/root/pricer/fp_merge.py"], "delta": "0:00:04.051777", "end": "2023-06-16 20:34:24.221526", "msg": "non-zero return code", "rc": 1, "start": "2023-06-16 20:34:20.169749", "stderr": "Traceback (most recent call last):\n File \"/root/pricer/fp_merge.py\", line 1, in <module>\n from fp2_14 import get_product_prices\n File \"/root/pricer/fp2_14.py\", line 248, in <module>\n df_new = get_product_prices(\"fp-20230425\", \"fp-subscription-NFL-zipcodes\", \"https://www.sayweee.com/en/product/Want-Want-Senbei-Rice-Crackers/17119\")\n File \"/root/pricer/fp2_14.py\", line 30, in get_product_prices\n driver = webdriver.Chrome(options=options)\n File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/chrome/webdriver.py\", line 49, in __init__\n super().__init__(\n File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/chromium/webdriver.py\", line 54, in __init__\n super().__init__(\n File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py\", line 206, in __init__\n self.start_session(capabilities)\n File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py\", line 291, in start_session\n response = self.execute(Command.NEW_SESSION, caps)[\"value\"]\n File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py\", line 346, in execute\n self.error_handler.check_response(response)\n File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/errorhandler.py\", line 245, in check_response\n raise exception_class(message, screen, stacktrace)\nselenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally.\n (unknown error: DevToolsActivePort file doesn't exist)\n (The process started from chrome location /snap/chromium/2497/usr/lib/chromium-browser/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)\nStacktrace:\n#0 0x55d090bec023 <unknown>\n#1 0x55d090931076 <unknown>\n#2 0x55d090959ac6 <unknown>\n#3 0x55d090955ea0 <unknown>\n#4 0x55d090993d8d <unknown>\n#5 0x55d09099356f <unknown>\n#6 0x55d09098af93 <unknown>\n#7 0x55d090960a14 <unknown>\n#8 0x55d090961a7e <unknown>\n#9 0x55d090baf32d <unknown>\n#10 0x55d090bb3d25 <unknown>\n#11 0x55d090bbd21e <unknown>\n#12 0x55d090bb47ac <unknown>\n#13 0x55d090b85462 <unknown>\n#14 0x55d090bd5d88 <unknown>\n#15 0x55d090bd5f2c <unknown>\n#16 0x55d090be54d5 <unknown>\n#17 0x7f6f57ce4609 start_thread", "stderr_lines": ["Traceback (most recent call last):", " File \"/root/pricer/fp_merge.py\", line 1, in <module>", " from fp2_14 import get_product_prices", " File \"/root/pricer/fp2_14.py\", line 248, in <module>", " df_new = get_product_prices(\"fp-20230425\", \"fp-subscription-NFL-zipcodes\", \"https://www.sayweee.com/en/product/Want-Want-Senbei-Rice-Crackers/17119\")", " File \"/root/pricer/fp2_14.py\", line 30, in get_product_prices", " driver = webdriver.Chrome(options=options)", " File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/chrome/webdriver.py\", line 49, in __init__", " super().__init__(", " File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/chromium/webdriver.py\", line 54, in __init__", " super().__init__(", " File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py\", line 206, in __init__", " self.start_session(capabilities)", " File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py\", line 291, in start_session", " response = self.execute(Command.NEW_SESSION, caps)[\"value\"]", " File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/webdriver.py\", line 346, in execute", " self.error_handler.check_response(response)", " File \"/usr/local/lib/python3.10/dist-packages/selenium/webdriver/remote/errorhandler.py\", line 245, in check_response", " raise exception_class(message, screen, stacktrace)", "selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally.", " (unknown error: DevToolsActivePort file doesn't exist)", " (The process started from chrome location /snap/chromium/2497/usr/lib/chromium-browser/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)", "Stacktrace:", "#0 0x55d090bec023 <unknown>", "#1 0x55d090931076 <unknown>", "#2 0x55d090959ac6 <unknown>", "#3 0x55d090955ea0 <unknown>", "#4 0x55d090993d8d <unknown>", "#5 0x55d09099356f <unknown>", "#6 0x55d09098af93 <unknown>", "#7 0x55d090960a14 <unknown>", "#8 0x55d090961a7e <unknown>", "#9 0x55d090baf32d <unknown>", "#10 0x55d090bb3d25 <unknown>", "#11 0x55d090bbd21e <unknown>", "#12 0x55d090bb47ac <unknown>", "#13 0x55d090b85462 <unknown>", "#14 0x55d090bd5d88 <unknown>", "#15 0x55d090bd5f2c <unknown>", "#16 0x55d090be54d5 <unknown>", "#17 0x7f6f57ce4609 start_thread"], "stdout": "Created a dataframe: Empty DataFrame\nColumns: [Product ID, Zip Code, Shown Price, Minimum Quantity, Total Price, Unit Price, Sales Count, Sold Status]\nIndex: []", "stdout_lines": ["Created a dataframe: Empty DataFrame", "Columns: [Product ID, Zip Code, Shown Price, Minimum Quantity, Total Price, Unit Price, Sales Count, Sold Status]", "Index: []"]}
```
**Expectation:**
The scraper should give us a dataframe and the hdf5 file listing the pricing information of those 10 items listed in the `url_list`.
**Research:**
I run fp2_10.py initially to ensure that zipcodes have been published to the topic, and then I run the playbook. Unfortunately, this error still exists. I think there should be some issues regarding the fp_merge.py itself.
The browser should be running and the port number matches the actual port on which the browser's DevTools protocol is available.
Issue:
The full traceback is:
Traceback (most recent call last):
File "/home/karen_zeng/.local/lib/python3.10/site-packages/ansible/executor/task_executor.py", line 165, in run
res = self._execute()
File "/home/karen_zeng/.local/lib/python3.10/site-packages/ansible/executor/task_executor.py", line 661, in _execute
result = self._handler.run(task_vars=vars_copy)
File "/home/karen_zeng/.local/lib/python3.10/site-packages/ansible/plugins/action/script.py", line 80, in run
source = parts[0]
IndexError: list index out of range
fatal: [docker_container]: FAILED! => {
"msg": "Unexpected failure during module execution: list index out of range",
"stdout": ""
}
`DevToolsActivePort` is an environment variable used by Chromium-based browsers and the ChromeDriver to establish a connection between them.
To troubleshoot this issue, I take the following steps:
1. Verify that the chromium browser is running and accessible at the specified port. Here is what I get:
```
{
"chromium_port.stdout": "Active Internet connections (only servers)\nProto Recv-Q Send-Q Local Address Foreign Address State PID/Program name \ntcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 107/systemd-resolve \ntcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 168/sshd: /usr/sbin \ntcp 0 0 0.0.0.0:631 0.0.0.0:* LISTEN 759/cupsd \ntcp6 0 0 :::22 :::* LISTEN 168/sshd: /usr/sbin \ntcp6 0 0 :::631 :::* LISTEN 759/cupsd "
}
```
2. Check the `DevToolsActivePort` value to ensure it matches the actual port.
3. Set up chrome options
```
chrome_options:
- "--headless"
- "--no-sandbox"
- "--disable-dev-shm-usage"
```
**Expectation:**
The browser should be running and the port number matches the actual port on which the browser's DevTools protocol is available.
**Issue:**
```
The full traceback is:
Traceback (most recent call last):
File "/home/karen_zeng/.local/lib/python3.10/site-packages/ansible/executor/task_executor.py", line 165, in run
res = self._execute()
File "/home/karen_zeng/.local/lib/python3.10/site-packages/ansible/executor/task_executor.py", line 661, in _execute
result = self._handler.run(task_vars=vars_copy)
File "/home/karen_zeng/.local/lib/python3.10/site-packages/ansible/plugins/action/script.py", line 80, in run
source = parts[0]
IndexError: list index out of range
fatal: [docker_container]: FAILED! => {
"msg": "Unexpected failure during module execution: list index out of range",
"stdout": ""
}
```
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Issue Summary:
The scraper runs but a failure shows that no zipcode returns (which may be caused as I have not upload the zipcode file to the pub/sub topic), and another exception appears as the chrome fails to start.
Reproduction:
Expectation:
The scraper should give us a dataframe and the hdf5 file listing the pricing information of those 10 items listed in the
url_list.Research:
I run fp2_10.py initially to ensure that zipcodes have been published to the topic, and then I run the playbook. Unfortunately, this error still exists. I think there should be some issues regarding the fp_merge.py itself.
Look up what DevToolsActivePort is, and figure out why chromedriver can't connect to it.
DevToolsActivePortis an environment variable used by Chromium-based browsers and the ChromeDriver to establish a connection between them.To troubleshoot this issue, I take the following steps:
Verify that the chromium browser is running and accessible at the specified port. Here is what I get:
{
"chromium_port.stdout": "Active Internet connections (only servers)\nProto Recv-Q Send-Q Local Address Foreign Address State PID/Program name \ntcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 107/systemd-resolve \ntcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 168/sshd: /usr/sbin \ntcp 0 0 0.0.0.0:631 0.0.0.0:* LISTEN 759/cupsd \ntcp6 0 0 :::22 :::* LISTEN 168/sshd: /usr/sbin \ntcp6 0 0 :::631 :::* LISTEN 759/cupsd "
}
```
Check the
DevToolsActivePortvalue to ensure it matches the actual port.Set up chrome options
chrome_options:
- "--headless"
- "--no-sandbox"
- "--disable-dev-shm-usage"
```
Expectation:
The browser should be running and the port number matches the actual port on which the browser's DevTools protocol is available.
Issue: