UC drivers overlapping on Multithreading

[BishoLv]

When executing this snippet of code below it opens the required number of browsers, but it executes the next lines in the first browser only.

so if the loop runs 4 times it opens 4 browsers but opens new tap and open the link 4 times on the first browser but if i didn't use undetected driver it works well

    def start_checking(self):
        combolist_files = self.checker_combolist_box.get(0, tk.END)
        t = threading.Thread(target=self.run_threads, args=(combolist_files,))
        t.start()

    def run_threads(self, combolist_files):
        with concurrent.futures.ThreadPoolExecutor() as executor:
            for combolist_file in combolist_files:
                executor.submit(self.gCheck, combolist_file)

    def gCheck(self, combolist_file):
        with open(combolist_file) as f, SB(uc=True, block_images=True, incognito=True) as sb:
            combolist = (line.strip() for line in f)
            for combo in combolist:
                    sb.driver.execute_script("window.open('');")
                    sb.open('https://accounts.google.com/signin')

idk if this is a bug or my code isn't correct but i want to find a solution

[mdmintz]

Multi-threading in UC Mode is possible if you use pytest multi-threading provided by pytest-xdist.

You'll need to use command-line options for this format, eg --uc to activate UC Mode, and -n4 (for 4 parallel processes, etc). You can go as high as the number of cores safely. Eg, since my computer has 8 cores, I can do -n8 for 8 parallel processes.

Below is a sample run command, with the options you need:

pytest --uc -n4

(For blocking images, add --block-images. For incognito mode, add --incognito.)

Here's a sample file that uses @pytest.mark.parametrize() to turn one test into four tests when run with pytest:

import pytest

@pytest.mark.parametrize("", [[]] * 4)
def test_multi_threaded(sb):
    sb.open("https://nowsecure.nl/#relax")
    try:
        sb.assert_text("OH YEAH, you passed!", "h1", timeout=5.25)
        sb.post_message("Selenium wasn't detected!", duration=2.8)
        sb._print("\n Success! Website did not detect Selenium! ")
    except Exception:
        sb.fail('Selenium was detected! Try using: "pytest --uc"')

Here's the output when running that file with pytest --uc -n4:

pytest test_multi_uc.py --uc -n4
============================ test session starts =============================
platform darwin -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0
rootdir: ~/github/SeleniumBase/examples
configfile: pytest.ini
plugins: html-2.0.1, rerunfailures-12.0, cov-4.1.0, metadata-3.0.0, ordering-0.6, xdist-3.3.1, seleniumbase-4.17.13
4 workers [4 items]     

 Success! Website did not detect Selenium! 

 Success! Website did not detect Selenium! 
..
 Success! Website did not detect Selenium! 
.
 Success! Website did not detect Selenium! 
.
============================= 4 passed in 9.38s ==============================

Some websites may block you if they detect multiple simultaneous connections like that. Be careful where you go.

Note that there are different syntax formats. See: https://github.com/seleniumbase/SeleniumBase/blob/master/help_docs/syntax_formats.md

Also note that pytest-xdist (the multi-processing library for pytest) is more advanced than https://docs.python.org/3/library/concurrent.futures.html for complex multi-threading of this nature.

[BishoLv]

but i dont wanna use pytest because this is one function in a big project contains GUI and there're shared resources, so it won't be easy if i use pytest to run multiple browsers

[mdmintz]

Unfortunately, pytest has the only working multiprocessing system capable of handling all the shared resources successfully at this time. When the other multiprocessing libraries make improvements, there’s already a TODO for utilizing them. In the meantime, you can try calling pytest from a subprocess, etc, in order to run multiple tests at the same time. Using pytest multiprocessing with SeleniumBase has proven itself to be quite capable of handling shared resources correctly for multiple parallel tests.

[mdmintz]

Upgrade to 4.17.12 (or newer) for the multithreading improvements: https://github.com/seleniumbase/SeleniumBase/releases/tag/v4.17.12

[MacMarde]

Upgrade to 4.17.12 (or newer) for the multithreading improvements: https://github.com/seleniumbase/SeleniumBase/releases/tag/v4.17.12

Will it now be possible to use the driver manager for Multithreading with uc=True ?
And if not, what would prevent it from using multithreading? I have been using original UC for multithreading without any problems.

[mdmintz]

@MacMarde That depends on how you implement thread-locking. It may work (some people claim it has worked since the 4.17.12 update), but only pytest multithreading is officially supported because it's the only way SeleniumBase knows how to handle the threads properly. If you've already converted chromedriver to uc_driver (one of the things that UC Mode does), then you will likely avoid file-overlap issues. Also, UC Mode picks a remote-debugging port at startup... if more than one thread starts at the exact same time, that port overlap could happen, causing issues if you manage your own multithreaded system. So if you make sure no thread starts at the exact same time as another, then you'll probably be safe too.

[MacMarde]

@MacMarde That depends on how you implement thread-locking. It may work (some people claim it has worked since the 4.17.12 update), but only pytest multithreading is officially supported because it's the only way SeleniumBase knows how to handle the threads properly. If you've already converted chromedriver to uc_driver (one of the things that UC Mode does), then you will likely avoid file-overlap issues. Also, UC Mode picks a remote-debugging port at startup... if more than one thread starts at the exact same time, that port overlap could happen, causing issues if you manage your own multithreaded system. So if you make sure no thread starts at the exact same time as another, then you'll probably be safe too.

Thanks so much. This sounds not bad. I will give it a try.

[suredarv06]

#2006 (comment)

In the above comment, Multithreading issue is solved ,but the UC mode is called from the command pytest test_multi_uc.py --uc -n4 ,
How can use rest of the configurations like headless,proxy,uc_cdp_events ..etc?

[mdmintz]

SeleniumBase/help_docs/customizing_test_runs.md

SeleniumBase/help_docs/customizing_test_runs.md

Line 104 in 77ee514

🎛️ SeleniumBase provides additional ``pytest`` command-line options for tests:

See seleniumbase/plugins/pytest_plugin.py for all pytest command-line options.

[suredarv06]

That's good ,But can I change User-Agent, proxy in the code after launching with pytest

[mdmintz]

After launching with pytest, use self.get_new_driver(OPTIONS):

SeleniumBase/help_docs/method_summary.md

Line 261 in 77ee514

self.get_new_driver(

[mdmintz]

Possible solution to preventing overlap when running multithreaded scripts without pytest:

Run this before you call SB() or Driver():

import sys
sys.argv.append("-n")

That way, the extra code for handling multithreading gets run: (Using locks to handle shared resources).

SeleniumBase/seleniumbase/core/browser_launcher.py

Line 3138 in edeb2db

if not ("-n" in sys.argv or " -n=" in args or args == "-c"):

(That's in a lot of places, but adding the "-n" to sys.argv should take care of it.)