#selenium

My message got lost in the troubleshooting so I thought I'd post it again

I am developing an application that uses sidekiq to asynchronously launch selenium processes to search for a product code in separate supplier pages. When I run this in my development environment (mac book pro) it executes relatively fast (2 seconds per search). However, when I deploy to an Amazon EC2 micro, it takes significantly longer (20-30 seconds per search). I think this has to do with my mac having 4 cores and the

Also noticed when running on my mac that if one of the browsers instances is loading a page for 5 seconds, during those 5 seconds, the others instances don't do anything- is there a way to have browsers operate concurrently on separate threads in the same process? Does anyone know what the 'best practice' approach to parallelizing selenium work?

[13selenium] 15shs96c pushed 1 new commit to 06master: 02https://git.io/vbbsl

13selenium/06master 14006aa17 15Simon Stewart: Bump the java libraries we depend on to their latest versions...

:ci

https://travis-ci.org/SeleniumHQ/selenium 04FAILED | https://ci.appveyor.com/project/SeleniumHQ/sele... 03SUCCESS

gseidner-sl The micro instance type is tiny. You can find the specs at https://aws.amazon.com/ec2/instance-types/ You could try increasing the size of the instance and see if that changes your results. Personally I use c4.xlarge for my Grid

but I also have some heavy performance requirements, justifying the c4.xlarge You should have CPU Data at your disposal for the instance in Cloudwatch - if you run your tests and see the Max CPU near 100%, you’ll probably want to consider increasing the type size. I use that as well as benchmark durations against different sizes

Thanks! I'll try that. Just so I understand (been struggling alot with this) 1) Does Selenium limit the number of browsers that can run concurrently to n or n-1 where n is the number of cores? I've noticed that in my mac (4 cores) when I queue 4 browsers to run, 3 run first and then the 4th runs. Also, 2) I've noticed that while a page is loading, other browsers are not doing anything. Is this part of Selenium's design?

BTW (did not see CPU usage go above 50%)

I’m not completely sure about that, it might depend on the test framework you’re using. You might benefit from using Selenium Grid if you’re trying to scale concurrently. https://github.com/SeleniumHQ/selenium/wiki/Grid2

Ok, I'll check it out. Does Grid need multiple cores to work? I'd think so, right?

It depends on your web application and the resources needed to load it. Also probably depends on the browser you’re using and the resources that takes.

For example, Chrome and Firefox will probably have different effects on your cpu utilization

My personal recommendation is trial and error :) Track the key metrics like CPU Util and see what happens.

If you’re using Sidekiq though and actually running the tests on the same box as your web app, that sounds kind of scary to me. If running tests hogs performance, it might have negative effects on your website performance.

Yea, definitely making the page website hang. Sounds like I need to use more than just the box my app is in. As a relative newby, I was trying to do everything in one box to keep things simpler, but might have to use more

Yeah you could use Selenium Grid to do the heavy lifting of firing up the browser and doing the things you need without affecting your webapp box

but all of this depends on your web app & infrastructure configuration. Tracking those key metrics I think will be the best so that you know when to scale and by how much

good luck :)

The docker containers for Selenium Grid are pretty sweet, if you don’t mind running Selenium on Linux

Ahh! So many new things to learn! Thanks so much for your help!

This got me going for the docker containers: https://www.youtube.com/watch?v=S4OkrnFb-YY

It might be a bit outdated though by now

will be good to learn the key concepts though

You're the man!

:) good luck, feel free to post here if you have questions