phantom JS unable to "findElements" when navigate to a URL requiring authentication

[SugarRayLua]

Hello.

I'm using the webdriver package in the Rcrawler package and on its own to try to log on and scrape websites for personal use that I have accounts with. I have no problem with the session$findElement() function in webdriver when I go to sites that don't require authentication but can't seem to find similar elements when I log onto sites requiring authentication.

For instance, I have a Facebook account and am interested in scraping Facebook marketplace to better locate some items I would like to purchase. Rcrawler has the useful function "LoginSession()" which lets one log into sites such as Facebook marketplace that require authentication if the user enters the css selectors for the entry fields for the user's name and password and the button to click when done when the authentication popup screen loads up. If I use the development tools from a standard web browser on the following site:

https://www.facebook.com/marketplace

I can tell that the input email element is:

"input[type='text']"

the input password element is:

"input[type='password']"

and the button to click when the user is done entering their credentials is (although not as certain):

"[data-visualcompletion='ignore']"    

And have verified those css selectors using document.querySelector() for all of those selectors with the console on the developmental tools site of the web browsers I used to access the Facebook Marketplace. However, Neither Rcrawler nor the webdriver package seemed to be able to locate those elements using the above selectors which prevents me from logging on an scraping that site.

Is there something I'm not doing correctly to be able to access elements to authenticate on sites with webdriver or Rcrawler that require authentication or is it some limitation of webdriver/phantomJS preventing me from doing so?

Thank you.

[wch]

PhantomJS has not been maintained for the last 7 years, so it is likely it will not work with web sites that use more modern features.

I suggest using Chromote instead, which lets you control the latest Chromium-based browsers.

[SugarRayLua]

Okay, thank you very much. I'll try that 👍😊

[SugarRayLua]

Dear @wch ,

I tried using Chromote but didn't find it nearly as easy to use as the webdriver package nor could I figure out how to usse Chromote with Rcrawler (which uses webdriver). From what I've tested so far, Rcrawler is the most efficient R automated web crawler + scraper. However, it needs webdriver to be able to scrape sites rendered with javascript and hence my continued interested in trying to figure out how to get better performance out of webdriver.

I figured out that the problem underlying my initial issue seemed more complicated than solely not using webdriver for sites with javascript. I realized that I should be able to use webdriver and phantomjs to visit sites that are dynamic by waiting for the dynamic parts of the DOM to be created with webdriver functions such as session$setTimeout(implicit = 10000), yet webdriver still could not find elements on the DOM I queried despite the default browser's developer tools console use of document.querySelector() with the same Css selectors indicating I should have been able to query those elements.

Next, I thus copied the example code from webdriver's website:

https://github.com/rstudio/webdriver

and found that all of the example element queries on the example cran website queried worked fine.

Yet, when I attempted to obtain elements from other sites beside the example site on the webdriver's webpage, such as the wikipedia R article site below, all of my webdriver/phantomjs element queries failed as demonstrated:

R version 4.4.3 (2025-02-28) -- "Trophy Case"

[Workspace loaded from ~/.RData]

> Sys.setenv("OPENSSL_CONF"="/dev/null")
> library(webdriver)
> pjs = run_phantomjs()
> ses = Session$new(port = pjs$port)
> ses$go("https://en.wikipedia.org/wiki/R_(programming_language")
> ses$getTitle()
[1] "R (programming language - Wikipedia"
> el = ses$findElement('#History')
Error in session_makeRequest(self, private, endpoint, data, params, headers) : 
  Unable to find element with css selector '#History'
> el = ses$findElement('#Packages')
Error in session_makeRequest(self, private, endpoint, data, params, headers) : 
  Unable to find element with css selector '#Packages'
> el = ses$executeScript('return document.querySelector("#History");')
> el
NULL
> el$getText()
Error: attempt to apply non-function
> el = ses$executeScript('return document.getElementByID("#History");')
Error in session_makeRequest(self, private, endpoint, data, params, headers) : 
  undefined is not a function (evaluating 'document.getElementByID("#History")')
> el = ses$executeScript("return document.getElementByID('#History');")
Error in session_makeRequest(self, private, endpoint, data, params, headers) : 
  undefined is not a function (evaluating 'document.getElementByID('#History')')

Although I am only a novice JS programmer, from examining the underlying html source of that
R Wikipedia page, it doesn't seem that the underlying html code is that complex and would have expected
that webdriver/phantomjs would be able to retrieve elements as the package is able to do for the
sample CRAN site that was described on the webdriver github site.

Am I trying to use webdriver incorrectly, or is there something that is not working as intended
with the webdriver package?

Thank you :-)

[wch]

I think that if you use PhantomJS, you will run into problems in the future as web technologies continues to evolve. I know that there are many websites that won't work with it today.

That said, I do see some errors in your code:

This URL is missing the closing ):

ses$go("https://en.wikipedia.org/wiki/R_(programming_language")

It should be:

ses$go("https://en.wikipedia.org/wiki/R_(programming_language)")

And when you call getElementByID(), the capitalization is wrong. It should be getElementById().

[SugarRayLua]

Thanks you very much, @wch , for you additional help.

I apologize; I did multiple tests using webdriver and didn't double check some of the runs and missed those typos. After re-installing phantomjs and paying attention to those typos, indeed, webdriver finds the elements on the rather straightforward html wikipedia page fine just as in the posted example on the webdriver page.

Reflecting more on your comments about PhantomJS not being able to handle newer websites as well, I re-tested the original website example I posted the issue about (Facebook marketplace) and found that webdriver using PhantomJS still had definitely finding the elements on that site. However, comparing the Facebook marketplace website that gives webdriver/PhantomJS trouble compared to the wikipedia and the example website on the webdriver webpage that don't give webdriver trouble, it is clear that the Facebook marketplace site incorporates much more javascript than the other sites, and I suspect that is the actual problem. I read it can be remarkably slow to scrape a website with PhantomJs that has a lot of javascript and the Facebook marketplace start page loads many scripts before it gets to the section of the DOM that contains the element I was looking for webdriver/PhantomJS to find. Thus, despite setting an added amount of timeout to element searches in webdriver/PhantomJS, I suspect that I still have not allowed enough time for PhantomJS to run all of the javascript when I've asked webdriver/PhantomJS to find the element and thus PhantomJS still hasn't loaded that element from the DOM so that webdriver returns from the element search saying the element doesn't exist. To support that supposition, asking webdriver/PhantomJS to find an element on the Facebook marketplace site that is at the top of the HTML DOM is successful (i.e. findElement() returns the correct element).

Thus, as you suggested, I will working more on getting a Chrome headless browser installed and learning how to drive that headless browser for my more complex web scraping projects in the future.

Have a good weekend :-)