Downloading a large query fails with "Too Many Requests (RFC 6585) (HTTP 429)"

[sampoves]

Hello all,

When trying to query a full table in the Statistics Finland PxWeb database with pxweb, the data tables statfin_raku_pxt_156f and statfin_ras_pxt_12fy fail with "Too Many Requests". These tables are large, over 4 million data cells but what confuses me is that these downloads worked before. Is it possible that the database configuration has changed?

library(jsonlite)
library(pxweb)

pxweb_url <- "https://pxdata.stat.fi/PxWeb/api/v1/en/StatFin/raku/statfin_raku_pxt_156f.px"
# example 2
#pxweb_url <- "https://statfin.stat.fi:443/PxWeb/api/v1/en/StatFin/ras/statfin_ras_pxt_12fy.px"

content <- jsonlite::fromJSON(pxweb_url)
code <- content$variables$code
values <- content$variables$values
dims <- setNames(as.list(values), code)
pxq <- pxweb::pxweb_query(dims)

res <- pxweb::pxweb_get_data(pxweb_url, pxq)

I get the error message usually roughly at 67 %:

Error in pxweb_advanced_get(url = url, query = query, verbose = verbose) :
Too Many Requests (RFC 6585) (HTTP 429).

Link to the database configuration: https://statfin.stat.fi/PXWeb/api/v1/en/?config

I run pxweb_0.17.0 with R version 4.4.3 (2025-02-28 ucrt) for Windows.

Thank you for your time.

[MansMeg]

Hi. This is the likely situation. You can change this for the specific API. I think it is part of the vignette, otherwise let me know and I can see if I can help. Unfortunately Im busy today.

[MansMeg]

It can also be that they use another config than what is communicated through the API. Yes, this is a bad habit, but it has happended before. Then you need to email them.

[pitkant]

It would seem that the issue is related to rate-limiting, i.e. making too many requests inside the 60 second time window? Maybe by adding httr::RETRY or httr2::req_retry logic into pxweb_advanced_get:

pxweb/R/pxweb_get.R

Lines 148 to 166 in 816e244

	if (length(pxqs) > 1 & verbose) {
	cat(" Downloading large query (in", length(pxqs), "batches):\n")
	pb <- utils::txtProgressBar(min = 0, max = length(pxqs), style = 3)
	}
	for (i in seq_along(pxqs)) {
	px <- pxweb_add_call(px)
	pxurl <- build_pxweb_url(px)
	pxqs[[i]] <- pxweb_remove_metadata_from_query(pxqs[[i]], pxmd)
	r <- httr::POST(pxurl, body = pxweb_as_json(x = pxqs[[i]]), pxweb_user_agent(), ...)
	pxweb_http_log_response(r)
	httr::stop_for_status(r)
	pxr[[i]] <- pxweb_parse_response(x = r)
	if (length(pxqs) > 1 & verbose) {
	utils::setTxtProgressBar(pb, i)
	}
	}
	if (length(pxqs) > 1 & verbose) {
	close(pb)
	}

this could be mitigated.

[sampoves]

Hello @MansMeg and @pitkant, thank you for looking into my issue.

pxweb vignette lead me to the Statistics Finland database description. They say it here:

Status codes (error messages):
429 - Too many queries within a minute. The API limit is now 30 queries within 10 seconds.

Maybe then pxweb_get does not wait long enough right now? I still believe Statistics Finland have changed their configuration on their end, but I can't find proof.

According to pxweb/inst/extdata/api.json, pxweb uses correct configuration for statfin.stat.fi. However, maybe there is a new constraint now, 30 queries within 10 seconds within a minute? The api description error message is confusing to me.

[sampoves]

It would seem that the issue is related to rate-limiting, i.e. making too many requests inside the 60 second time window? Maybe by adding httr::RETRY or httr2::req_retry logic into pxweb_advanced_get:

pxweb/R/pxweb_get.R

Lines 148 to 166 in 816e244

if (length(pxqs) > 1 & verbose) {
cat(" Downloading large query (in", length(pxqs), "batches):\n")
pb <- utils::txtProgressBar(min = 0, max = length(pxqs), style = 3)
}
for (i in seq_along(pxqs)) {
px <- pxweb_add_call(px)
pxurl <- build_pxweb_url(px)
pxqs[[i]] <- pxweb_remove_metadata_from_query(pxqs[[i]], pxmd)
r <- httr::POST(pxurl, body = pxweb_as_json(x = pxqs[[i]]), pxweb_user_agent(), ...)
pxweb_http_log_response(r)
httr::stop_for_status(r)
pxr[[i]] <- pxweb_parse_response(x = r)
if (length(pxqs) > 1 & verbose) {
utils::setTxtProgressBar(pb, i)
}
}
if (length(pxqs) > 1 & verbose) {
close(pb)
}
this could be mitigated.

I have now proposed a fix to this issue as a pull request. I successfully tested the function after wrapping httr::POST in httr::RETRY. After your consideration, may we move forward with this fix?

[MansMeg]

Hi!

I have replicated your results. Sorry for taking so long time - was a crazy week. I actually start to think the problem might be on the statfi side.

I changed the config for the call:

library(jsonlite)
library(pxweb)

pxweb_url <- "https://pxdata.stat.fi/PxWeb/api/v1/en/StatFin/raku/statfin_raku_pxt_156f.px"
# example 2
#pxweb_url <- "https://statfin.stat.fi:443/PxWeb/api/v1/en/StatFin/ras/statfin_ras_pxt_12fy.px"

content <- jsonlite::fromJSON(pxweb_url)
code <- content$variables$code
values <- content$variables$values
dims <- setNames(as.list(values), code)
pxq <- pxweb::pxweb_query(dims)

pxo <- pxweb(pxweb_url)
pxo$config$calls_per_period <- 10 (is 30 in the formal config)
res <- pxweb::pxweb_advanced_get(pxo, pxq)

And I still get the same error.

I would reach out to StatFi and ask. I know that this config is not always the exact one that is used by the server (don't ask me why). And if this worked before (and we haven't changed the code), it is most likely an error on their side. Could you reach out to them?